×

Identification of spikes in time series. (English) Zbl 1445.62225

Summary: Researchers interested in the effects of exposure spikes on an outcome need tools to identify unexpectedly high values in a time series. However, the best method to identify spikes in time series is not known. This paper aims to fill this gap by testing the performance of several spike detection methods in a simulation setting. We created simulations parameterized by monthly violence rates in nine California cities that represented different series features, and randomly inserted spikes into the series. We then compared the ability to detect spikes of the following methods: ARIMA modeling, Kalman filtering and smoothing, wavelet modeling with soft thresholding, and an iterative outlier detection method. We varied the magnitude of spikes from 10 to 50 % of the mean rate over the study period and varied the number of spikes inserted from 1 to 10. We assessed performance of each method using sensitivity and specificity. The Kalman filtering and smoothing procedure had the best overall performance. We applied each method to the monthly violence rates in nine California cities and identified spikes in the rate over the 2005–2012 period.

MSC:

62M10 Time series, auto-correlation, regression, etc. in statistics (GARCH)
62G32 Statistics of extreme values; tail inference
62P25 Applications of statistics to social sciences
PDF BibTeX XML Cite
Full Text: DOI arXiv

References:

[1] Ahern, J., Cerdá, M., Lippman, S. A., Tardiff, K. J., Vlahov, D., and Galea, S. (February 2013). Navigating non-positivity in neighbourhood studies: An analysis of collective efficacy and violence. Journal of Epidemiology and Community Health, 67(2):159-165.; Ahern, J.; Cerdá, M.; Lippman, S. A.; Tardiff, K. J.; Vlahov, D.; Galea, S., Navigating non-positivity in neighbourhood studies: An analysis of collective efficacy and violence, Journal of Epidemiology and Community Health, 67, 2, 159-165 (2013)
[2] Ahern, J., Matthay, E. C., Goin, D. E., Farkas, K., and Rudolph, K. E. (2018). Acute changes in community violence and increases in hospital visits and deaths from stress-responsive diseases. Epidemiology, 29(5):684-691.; Ahern, J.; Matthay, E. C.; Goin, D. E.; Farkas, K.; Rudolph, K. E., Acute changes in community violence and increases in hospital visits and deaths from stress-responsive diseases, Epidemiology, 29, 5, 684-691 (2018)
[3] Apter, A. J., Garcia, L. A., Boyd, R. C., Wang, X., Bogen, D. K., and Ten Have, T. (September 2010). Exposure to community violence is associated with asthma hospitalizations and emergency department visits. The Journal of Allergy and Clinical Immunology, 126(3):552-557.; Apter, A. J.; Garcia, L. A.; Boyd, R. C.; Wang, X.; Bogen, D. K.; Ten Have, T., Exposure to community violence is associated with asthma hospitalizations and emergency department visits, The Journal of Allergy and Clinical Immunology, 126, 3, 552-557 (2010)
[4] Baird, S., Friedman, J., and Schady, N. (2011). Aggregate income shocks and infant mortality in the developing world. Review of Economics and Statistics, 93(3):847-856.; Baird, S.; Friedman, J.; Schady, N., Aggregate income shocks and infant mortality in the developing world, Review of Economics and Statistics, 93, 3, 847-856 (2011)
[5] Balke, N. S. (1993). Detecting level shifts in time series. Journal of Business & Economic Statistics, 11(1):81-92.; Balke, N. S., Detecting level shifts in time series, Journal of Business & Economic Statistics, 11, 1, 81-92 (1993)
[6] Besculides, M., Heffernan, R., Mostashari, F., and Weiss, D. (2005). Evaluation of school absenteeism data for early outbreak detection, New York city. BMC Public Health, 5(1):105.; Besculides, M.; Heffernan, R.; Mostashari, F.; Weiss, D., Evaluation of school absenteeism data for early outbreak detection, New York city, BMC Public Health, 5, 1, 105 (2005)
[7] Bhattacharya, J., DeLeire, T., Haider, S., and Currie, J. (2003). Heat or eat? Cold-weather shocks and nutrition in poor American families. American Journal of Public Health, 93(7):1149-1154.; Bhattacharya, J.; DeLeire, T.; Haider, S.; Currie, J., Heat or eat? Cold-weather shocks and nutrition in poor American families, American Journal of Public Health, 93, 7, 1149-1154 (2003)
[8] Brown, J. R., Hill, H. M., and Lambert S. F. (2005). Traumatic stress symptoms in women exposed to community and partner violence. Journal of Interpersonal Violence, 20(11):1478-1494.; Brown, J. R.; Hill, H. M.; Lambert, S. F., Traumatic stress symptoms in women exposed to community and partner violence, Journal of Interpersonal Violence, 20, 11, 1478-1494 (2005)
[9] Chan, K. S., and Ripley, B. (2012). Tsa: Time series analysis. R Package Version 1.01. URL: . R-project. org/package= TSA.; Chan, K. S.; Ripley, B., Tsa: Time series analysis (2012)
[10] Chang, Ih, Tiao, G. C., and Chen, C. (1988). Estimation of time series parameters in the presence of outliers. Technometrics, 30(2):193-204.; Chang, Ih; Tiao, G. C.; Chen, C., Estimation of time series parameters in the presence of outliers, Technometrics, 30, 2, 193-204 (1988)
[11] Chen, C., and Liu, L.-M. (1993). Joint estimation of model parameters and outlier effects in time series. Journal of the American Statistical Association, 88(421):284-297.; Chen, C.; Liu, L.-M., Joint estimation of model parameters and outlier effects in time series, Journal of the American Statistical Association, 88, 421, 284-297 (1993) · Zbl 0775.62229
[12] Clark, C., Ryan, L., Kawachi, I., Canner, M. J., Berkman, L., and Wright, R. J. (January 2008). Witnessing community violence in residential neighborhoods: A mental health hazard for urban women. Journal of Urban Health, 85(1):22-38.; Clark, C.; Ryan, L.; Kawachi, I.; Canner, M. J.; Berkman, L.; Wright, R. J., Witnessing community violence in residential neighborhoods: A mental health hazard for urban women, Journal of Urban Health, 85, 1, 22-38 (2008)
[13] Cryer, J. D., and Chan, K.-S. (2008). Time Series Analysis. Springer Texts in Statistics. New York, NY: Springer New York.; Cryer, J. D.; Chan, K.-S., Springer Texts in Statistics (2008)
[14] Curry, A., Latkin, C., and Davey-Rothwell, M. (July 2008). Pathways to depression: The impact of neighborhood violent crime on inner-city residents in Baltimore, Maryland, USA. Social Science & Medicine, 67(1):23-30.; Curry, A.; Latkin, C.; Davey-Rothwell, M., Pathways to depression: The impact of neighborhood violent crime on inner-city residents in Baltimore, Maryland, USA, Social Science & Medicine, 67, 1, 23-30 (2008)
[15] Dafni, U. G., Tsiodras, S., Panagiotakos, D., Gkolfinopoulou, K., Kouvatseas, G., Tsourti, Z., and Saroglou, G. (2004). Algorithm for statistical detection of peaks-syndromic surveillance system for the athens 2004 olympic games. MMWR Morb Mortal Wkly Rep, 53(Suppl):86-94.; Dafni, U. G.; Tsiodras, S.; Panagiotakos, D.; Gkolfinopoulou, K.; Kouvatseas, G.; Tsourti, Z.; Saroglou, G., Algorithm for statistical detection of peaks-syndromic surveillance system for the athens 2004 olympic games, MMWR Morb Mortal Wkly Rep, 53, Suppl, 86-94 (2004)
[16] De Jong, P., and Penzer, J. (August 2000). Diagnosing shocks in time series. Technical Report, London School of Economics.; De Jong, P.; Penzer, J., Diagnosing shocks in time series (2000) · Zbl 0926.62079
[17] Durbin, J., and Koopman, S. J. (2012). Time Series Analysis by State Space Methods, Volume 38. Oxford: OUP.; Durbin, J.; Koopman, S. J., Time Series Analysis by State Space Methods, 38 (2012) · Zbl 1270.62120
[18] Faragher, R. (2012). Understanding the basis of the kalman filter via a simple and intuitive derivation [lecture notes]. IEEE Signal Processing Magazine, 29(5):128-132.; Faragher, R., Understanding the basis of the kalman filter via a simple and intuitive derivation [lecture notes], IEEE Signal Processing Magazine, 29, 5, 128-132 (2012)
[19] Helske, J. (2014). Kfas: Kalman filter and smoothers for exponential family state space models. R Package Version, 1:4-1.; Helske, J., Kfas: Kalman filter and smoothers for exponential family state space models, R Package Version, 1, 4-1 (2014) · Zbl 1391.62002
[20] Helske, J. (2017). KFAS: Exponential family state space models in R. Journal of Statistical Software, 78(10):1-39.; Helske, J., KFAS: Exponential family state space models in R, Journal of Statistical Software, 78, 10, 1-39 (2017) · Zbl 1391.62002
[21] Humphreys, D. K., Panter, J., Sahlqvist, S., Goodman, A., and Ogilvie, D. (2016). Changing the environment to improve population health: A framework for considering exposure in natural experimental studies. Journal of Epidemiology and Community Health, 70(9):941-946.; Humphreys, D. K.; Panter, J.; Sahlqvist, S.; Goodman, A.; Ogilvie, D., Changing the environment to improve population health: A framework for considering exposure in natural experimental studies, Journal of Epidemiology and Community Health, 70, 9, 941-946 (2016)
[22] Hyndman, R, Athanasopoulos, G., Bergmeir, C., Caceres, G., Chhay, L., O’Hara-Wild, M., Petropoulos, F., Razbash, S., Wang, E., and Yasmeen, F. (2019). Forecast: Forecasting functions for time series and linear models, 2019. R Package Version 8.5.; Hyndman, R.; Athanasopoulos, G.; Bergmeir, C.; Caceres, G.; Chhay, L.; O’Hara-Wild, M.; Petropoulos, F.; Razbash, S.; Wang, E.; Yasmeen, F., Forecast: Forecasting functions for time series and linear models, 2019, R Package Version, 8.5 (2019)
[23] Hyndman, R. J., and Khandakar, Y. (2008). Automatic time series forecasting: The forecast package for R. Journal of Statistical Software, 26(3):1-22.; Hyndman, R. J.; Khandakar, Y., Automatic time series forecasting: The forecast package for R, Journal of Statistical Software, 26, 3, 1-22 (2008)
[24] Hyndman, R. J., Khandakar, Y. (2007). Automatic time series for forecasting: The forecast package for R. Technical Report, Monash University, Department of Econometrics and Business Statistics.; Hyndman, R. J.; Khandakar, Y., Automatic time series for forecasting: The forecast package for R. Technical Report (2007)
[25] Jacob, B., Lefgren, L., and Moretti, E. (2007). The dynamics of criminal behavior evidence from weather shocks. Journal of Human Resources, 42(3):489-527.; Jacob, B.; Lefgren, L.; Moretti, E., The dynamics of criminal behavior evidence from weather shocks, Journal of Human Resources, 42, 3, 489-527 (2007)
[26] Kane, R. J. (2011). The ecology of unhealthy places: Violence, birthweight, and the importance of territoriality in structurally disadvantaged communities. Social Science & Medicine, 73(11):1585-1592.; Kane, R. J., The ecology of unhealthy places: Violence, birthweight, and the importance of territoriality in structurally disadvantaged communities, Social Science & Medicine, 73, 11, 1585-1592 (2011)
[27] Koppensteiner, M. F., and Manacorda, M. (2016). Violence and birth outcomes: Evidence from homicides in brazil. Journal of Development Economics, 119:16-33.; Koppensteiner, M. F.; Manacorda, M., Violence and birth outcomes: Evidence from homicides in brazil, Journal of Development Economics, 119, 16-33 (2016)
[28] Margerison-Zilko, C. E., Catalano, R., Hubbard, A., and Ahern, J. (2011). Maternal exposure to unexpected economic contraction and birth weight for gestational age. Epidemiology (Cambridge, Mass.), 22(6):855.; Margerison-Zilko, C. E.; Catalano, R.; Hubbard, A.; Ahern, J., Maternal exposure to unexpected economic contraction and birth weight for gestational age, Epidemiology (Cambridge Mass.), 22, 6, 855 (2011)
[29] Martinez, P., and Richters, J. E. (February 1993). The NIMH community violence project: II. Children’s distress symptoms associated with violence exposure. Psychiatry, 56(1):22-35.; Martinez, P.; Richters, J. E., The NIMH community violence project: II, Children’s distress symptoms associated with violence exposure. Psychiatry, 56, 1, 22-35 (1993)
[30] Masi, C. M., Hawkley, L. C., Harry Piotrowski, Z., and Pickett, K. E. (December 2007). Neighborhood economic disadvantage, violent crime, group density, and pregnancy outcomes in a diverse, urban population. Social Science & Medicine, 65(12):2440-2457.; Masi, C. M.; Hawkley, L. C.; Harry Piotrowski, Z.; Pickett, K. E., Neighborhood economic disadvantage, violent crime, group density, and pregnancy outcomes in a diverse, urban population, Social Science & Medicine, 65, 12, 2440-2457 (2007)
[31] Metcalfe, A. V., and Cowpertwait, P. S. P. (2009). Introductory Time Series with R. New York, NY: Springer New York.; Metcalfe, A. V.; Cowpertwait, P. S. P., Introductory Time Series with R (2009) · Zbl 1179.62115
[32] Michael Oakes, J. (June 2006). Commentary: Advancing neighbourhood-effects research—selection, inferential support, and structural confounding. International Journal of Epidemiology, 35(3):643-647.; Michael Oakes, J., Commentary: Advancing neighbourhood-effects research—selection, inferential support, and structural confounding, International Journal of Epidemiology, 35, 3, 643-647 (2006)
[33] Miguel, E., Satyanath, S., and Sergenti, E. (2004). Economic shocks and civil conflict: An instrumental variables approach. Journal of Political Economy, 112(4):725-753.; Miguel, E.; Satyanath, S.; Sergenti, E., Economic shocks and civil conflict: An instrumental variables approach, Journal of Political Economy, 112, 4, 725-753 (2004)
[34] Miller, L. S., Wasserman, G. A., Neugebauer, R., Gorman-Smith, D., and Kamboukos, D. (March 1999). Witnessed community violence and antisocial behavior in high-risk, urban boys. Journal of Clinical Child Psychology, 28(1):2-11.; Miller, L. S.; Wasserman, G. A.; Neugebauer, R.; Gorman-Smith, D.; Kamboukos, D., Witnessed community violence and antisocial behavior in high-risk, urban boys, Journal of Clinical Child Psychology, 28, 1, 2-11 (1999)
[35] Nason, G. P., editor. (2008). Wavelet Methods in Statistics with R. New York, NY: Springer New York.; Nason, G. P., Wavelet Methods in Statistics with R (2008) · Zbl 1165.62033
[36] Nason, G. P. (2010). Wavethresh: Wavelets statistics and transforms. R Package Version, 4:163.; Nason, G. P., Wavethresh: Wavelets statistics and transforms, R Package Version, 4, 163 (2010)
[37] Nason, G. P., and Silverman, B. W. (1994). The discrete wavelet transform in s. Journal of Computational and Graphical Statistics, 3(2):163-191.; Nason, G. P.; Silverman, B. W., The discrete wavelet transform in s, Journal of Computational and Graphical Statistics, 3, 2, 163-191 (1994) · Zbl 0828.62038
[38] R Core Team. (2017). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria.; Core Team, R., R: A language and environment for statistical computing (2017)
[39] Tsay, R. S. (January 1988). Outliers, level shifts, and variance changes in time series. Journal of Forecasting, 7(1):1-20.; Tsay, R. S., Outliers, level shifts, and variance changes in time series, Journal of Forecasting, 7, 1, 1-20 (1988)
[40] Unkel, S., Paddy Farrington, C., Garthwaite, P. H., Robertson, C., and Andrews, N. (2012). Statistical methods for the prospective detection of infectious disease outbreaks: A review. Journal of the Royal Statistical Society: Series A (Statistics in Society), 175(1):49-82.; Unkel, S.; Paddy Farrington, C.; Garthwaite, P. H.; Robertson, C.; Andrews, N., Statistical methods for the prospective detection of infectious disease outbreaks: A review, Journal of the Royal Statistical Society: Series A (Statistics in Society), 175, 1, 49-82 (2012)
[41] Wilde, J., Apouey, B. H., and Jung, T. (2017). The effect of ambient temperature shocks during conception and early pregnancy on later life outcomes. European Economic Review.; Wilde, J.; Apouey, B. H.; Jung, T., The effect of ambient temperature shocks during conception and early pregnancy on later life outcomes (2017)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.