Deep learning at the interface of agricultural insurance risk and spatio-temporal uncertainty in weather extremes. (English) Zbl 1429.91278

Summary: Challenges in risk estimation for agricultural insurance bring to the fore statistical problems of modeling complex weather and climate dynamics, analyzing massive multi-resolution, multi-source data. Nonstationary space-time structure of such data also introduces greater complexity when assessing the highly nonlinear relationship between weather events and crop yields. In this setting, conventional parametric statistical and actuarial models may no longer be appropriate. In turn, modern machine learning and artificial intelligence procedures, which allow fast and automatic learning of hidden dependencies and structures, offer multiple operational benefits and now prove to deliver a highly competitive performance in a variety of applications, from credit card fraud detection to the next best product offer and customer segmentation. Yet their potential in actuarial sciences, and particularly agricultural insurance, remains largely untapped. In this project, we introduce a modern deep learning methodology into the assessment of climate-induced risks in agriculture and evaluate its potential to deliver a higher predictive accuracy, speed, and scalability. We present a pilot study of deep learning algorithms – specifically, deep belief networks – using historical crop yields, weather station-based records, and gridded weather reanalysis data for Manitoba, Canada from 1996 to 2011. Our findings show that deep learning can attain higher prediction accuracy, based on benchmarking its performance against more conventional approaches, especially in multiscale, heterogeneous data environments of agricultural risk management.


91G05 Actuarial mathematics
62P05 Applications of statistics to actuarial sciences and financial mathematics


gamair; R; darch; glmnet
Full Text: DOI


[1] Andrejko, E., Deep Learning in Agriculture, PowerPoint presentation at the Silicon Valley Machine Learning Meetup (2014)
[2] Bengio, Y., Learning Deep Architectures for AI, Foundations and Trends in Machine Learning, 2, 1, 1-127 (2009) · Zbl 1192.68503
[3] Bengio, Y.; Montavon, G.; Orr, G. B.; Müller, K.-R., Neural networks: Tricks of the trade, Practical recommendations for gradient-based training of deep architectures, 437-478 (2012), New York: Springer, New York
[4] Bergstra, J.; Bengio, Y., Random search for hyper-parameter optimization, Journal of Machine Learning Research, 13, 281-305 (2012) · Zbl 1283.68282
[5] Berrisford, P.; Dee, Poli, D. P.; Brugge, R.; Fielding, M.; Fuentes, M.; Kållberg, P. W.; Kobayashi, S.; Uppala, S.; Simmons, A. (2011)
[6] Breiman, L., Random forests, Machine Learning, 45, 1, 5-32 (2001) · Zbl 1007.68152
[7] Brockwell, P. J.; Davis, R. A., Time series: Theory and methods (2009), New York: Springer, New York · Zbl 1169.62074
[8] Candel, A.; Lanford, J.; Ledell, E.; Parmer, V.; Arora, A., Deep learning with h2o (2015)
[9] Cheng, Z.; Chow, M. Y.; Jung, D.; Jeon, J., A big data based deep learning approach for vehicle speed prediction, IEEE 26th International Symposium on Industrial Electronics (2017)
[10] Cressie, N.; Wikle, C. K., Statistics for spatio-temporal data (2011), Hoboken, NJ: John Wiley & Sons, Hoboken, NJ · Zbl 1273.62017
[11] Daron, J. D.; Stainforth., D. A., Assessing pricing assumptions for weather index insurance in a changing climate, Climate Risk Management, 1, 76-91 (2014)
[12] Dee, D. P., The ERA-Interim reanalysis: Configuration and performance of the data assimilation system, Quarterly Journal of the Royal Meteorological Society, 137, 656, 553-97 (2011)
[13] De Leeuw, J.; Vrieling, A.; Shee, A.; Atzberger, C.; Hadgu, K. M.; Biradar, C. M.; Keah, H.; Turvey., C., The potential and uptake of remote sensing in insurance: A review, Remote Sensing, 6, 11, 10888-912 (2014)
[14] Elidan, G., Copulas in machine learning, CRM-CANSSI Workshop on New Horizons in Copula Modeling, 1-10 (2014)
[15] Erhan, D.; Bengio, Y.; Courville, A.; Manzagol, P.-A.; Vincent, P.; Bengio, S., Why does unsupervised pre-training help deep learning?, Journal of Machine Learning Research, 11, 625-60 (2010) · Zbl 1242.68219
[16] Francesco, S.; Franzin, A. (2016)
[17] Francis, L. (2015)
[18] Friedman, J. J. H., Stochastic gradient boosting, Computational Statistics & Data Analysis, 38, 4, 367-78 (2002) · Zbl 1072.65502
[19] Friedman, J. J. H.; Hastie, T.; Tibshirani., R., Regularization paths for generalized linear models via coordinate descent, Journal of Statistical Software, 33, 1 (2009)
[20] Ghahari, A.; Gel, Y. R.; Lyubchich, V.; Chun, Y.; Uribe, D., On employing multi-resolution weather data in crop insurance, Proceedings of the SIAM International Conference on Data Mining (SDM17) Workshop on Mining Big Data in Climate and Environment (2017)
[21] Glorot, X.; Bengio, Y., Understanding the difficulty of training deep feedforward neural networks, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, 9, 249-56 (2010)
[22] Goodfellow, I.; Bengio, Y.; Courville, A., Deep learning (2016), Cambridge, MA: MIT Press, Cambridge, MA · Zbl 1373.68009
[23] Goodwin, B. K.; Hungerford., A., Copula-based models of systemic risk in U.S. agriculture: Implications for crop insurance and reinsurance contracts, American Journal of Agricultural Economics, 97, 3, 879-96 (2014)
[24] Grover, A.; Kapoor, A.; Horvitz, E., A deep hybrid model for weather forecasting, Proceedings of the KDD’15 (2015)
[25] Halcrow, H. G., Actuarial structures for crop insurance, Journal of Farm Economics, 31, 3, 418-43 (1949)
[26] Hastie, T. J.; Tibshirani, R. J., Generalized additive models (1990), New York: Chapman and Hall/CRC, New York
[27] Hastie, T.; Tibshirani, R.; Friedman, J. J. H., The elements of statistical learning (2001), New York: Springer, New York
[28] Hinton, G. E. (2002)
[29] Hinton, G. E. (2010)
[30] Hinton, G. E.; Osindero, S.; Teh., Y. W., A fast learning algorithm for deep belief nets, Neural Computation, 18, 7, 1527-54 (2006) · Zbl 1106.68094
[31] Hinz, T.; Navarro-Guerrero, N.; Magg, S.; Wermter., S., Speeding up the hyperparameter optimization of deep convolutional neural networks, International Journal of Computational Intelligence and Applications, 17, 2 (2018)
[32] Hochreiter, S.; Schmidhuber., J., Long short-term memory, Neural Computation, 9, 8, 1735-80 (1997)
[33] Ibarra, H.; Skees., J., Innovation in risk transfer for natural hazards impacting agriculture, Environmental Hazards, 7, 62-9 (2007)
[34] Krauss, C.; Do, X. A.; Huck, N., Deep neural networks, gradient-boosted trees, random forests: Statistical arbitrage on the S&P 500, European Journal of Operational Research, 259, 689-702 (2017) · Zbl 1395.91514
[35] Lv, Y.; Duan, Y.; Kang, W.; Li, Z.; Wang, F.-Y., Traffic flow prediction with big data: A deep learning approach, IEEE Transactions on Intelligent Transportation Systems, 16, 865-73 (2015)
[36] Lyubchich, V.; Newlands, N. K.; Ghahari, A.; Mahdi, T.; Gel., Y. R., Insurance risk assessment in the face of climate change: Integrating data science and statistics, Wiley Interdisciplinary Reviews: Computational Statistics, 11, 4, e1462 (2019)
[37] Martinich, J.; Crimmins, A.; Beach, R. H.; Thomson, A.; Mcfarland., J., Focus on agriculture and forestry benefits of reducing climate change impacts, Environmental Research Letters, 12, 6, 060301 (2017)
[38] Miranda, M.; Farrin., K., Index insurance for developing countries, Applied Economic Perspectives and Policy, 34, 3, 391-427 (2012)
[39] Mudelsee, M., Climate time series analysis (2014), Basel, Switzerland: Springer, Basel, Switzerland · Zbl 1300.86001
[40] Nadolnyak, D.; Vedenov., D., Information value of climate forecasts for rainfall index insurance for pasture, rangeland, and forage in the southeast United States, Journal of Agricultural & Applied Economics, 45, 109-24 (2013)
[41] Newlands, N. K., Future sustainable ecosystems: Complexity, risk, uncertainty. Applied Environmental Statistics (2017), Baton Rouge, FL: Chapman & Hall/CRC, Baton Rouge, FL
[42] Newlands, N. K.; Townley-Smith, L., Predicting energy crop yield using Bayesian networks, Proceedings of the Fifth IASTED International Conference, 711, 14-106 (2010)
[43] Porth, L.; Tan., K. S., Agricultural insurance—More room to grow?, The Actuary Magazine, 12, 2, 35-41 (2015)
[44] Porth, L.; Villeneuve, R., Issues in agricultural insurance, Proceedings of the 2015 SOA Annual Meeting and Exhibit (2015)
[45] R Core Team, R: A language and environment for statistical computing. Version 3.4.2 (2017), R Foundation for Statistical Computing: R Foundation for Statistical Computing, Vienna, Austria
[46] Rong, X. (2015)
[47] Skees, J. (2010)
[48] Skees, J.; Murphy, A.; Collier, B.; Mccord, M. J.; Roth, J. (2007)
[49] Smith, S. L.; Kindermans, P.-J.; Ying, C.; Le., Q. V., Don’t decay the learning rate, Increase the batch size, ArXiv e-prints (2017)
[50] Spedicato, G. A., Machine learning methods to perform pricing optimization. A comparison with standard GLMs, Variance, 12, 69-89 (2018)
[51] Srivastava, N.; Hinton, G. E.; Krizhevsky, A.; Sutskever, I.; Salakhutdinov, R., Dropout: A simple way to prevent neural networks from overfitting, Journal of Machine Learning Research, 15, 1, 1929-1958 (2014) · Zbl 1318.68153
[52] The H2O.ai team (2017)
[53] Tibshirani, R., Regression shrinkage and selection via the Lasso, Journal of the Royal Statistical Society. Series B, 58, 1, 267-88 (1996) · Zbl 0850.62538
[54] Vyushin, D.; Kushner, P.; Zwiers., F., Modeling and understanding persistence of climate variability, Journal of Geophysical Research: Atmospheres, 117, D21 (2012)
[55] Wang, G.; Peng, J.; Luo, P.; Wang, X.; Lin., L., Batch Kalman normalization: Towards training deep neural networks with micro-batches, ArXiv e-prints (2018)
[56] Wood, S. N., Generalized additive models: An introduction with R (2006), New York: Chapman and Hall/CRC, New York · Zbl 1087.62082
[57] Woodward, J. D., Impacts of weather and time horizon selection on crop insurance ratemaking: A conditional distribution approach, North American Actuarial Journal, 18, 2, 279-93 (2014) · Zbl 1414.91243
[58] Xie, J.; Girshick, R. B.; Farhadi, A., Proceedings of the 33rd International Conference on International Conference on Machine Learning, 48, Unsupervised deep embedding for clustering analysis, 478-87 (2016), New York: ACM Press, New York
[59] Yao, C.; Cai, D.; Bu, J.; Chen., G., Pre-training the deep generative models with adaptive hyperparameter optimization, Neurocomputing, 247, 144-55 (2017)
[60] You, J.; Li, X.; Low, M.; Lobell, D.; Ermon., S., Deep Gaussian process for crop yield prediction based on remote sensing data, AAAI Conference on Artificial Intelligence, 4559-66 (2017), Palo Alto, CA: AAAI, Palo Alto, CA
[61] Zaidi, N. A.; Webb, G. I.; Carman, M. J.; Petitjean, F.; Buntine, W.; Hynes, M.; Sterck., H., Efficient parameter learning of Bayesian network classifiers, Journal of Machine Learning Research, 106, 1289-1329 (2017) · Zbl 1454.62183
[62] Zhang, J., Machine learning and Its applicability to insurance, CAS Continuing Education Webinars (2015)
[63] Zheng, H.; Chen, L.; Han, Z.; Yan., M., Classification and regression tree (CART) for analysis of soybean yield variability among fields in Northeast China: The importance of phosphorus application rates under drought conditions, Agriculture Ecosystems & Environment, 132, 98-105 (2009)
[64] Zhu, W.; Porth, L.; Tan., K. S., A credibility-based yield forecasting model for crop reinsurance pricing and weather risk management, Agricultural Finance Review (2015)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.