×

The characterization of Monte Carlo errors for the quantification of the value of forensic evidence. (English) Zbl 07192020

Summary: Recent developments in forensic science have lead to a proliferation of methods for quantifying the probative value of evidence by constructing a Bayes Factor that allows a decision-maker to select between the prosecution and defense models. Unfortunately, the analytical form of a Bayes Factor is often computationally intractable. A typical approach in statistics uses Monte Carlo integration to numerically approximate the marginal likelihoods composing the Bayes Factor. This article focuses on developing a generally applicable method for characterizing the numerical error associated with Monte Carlo integration techniques used in constructing the Bayes Factor. The derivation of an asymptotic Monte Carlo standard error (MCSE) for the Bayes Factor will be presented and its applicability to quantifying the value of evidence will be explored using a simulation-based example involving a benchmark data set. The simulation will also explore the effect of prior choice on the Bayes Factor approximations and corresponding MCSEs.

MSC:

62F15 Bayesian inference
62P99 Applications of statistics
65C05 Monte Carlo methods
65C60 Computational problems in statistics (MSC2010)

Software:

BayesDA; Flury; MCMCglmm
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] European Network of Forensic Science Institutes. ENFSI guideline for evaluative reporting in forensic science. Strengthening the evaluation of forensic results across Europe (STEOFRAE); 2015. [Google Scholar]
[2] National Research Council Committee on Identifying the Needs of the Forensic Sciences Community. Strengthening forensic science in the United States: a path forward. Washington (DC): The National Academies Press; 2009. [Google Scholar]
[3] Aitken C, Roberts P, Jackson G. Fundamentals of probability and statistical evidence in criminal proceedings; guidance for judges, lawyers, forensic scientists and expert witnesses. 1st ed.London: Royal Statistical Society’s Working Group on Statistics and the Law; 2010. [Google Scholar]
[4] Aitken C, Taroni F. Statistics and the evaluation of evidence for forensic scientists. 2nd ed.Chichester: John Wiley and Sons, Ltd.; 2004. [Crossref], [Google Scholar] · Zbl 1057.62118
[5] Cook R, Evett IW, Jackson G, et al. A hierarchy of propositions: deciding which level to address in casework. Sci Justice. 1998;38(4):231-239. doi: 10.1016/S1355-0306(98)72117-3[Crossref], [Web of Science ®], [Google Scholar]
[6] Cereda G. Bayesian approach to LR assessment in case of rare type match: careful derivation and limits. Report No.: arXiv:1502.02406v9. University of Lausanne/Leiden University; 2016. [Google Scholar]
[7] Cereda G. Impact of model choice on LR assessment in case of rare haplotype match (Frequentist Approach). Scand J Statist. 2016. doi: 10.1111/sjos.12250[Crossref], [Google Scholar] · Zbl 1361.62055
[8] Lunn D, Jackson C, Best N, et al. The BUGS book: a practical introduction to Bayesian analysis. Texts in Statistical Science. Boca Raton (FL): CRC Press; 2013. [Google Scholar] · Zbl 1281.62009
[9] Han C, Carlin BP. MCMC methods for computing Bayes factors: a comparative review. J Amer Statist Assoc. 2001;96:1122-1132. doi: 10.1198/016214501753208780[Taylor & Francis Online], [Web of Science ®], [Google Scholar]
[10] Taroni F, Bozza S, Biedermann A, et al. Dismissal of the illusion of uncertainty in the assessment of a likelihood ratio. Law Probab Risk. 2016;15(1):1-16. doi: 10.1093/lpr/mgv008[Crossref], [Web of Science ®], [Google Scholar]
[11] Curran JM, Buckleton JS, Triggs CM, et al. Assessing uncertainty in DNA evidence caused by sampling effects. Sci Justice. 2002;42(1):29-37. doi: 10.1016/S1355-0306(02)71794-2[Crossref], [PubMed], [Web of Science ®], [Google Scholar]
[12] Buckleton J, Triggs CM, Walsh SJ, editors. Forensic DNA evidence interpretation. Boca Raton (FL): CRC Press; 2005. [Google Scholar]
[13] Morrison GS. Measuring the validity and reliability of forensic likelihood-ratio systems. Sci Justice. 2011;51:91-98. doi: 10.1016/j.scijus.2011.03.002[Crossref], [PubMed], [Web of Science ®], [Google Scholar]
[14] Sjerps MJ, Alberink I, Bolck A, et al. Uncertainty and LR: to integrate or not to integrate, that’s the question. Law Probab Risk. 2016;15(1):23-29. doi: 10.1093/lpr/mgv005[Crossref], [Web of Science ®], [Google Scholar]
[15] Beecham GW, Weir BS. Confidence interval of the likelihood ratio associated with mixed stain DNA evidence. J Forensic Sci. 2011;56(S1):166-171. doi: 10.1111/j.1556-4029.2010.01600.x[Crossref], [Web of Science ®], [Google Scholar]
[16] Morrison GS, Enzinger E. What should a forensic practitioner’s likelihood ratio be? Sci Justice. 2016;56(5):374-379. doi: 10.1016/j.scijus.2016.05.007[Crossref], [PubMed], [Web of Science ®], [Google Scholar]
[17] Curran JM. Admitting to uncertainty in the LR. Sci Justice. 2016;56(5):380-382. doi: 10.1016/j.scijus.2016.05.005[Crossref], [PubMed], [Web of Science ®], [Google Scholar]
[18] Ommen DM, Saunders CP, Neumann C. An argument against presenting interval quantifications as a surrogate for the value of evidence. Sci Justice. 2016;56(5):383-387. doi: 10.1016/j.scijus.2016.07.001[Crossref], [PubMed], [Web of Science ®], [Google Scholar]
[19] Berger CEH, Slooten K. The LR does not exist. Sci Justice. 2016;56(5):388-391. doi: 10.1016/j.scijus.2016.06.005[Crossref], [PubMed], [Web of Science ®], [Google Scholar]
[20] Biedermann A, Bozza S, Taroni F, et al. Reframing the debate: a question of probability, not of likelihood ratio. Sci Justice. 2016;56(5):392-396. doi: 10.1016/j.scijus.2016.05.008[Crossref], [PubMed], [Web of Science ®], [Google Scholar]
[21] van den Hout A, Alberink I. Posterior distributions for likelihood ratios in forensic science. Sci Justice. 2016;56(5):397-401. doi: 10.1016/j.scijus.2016.06.011[Crossref], [PubMed], [Web of Science ®], [Google Scholar]
[22] Taylor D, Hicks T, Champod C. Using sensitivity analyses in Bayesian networks to highlight the impact of data paucity and direct future analyses: a contribution to the debate on measuring and reporting the precision of likelihood ratios. Sci Justice. 2016;56(5):402-410. doi: 10.1016/j.scijus.2016.06.010[Crossref], [PubMed], [Web of Science ®], [Google Scholar]
[23] Kass RE, Raftery AE. Bayes factors. J Amer Statist Assoc. 1995;90(430):773-795. doi: 10.1080/01621459.1995.10476572[Taylor & Francis Online], [Web of Science ®], [Google Scholar] · Zbl 0846.62028
[24] Newton MA, Raftery AE. Approximate Bayesian inference with the weighted likelihood bootstrap. J R Stat Soc Ser B Methodol. 1994;56(1):3-48. [Google Scholar] · Zbl 0788.62026
[25] Tanner MA. Tools for statistical inference: methods for the exploration of posterior distributions and likelihood functions. 3rd ed.New York (NY): Springer; 1996. [Crossref], [Google Scholar] · Zbl 0846.62001
[26] Geweke J. Bayesian inference in econometric models using Monte Carlo integration. Econometrica. 1989;57(6):1317-1339. doi: 10.2307/1913710[Crossref], [Web of Science ®], [Google Scholar] · Zbl 0683.62068
[27] Serfling RJ. Approximation theorems of mathematical statistics. Wiley Series in Probability and Statistics. New York: John Wiley and Sons, Ltd.; 1980. [Crossref], [Google Scholar] · Zbl 0538.62002
[28] Neal R. The harmonic mean of the likelihood: worst Monte Carlo method ever; 2008 [cited 2014 May 15]. Available from: http://radfordneal.wordpress.com/2008/08/17/the-harmonic-mean-of-the-likelihood-worst-monte-carlo-method-ever/[Google Scholar]
[29] Aitken CGG, Lucy D. Evaluation of trace evidence in the form of multivariate data. J Roy Statist Soc Ser C. 2004;53(1):109-122. doi: 10.1046/j.0035-9254.2003.05271.x[Crossref], [Web of Science ®], [Google Scholar] · Zbl 1111.62390
[30] Dataset from the Journal of the Royal Statistical Society: Series C; 2004 [cited 2016 Apr 20]. Available from: http://onlinelibrary.wiley.com/store/10.1111/(ISSN)1467-9876/asset/homepages/glass-data.txt?v=1&s=e9bb566557f519b28a40918e8016affc52981802&isAguDoi&false[Google Scholar]
[31] Ferguson TS. Mathematical statistics: a decision theoretic approach. New York: Academic Press; 1967. [Google Scholar] · Zbl 0153.47602
[32] Gelman A, Carlin JB, Stern HS, et al. Bayesian data analysis. 3rd ed.Texts in Statistical Science. Boca Raton (FL): CRC Press; 2014. [Google Scholar] · Zbl 1279.62004
[33] Izenman AJ. Modern multivariate statistical techniques: regression, classification, and manifold learning. New York: Springer Texts in Statistics; 2013. [Google Scholar] · Zbl 1155.62040
[34] Anderson TW. An introduction to multivariate statistical analysis. 3rd ed.Hoboken (NJ): John Wiley and Sons, Ltd.; 2003. [Google Scholar] · Zbl 1039.62044
[35] Miller JJ. Asymptotic properties of maximum likelihood estimates in the mixed model of the analysis of variance. Ann Statist. 1977;5(4):746-762. doi: 10.1214/aos/1176343897[Crossref], [Web of Science ®], [Google Scholar] · Zbl 0406.62017
[36] Dettman JR, Cassabaum AA, Saunders CP, et al. Forensic discrimination of copper wire using trace element concentrations. Anal Chem. 2014;86:8176-8182. doi: 10.1021/ac5013514[Crossref], [PubMed], [Web of Science ®], [Google Scholar]
[37] Hadfield JD. MCMC methods for multi-response generalised linear mixed models: the MCMCglmm R package. J Statist Softw. 2010;33(2):1-22. doi: 10.18637/jss.v033.i02[Crossref], [PubMed], [Web of Science ®], [Google Scholar]
[38] Hepler AB, Saunders CP, Davis LJ, et al. Score-based likelihood ratios for handwriting evidence. Forensic Sci Int. 2012;219(1):129-140. doi: 10.1016/j.forsciint.2011.12.009[Crossref], [PubMed], [Web of Science ®], [Google Scholar]
[39] Berger JO, Bernardo JM, Sun D. The formal definition of reference priors. Ann Statist. 2009;37(2):905-938. doi: 10.1214/07-AOS587[Crossref], [Web of Science ®], [Google Scholar] · Zbl 1162.62013
[40] Robert CP. The Bayesian choice: from decision-theoretic foundations to computational implements. Springer Texts in Statistics. New York: Springer; 2001. [Google Scholar] · Zbl 0980.62005
[41] Flury B. A first course in multivariate statistics. Springer Texts in Statistics. New York: Springer-Verlag; 1997. [Crossref], [Google Scholar] · Zbl 0879.62052
[42] Saunders CP. Empirical processes for estimated projections of multivariate normal vectors with applications to E.D.F. and correlation type goodness of fit tests [dissertation]. University of Kentucky; 2006. [Google Scholar]
[43] Fisher RA. Statistical methods for research workers. 5th ed.Edinburgh: Oliver and Boyd; 1934. [Google Scholar] · JFM 60.1162.01
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.