horserule swMATH ID: 27913 Software Authors: Nalenz, Malte; Villani, Mattias Description: R package horserule: Tree ensembles with rule structured horseshoe regularization. We propose a new Bayesian model for flexible nonlinear regression and classification using tree ensembles. The model is based on the RuleFit approach in Friedman and Popescu [Ann. Appl. Stat.2 (2008) 916–954] where rules from decision trees and linear terms are used in a L1-regularized regression. We modify RuleFit by replacing the L1-regularization by a horseshoe prior, which is well known to give aggressive shrinkage of noise predictors while leaving the important signal essentially untouched. This is especially important when a large number of rules are used as predictors as many of them only contribute noise. Our horseshoe prior has an additional hierarchical layer that applies more shrinkage a priori to rules with a large number of splits, and to rules that are only satisfied by a few observations. The aggressive noise shrinkage of our prior also makes it possible to complement the rules from boosting in RuleFit with an additional set of trees from Random Forest, which brings a desirable diversity to the ensemble. We sample from the posterior distribution using a very efficient and easily implemented Gibbs sampler. The new model is shown to outperform state-of-the-art methods like RuleFit, BART and Random Forest on 16 datasets. The model and its interpretation is demonstrated on the well known Boston housing data, and on gene expression data for cancer classification. The posterior sampling, prediction and graphical tools for interpreting the model results are implemented in a publicly available \( t R\) package. Homepage: https://cran.r-project.org/web/packages/horserule/index.html Source Code: https://github.com/cran/horserule Dependencies: R Keywords: nonlinear regression; classification; decision trees; Bayesian; prediction; mcmc; interpretation Related Software: VBayesLab; BSL; synlik; abcrf; abc; QRM; GAMLSS; tokenizers; Medlda; BhGLM; GHS; BayesReg; horseshoe; sparsenet; PMA; softImpute; ISLR; EBayesThresh; ElemStatLearn; hierNet Cited in: 5 Publications Standard Articles 1 Publication describing the Software, including 1 Publication in zbMATH Year Tree ensembles with rule structured horseshoe regularization. Zbl 1412.62169Nalenz, Malte; Villani, Mattias 2018 all top 5 Cited by 13 Authors 2 Villani, Mattias 1 Bhadra, Anindya 1 Datta, Jyotishka 1 Jonsson, Leif 1 Klein, Nadja 1 Li, Longhai 1 Magnusson, Måns 1 Nalenz, Malte 1 Nott, David John 1 Polson, Nicholas G. 1 Smith, Michael Stanley 1 Willard, Brandon T. 1 Yao, Weixin Cited in 5 Serials 1 Statistical Science 1 Computational Statistics 1 Journal of Statistical Computation and Simulation 1 The Annals of Applied Statistics 1 Journal of Computational and Graphical Statistics Cited in 2 Fields 5 Statistics (62-XX) 1 Computer science (68-XX) Citations by Year