In 2001, a very influential paper authored by Leo Breiman was published (with discussion and a rejoinder) in Statistical Science (vol.16, pp. 199–231) under the title “Statistical Modeling: The Two Cultures”-this paper has subsequently been cited more than 6,000 times. In this paper, Breiman considered a situation in which each of a number of known values (some of which may be identical) of a vector x of input variables gives rise to an observable value of an output variable y (“independently” via a process that is assumed to remain stable over the relevant period). And he defined two cultures among those in the relevant population based on how they approach these kinds of situations and their objectives in doing so. The first culture included those who adopted a statistical model for the relationship between y and x with the primary objective of using the model as a basis for making inferences about the underlying process, while the second culture included those who adopted an “algorithmic” approach with the primary objective of devising a procedure for predicting future values of y from the corresponding values of x—the inferences made by those in the first culture might include “predictive inferences.” Brieman went on to indicate that the first culture included most statisticians (98% of them) and that the second was populated primarily by non-statisticians (and included only 2% of statisticians).
The algorithmic approach leads to methods known as machine-learning methods and include neural nets as well as “tree-based” methods like random forests. At least in situations where little is known about the underlying process (which appears to be the kind of situations Breiman had in mind), the prediction methods obtained in this way appear to be markedly superior to the kind of prediction methods that would be obtained via a model-based approach, which (in the absence of any knowledge about the underlying process and in the case of a continuous y variable) might take the form of linear regression methods.In recent years, the algorithmic-based prediction methods have been widely applied with considerable success and have become quite popular. The response of the statistical community to this phenomenon has been one that could be characterized as “big tent”; that is, they have taken the position that these methods should be regarded as part of statistics and have embraced them accordingly. Not surprisingly, computer scientists and other non-statisticians (and the organizations that represent them) do not appear to be fully “on board” with this view.
There are many important prediction problems where there are meaningful relationships among the relevant quantities (both the quantities that are observable and those that are to be predicted) and about which there may be significant information. Even in the relatively simple kinds of situations considered by Breiman, it is easy to envision cases where there could exist meaningful relationships between various of the observed and/or future values of y and where there could exist significant information about those relationships. The relationships could be genetic relationships or as discussed by Noel Cressie in 2020 in his comments in International Statistical Review (vol. 88, pp. S65–S69) on a paper in that journal by Bradley Efron, they could be spatial and/or temporal in nature. They could also be of a kind that reflect what in a Bayesian formulation might be regarded as prior information. Such relationships can be expressed in the form of a hierarchical model—in my view of the world, a prior distribution is considered to be part of a model that is hierarchical in nature—and one can attempt to use that model to exploit the information inherent in the relationships by adopting a model-based approach. In fact, as I pointed out in a paper published (with discussion and a rejoinder) in 2014 in The American Statistician (vol. 68, pp. 71–92) under the title “The Need for More Emphasis on Prediction: A ‘Nondenominational’ Model-Based Approach,” such an approach has been widely adopted in many areas of application (e.g., animal breeding) and has proven to be quite successful.
It has long been the case (and continues to be the case) that the primary purpose of mainstream statistics is considered to be (as stated in an online account) to make inferences about a population based on a sample. That predictive inference should be regarded as central to mainstream statistics is a view promoted (so far not very successfully) by some past and present members of the statistical community including Seymour Geisser (who among other things authored the book Predictive Inference: An Introduction), W. E. Deming (who is said to have made some comments to the effect that the only useful function of a statistician is to make predictions and thus to provide a basis for action), and Bertrand Clarke and Jennifer Clarke (who recently authored a book entitled Predictive Statistics). My 2014 paper in The American Statistician likewise represented an attempt to promote such a view. That some of the key ideas in the development of the model-based approach to prediction originated with those in areas of application (e.g., C. R. Henderson in animal breeding) rather than within the mainstream statistical community could conceivably be regarded as reflecting a lack of interest on the part of that community. And the popularity of machine learning may have been enhanced and that of mainstream statistics diminished by the fact that prediction is considered to be central to machine learning—that is, “at least the machine learners are addressing the right problem.”
While prediction procedures of the kind obtained via an algorithmic approach may have been accepted and even embraced and adopted by much of the statistical community, there does not appear to be any greater awareness among those in this community of the many successes achieved in the use of prediction procedures of the kind obtained via a model-based approach nor any greater acceptance of the view that predictive inference should be regarded as an important part of mainstream statistics. What seems to be badly needed is a coherent presentation of predictive inference in a framework that is broad enough to encompass all of the different flavors of prediction (and that employs a single collection of uniform and well thought-out terminology and notation). What also seems to be badly needed is a broad spectrum of comparative studies that serves to identify the sorts of applications for which the different kinds of predictors are well-suited. The book by the Clarkes provides some coverage of the model-based approach to prediction as well as the algorithmic approach, however their presentation seems to reflect a rather limited knowledge of the literature on the model-based approach and its application and (perhaps as a result) leads to what strikes me as an overly pessimistic assessment of the range of applications for which a model-based approach is appropriate.
The prediction of basketball or football games is a prediction problem of a kind that can be of widespread interest, that can be very challenging, and that would seem to lend itself to a wide variety of approaches. Over the years, I have devised and occasionally refined or otherwise revised a procedure for predicting the outcomes of high school, college, or professional basketball or (American) football games. And from time to time, I have tested the accuracy of that procedure by applying it over the course of an entire basketball or football season. In recent years, I have focused primarily on college football. My college football predictions have been posted on the present website (davidharville.com) and have been based on an implementation of my procedure outlined below (under the heading Methodology).
My college football predictions are among a large number of such predictions posted and evaluated weekly by Todd Beck on the website thepredictiontracker.com. My predictions are model-based, as appears to be the case with a number of the other predictions posted on the prediction tracker website including those attributed to Jeff Sagarin. The prediction tracker website also includes the predictions produced by ESPN’s FPI (college Football Prediction Index)—while general descriptions of the FPI are available, the details are considered to be proprietary; nevertheless, based on what is known about the “founding fathers,” it can be conjectured that the FPI makes at least some use of machine learning methods. Among the other predictions included in the coverage are some obtained from an implementation of the Elo rating system (an implementation devised and maintained by Todd Beck)—the Elo rating system (which originated in the world of chess) could be regarded as a procedure that (while identified with sports-related applications and while somewhat limited in scope) is algorithmic in nature. Perhaps more significantly, the coverage of the prediction tracker website includes the predictions inherent in the betting lines maintained by a sports book. When it comes to sporting events (or other kinds of events that are subject to betting), the level of accuracy achieved by the predictions inherent in the betting lines (or the moneylines) is considered to be the “gold standard.” This form of prediction seems fundamentally different from other forms. Its relatively high level of accuracy is sometimes attributed to a phenomenon known as “the wisdom of the crowd.”
When it comes to comparing the accuracy of alternative predictors of the outcomes of events like college basketball or football games, there are considerations of “meaningfulness” or “comparability.” Once the predictions generated by any particular procedure become available, they could conceivably be included as “inputs” in other procedures. Thus, in making comparisons among prediction procedures that differ in various ways, one needs to insure that the comparisons are meaningful in the sense that the procedures being compared are the procedures that one truly wishes to compare. In the case of my college football predictions, I have chosen to limit the inputs to the outcomes of past games (including games from past years), to the identities of the opposing teams in each game, to the date on which each game was played, and to information for each game as to whether it was played at a “neutral” site and if not information as to which team was the “home” team—the outcome of each game is taken to consist of the difference in score between the opposing teams at the end of regulation play and when that difference is 0 (i.e., in the case of overtime) the identity of the winning team.