Model Invariants and Functional Regularization
Abstract
When modeling data, we would like to know that our models are extracting facts about the data itself, and not about something arbitrary, like the order of the factors used in the modeling. Formally speaking, this means we want the model to be invariant with respect to certain transformations. Here we look at different models and the nature of their invariants. We find that regression, MLE and Bayesian estimation all are invariant with respect to linear transformations, whereas regularized regressions have a far more limited set of invariants. As a result, regularized regressions produce results that are less about the data itself and more about how it is parameterized. To correct this, we propose an alternative expression of regularization which we call functional regularization. Ridge regression and lasso are special cases of functional regularization, as is Bayesian estimation. But functional regularization gives a framework under which the models become invariant with respect to linear transformations. It is also more flexible, and easier to understand, thus yielding a number of advantages over ridge regression and lasso.
Watch recording ↗About the speaker
Harvey Stein is a senior VP in the Labs group in Two Sigma where he works on various research projects around the firm. From 1993 to 2022, Dr. Stein was at Bloomberg, where he served as the head of several departments including Quantitative Risk Analytics, Counterparty and Credit Risk, Interest Rates Derivatives, and Quantitative Finance R&D. Harvey is well known in the industry, having published and lectured on credit risk modeling, financial regulation, interest rate and FX modeling, CVA calculations, mortgage backed security valuation, COVID-19 data analysis, and other subjects. Dr. Stein is on the board of directors of the IAQF, a board member of the Rutgers University Mathematical Finance program, an adjunct professor at Columbia University, and organizer of the IAQF/Thalesians financial seminar series. He's also worked as a quant researcher on the Bloomberg for President campaign. Harvey holds a Ph.D. in Mathematics from the University of California, Berkeley (1991) and a B.S. in Mathematics from Worcester Polytechnic Institute (1982).