Biostatistics Seminar: Jacob Bien

Presentation: High-Dimensional Variable Selection When Features are Sparse

Speaker: Jacob Bien, Ph.D., Assistant Professor of Data Sciences and Operations, USC Marshall School of Business

Abstract: It is common in modern prediction problems for many predictor variables to be counts of rarely occurring events. This leads to design matrices in which a large number of columns are highly sparse. The challenge posed by such “rare features” has received little attention despite its prevalence in diverse areas, ranging from biology (e.g., rare species) to natural language processing (e.g., rare words). We show, both theoretically and empirically, that not explicitly accounting for the rareness of features can greatly reduce the effectiveness of an analysis. We next propose a framework for aggregating rare features into denser features in a flexible manner that creates better predictors of the response. An application to online hotel reviews demonstrates the gain in accuracy achievable by proper treatment of rare words.

This is joint work with Xiaohan Yan.

Thu, Feb 8, 2018, 3:30pm to 5:00pm
Room T-639 (HST)

Faculty Coordinator: Amy Willis,

Seminar Coordinator: Sandra Coke,