Inference for model-agnostic variable importance

class: center, middle, title-slide

# Inference for model-agnostic variable importance
### Brian D. Williamson, PhD<br> <span style="font-size: 75%;"> Fred Hutchinson Cancer Research Center </span>
### 4 May, 2021<br> <a href = 'https://bdwilliamson.github.io/#talks' style = 'color: white;'>https://bdwilliamson.github.io/#talks</a>

---

## Research team

This work was done in collaboration with:
<img src="img/people1.PNG" width="65%" style="display: block; margin: auto;" />
<img src="img/people2.PNG" width="55%" style="display: block; margin: auto;" />

---

## Motivation: HIV envelope and antibody targets

.small[Source: Koff and Berkley (2010)]

---

## Motivation: AMP

AMP overall objective: assess .blue2[VRC01] .blue1[prevention efficacy] (PE) against HIV-1

* &zwj;VRC01: broadly neutralizing antibody (bnAb) isolated from donor

Key secondary question:

.green[Which genetic mutations] make HIV-1 .purple[susceptible] to neutralization?

&zwj;Challenges:
* How should we measure susceptibility?
--

* How do we determine if a mutation has a real effect?
--

at .red[many] positions?
--

* Can we use .mutedred[machine learning]?

---

## Variable importance: what and why

**What is variable importance?**
* .blue1[Quantification of "contributions" of a variable] (or a set of variables)

Traditionally: contribution to .blue2[predictions]
--

* Useful to distinguish between contributions of predictions...
--

* (.blue1[extrinsic importance]) ... .blue1[by a given (possibly black-box) algorithm]
  .small[ [e.g., Breiman, (2001)] ]
--

* (.blue1[intrinsic importance]) ... .blue1[by best possible (i.e., oracle) algorithm]
  .small[ [e.g., van der Laan (2006)] ]
--

* Our work focuses on .blue1[interpretable, model-agnostic intrinsic importance]

Example uses of .blue2[intrinsic] variable importance:
* is it worth extracting text from notes in the EHR for the sake of predicting hospital readmission?

* is it worth collecting a given covariate for the sake of predicting neutralization sensitivity?

---

## Case study: ANOVA importance

Data unit `\((X, Y)\)` with:
* outcome `\(Y\)` 
* covariate `\(X := (X_1, X_2, \ldots, X_p)\)`

**Goals:** 
* .green[estimate]
* .blue1[and do inference on]

the importance of `\((X_j: j \in s)\)` in predicting `\(Y\)`

How do we typically do this in **linear regression**?

---

## Case study: ANOVA importance

How do we typically do this in **linear regression**?

* Fit a linear regression of `\(Y\)` on `\(X\)` `\(\rightarrow \color{magenta}{\hat{\mu}(X)}\)`
--

* Fit a linear regression of `\(Y\)` on `\(X_{-s}\)` `\(\rightarrow \color{magenta}{\hat{\mu}_s(X)}\)`
--

* .green[Compare the fitted values] `\([\hat{\mu}(X_i), \hat{\mu}_s(X_i)]\)`

Many ways to compare fitted values, including:
* ANOVA decomposition
* Difference in `\(R^2\)`

Can extend this to a .blue1[nonparametric] measure using .blue2[population quantities]

Can do .blue1[inference] on this parameter, even if we used .green[machine learning]

.small[[details in Williamson et al. (2020a)]]

---

## A unified framework

Our proposed general framework for inference on **model-agnostic** variable importance:
--

1. Choose a .blue2[scientifically relevant] measure of .blue1[predictiveness] `\(V\)`

* Large `\(V\)` = good predictions
  * e.g., classification accuracy, AUC, `\(R^2\)`

2. Define variable importance as

`\(V(\)` .blue1[best possible] prediction function using .green[all variables] `\() -\)`
  
  `\(V(\)` .blue1[best possible] prediction function .red[excluding the variables of interest] `\()\)`

.blue1[Inference] possible even with machine learning! .small[ [Details in Williamson et al. (2020b)] ]

Implemented in `R` package `vimp` (available on `CRAN`!)

---

## Preparing for AMP

&zwj;Implications:

* All sites in the VRC01 binding footprint, CD4 binding sites appear important

* Results may differ based on chosen measure

.small[ _Data sourced from the CATNAP database (Yoon et al., 2015)_ ]

---

## Extensions and future work

Handling potentially correlated features:

Assessing neutralization potential of combination bnAb regimens:

Variable selection using population importance: my postdoctoral focus

---

## Working towards equity, diversity, and inclusion

My personal goal: work towards EDI in biomedical research and practice

**Educational outreach:** [Hutch United Outreach Committee](https://www.fredhutch.org/en/research/education-training/hutch-united.html)

**Mentoring:** [Cientifico Latino](https://www.cientificolatino.com/gsmi), [Pomona College alumni mentor](https://www.sagepost47.com/)

**Research:**
  * working with historically underserved communities
  * working to reduce health disparities
  * increasing access to high-quality preventive care

---

## HU + SCC MESA Partnership

.small[ _Images courtesy of Stephanie Shadbolt_ ]
---