Perform a hypothesis test against the null hypothesis of \(\delta\) importance

Perform a hypothesis test against the null hypothesis of zero importance by: (i) for a user-specified level \(\alpha\), compute a \((1 - \alpha)\times 100\)% confidence interval around the predictiveness for both the full and reduced regression functions (these must be estimated on independent splits of the data); (ii) if the intervals do not overlap, reject the null hypothesis.

vimp_hypothesis_test(
  predictiveness_full,
  predictiveness_reduced,
  se,
  delta = 0,
  alpha = 0.05
)

Arguments

predictiveness_full

the estimated predictiveness of the regression including the covariate(s) of interest.

predictiveness_reduced

the estimated predictiveness of the regression excluding the covariate(s) of interest.

se

the estimated standard error of the variable importance estimator

delta

the value of the \(\delta\)-null (i.e., testing if importance < \(\delta\)); defaults to 0.

alpha

the desired type I error rate (defaults to 0.05).

Value

a list, with: the hypothesis testing decision (TRUE if the null hypothesis is rejected, FALSE otherwise); the p-value from the hypothesis test; and the test statistic from the hypothesis test.

Details

See the paper by Williamson, Gilbert, Simon, and Carone for more details on the mathematics behind this function and the definition of the parameter of interest.