Why?

RCTs are expensive:

• Median cost of Phase 3 trials: $19 million (IQR: $12.2 - $33.1M)
• Avg. cost of bringing new drug to market ~ 1 - 3 billion
• May not generalize to larger “real world” population

May not be practical or ethically feasible

Real World

Real World Data RWD:

• Observational, collected in “real world” setting
• EHR database, hospital visit notes, wearable devices
• Medical claims/billing database, disease registries

Real World Evidence RWE:

• Clinical evidence about the benefits/harms of medical products derived from analyzing RWD (Franklin et al. 2021)
• Poor quality RWD: garbage-in-garbage-out
• 90 examples of RWE used by FDA

Potential outcomes causal framework

• 2 exposure groups: e.g. Treatment vs Control
• How does the exposure affect a given outcome \(Y\)?
• The \(i\)th subject has 2 potential outcomes: \[Y_i(T) \text{ and } Y_i(C)\]
• For each subject, the treatment effect is defined as: \[Y_i(T) - Y_i(C)\]
  • Only 1 of these is observed in reality

(Little and Rubin 2000; Rubin 2005)

Example

• In RCT, \(\tau = \bar{Y}_i(T) - \bar{Y}_i(C)\) estimates a causal effect

• In general, \(\tau\) is not causal for observational studies (OS)

  • Many methods try to change this (Austin 2011a; Varga et al. 2023)

Propensity scores (PS)

“Probability of treatment assignment based on observed baseline covariates” (Rosenbaum and Rubin 1983)

• Given treatment variable \(X_i \in \{0,1\}\) and baseline covariates \(\boldsymbol{Z}_i\), then estimate PS:

\[ PS_i = Pr(X_i = 1) = f(\boldsymbol{Z}_i) + \epsilon_i \qquad(1)\]

Logistic regression or machine learning to estimate Equation 1

Why model treatment assignment?

Without a model for how treatments are assigned to units, formal causal inference is impossible (Little and Rubin 2000)

4 methods of using PS

The PS is a balancing score: patients with similar PS should have similar baseline covariates (Austin 2011a)

Matching: many choices

Recommendations from simulations of (Austin 2014a):

• Optimal vs greedy nearest neighbor matching?

Caliper or no caliper?

https://help.easymedstat.com/support/solutions/articles/77000538175-caliper-in-propensity-score-matching

• How wide should caliper be? 0.2*SD of logit PS (Austin 2011b)

• Match with or without replacement?

• For greedy matching, what order should you select the treated subjects?
  • Lowest to highest PS, highest to lowest, best match first, or random order

Types of treatment effects

ATE: \(E\big[Y_i(T) - Y_i(C)\big]\)

• Average effect for ALL patients in the population

ATT: \(E\big[Y_i(T) - Y_i(C)\big | T]\)

• Among patients who were Treated, how would their outcomes have changed if they were a Control?

ATC: \(E\big[Y_i(T) - Y_i(C)\big | C]\)

• Among patients who were Controls, how would their outcomes have changed if they were Treated?

(Deb et al. 2016)

Employment training and income

• Treatment (N = 185): National Supported Work Demonstration (NSW) employment training program
• Control (N = 429): from the Population Survey of Income Dynamics (PSID)
• Goal: Does training program increase mean income (1978)?
• Baseline covariates: age, education, race, marital status, pre-treatment income (1974/1975)

Nearest neighbor caliper matching

library(MatchIt);

set.seed(1234);
match.nnc.logit <- matchit(
    treat ~.,
    data = psdata,
    method = 'nearest',
    distance = 'glm',
    replace = FALSE,
    caliper = 0.2,
    std.caliper = TRUE,
    ratio = 1 # 1:1 matching
    );

PS distribution before/after matching

• Notice poor overlap before matching ➔ won’t match all Trt patients, unless \(N_{control}\) is large

• ⬆ unmatched Trt patients = ⬆ biased \(\widehat{\text{ATT}}\), instead estimates “Average treatment effect in the Overlap population (ATO)” (Varga et al. 2023)

Covariate balance

Standardized differences

Estimated treatment effect

• Interestingly, PS estimate of trt effect ($1572) is close to RCT estimate ($1641) (LaLonde 1986)

Optimal matching

• Matches 100% of Trt patients (unlike greedy caliper match)
• Attempts to optimize the total distance between all pairs
• However, resulted in much worse covariate balance

Summary

• Goal: estimate causal effect of exposure \(X\) on outcome \(Y\), using observational data
• PS matching balances measured confounders between exposure groups (similar to RCT)
  • Intuitive: show covariates are balanced after matching, like RCT
  • Of course, unmeasured confounders may remain
• If estimating ATT (ATC), then need to match all Trt (Control) patients. If high % unmatched, consider other methods.

Alternatives

• Stratification on PS uses all patients, but does not work well with survival data (Austin 2014b, 2016)

• Inverse probability weighting and regression adjustment are very flexible, but generally not accepted by FDA (unlike matching) (Lu 2019)

• Causal Random Forests look promising: https://grf-labs.github.io/grf/