"Cracking Two-Sided Non-Compliance Challenges"

Two-sided non-compliance in field experiments occurs when participants in both treatment and control groups deviate from their assigned conditions, complicating causal inference. Researchers employ strategies like Intention-to-Treat (ITT) analysis, Complier Average Causal Effect (CACE), and Instrumental Variables (IV) to address this challenge and ensure valid conclusions.


Two-Sided Non-Compliance in Field Experiments

Field experiments are a powerful method used by researchers to estimate causal relationships in real-world settings. However, in practice, such experiments often face challenges, one of which is non-compliance. Non-compliance occurs when participants do not adhere to their assigned treatment conditions. While one-sided non-compliance involves only one group deviating from their assignment, two-sided non-compliance occurs when participants from both the treatment and control groups fail to follow their assignments. This phenomenon can complicate the interpretation and analysis of experimental results.

Key Concepts Description
Definition of Two-Sided Non-Compliance

Two-sided non-compliance arises when participants from both the treatment and control groups deviate from their assigned conditions. For example, some individuals assigned to the treatment group may fail to receive or adopt the intervention (non-compliers), while some individuals in the control group may gain access to the intervention despite not being assigned to it (crossovers). This dual deviation creates significant challenges for causal inference.

Example Scenario

Consider a field experiment designed to evaluate the effectiveness of a new educational program. Students are randomly assigned to either the treatment group (who participate in the program) or the control group (who do not). In the case of two-sided non-compliance:

  • Some students in the treatment group may skip or fail to attend the program sessions (non-compliance).
  • Some students in the control group may gain access to the program through other means, such as enrolling independently (crossover).
Implications of Two-Sided Non-Compliance

Two-sided non-compliance poses challenges for data analysis and interpretation, as it undermines the clarity of treatment-control group comparisons. The treatment effect estimated under non-compliance may not reflect the true causal impact of the intervention. Specifically:

  • Non-compliance dilutes the estimated treatment effect, potentially making the intervention appear less effective than it actually is.
  • Crossovers in the control group introduce bias, as the control group no longer serves as a pure baseline for comparison.
Analytical Strategies to Address Two-Sided Non-Compliance

Researchers employ several strategies to address and account for two-sided non-compliance in field experiments:

  • Intention-to-Treat (ITT) Analysis: This approach estimates the treatment effect based on the original random assignments, regardless of whether participants complied with their assignments. ITT analysis provides a conservative estimate of the intervention's impact.
  • Complier Average Causal Effect (CACE): Also known as the Local Average Treatment Effect (LATE), this method focuses on the subset of participants who complied with their assignments. CACE provides an estimate of the treatment effect for compliers.
  • Instrumental Variables (IV): Researchers can use random assignment as an instrumental variable to estimate the causal effect of the treatment on the treated population, accounting for non-compliance.
  • Sensitivity Analysis: Conducting sensitivity analyses can help assess the robustness of the estimated treatment effect under different assumptions about non-compliance.
Conclusion

Two-sided non-compliance is a common challenge in field experiments, particularly in real-world settings where participant behavior cannot be strictly controlled. Recognizing and addressing this issue is critical to maintaining the validity of causal inferences. By employing robust analytical strategies, researchers can mitigate the impact of non-compliance and derive meaningful insights from their experiments.


Infographic: Two-Sided Noncompliance

THE ENCOURAGEMENT DILEMMA

In many experiments, we can't force people to act. We can only randomize an *encouragement*. Some who are encouraged won't act, and some who *aren't* encouraged will act anyway. This is **Two-Sided Noncompliance**, and it makes finding the true effect of a treatment tricky.

Who's Who in the Population?

To solve this, we imagine everyone belongs to one of four hidden groups (**Principal Strata**) based on how they'd react to being encouraged or not.

✓

Compliers

Take the treatment *if and only if* encouraged. They follow directions.

→

Always-Takers

Take the treatment *no matter what*. They are self-motivated.

✗

Never-Takers

Will *never* take the treatment, even if encouraged.

?

Defiers

Do the *opposite* of what they are encouraged to do. (Assumed to be rare).

What Are We Actually Measuring?

With noncompliance, we must choose our question carefully. We can measure the effect of the *offer* or the effect of the *treatment itself*—but only for one group.

Intention-to-Treat (ITT)

The effect of the ENCOURAGEMENT.

This compares everyone assigned to the encouragement group vs. everyone assigned to control. It's a real-world policy effect, but it's "diluted" by people who don't follow their assignment.

Complier Average Causal Effect (CACE)

The effect of the TREATMENT for Compliers.

This estimates the true effect of the treatment itself, but only for the Compliers—the group whose behavior was actually changed by the encouragement.

Case Study: The Vietnam Draft Lottery

A classic example used a random lottery number as an "encouragement" (instrument) for military service. This allows us to estimate the causal effect of serving in the military on later-life mortality.

Effect of Draft Eligibility on Mortality

This is the Intention-to-Treat effect on the outcome ($ITT_Y$). It shows the raw effect of the encouragement.

Effect of Draft Eligibility on Service Rate

This is the Intention-to-Treat effect on participation ($ITT_D$). It shows how much the encouragement changed behavior.

The CACE Formula: Un-diluting the Effect

The logic to find the true effect on Compliers (CACE) is to simply "inflate" the diluted ITT effect by the compliance rate. This is called the Wald Estimator.

Effect on Mortality ($ITT_Y$)

Effect on Service Rate ($ITT_D$)

=

Effect for Compliers (CACE)

The Final Results

Plugging in the numbers from the draft lottery reveals the true impact of military service for those induced to serve.

Intention-to-Treat (ITT) Effect

+0.6%

Being draft-eligible increased mortality by 0.6 percentage points. This is the diluted policy effect.

Complier Average Causal Effect (CACE)

+4.0%

For Compliers (men who served *because* of the draft), military service increased mortality by 4.0 percentage points.

2-causal-inference    3-hypothesis-testing    4-covariates    5-one-sided-compliance    6-two-sided-compliance    7-attrition    8-mediation    9-spillover    Field-experiment-protection-o