Research Methodology | Experimentation

Discover research methods like A/B testing, field experiments, and observation. Learn their pros, cons, and best use cases with this visual guide.


Infographic: A Guide to Research Methodologies

The Researcher's Compass

A visual guide to navigating your testing options, from passive observation to active experimentation.

🔍

Observational Study

Understand user behavior in its natural context without intervention.

Key Characteristics

✓Pros

  • High ecological validity
  • Excellent for discovery
  • Identifies real-world pain points

✗Cons

  • Cannot determine causality
  • Prone to observer bias
  • Can be time-consuming

Implementation Flow

Define Goals
→
Collect Data
→
Analyze & Synthesize
→
Form Hypotheses
🌍

Field Experiment

Test the causal impact of a change in a real-world setting.

Key Characteristics

✓Pros

  • Strong causal inference
  • High ecological validity
  • Results are generalizable

✗Cons

  • Less control than a lab
  • Can be complex & costly
  • Ethical considerations

Implementation Flow

Define Hypothesis
→
Randomize Users
→
Deploy Change
→
Measure & Analyze
📊

A/B Testing

Directly compare two versions to see which performs better.

Key Characteristics

✓Pros

  • Very strong causal inference
  • Highly scalable & quantitative
  • Low cost per test

✗Cons

  • Focuses on local maxima
  • Requires significant traffic
  • Doesn't explain "why"

Implementation Flow

Create Variations
→
Split Traffic
→
Run & Collect Data
→
Analyze & Implement

Method Comparison

A visual summary of the trade-offs. Each method is rated on a scale of 1 (Low) to 5 (High).

Key Decision Factors

  • Causal Strength: Need to prove A caused B? Choose an experiment.
  • Ecological Validity: Need behavior in a real context? Choose field studies.
  • Control: Need to isolate variables? A/B tests offer the most control.
  • Scalability: Have lots of users? Digital methods like A/B testing scale well.
  • Cost & Time: Limited resources? Start with observation, which is often cheaper.

Other Methods

Lab Experiment

Maximum control, but artificial setting. Best for isolating specific usability issues.

Survey / Questionnaire

Gathers self-reported attitudes. Good for 'what people think', not 'what they do'.

User Interview

Deep qualitative insights. Best for understanding the 'why' behind behaviors.

```

Asociation-to-understanding    Association-to-understanding    Observationstudy-fieldexperim    Research-methodology