ML10.6 · LESSON 6 OF 6

Causal reasoning

Distinguish sequential decision-making, temporal prediction, recommendation and causal estimation.

THE IDEA

Seeing versus doing

A prediction asks: given that I observe this, what is likely? A causal question asks: what happens if I change this? They differ whenever something else influences both. Umbrellas and rain are strongly correlated, and banning umbrellas would not stop the rain, because rain causes umbrellas, not the other way around.

Arrows point from cause to effect. Course difficulty points to both tutoring and passing, so it confounds the comparison of tutored and untutored students.

Predict first. Umbrella sales and rainfall are correlated. Does that show umbrellas cause rain?

Simpson's reversal: the confounder flips the answer

Suppose students choose tutoring, and students in hard courses are more likely to choose it. Tutored students may then do worse overall simply because they took harder courses, even though tutoring helps inside every course. Comparing the groups directly is misleading. Stratifying compares like with like: measure the difference inside each course and average the results.

group	tutored passed	not tutored passed	gap
all 400 students	108/200 = 54%	150/200 = 75%	-21 points

Tutored students pass less often overall: a gap of -21 points. Does tutoring hurt?

The same counts, read two ways. The overall comparison and the within-course comparison point in opposite directions.

Chapter context

These topics change the structure of the learning problem. Their first lesson is to identify what a prediction or action is allowed to depend on and how feedback is produced. Each section is a bridge to a larger specialist subject.

TRY IT YOURSELF

Part 1. Do it hands-on

You decide how students are assigned to tutoring and which estimate to read. Build a reversal, then remove the confounding.

True pass rates never change: easy course 90% tutored vs 80% not, hard course 50% tutored vs 30% not. You only choose who gets tutored.

hard-course share among tutored50%hard-course share among not tutored50%

tutored pass rate = 0.5 x 0.5 + 0.5 x 0.9 = 0.7

not tutored pass rate = 0.5 x 0.3 + 0.5 x 0.8 = 0.55

naive gap = 0.15

adjusted (0.5 x 0.10 + 0.5 x 0.20) = 0.15

Estimate shown (naive): +15 points - it says tutoring helps

To do: Recreate Simpson's reversal: assign tutoring so the naive comparison is negative while the true effect inside each course is positive
To do: With the reversal in place, switch to the stratified (adjusted) estimate and read the sign of the true effect
To do: Remove the confounding by assignment: give both groups the same share of hard-course students, but not 50%. The naive gap is then just a blend of the two within-course effects

Part 2. Do it in code

Each record is a tuple (stratum, treated, outcome) with treated and outcome equal to 0 or 1. Write naive_effect(records): mean outcome of treated minus mean outcome of control, ignoring strata. Return None if either group is empty.
Write stratified_effect(records): compute the treated-minus-control difference inside every stratum that has both a treated and a control record, then average them weighted by the number of records in each stratum (among the usable strata). Strata missing a group are skipped. Return None if no stratum is usable.
Write group_rates(records): a dict stratum -> (treated rate, control rate), with None for an empty group.

def naive_effect(records):
    pass

def stratified_effect(records):
    pass

def group_rates(records):
    pass

demo = [("a", 1, 1), ("a", 0, 0)]
print(naive_effect(demo))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does a correlation between umbrellas and rain imply umbrellas cause rain?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

G_t=\sum_{k\ge0}\gamma^k r_{t+k+1},\quad Q(s,a)\leftarrow Q(s,a)+\alpha[r+\gamma\max_{a'}Q(s',a')-Q(s,a)].

For a terminal transition the continuation term is zero. A policy-gradient estimator uses a term proportional to $\nabla\log\pi_\theta(a\mid s)$ times a return or advantage estimate.

Worked example

For a nonterminal transition, current Q=2, reward=1, discount=0.9 and best next Q=4 give target 1+0.9×4=4.6. With alpha=0.5, updated Q is 2+0.5×(4.6−2)=3.3. Reusing the same target for a terminal transition would be incorrect.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
q, reward, discount, next_q, rate = 2, 1, .9, 4, .5
target = reward + discount*next_q
assert isclose(target, 4.6)
assert isclose(q+rate*(target-q), 3.3)
terminal_target = reward
assert q+rate*(terminal_target-q) == 1.5

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Prediction is not intervention. Randomly splitting overlapping time windows can leak future information. An environment reward is a modeling choice and may omit important real-world consequences.