Code
library(dplyr)
library(tidyr)
library(ggplot2)
library(broom)
library(purrr)
set.seed(1985)An explanation with a simulation
Ashenfelter and Card (1985) ask how to estimate the effect of a job-training program on earnings when participants are not randomly assigned. Their data are longitudinal: they have annual earnings histories for trainees and for a comparison group that are drawn from the general population.
Two features of participant earnings make the problem hard:
If the transitory component of earnings is serially correlated (a temporary earnings shock today tends to carry over into tomorrow), then a low draw in the selection year (the worker happens to receive an unusually negative temporary earnings shock in that year) predicts low earnings in neighbouring years too. Any estimator that uses the selection year (or its neighbours) as the “before” period will mix up recovery from a bad shock with the treatment effect.
As previously stated, the evaluation of the training program in the paper does not correspond to a randomized experiment. Nobody assigned people to the program by lottery, so the authors had to work with observational data.
Suppose the earnings process is
\[ Y_{it} = \alpha_i + d_t + \beta D_{it} + \epsilon_{it}, \]
where
Participation depends on earnings in a selection year \(T-k\), so it depends on \(\alpha_i\) and on \(\epsilon_{i,T-k}\). Under the AR(1) process,
\[ E[\epsilon_{it} \mid \epsilon_{i,T-k}] = \rho^{|t-(T-k)|}\,\epsilon_{i,T-k}, \]
so among participants (who were selected for having low \(\epsilon_{i,T-k}\)) the expected transitory component is negative around the selection year and gradually returns to zero.
Suppose the transitory component follows an AR(1) process, \(\varepsilon_{it}=\rho\,\varepsilon_{i,t-1}+u_{it}\) with \(\rho>0\): a temporary shock fades gradually instead of disappearing after one year.
Take a worker who enters training in 1976 because 1975 was unusually bad (illness, cut hours, a struggling employer), say \(\varepsilon_{i,1975}=-1000\). With \(\rho=0.7\), the expected shock is \(-700\) in 1974 and 1976, and \(-490\) in 1973 and 1977.
In both directions, \[ E[\varepsilon_{it}\mid\varepsilon_{i,1975}]=\rho^{|t-1975|}\,\varepsilon_{i,1975}. \]
Two problems arise:
The authors suggest a symmetric fix: 1973 and 1977 are equally far from 1975 and carry the same expected shock (\(-490\)), so comparing them cancels it.
Let us consider three stages, each showing a specific estimator.
The naive estimator compares participants’ average earnings after the program with their average earnings before it. It needs no comparison group, but it attributes to the program everything that changed over time for participants, including aggregate growth and the rebound from the pre-program dip.
\[ \hat\beta_{\text{naive}} = E(Y_{\text{post}} - Y_{\text{pre}} \mid D=1). \]
This ignores aggregate growth (\(d_t\)) and, if “pre” is the dip year, attributes the rebound from the dip to the program.
The conventional estimator subtracts from the participants’ before-after change the same change for a comparison group. This removes common time effects and permanent differences between the groups. However, if the base year is close to the selection year, the participants’ rebound from the bad draw remains in the estimate.
\[ \hat\beta_{\text{DiD}} = E(Y_{\text{post}} - Y_{\text{pre}} \mid D=1) - E(Y_{\text{post}} - Y_{\text{pre}} \mid D=0). \]
This removes common time effects and permanent differences \(\alpha_i\). But with a serially correlated \(\epsilon_{it}\) and a base year close to the selection year, the trainees’ rebound is still in the estimate. The paper shows that one can compute a different DiD estimate for each possible pre-program base year, and these estimates differ.
The symmetric estimator also uses a comparison group, but it chooses the pre-program year so that it is as far before the selection year as the post-program year is after it. With selection in period \(T-k\), this gives
\[ \boxed{ \hat\beta_{AC} = \left[E(Y_{T+1} - Y_{T-2k-1} \mid D=1)\right] - \left[E(Y_{T+1} - Y_{T-2k-1} \mid D=0)\right] } \]
The selection year \(T-k\) is the midpoint of \([T-2k-1,\,T+1]\), so both endpoints are \(k+1\) years away from it. Under stationarity, the selection shock then affects both years equally and cancels out of the change: \(E[\epsilon_{T+1}-\epsilon_{T-2k-1}\mid\epsilon_{T-k}]=(\rho^{k+1}-\rho^{k+1})\,\epsilon_{T-k}=0.\)
The paper also discusses a scaling factor \(1/(1-p)\), where \(p\) is the fraction of the comparison population that participates. It is needed when the comparison group is the whole population, trainees included. See the caveat in Section 6.4.
Ashenfelter and Card go further. They estimate a components-of-variance model on the comparison group, use a model of participation to predict what trainees’ earnings histories would have looked like, and take the gap between predicted and actual post-training earnings as the program effect. The symmetric DiD is an intermediate step. Simulating the full prediction-based estimator is the natural extension (see Section 9).
We simulate 5,000 individuals over 1970-1978. Selection is on 1975 earnings, so \(T-k = 1975\) with \(T = 1976\) and \(k = 1\). Training takes effect from 1977. The true effect is $500 per year.
library(dplyr)
library(tidyr)
library(ggplot2)
library(broom)
library(purrr)
set.seed(1985)N <- 5000
years <- 1970:1978
beta <- 500 # true annual treatment effect ($)
rho <- 0.70 # persistence of transitory shocks
sigma_eps <- 500 # sd of innovations
individuals <- tibble(
id = 1:N,
alpha_i = rnorm(N, mean = 0, sd = 1200), # permanent component
age = round(rnorm(N, mean = 31, sd = 6)),
educ = pmax(8, pmin(18, round(rnorm(N, mean = 12, sd = 2))))
)data <- individuals |>
crossing(year = years) |>
arrange(id, year)
# Economy-wide earnings growth
year_effects <- tibble(
year = years,
d_t = c(0, 100, 250, 450, 650, 700, 800, 950, 1100)
)
data <- data |> left_join(year_effects, by = "year")
# AR(1) transitory component, started from its stationary distribution
data <- data |>
group_by(id) |>
mutate(u_it = rnorm(n(), mean = 0, sd = sigma_eps)) |>
mutate(
eps_it = {
eps <- numeric(n())
eps[1] <- u_it[1] / sqrt(1 - rho^2)
for (j in 2:n()) eps[j] <- rho * eps[j - 1] + u_it[j]
eps
}
) |>
ungroup()data <- data |>
mutate(
base_earnings = 3500 + 250 * (educ - 12) + 20 * (age - 31),
earnings_0 = pmax(0, base_earnings + alpha_i + d_t + eps_it)
)Lower 1975 earnings raise the probability of participating. Treatment status is a person-level constant.
selection_data <- data |>
filter(year == 1975) |>
select(id, earnings_1975 = earnings_0)
data <- data |>
left_join(selection_data, by = "id") |>
mutate(p_treat = plogis(-0.5 - 0.0015 * (earnings_1975 - 3500))) |>
group_by(id) |>
mutate(treated = rbinom(1, size = 1, prob = first(p_treat))) |>
ungroup()
data |> distinct(id, treated) |> count(treated)# A tibble: 2 × 2
treated n
<int> <int>
1 0 3521
2 1 1479
The dip is an additional $600 drop in 1975 for participants. The treatment adds $500 from 1977 on.
data <- data |>
mutate(
ash_dip = if_else(year == 1975 & treated == 1, -600, 0),
earnings_0_dip = pmax(0, earnings_0 + ash_dip),
treatment_effect = if_else(treated == 1 & year >= 1977, beta, 0),
earnings = earnings_0_dip + treatment_effect
)Our simulated dataset is a panel dataset where each row gives the earnings of an individual in a given year, as well as their characteristics
data# A tibble: 45,000 × 17
id alpha_i age educ year d_t u_it eps_it base_earnings earnings_0
<int> <dbl> <dbl> <dbl> <int> <dbl> <dbl> <dbl> <dbl> <dbl>
1 1 508. 30 13 1970 0 -626. -876. 3730 3361.
2 1 508. 30 13 1971 100 630. 17.0 3730 4355.
3 1 508. 30 13 1972 250 -746. -734. 3730 3754.
4 1 508. 30 13 1973 450 605. 91.4 3730 4779.
5 1 508. 30 13 1974 650 358. 422. 3730 5310.
6 1 508. 30 13 1975 700 -440. -144. 3730 4793.
7 1 508. 30 13 1976 800 -828. -929. 3730 4109.
8 1 508. 30 13 1977 950 521. -129. 3730 5059.
9 1 508. 30 13 1978 1100 -290. -380. 3730 4957.
10 2 -1336. 27 16 1970 0 -149. -209. 4420 2875.
# ℹ 44,990 more rows
# ℹ 7 more variables: earnings_1975 <dbl>, p_treat <dbl>, treated <int>,
# ash_dip <dbl>, earnings_0_dip <dbl>, treatment_effect <dbl>, earnings <dbl>
mean_earnings <- data |>
group_by(year, treated) |>
summarise(mean_earnings = mean(earnings), .groups = "drop")
mean_earnings |>
pivot_wider(names_from = "treated", values_from = "mean_earnings",
names_prefix = "treated_")# A tibble: 9 × 3
year treated_0 treated_1
<int> <dbl> <dbl>
1 1970 3936. 2381.
2 1971 4036. 2464.
3 1972 4190. 2627.
4 1973 4399. 2761.
5 1974 4623. 2869.
6 1975 4720. 2215.
7 1976 4791. 2983.
8 1977 4929. 3704.
9 1978 5061. 3919.
ggplot(mean_earnings,
aes(x = year, y = mean_earnings, group = treated,
linetype = factor(treated))) +
geom_line(linewidth = 1) +
geom_point() +
geom_vline(xintercept = 1976, linetype = "dashed") +
labs(x = "Year", y = "Mean annual earnings",
linetype = "Program participant",
title = "Simulated earnings histories") +
theme_minimal()The dashed line marks the start of training. Participants’ earnings fall before it and rebound afterwards, even without any treatment effect. That rebound is what the estimators must not mistake for a program effect.
\(\hat\beta_{\text{naive}} = E[Y_{i,1978} - Y_{i,1975} \mid D_i = 1]\)
naive <- data |>
filter(year %in% c(1975, 1978)) |>
select(id, treated, year, earnings) |>
pivot_wider(names_from = year, values_from = earnings,
names_prefix = "earnings_") |>
filter(treated == 1) |>
summarise(naive_effect = mean(earnings_1978 - earnings_1975))
naive# A tibble: 1 × 1
naive_effect
<dbl>
1 1703.
Because 1975 is the dip year, this is much larger than the true $500.
We simply compute, for each treated, the change in earnings between 1975 and 1978 and then calculate the average over each group (treated and untreated).
did_1975 <- data |>
filter(year %in% c(1975, 1978)) |>
select(id, treated, year, earnings) |>
pivot_wider(names_from = year, values_from = earnings,
names_prefix = "earnings_") |>
mutate(change = earnings_1978 - earnings_1975) |>
group_by(treated) |>
summarise(mean_change = mean(change), .groups = "drop")data |>
filter(year %in% c(1975, 1978)) |>
select(id, treated, year, earnings) |>
pivot_wider(names_from = year, values_from = earnings,
names_prefix = "earnings_") |>
mutate(change = earnings_1978 - earnings_1975) |>
group_by(treated)# A tibble: 5,000 × 5
# Groups: treated [2]
id treated earnings_1975 earnings_1978 change
<int> <int> <dbl> <dbl> <dbl>
1 1 0 4793. 4957. 164.
2 2 0 4059. 3941. -118.
3 3 0 3775. 4053. 278.
4 4 1 1944. 4163. 2218.
5 5 0 5155. 5073. -81.1
6 6 0 3174. 3679. 505.
7 7 0 5523. 6609. 1086.
8 8 0 4312. 4372. 59.7
9 9 0 3552. 3405. -147.
10 10 0 3143. 3403. 260.
# ℹ 4,990 more rows
did_1975# A tibble: 2 × 2
treated mean_change
<int> <dbl>
1 0 341.
2 1 1703.
did_estimate <- did_1975 |>
summarise(DID = mean_change[treated == 1] - mean_change[treated == 0])
did_estimate# A tibble: 1 × 1
DID
<dbl>
1 1362.
This is better than the naive estimate: it removes aggregate growth. But it still uses the selection/dip year as its base, so it is still biased upward.
With \(T = 1976\) and \(k = 1\): post period \(T+1 = 1977\), pre period \(T-2k-1 = 1973\). Both are two years from the selection year 1975.
symmetric_did <- data |>
filter(year %in% c(1973, 1977)) |>
select(id, treated, year, earnings) |>
pivot_wider(names_from = year, values_from = earnings,
names_prefix = "earnings_") |>
mutate(change = earnings_1977 - earnings_1973) |>
group_by(treated) |>
summarise(mean_change = mean(change), .groups = "drop")data |>
filter(year %in% c(1973, 1977)) |>
select(id, treated, year, earnings) |>
pivot_wider(names_from = year, values_from = earnings,
names_prefix = "earnings_") |>
mutate(change = earnings_1977 - earnings_1973) |>
group_by(treated)# A tibble: 5,000 × 5
# Groups: treated [2]
id treated earnings_1973 earnings_1977 change
<int> <int> <dbl> <dbl> <dbl>
1 1 0 4779. 5059. 280.
2 2 0 2523. 3541. 1018.
3 3 0 3927. 4011. 83.5
4 4 1 2360. 3449. 1089.
5 5 0 4359. 5186. 827.
6 6 0 2482. 3667. 1185.
7 7 0 4715. 6765. 2049.
8 8 0 3858. 4571. 714.
9 9 0 1988. 3340. 1352.
10 10 0 3677. 3291. -386.
# ℹ 4,990 more rows
symmetric_did# A tibble: 2 × 2
treated mean_change
<int> <dbl>
1 0 530.
2 1 943.
symmetric_estimate <- symmetric_did |>
summarise(symmetric_DID = mean_change[treated == 1] - mean_change[treated == 0])
symmetric_estimate# A tibble: 1 × 1
symmetric_DID
<dbl>
1 413.
The paper’s comparison group is a general population sample, which includes some trainees. Because those trainees pull the comparison group’s average toward the participants’, the estimated gap is too small. Ashenfelter and Card correct for this by dividing the estimate by \((1-p)\), where \(p\) is the share of participants in the population.
Take 100 people, 30 of whom join the program, so \(p = 0.3\). Suppose the change in earnings between 1973 and 1977 is \(+500\) for participants and \(0\) for non-participants. The gap we want to measure is \(500 - 0 = 500\).
If the comparison group is the whole population, its average change mixes both groups:
\[ 0.3 \times 500 + 0.7 \times 0 = 150 . \]
Comparing participants with this group gives \(500 - 150 = 350\), which is \((1-p) = 0.7\) times the true gap. Dividing by \((1-p)\) recovers it:
\[ \frac{350}{1-p} = \frac{350}{0.7} = 500 . \]
In our simulation, the controls are non-participants only, so the comparison group is not contaminated by trainees and the unadjusted symmetric DiD already targets the true effect.
Dividing it by \((1-p)\) would overshoot here. Hence, we will do so for illustrative purposes only.
The proportion of treated:
p <- data |> distinct(id, treated) |> summarise(p = mean(treated)) |> pull(p)
p[1] 0.2958
Then the symmetric DiD with each comparison group:
change_1973_1977 <- data |>
filter(year %in% c(1973, 1977)) |>
select(id, treated, year, earnings) |>
pivot_wider(names_from = year, values_from = earnings,
names_prefix = "earnings_") |>
mutate(change = earnings_1977 - earnings_1973)
avg_change <- change_1973_1977 |>
summarise(
participants = mean(change[treated == 1]),
non_participants = mean(change[treated == 0]),
whole_population = mean(change)
)
avg_change |>
transmute(
`Non-participants only` = participants - non_participants,
`Whole population` = participants - whole_population,
`Whole population, adjusted` = (participants - whole_population) / (1 - p)
) |>
pivot_longer(everything(), names_to = "comparison_group", values_to = "estimate")# A tibble: 3 × 2
comparison_group estimate
<chr> <dbl>
1 Non-participants only 413.
2 Whole population 291.
3 Whole population, adjusted 413.
The same three estimators can be written as regressions. The relevant coefficient is on post (naive) or on treated:post (DiD variants).
# Naive
naive_reg_data <- data |>
filter(year %in% c(1975, 1978), treated == 1) |>
mutate(post = as.integer(year == 1978))
tidy(lm(earnings ~ post, data = naive_reg_data))# A tibble: 2 × 5
term estimate std.error statistic p.value
<chr> <dbl> <dbl> <dbl> <dbl>
1 (Intercept) 2215. 29.7 74.6 0
2 post 1703. 42.0 40.6 1.36e-286
# Conventional DiD
did_reg_data <- data |>
filter(year %in% c(1975, 1978)) |>
mutate(post = as.integer(year == 1978))
tidy(lm(earnings ~ treated + post + treated:post, data = did_reg_data))# A tibble: 4 × 5
term estimate std.error statistic p.value
<chr> <dbl> <dbl> <dbl> <dbl>
1 (Intercept) 4720. 20.7 228. 0
2 treated -2505. 38.1 -65.7 0
3 post 341. 29.3 11.6 4.15e- 31
4 treated:post 1362. 53.9 25.3 1.47e-136
# Symmetric DiD
symmetric_reg_data <- data |>
filter(year %in% c(1973, 1977)) |>
mutate(post = as.integer(year == 1977))
tidy(lm(earnings ~ treated + post + treated:post, data = symmetric_reg_data))# A tibble: 4 × 5
term estimate std.error statistic p.value
<chr> <dbl> <dbl> <dbl> <dbl>
1 (Intercept) 4399. 21.0 210. 0
2 treated -1639. 38.6 -42.5 0
3 post 530. 29.7 17.9 3.26e-70
4 treated:post 413. 54.6 7.57 4.03e-14
comparison <- tibble(
estimator = c("True treatment effect",
"Naive before-after",
"Conventional DiD: 1975-1978",
"Symmetric DiD: 1973-1977"),
estimate = c(beta,
naive$naive_effect,
did_estimate$DID,
symmetric_estimate$symmetric_DID)
)
comparison# A tibble: 4 × 2
estimator estimate
<chr> <dbl>
1 True treatment effect 500
2 Naive before-after 1703.
3 Conventional DiD: 1975-1978 1362.
4 Symmetric DiD: 1973-1977 413.
Note that the true effect is $500 in each post year, and the 1977 and 1978 effects are both $500, so the symmetric estimate (post = 1977) and the conventional one (post = 1978) target the same number.
Instead of fixing 1975, compute
\[ \hat\beta_j = E(Y_{1978} - Y_j \mid D=1) - E(Y_{1978} - Y_j \mid D=0) \]
for each pre-treatment base year \(j\).
base_years <- 1970:1975
did_by_base_year <- base_years |>
map_dfr(
~ data |>
filter(year %in% c(.x, 1978)) |>
select(id, treated, year, earnings) |>
pivot_wider(names_from = year, values_from = earnings) |>
mutate(change = `1978` - .data[[as.character(.x)]]) |>
group_by(treated) |>
summarise(mean_change = mean(change), .groups = "drop") |>
summarise(
base_year = .x,
DID = mean_change[treated == 1] - mean_change[treated == 0]
)
)
did_by_base_year# A tibble: 6 × 2
base_year DID
<int> <dbl>
1 1970 413.
2 1971 429.
3 1972 420.
4 1973 496.
5 1974 612.
6 1975 1362.
ggplot(did_by_base_year, aes(x = base_year, y = DID)) +
geom_line() +
geom_point(size = 3) +
geom_hline(yintercept = beta, linetype = "dashed") +
labs(x = "Pre-program base year",
y = "Estimated treatment effect",
title = "Sensitivity of DiD to the choice of pre-program base year",
subtitle = "Dashed line = true treatment effect in the simulation") +
theme_minimal()The pattern shows the core message: the answer depends on which “before” year you pick, and base years near the selection shock give the most misleading estimates. A base year is only “safe” if it sits at the symmetric distance from the selection year that matches the post year (here 1973 for a 1977 post period). With \(\rho = 0.7\), base years farther back (1970-1972) are also close to unbiased, because the shock has largely faded.
| Estimator | Removes common trends | Removes permanent differences | Handles serially correlated selection shock |
|---|---|---|---|
| Naive before-after | no | (yes, within person) | no |
| Conventional DiD | yes | yes | no |
| Symmetric DiD | yes | yes | yes, under covariance stationarity |
To reproduce the paper’s final estimator in this simulation, one would: