Treatment-control difference in mean active days after assignment
1 The gist · for everyone
Among users kept by the saved analysis, users had 0.21 more active days than users in days 1-28 after they were assigned.
Control users averaged 6.42 active days; treatment users averaged 6.63.
For every 100 users in the analysis table, Home v2 adds about 21 active user-days over the 28-day follow-up window.
"In the saved analysis, treatment users averaged 0.21 more active days than control users over days 1-28 after assignment."
"Home v2 caused all new users to be active 0.21 more days per month."
The query is not pure : it keeps only users joined to an , excludes China, and drops users with zero recorded activity days in the outcome window.
2 How it's built · for the curious
Alex is assigned to on 10 May 2026, sees the variant screen, is outside China, and has app activity on several later dates.
Result: Alex contributes 2 to the average. Assignment-day activity and day-29 activity do not count.
Now the general rule, one step at a time. Each step has the plain version, with the exact version one click away.
Step 1 · Who is counted
One app user in the Home v2 experiment analysis table.
One user_id from exp.assignments for experiment_id = 'home_v2', after joining to exp.exposures with saw_variant_screen = 1, excluding country_code = 'CN', and requiring at least one joined app.events activity_date in the outcome window.
S3Step 2 · When the clock starts
Each user's clock starts on their assignment date.
assigned_date = CAST(a.assigned_at AS DATE).
S3Step 3 · What counts
Counts how many different dates had any app activity in the outcome window.
COUNT(DISTINCT act.activity_date), where activity_date is DATE(e.event_ts) from app.events grouped by user_id and UTC-derived activity_date.
S3Step 4 · Combine
Average the active-day counts within each experiment arm.
AVG(active_days) grouped by arm; denominator is COUNT(*) from per_user.
S3Step 5 · Compare
Subtract the average from the treatment average.
mean_active_days_treatment - mean_active_days_control
With the real numbers: 6.63 − 6.42 = 0.210
Whose effect? No causal target; a comparison of observed numbers.
diff = treatment.mean() - control.mean(); from the summary table, 6.63 - 6.42 = 0.21 days.
S4S5Step 6 · How sure
The saved interval uses the spread of per-user active days in each arm to estimate sampling uncertainty around the treatment-control difference.
Not covered by this range: The interval does not cover bias from post-assignment filtering, UTC-vs-local date mismatch, missing events, or whether zero-activity assigned users were dropped.
se = sqrt(var_treatment / n_treatment + var_control / n_control); interval = diff +/- 1.96 * se. p comes from scipy.stats.ttest_ind(..., equal_var=False).
S4S53 Keep in mind · before you act on it
The readout says every assigned user is analyzed in their assigned arm, but the query keeps only users with a matching and at least one post-assignment activity day. If changes exposure or whether a user has any activity at all, the reported mean is conditional on a post-randomization outcome.
Because per_user uses an inner join to activity, users with no recorded in days 1-28 are absent rather than counted as zero active days. This likely overstates each arm's average active days; the direction for the treatment-control difference is unknown.
The readout says activity is counted in each user's local time zone. The query turns event times into dates before counting active days.
The implemented population excludes China and requires seeing the variant screen. The saved result should not be generalized to all new users without checking those exclusions.
| Tempting, but wrong | Accurate |
|---|---|
| Home v2 caused all new users to be active 0.21 more days per month. | The saved analysis shows a 0.21-day difference among exposed, non-CN users retained in the per_user table over days 1-28 after assignment. |
| 0.21 days is a 0.21% lift. | It is an absolute difference in days. Relative to the 6.42-day average, it is about a 3.3% lift. |
| The denominator is all assigned users. | The SQL denominator is users in per_user, after the exposure join, China exclusion, and inner join to activity. |
| Days 1-28 are local-calendar days. | The SQL derives dates with DATE(event_ts), and the source comment says event_ts is stored in UTC. |
Chose: Use the 0.21-day treatment-control difference, because both readouts emphasize that number and the CSV stores the matching difference and interval.
Could have chosen: A card could instead headline the mean (6.63 days), the (+3.3%), or an rerun if zero-activity assigned users were available.
Why: The user asked for the headline number readers see, and the readout repeats +0.21 days.
What it changes: The card focuses on the difference in means and treats the relative lift as a secondary restatement.
Chose: Mark the saved result as associational, even though the experiment assignment was randomized.
Could have chosen: If the analysis were rerun on all assigned users as true intent-to-treat, the claim strength could be causal for the randomized population.
Why: The implemented SQL applies post-assignment exposure and activity filters, so the analyzed denominator may no longer preserve the original randomization.
What it changes: This prevents the card from repeating the readout's broader causal claim until the denominator is verified or rerun.
4 Reproduce it · for specialists
| Population | Users assigned to home_v2 from 2026-05-04 through 2026-05-17, excluding China, joined to a saw_variant_screen exposure row, and present in app.events on at least one date in days 1-28. |
|---|---|
| Exposure / contrast | Home v2 treatment assignment versus control assignment, as labeled by arm. |
| Time zero | CAST(assigned_at AS DATE) for each user. |
| Variable | Count of distinct UTC-derived app activity dates greater than assigned_date and less than or equal to assigned_date plus 28 days. |
| Intercurrent events | The saved SQL does not specify special handling for uninstall, churn, travel, or missing events. Users with no activity in the window are excluded by the inner join rather than retained with zero. |
| Summary measure | Difference in arm-level means, treatment minus control, with a Welch t-test confidence interval. |
| Target | descriptive: No causal target; a comparison of observed numbers |
README says every assigned user is analyzed in the arm they were assigned to.
assigned joins to exposures with saw_variant_screen = 1, then per_user inner joins to activity, excluding assigned users without a matching exposure row or without activity in days 1-28.
Impact: Narrows the denominator from all assigned users to exposed active users. The impact on the +0.21-day difference is unknown from saved outputs.
S1S3The readouts describe randomized users or all new users without a country exclusion.
The SQL filters a.country_code <> 'CN'.
Impact: Results do not cover China. Direction and size of any difference are unknown.
S2S3README says active days use the user's local time zone.
activity_date is DATE(e.event_ts), and the SQL comment says event_ts is stored in UTC.
Impact: Activity near local midnight can land on a different date. The effect on distinct active-day counts is unknown.
S1S3Readout says Home v2 causes users to be active 0.21 more days a month and applies to all new users.
The code estimates a treatment-control difference over days 1-28 among exposed, non-CN users who had at least one activity day in the outcome window.
Impact: The implemented result is narrower than the readout claim. A true all-assigned ITT estimate could be smaller or larger.
S2S3| Stage | Exact rule | Source |
|---|---|---|
| Eligibility 1 | a.experiment_id = 'home_v2' and assigned_at >= TIMESTAMP '2026-05-04 00:00:00' and assigned_at < TIMESTAMP '2026-05-18 00:00:00'. | S3 |
| Eligibility 2 | INNER JOIN exp.exposures on user_id and experiment_id with saw_variant_screen = 1; country_code <> 'CN'. | S3 |
| Eligibility 3 83,482 remain | per_user is built with an INNER JOIN from assigned to activity on act.activity_date > assigned_date and act.activity_date <= DATE_ADD(assigned_date, 28), so users with no activity_date in that interval are not in the final COUNT(*). | S3S4 |
| Time zero | assigned_date = CAST(a.assigned_at AS DATE). | S3 |
| Assignment window [0, 14) days | assigned_at >= 2026-05-04 00:00:00 and assigned_at < 2026-05-18 00:00:00. | S3 |
| Days 1-28 after assignment [1, 29) days | act.activity_date > assigned_date AND act.activity_date <= DATE_ADD(assigned_date, 28). | S3 |
| Active days count | COUNT(DISTINCT act.activity_date), where activity_date is DATE(e.event_ts) from app.events grouped by user_id and UTC-derived activity_date. | S3 |
| Aggregate | AVG(active_days) grouped by arm; denominator is COUNT(*) from per_user. | S3 |
| Comparison | diff = treatment.mean() - control.mean(); from the summary table, 6.63 - 6.42 = 0.21 days. | S4S5 |
| Uncertainty | se = sqrt(var_treatment / n_treatment + var_control / n_control); interval = diff +/- 1.96 * se. p comes from scipy.stats.ttest_ind(..., equal_var=False). | S4S5 |
| Group | Definition | Value | n |
|---|---|---|---|
| Control | arm = 'control' in the per_user export. | 6.42 | 41,880 |
| Home v2 | arm = 'treatment' in the per_user export. | 6.63 | 41,602 |
Check: 6.63 − 6.42 = 0.210 (headline 0.21).
| Name | Value | Meaning |
|---|---|---|
experiment_id | home_v2 | Experiment analyzed |
assignment_start | 2026-05-04 00:00:00 | Inclusive assignment start |
assignment_end | 2026-05-18 00:00:00 | Exclusive assignment end |
outcome_start | 1 | First included day after assignment |
outcome_end | 28 | Last included day after assignment |
excluded_country | CN | Country excluded in SQL |
exp.assignmentsExperiment assignment and arm labels
Grain: user_id x experiment assignment · Filters: experiment_id = home_v2; assigned_at in [2026-05-04, 2026-05-18); country_code <> CN
exp.exposuresExposure filter
Grain: user_id x experiment exposure · Filters: same experiment_id; saw_variant_screen = 1
app.eventsDaily app activity
Grain: event · Filters: grouped to one activity_date per user_id using DATE(event_ts)
results/ab_summary.csvSaved arm summaries, difference, interval, p-value, and data completeness date
Grain: arm summary plus estimate rows
FROM exp.assignments a
JOIN exp.exposures x
ON x.user_id = a.user_id
AND x.experiment_id = a.experiment_id
AND x.saw_variant_screen = 1
WHERE a.experiment_id = 'home_v2'
AND a.assigned_at >= TIMESTAMP '2026-05-04 00:00:00'
AND a.assigned_at < TIMESTAMP '2026-05-18 00:00:00'
AND a.country_code <> 'CN'SELECT e.user_id, DATE(e.event_ts) AS activity_date -- event_ts is stored in UTC
FROM app.events e
GROUP BY e.user_id, DATE(e.event_ts)
...
AND act.activity_date > s.assigned_date
AND act.activity_date <= DATE_ADD(s.assigned_date, 28)diff = treatment.mean() - control.mean()
se = np.sqrt(treatment.var(ddof=1) / len(treatment) + control.var(ddof=1) / len(control))
t, p = stats.ttest_ind(treatment, control, equal_var=False)
return {"diff": diff, "ci_low": diff - 1.96 * se, "ci_high": diff + 1.96 * se, "p": p}Data complete through: 2026-06-20
Receipts:
Tests: No executable test files were present in the fixture. The card was linted and rendered with the mcard tooling.
eval/fixtures/syn-b/README.md lines 1-15 · Experiment design, stated ITT, primary metric definition, and reported resulteval/fixtures/syn-b/readout.md lines 1-4 · Headline leadership readouteval/fixtures/syn-b/sql/experiment_metric.sql lines 1-30 · Implemented population, activity-date derivation, outcome window, and aggregationeval/fixtures/syn-b/results/ab_summary.csv lines 1-8 · Saved group means, sample sizes, interval, p-value, and data completeness dateeval/fixtures/syn-b/src/analysis.py lines 1-9 · Welch t-test calculationrecompute estimate reproduced from cells: 6.63 − 6.42 = 0.210Check yourself
Write one or two sentences as if telling a colleague. Then compare with the checklist.
A complete explanation mentions:
One distinct DATE(event_ts) in app.events after grouping by user_id.