← Full explainer ·

Metric Facts

Home v2 active-days lift · provisional
Headline
more active days per treatment user+0.21 days
Plausible range +0.05 days to +0.37 days · Welch t-test normal-approximation interval

Among users kept by the saved analysis, Home v2 users had 0.21 more active days than control users in days 1-28 after they were assigned.

CountedUsers who: assigned to the Home v2 experiment from 4 May through 17 May 2026; saw the variant screen and was not in China; had at least one recorded activity day in days 1-28 after assignment. n = 83,482
Clock startsEach user's clock starts on their assignment date.
Assignment windowDays 0–13 (eligibility). The experiment admitted users assigned from 4 May through 17 May 2026.
Days 1-28 after assignmentDays 1–28 (outcome). The 28 calendar dates after the assignment date, excluding assignment day.
What countsCounts how many different dates had any app activity in the outcome window.
Combined asAverage the active-day counts within each experiment arm.
Compared bySubtract the control average from the treatment average. mean_active_days_treatment - mean_active_days_control · descriptive
Kind of claimShows a link. Groups are compared, but nothing rules out other differences between them. Treat it as a link, not a cause.
CompletenessThe latest assignment date is 17 May 2026; its days 1-28 after assignment ends 14 Jun 2026. Data are complete through 20 Jun 2026, so every window is fully observed. This number should not move as more data arrive.
Before you repeat it

The query is not pure intent-to-treat: it keeps only users joined to an exposure row, excludes China, and drops users with zero recorded activity days in the outcome window.

Caveats
  • ◎Not pure intent-to-treat
  • ↕Zero-activity users are not in the denominator
  • ↕Time zone mismatch
  • ◎Scope is narrower than all new users
Don't say
  • Home v2 caused all new users to be active 0.21 more days per month.
  • 0.21 days is a 0.21% lift.
  • The denominator is all assigned users.
  • Days 1-28 are local-calendar days.
4 known gap(s) between description and code: Intent-to-treat denominator; Geographic scope; Time zone for active dates; Breadth of causal claim.
↑ may overstate · ↓ may understate · ± uncertainty understated · ↕ either way · ◎ limits scope. Owner: Product analytics. Updated 2026-10-04. Data as of 2026-06-20. Sources: S1 README.md, S2 readout.md, S3 experiment_metric.sql, S4 ab_summary.csv, S5 analysis.py.