scripts: reconcile telemetry against external acquisition denominators - #433
Merged
Conversation
`anonymous_installs` cannot distinguish tiny acquisition from disclosure
abandonment from an instrument that barely reaches anyone -- three situations
that call for completely different work. Only outside numbers separate them.
Read-only everywhere: GitHub release/traffic reads and ASC GETs. Creates no ASC
resource; the analytics report requests it reads were already provisioned.
Needs no new telemetry and no schema change.
Two contamination traps are handled inline rather than in a comment nobody
reads, because both produce a confident wrong number:
- Apple suppresses days below a privacy threshold (observed floor: nothing
under 5) and rounds what survives. "0 first-time Mac downloads" therefore
means "never once crossed the threshold on any single day", NOT zero. Every
printed Apple figure carries that caveat in its own output line.
- GitHub `download_count` includes CI, owner testing, bots and mirrors, and tap
`clones` includes crawlers. Both are upper bounds on humans, never counts.
The script refuses to compute a telemetry-to-Apple ratio for the same reason: a
ratio built on thresholded, rounded inputs carries a precision they do not have.
`gh api` failures exit non-zero rather than returning {}. A silent empty result
here would render as "zero downloads", which is precisely the green-but-lying
status signal this analysis exists to avoid.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a new analysis script under scripts/ to reconcile the existing anonymous telemetry funnel with external acquisition denominators from GitHub (release assets + traffic) and App Store Connect (App Downloads Detailed), explicitly accounting for Apple privacy-threshold suppression and GitHub “contaminated” counts.
Changes:
- Introduces
scripts/acquisition_reconcile.pyto pull GitHub release/traffic denominators and (optionally) ASC “App Downloads Detailed” denominators. - Implements “fail loudly” behavior for
gh apiand ASC GET calls, and prints inline caveats about privacy thresholds / contaminated counts. - Provides a “HOW TO READ THIS” section tying the output back to the Phase 1 SQL analysis.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+143
to
+148
| return jwt.encode( | ||
| {"iss": ISSUER, "exp": int(time.time()) + 1200, "aud": "appstoreconnect-v1"}, | ||
| open(path).read(), | ||
| algorithm="ES256", | ||
| headers={"kid": KEY_ID}, | ||
| ) |
Comment on lines
+170
to
+175
| raw = requests.get(s["attributes"]["url"], timeout=180).content | ||
| try: | ||
| text = gzip.decompress(raw).decode("utf-8") | ||
| except OSError: | ||
| text = raw.decode("utf-8") | ||
| rows += list(csv.DictReader(io.StringIO(text), delimiter="\t")) |
Comment on lines
+218
to
+223
| def count(r: dict) -> int: | ||
| try: | ||
| return int(r["Counts"]) | ||
| except (ValueError, KeyError): | ||
| return 0 | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
anonymous_installscan only answer "of the people who reached the disclosure card, how many reached a provider?". It cannot distinguish:Those three call for completely different work, and nothing in the telemetry separates them. Only outside numbers do — this pulls the outside numbers.
Read-only everywhere (GitHub reads + ASC GETs). Creates no ASC resource — the analytics report requests it reads were already provisioned. No new telemetry, no schema change.
Two contamination traps, handled in the output rather than in a comment
Both produce a confident wrong number, which is this project's recurring failure mode:
0 first-time Mac downloadsmeans "never once crossed the threshold on any single day", not zero. Every printed Apple figure carries that caveat on its own line.download_countincludes CI, owner testing, bots, mirrors; tapclonesincludes crawlers. Both are upper bounds on humans, never counts.The script also refuses to compute a telemetry-to-Apple ratio — a ratio built on thresholded, rounded inputs carries a precision they do not have.
Failure behaviour
gh apifailuresexitnon-zero rather than returning{}. A silent empty result would render as "zero downloads" — the exact green-but-lying signal this analysis exists to avoid. Same for ASC GETs, and an empty ASC result prints "that is not 'no downloads'" rather than a bare zero.Verification
Ran end to end against live GitHub + ASC,
exit=0, all sections populated. Pairs withbackend/supabase/analysis/phase1_menu_open_to_provider.sqlfrom #431.--skip-ascruns the GitHub half with no Apple credentials.Findings from the run are business data and live in
cli-pulse-internal, not here.🤖 Generated with Claude Code