Repository navigation
Capture tweet context: long-form truncation, Articles, quotes, polls, replies - #612
Open
fredrivett wants to merge 5 commits into
Open
fredrivett wants to merge 5 commits into
fredrivett wants to merge 5 commits into
Conversation
Long-form truncation flag, quoted-tweet and poll snapshots (JSON, shape owned by zod schemas in code) and the replied-to post as flat columns. All nullable or defaulted, so the migration is additive. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Long-form posts, quotes, polls, replies and X Articles all arrive in the syndication payload we already fetch, but we dropped them: long posts were silently cut at ~280 chars, Article posts saved as a bare t.co link, and text kept X's HTML escaping and trailing media links. - transformTweetData moves to src/lib/twitter and now normalises text (display range, expanded links, decoded entities) and extracts the truncation flag, quoted tweet, poll, reply context and Article card - Quote/poll JSON is validated by zod on write and on read; every renderer reads through one shared select + mapper, which also fixes re-hosted avatars never being served (the key was never selected) - Article posts are titled and described from the Article - A tombstoned tweet (deleted, protected or marked sensitive) fails as source_blocked instead of a retryable unknown error Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The detail view now shows what X shows around a tweet: the post it replies to (or the thread it continues), poll results, an X Article's cover/title/opening, and the quoted tweet. A long-form post ends with "… Show more on X" (tracked as tweet_show_more_clicked), and the grid card marks it with an ellipsis. A cover-less Article previews its title in the grid, and the card height estimate uses the same text. Each new piece has stories per variant and RTL tests per branch. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Re-fetches every saved tweet from X's embed endpoint and fills in the new context columns (plus an Article's card, title and description). Leaves media, avatars, existing cards and storage accounting alone, skips tweets X no longer serves, and compare-and-sets on tweetId so a concurrent re-capture wins. Run backfill-tweet-images afterwards to re-host new Article covers. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
6 issues found across 49 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="app/src/components/twitter/tweet-link-card.tsx">
<violation number="1" location="app/src/components/twitter/tweet-link-card.tsx:22">
P1: This card turns untrusted stored `card.url` into a navigable link without an HTTP(S) validation gate. Malformed or non-HTTP values can produce a broken link or allow unsafe schemes such as `javascript:`; validate with `isValidUrl` and render a non-anchor card when the URL is rejected.</violation>
</file>
<file name="app/src/lib/items/tweet-context-backfill.ts">
<violation number="1" location="app/src/lib/items/tweet-context-backfill.ts:59">
P1: This compare-and-set does not protect the card state that `addArticleCard` was based on. If capture or image re-hosting fills the card between the initial read and this update, the backfill overwrites its `imageFileKey`, so the Article cover becomes hotlinked and the durable storage reference is lost; include the card-null state in the claim (handling both SQL and JSON null) and skip or retry when it changed.</violation>
<violation number="2" location="app/src/lib/items/tweet-context-backfill.ts:70">
P2: This replaces titles users edited whenever an Article card is added. Read `titleEditedByUser` and preserve the title when it is true.</violation>
</file>
<file name="app/src/components/twitter/story-fixtures.ts">
<violation number="1" location="app/src/components/twitter/story-fixtures.ts:13">
P3: The QuoteTweet fixture reuses the quoting post’s ID for the quoted post, so the story models a self-quote and its quote link targets the same status. Give the quoted tweet a distinct ID.</violation>
</file>
<file name="app/trigger/backfill-tweet-context.ts">
<violation number="1" location="app/trigger/backfill-tweet-context.ts:71">
P2: The worker lets fetch, transform, and database failures escape without logging or `captureServerException`. Wrap the run body in the standard error handler, capture the failure, and rethrow so exhausted backfill failures remain observable.</violation>
</file>
<file name="app/src/components/twitter/quoted-tweet.tsx">
<violation number="1" location="app/src/components/twitter/quoted-tweet.tsx:60">
P2: Use a position-qualified key here because quote media URLs can repeat. Duplicate keys trigger React warnings and can cause the wrong image instance to be reused when the quote data changes.</violation>
</file>
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Re-trigger cubic
|
|
||
| return ( | ||
| <a | ||
| href={card.url} |
There was a problem hiding this comment.
P1: This card turns untrusted stored card.url into a navigable link without an HTTP(S) validation gate. Malformed or non-HTTP values can produce a broken link or allow unsafe schemes such as javascript:; validate with isValidUrl and render a non-anchor card when the URL is rejected.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At app/src/components/twitter/tweet-link-card.tsx, line 22:
<comment>This card turns untrusted stored `card.url` into a navigable link without an HTTP(S) validation gate. Malformed or non-HTTP values can produce a broken link or allow unsafe schemes such as `javascript:`; validate with `isValidUrl` and render a non-anchor card when the URL is rejected.</comment>
<file context>
@@ -0,0 +1,73 @@
+
+ return (
+ <a
+ href={card.url}
+ target="_blank"
+ rel="noopener noreferrer"
</file context>
| // Compare-and-set on tweetId: if the item was re-captured as a different | ||
| // tweet meanwhile, its newer row wins | ||
| const { count } = await tx.itemTwitterDetails.updateMany({ | ||
| where: { itemId, tweetId: existing.tweetId }, |
There was a problem hiding this comment.
P1: This compare-and-set does not protect the card state that addArticleCard was based on. If capture or image re-hosting fills the card between the initial read and this update, the backfill overwrites its imageFileKey, so the Article cover becomes hotlinked and the durable storage reference is lost; include the card-null state in the claim (handling both SQL and JSON null) and skip or retry when it changed.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At app/src/lib/items/tweet-context-backfill.ts, line 59:
<comment>This compare-and-set does not protect the card state that `addArticleCard` was based on. If capture or image re-hosting fills the card between the initial read and this update, the backfill overwrites its `imageFileKey`, so the Article cover becomes hotlinked and the durable storage reference is lost; include the card-null state in the claim (handling both SQL and JSON null) and skip or retry when it changed.</comment>
<file context>
@@ -0,0 +1,80 @@
+ // Compare-and-set on tweetId: if the item was re-captured as a different
+ // tweet meanwhile, its newer row wins
+ const { count } = await tx.itemTwitterDetails.updateMany({
+ where: { itemId, tweetId: existing.tweetId },
+ data: {
+ ...tweetContextData(details),
</file context>
| await tx.item.update({ | ||
| where: { id: itemId }, | ||
| data: { | ||
| title: tweetItemTitle(details), |
There was a problem hiding this comment.
P2: This replaces titles users edited whenever an Article card is added. Read titleEditedByUser and preserve the title when it is true.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At app/src/lib/items/tweet-context-backfill.ts, line 70:
<comment>This replaces titles users edited whenever an Article card is added. Read `titleEditedByUser` and preserve the title when it is true.</comment>
<file context>
@@ -0,0 +1,80 @@
+ await tx.item.update({
+ where: { id: itemId },
+ data: {
+ title: tweetItemTitle(details),
+ description: tweetDescriptionSource(details)?.slice(0, 200) ?? null,
+ },
</file context>
| maxDuration: 60, | ||
| run: async (payload: { itemId: string }) => { | ||
| const { itemId } = payload; | ||
| const result = await refreshTweetContext({ itemId, fetch: fetchTweet }); |
There was a problem hiding this comment.
P2: The worker lets fetch, transform, and database failures escape without logging or captureServerException. Wrap the run body in the standard error handler, capture the failure, and rethrow so exhausted backfill failures remain observable.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At app/trigger/backfill-tweet-context.ts, line 71:
<comment>The worker lets fetch, transform, and database failures escape without logging or `captureServerException`. Wrap the run body in the standard error handler, capture the failure, and rethrow so exhausted backfill failures remain observable.</comment>
<file context>
@@ -0,0 +1,86 @@
+ maxDuration: 60,
+ run: async (payload: { itemId: string }) => {
+ const { itemId } = payload;
+ const result = await refreshTweetContext({ itemId, fetch: fetchTweet });
+
+ if (result.refreshed) {
</file context>
| > | ||
| {stills.map((src, index) => ( | ||
| <Img | ||
| key={src} |
There was a problem hiding this comment.
P2: Use a position-qualified key here because quote media URLs can repeat. Duplicate keys trigger React warnings and can cause the wrong image instance to be reused when the quote data changes.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At app/src/components/twitter/quoted-tweet.tsx, line 60:
<comment>Use a position-qualified key here because quote media URLs can repeat. Duplicate keys trigger React warnings and can cause the wrong image instance to be reused when the quote data changes.</comment>
<file context>
@@ -0,0 +1,74 @@
+ >
+ {stills.map((src, index) => (
+ <Img
+ key={src}
+ src={src}
+ alt={tweetImageAlt(
</file context>
Suggested change
| key={src} | |
| // biome-ignore lint/suspicious/noArrayIndexKey: Static, never-reordered list; src can repeat | |
| key={`${src}-${index}`} |
| const avatar = (seed: string) => `https://picsum.photos/seed/${seed}/96/96`; | ||
|
|
||
| export const storyTweet: TwitterDetails = { | ||
| tweetId: "1900000000000000001", |
There was a problem hiding this comment.
P3: The QuoteTweet fixture reuses the quoting post’s ID for the quoted post, so the story models a self-quote and its quote link targets the same status. Give the quoted tweet a distinct ID.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At app/src/components/twitter/story-fixtures.ts, line 13:
<comment>The QuoteTweet fixture reuses the quoting post’s ID for the quoted post, so the story models a self-quote and its quote link targets the same status. Give the quoted tweet a distinct ID.</comment>
<file context>
@@ -0,0 +1,91 @@
+const avatar = (seed: string) => `https://picsum.photos/seed/${seed}/96/96`;
+
+export const storyTweet: TwitterDetails = {
+ tweetId: "1900000000000000001",
+ authorName: "Jonah Pierce",
+ authorUsername: "jonahpierce",
</file context>
Suggested change
| tweetId: "1900000000000000001", | |
| tweetId: "1900000000000000002", |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
X's embed (syndication) payload, which we already fetch for every saved tweet, carries much more than we kept. We were silently dropping it:
note_tweetmarker)t.colink; the Article's title, opening and cover were ignoredquotedTweetIdkept, never rendered)&), trailing mediat.colinks and leading reply@mentionsunknownerrorThis PR uses only that existing free payload. Full long-form text, full Article bodies and whole threads need another source (official X API / an ilo-style service), deliberately out of scope here.
Changes
is_truncated,quoted_tweet+poll(JSON),in_reply_to_*(flat columns). Additive migration.twitterDetailsSelect+mapTwitterDetailsreplaces four copy-pasted mappers (items, rooms, search, public page). This also fixes re-hosted author avatars never being served: the key was never selected.transformTweetDatamoved tosrc/lib/twitter; it normalises text and extracts truncation, quote, poll, reply and Article card. Articles reuse the existingcardcolumn (type: "article"), so their cover goes through the existing re-hosting, cover analysis and file-key inventory. Article posts are titled/described from the Article; enrichment source text includes the Article, quote and poll.source_blocked, not-found →source_not_found(non-retryable copy).backfill-tweet-contextre-fetches every saved tweet and fills only the new columns (+ Article card/title/description). It leaves media/avatars/existing cards/storage alone, skips tweets X no longer serves, compare-and-sets ontweetId, and is concurrency-limited to 5.Checklist
tweet_show_more_clickedPostHog event on the long-form "Show more on X" linkProcessingFailure(existing reporting path)Notes
20261005204621_tweet_context, needsprisma migrate deployon deploy.backfill-tweet-contextfrom the Trigger.dev dashboard, thenbackfill-tweet-imagesto re-host the Article covers it added. The backfill doesn't re-run enrichment (tags/embeddings keep the old text) to avoid AI spend.x.com/i/article/<id>links (no post ID) still go throughhandle-twitter-articleas before; the embed endpoint needs a post ID.t.co/&.🤖 Generated with Claude Code
Summary by cubic
Saves the tweet context X's embed payload already carries — truncated long-form posts, X Articles, quotes, polls and reply context — and shows it in the app instead of silently dropping it.
t.colinks clamped, leading reply mentions kept) and tombstoned tweets classify assource_blockedinstead of a retryableunknownerror.backfill-tweet-contextre-fetches saved tweets to fill the new columns without touching media, avatars, existing cards or enrichment.Migration
20261005204621_tweet_contextis additive and needsprisma migrate deploy; afterwards runbackfill-tweet-contextfrom Trigger.dev, thenbackfill-tweet-imagesto re-host new Article covers.Written for commit 6f3a7da. Summary will update on new commits.