Skip to content

Add fine-grained partition properties to replace Kind() checks#9900

Open
rkannan82 wants to merge 4 commits intomainfrom
kannan/ephemeral-partition-property
Open

Add fine-grained partition properties to replace Kind() checks#9900
rkannan82 wants to merge 4 commits intomainfrom
kannan/ephemeral-partition-property

Conversation

@rkannan82
Copy link
Copy Markdown
Contributor

@rkannan82 rkannan82 commented Apr 10, 2026

What

Add property methods to the Partition interface — HasTTLExpiry(), SupportsFairness(), SupportsVersioning(), SupportsPartitions() — and use them in place of Kind() == STICKY checks where the behavior isn't sticky-specific.

Describe task queue checks use Kind() == NORMAL since only user-facing queues can be described.

Why

Kind() == STICKY checks conflated multiple independent concerns. Fine-grained properties make each check self-documenting and make it easier to set correct values for new partition types.

How did you test it?

Unit tests (go test ./service/matching/... ./common/tqid/...). No behavioral changes — pure refactor.

🤖 Generated with Claude Code

Replace Kind() == STICKY checks with IsEphemeral() where the behavior
applies to all ephemeral partition types (sticky, and future types like
worker-commands), not just sticky queues specifically.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@rkannan82 rkannan82 force-pushed the kannan/ephemeral-partition-property branch 3 times, most recently from 9f1f466 to 526bf30 Compare April 10, 2026 04:22
@rkannan82 rkannan82 requested review from ShahabT and dnr April 10, 2026 04:23
@rkannan82 rkannan82 force-pushed the kannan/ephemeral-partition-property branch from 526bf30 to a163283 Compare April 10, 2026 04:25
@rkannan82 rkannan82 force-pushed the kannan/ephemeral-partition-property branch from a163283 to 2b99d6a Compare April 10, 2026 04:54
…ants

These are used in IsEphemeral() code paths that apply to all ephemeral
partition types, not just sticky queues.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@rkannan82 rkannan82 force-pushed the kannan/ephemeral-partition-property branch from 2b99d6a to 8e2ff4f Compare April 10, 2026 17:22
@dnr
Copy link
Copy Markdown
Contributor

dnr commented Apr 13, 2026

Several Kind() == STICKY checks in the matching service guard behaviors that apply to any ephemeral/transient partition

I think I'd like to split this up further:

TTL-based expiry

The goal is we want to use metadata TTLs for any queue that has the property: if the queue is idle (no poller or tasks) for 48 hours, it's okay to drop all tasks in it. Sticky fits that description, as do worker command queues. I think basically any worker-specific task queue would, right?

no fairness

This one feels a little different.. for more "scheduling"/"placement" approaches to matching where we send a high volume of tasks over a worker-specific channel, we may definitely want fairness there even though it's a worker-specific queue. (The problem with fairness + TTLs is for tasks, it doesn't apply to metadata. Though yeah, maybe it's silly to TTL metadata without TTLs on tasks. We'd eventually want a scavenger.)

no versioning

I'm not sure here, I guess by definition worker-specific doesn't need versioning. But maybe some other kind of "ephemeral" queues could be versioned?

no multi-partition fan-out

Also potentially different.. in theory we could have a high volume worker-specific queue accepting/dispatching tasks enough that want to spread them over a few partitions. After dynamic partition scaling maybe we should relax this for all queues, including sticky, and just default to 1 but let them scale?

no enhanced describe

I think this mostly has to do with versioning?

More behavior differences:

Sticky queues get userdata (and thus config) from a normal "parent". Should other ephemeral queues do that too?

"Return error on Add*Task if no poller in last 10s". That could apply to worker commands, maybe with a different timeout though? 5m?

Priority backlog forwarding: this is tied to having a normal "parent" partition(s), should probably match that.

Migration to v2 table: this is tied to fairness. Can be relaxed eventually.

So we could have (hand-waving draft):

SupportsFairness() bool
SupportsPartitions() bool
SupportsVersioning() bool
NormalParent() (string, bool)
RejectTaskIfNoPollerInterval() time.Duration // 0 means never reject
DeleteIfIdleTimeout() time.Duration // 0 means never
OverrideNameInMetrics() string // "" means use name as-is

@dnr
Copy link
Copy Markdown
Contributor

dnr commented Apr 13, 2026

Oh, and the original one that started this:

Appears in metric labels

@rkannan82 rkannan82 force-pushed the kannan/ephemeral-partition-property branch 2 times, most recently from f2e771c to 3591c16 Compare April 14, 2026 22:43
@rkannan82 rkannan82 changed the title Add IsEphemeral() property to Partition interface Replace IsEphemeral() with fine-grained partition properties Apr 14, 2026
@rkannan82 rkannan82 force-pushed the kannan/ephemeral-partition-property branch from a1a9cca to 994395c Compare April 14, 2026 22:49
Replace the single IsEphemeral() method on the Partition interface with
four specific property methods: HasTTLExpiry(), SupportsFairness(),
SupportsVersioning(), and SupportsPartitions(). Each call site now uses
the property that governs its behavior, making it clearer why a check
exists and easier to set correct values for new partition types.

PartitionCount() returns 1 when SupportsPartitions() is false, enforcing
the single-partition constraint explicitly.

Describe task queue checks use Kind() == NORMAL since only user-facing
queues can be described — this is about the queue's visibility, not a
specific property.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@rkannan82 rkannan82 force-pushed the kannan/ephemeral-partition-property branch from 994395c to 2a7094c Compare April 14, 2026 22:53
@rkannan82 rkannan82 changed the title Replace IsEphemeral() with fine-grained partition properties Add fine-grained partition properties to replace Kind() checks Apr 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants