Add an option to spread indexing pipelines over time - #6655
Conversation
927f22c to
2262d4a
Compare
loutPhilipps
changed the title
| let indexing_cycle_opt: Option<IndexingCycle> = | ||
| (spread_indexing_pipelines || cooperative_indexing_permits_opt.is_some()).then(|| { | ||
| IndexingCycle::new( | ||
| &pipeline_id, | ||
| indexing_settings.commit_timeout(), | ||
| cooperative_indexing_permits, | ||
| cooperative_indexing_permits_opt, | ||
| ) | ||
| }); |
There was a problem hiding this comment.
This is where we enable the IndexingCycle, either if spreading pipelines or if cooperative indexing is enabled
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 927f22cfca
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
eb67cf2 to
815d873
Compare
815d873 to
e84a525
Compare
| let num_pipelines = 100; | ||
| let num_steps = 15; | ||
| let semaphore = Arc::new(Semaphore::new(num_threads)); |
There was a problem hiding this comment.
My two cents is that having the semaphore in the reconcile test doesn't change anything to the test so I decided to test the general case instead. I can add another test for nudging to phase with cooperative indexing if needed.
|
@loutPhilipps Your current approach only adds the phase at the start of the pipeline.
ctx.schedule_self_msg(
self.indexing_settings.commit_timeout(),
commit_timeout_message,
);as well. Also we cannot merge something like that without testing! |
| | `enable_otlp_endpoint` | If true, enables the OpenTelemetry exporter endpoint to ingest logs and traces via the OpenTelemetry Protocol (OTLP). | `false` | | ||
| | `cpu_capacity` | Advisory parameter used by the control plane. The value can expressed be in threads (e.g. `2`) or in term of millicpus (`2000m`). The control plane will attempt to schedule indexing pipelines on the different nodes proportionally to the cpu capacity advertised by the indexer. It is NOT used as a limit. All pipelines will be scheduled regardless of whether the cluster has sufficient capacity or not. The control plane does not attempt to spread the work equally when the load is well below the `cpu_capacity`. Users who need a balanced load on all of their indexer nodes can set the `cpu_capacity` to an arbitrarily low value as long as they keep it proportional to the number of threads available. | `num threads available` | | ||
| | `enable_cooperative_indexing` | Enable sharing resources more efficiently when the number of indexes actively written to is significantly higher than the number of cores but might decrease the overall indexing throughput. | `false` | | ||
| | `enable_spread_indexing_pipelines` | Enable spreading indexing pipelines in time to make sure they are not in sync. | `false` | |
There was a problem hiding this comment.
can we make that an environment flag? (the goal is just to make it as light as possible).
2 participants
Tip
This PR is better reviewed commit-by-commit.
Problem
An indexer node's pipelines are started at the same time with the same commit timeout, leading to them committing at the same time. This is not an efficient usage of resources when there are many pipelines on a single indexer node.
This problem was addressed in #4877 with the cooperative indexing feature by introducing both a time spread of pipelines, as well as a semaphore to prevent running too many pipelines concurrently.
When having a large index though, we don't really want to have a semaphore, but do want to spread indexing pipelines.
What this PR does
Add an option
enable_spread_indexing_pipelineson the indexer's config. For backwards compatibility reasons, enabling cooperative indexing also enables this feature, with the addition of a semaphore.The implementation of
CooperativeIndexingCyclehas been mostly left intact, with theindexing_permitsbecoming optional.