Skip to content

feat(http): migrate ClickHouse sink to Hyper 1 - #26157

Open
pront wants to merge 6 commits into
masterfrom
pront-http-v1-honeycomb
Open

feat(http): migrate ClickHouse sink to Hyper 1#26157
pront wants to merge 6 commits into
masterfrom
pront-http-v1-honeycomb

Conversation

@pront

@pront pront commented Aug 19, 2026

Copy link
Copy Markdown
Member

Summary

Motivation

The HTTP v1 migration needs a real production sink caller so the native client can be validated beyond isolated transport tests. This PR migrates the ClickHouse sink, which has Docker-backed integration coverage, while keeping the existing request, retry, delivery, and internal telemetry contracts.

The earlier Honeycomb migration is reverted in this PR's history; Honeycomb remains on the existing client. No configuration surface is added, and the migration is limited to ClickHouse so behavior can be compared against the released binary before more sinks move.

Changes

  • Migrate ClickHouse healthchecks and event batches to the private native Hyper 1 client.
  • Add a shared native Hyper 1 HTTP sink service for buffered request bodies.
  • Apply authentication directly to native HTTP 1 request headers; no V0-to-V1 request bridge remains on the migrated ClickHouse path.
  • Use version-neutral names at the migrated sink boundary; retain the legacy client only for ClickHouse Arrow schema resolution.
  • Adapt Hyper 1 request/response metadata to the existing HTTP internal telemetry events.
  • Generalize HTTP status retry logic so the native client preserves retriable transport-error behavior.

References

Vector configuration

No production configuration changes. The exact YAML used for the local A/B comparison was:

api:
  enabled: true
  address: 127.0.0.1:8686

sources:
  input:
    type: stdin
    framing:
      method: newline_delimited
    decoding:
      codec: json
  internal:
    type: internal_metrics
    scrape_interval_secs: 0.5

sinks:
  clickhouse:
    type: clickhouse
    inputs:
      - input
    endpoint: http://127.0.0.1:18123
    database: default
    table: events
    compression: none
    batch:
      max_events: 1
      timeout_secs: 1
  telemetry:
    type: console
    inputs:
      - internal
    target: stdout
    encoding:
      codec: json

The same configuration was run against a deterministic local mock ClickHouse endpoint with the custom branch binary and the released Vector 0.57.0 binary.

How did you test this PR?

  • make fmt
  • make check-fmt
  • make check-clippy
  • make check-events
  • make build-licenses
  • Native HTTP auth unit test: 1 passed.
  • HTTP transport/unit tests: 47 passed.
  • ClickHouse-focused unit tests: 39 passed.
  • Full Vector workspace suite: 3499 passed, 13 skipped.
  • ClickHouse integration suite: 51 passed, 377 skipped.
  • HTTP client integration suite: 26 passed, 443 skipped.
  • Deterministic custom-vs-release A/B comparison:
    • both binaries sent one SELECT 1 healthcheck and one ClickHouse JSONEachRow event batch;
    • both delivered one event, with receivedEventsTotal=1, sentEventsTotal=1, and sentBytesTotal=119 for the ClickHouse sink;
    • the request method, query parameters, content type, and event body matched;
    • component telemetry was queried through the local gRPC API and the ClickHouse delivery counters matched.

The A/B comparison exposed one expected wire-level difference for follow-up: the custom client sends Accept-Encoding: identity, while Vector 0.57.0 advertises zstd,gzip,deflate,br. User-agent differences are expected because the binaries have different versions.

Kubernetes E2E was not run locally because Minikube was stopped.

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. This is an internal HTTP client migration with no configuration change or intended user-facing behavior change.

Contributor Guidelines

  • Please read our Vector contributor resources.
  • Do not hesitate to use @vectordotdev/vector to reach out to us regarding the PR.
  • Some CI checks run only after we manually approve them. To catch issues early, add a pre-push hook (template) or run the following locally before pushing:
    • make fmt
    • make check-clippy (auto-fix with make clippy-fix)
    • make test
  • After a review is requested, please avoid force pushes to help us review incrementally.
    • Feel free to push as many commits as you want. They will be squashed into one before merging.
    • For example, you can run git merge origin master and git push.
  • If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
    run make build-licenses to regenerate the license inventory (https://github.com/vectordotdev/vrl/blob/main/LICENSE-3rdparty.csv) and commit the changes (if any). More details on the dd-rust-license-tool (https://crates.io/crates/dd-rust-license-tool).

@github-actions github-actions Bot added the domain: sinks Anything related to the Vector's sinks label Aug 19, 2026
@pront pront added the no-changelog Changes in this PR do not need user-facing explanations in the release changelog label Aug 19, 2026
@pront pront changed the title chore(http): migrate Honeycomb sink to Hyper 1 client feat(http): migrate ClickHouse sink to Hyper 1 Aug 19, 2026
@pront
pront marked this pull request as ready for review August 19, 2026 19:29
@pront
pront requested a review from a team as a code owner August 19, 2026 19:29
@pront

pront commented Aug 19, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 2900b4bba2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain: sinks Anything related to the Vector's sinks no-changelog Changes in this PR do not need user-facing explanations in the release changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant