Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 121 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# AGENTS.md — Red Hat Vector Fork Context

This file provides context for AI agents and developers working in this repository. It covers only what is specific to this Red Hat fork. For generic Vector development commands, project structure, and Rust conventions, see the [upstream AGENTS.md](https://github.com/vectordotdev/vector/blob/master/AGENTS.md).

## What This Repo Is

A fork of [vectordotdev/vector](https://github.com/vectordotdev/vector) maintained by Red Hat for OpenShift Logging. Vector runs as a DaemonSet on every OpenShift node, collecting application and infrastructure logs. It is deployed and configured by the [cluster-logging-operator](https://github.com/openshift/cluster-logging-operator) — Vector is never run standalone in production.

**Upstream remote:** `vectordev` (git@github.com:vectordotdev/vector.git)
**Fork origin:** `origin` (git@github.com:ViaQ/vector.git)

## Critical: Branch Selection

**Never commit code to `rh-main`.** It contains only documentation.

| Branch | Vector Version | Rust | Use |
|--------|---------------|------|-----|
| `v0.47.0-rh` | v0.47.0 | 1.85 | Stable — OpenShift 6.2–6.5 |
| `v0.54.0-rh` | v0.54.0 | 1.92 | Next — future OpenShift releases |

Always check out the appropriate version branch before making changes:

```bash
git checkout v0.47.0-rh # stable
git checkout v0.54.0-rh # next
```

When a fix applies to both branches, submit separate PRs per branch.

## Build Quick Reference

The Makefile defaults to `FEATURES=ocp-logging` on Linux (not upstream's `default` feature set).

```bash
make build # cargo build --release --no-default-features --features ocp-logging
make test # cargo nextest run ... --features ocp-logging --test-threads 1
make fmt # rustfmt
make check-clippy # clippy lints
```

Key differences from upstream builds:
- **`--features ocp-logging`** — curated subset of Vector, not the full feature set
- **`--test-threads 1`** — single-threaded tests to avoid OOM in CI
- **`lld` linker** — configured in `.cargo/config.toml` (GNU ld OOMs during linking)
- **`CFLAGS=-g0 -O3`** — set in the build target for release optimization

## `ocp-logging` Feature Scope

This is the curated subset of Vector components enabled for OpenShift. Only these are compiled into the Red Hat binary.

**Sources:** stdin, file_descriptor, file, journald, kubernetes_logs, prometheus, internal_metrics, demo_logs, http_server, syslog, opentelemetry

**Transforms:** route, dedupe, filter, remap, log_to_metric, lua, throttle, reduce, detect_exceptions

**Sinks (v0.47.0-rh):** aws_cloudwatch_logs, aws_s3, azure_monitor_logs, elasticsearch, file, kafka, loki, console, prometheus, gcp, splunk_hec, http, socket, opentelemetry

**Sinks (v0.54.0-rh adds):** azure_logs_ingestion

**Also enabled:** api, api-client, unix

Components not in this list are **not available** in the Red Hat binary even though they exist in the source tree.

## Fork-Specific Files

These files exist in this fork but not in upstream:

| Path | Purpose |
|------|---------|
| `patch/openssl/` | Forked openssl crate — enables FIPS compliance and TLS security profile enforcement (LOG-2552, LOG-3398) |
| `patch/hyper/` | Forked hyper crate — HTTP behavior modifications |
| `thirdparty/` | Bundled binaries: `protoc`, `cargo-nextest`, `cargo-deny` (no external downloads in CI) |
| `Dockerfile` | Developer/local build image (UBI 9 + rustup) |
| `Dockerfile.art` | ART production build (UBI 9 minimal + RPM rust-toolset, `make build-offline`) |
| `Dockerfile.in` | OSBS/Brew template with `REMOTE_SOURCE` markers for Cachito |
| `Dockerfile.unit` | CI unit test runner |
| `OWNERS` | Kubernetes-style code review assignments |
| `CLAUDE.md` | Stub pointing to this file |

## Patched Crates (`[patch.crates-io]`)

**Both branches:**
- `openssl = { path = "patch/openssl" }` — Red Hat's forked openssl crate for FIPS and TLS security profiles
- `hyper = { path = "patch/hyper" }` — HTTP behavior patches
- `ntapi` — pinned revision for Windows alignment bug (inherited from upstream)

**v0.47.0-rh only** (removed in v0.54.0-rh):
- `tokio-util` — vectordotdev fork for framed-read-continue-on-error
- `nix` — vectordotdev fork for memfd gnu/musl support

When modifying TLS or HTTP behavior, check `patch/openssl/` and `patch/hyper/` — the standard crate.io versions are not used.

## Jira Conventions

Issues are tracked in [Red Hat JIRA](https://issues.redhat.com) under the **LOG** project.

- Prefix commit messages with the Jira ticket: `LOG-XXXX: description`
- Reference tickets in PR descriptions
- Use conventional commit format for PR titles: `fix(scope): description`

## Code Review (OWNERS)

```
approvers: jcantrill, alanconway, xperimental
reviewers: jcantrill, Clee2691, syedriko, vparfonov, cahartma
```

Dockerfile changes get the `midstream/Dockerfile` label automatically.

## Upstream References

For generic Vector development, use these upstream docs (they apply to this fork's Rust code):

| Topic | Upstream Document |
|-------|-------------------|
| Dev commands, project structure, common patterns | [AGENTS.md](https://github.com/vectordotdev/vector/blob/master/AGENTS.md) |
| Contribution workflow, PR format | [CONTRIBUTING.md](https://github.com/vectordotdev/vector/blob/master/CONTRIBUTING.md) |
| Internal engine architecture (topology, fanout, buffering) | [docs/ARCHITECTURE.md](https://github.com/vectordotdev/vector/blob/master/docs/ARCHITECTURE.md) |
| Rust coding conventions (agent-oriented) | [docs/RUST_STYLE.md](https://github.com/vectordotdev/vector/blob/master/docs/RUST_STYLE.md) |
| Code style (logging, metrics, error handling, concurrency) | [STYLE.md](https://github.com/vectordotdev/vector/blob/master/STYLE.md) |
| Developer setup, testing, benchmarking | [docs/DEVELOPING.md](https://github.com/vectordotdev/vector/blob/master/docs/DEVELOPING.md) |
| Component specification | [docs/specs/component.md](https://github.com/vectordotdev/vector/blob/master/docs/specs/component.md) |
130 changes: 130 additions & 0 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
# Architecture — Red Hat Vector Fork

This document covers what is architecturally specific to this fork. For Vector's internal engine architecture (topology, fanout, buffering, component construction), see the upstream [docs/ARCHITECTURE.md](https://github.com/vectordotdev/vector/blob/master/docs/ARCHITECTURE.md).

## Deployment Model

Vector runs as a **DaemonSet** on every OpenShift node, managed by the [cluster-logging-operator](https://github.com/openshift/cluster-logging-operator). It is not deployed standalone.

- Configuration is injected via ConfigMap by the operator
- The operator provides a `run.sh` entrypoint script
- Vector collects container logs from `/var/log/containers/` (file source) and infrastructure logs from journald
- Pod metadata is extracted from file paths and enriched by transforms

## `ocp-logging` Feature

The fork defines a custom Cargo feature `ocp-logging` in `Cargo.toml` that compiles only the Vector components needed for OpenShift. The Makefile sets `FEATURES=ocp-logging` by default on Linux.

### Enabled Components

**Sources:** stdin, file_descriptor, file, journald, kubernetes_logs, prometheus, internal_metrics, demo_logs, http_server, syslog, opentelemetry

**Transforms:** route, dedupe, filter, remap, log_to_metric, lua, throttle, reduce, detect_exceptions

**Sinks:**

| Sink | v0.47.0-rh | v0.54.0-rh |
|------|:----------:|:----------:|
| aws_cloudwatch_logs | yes | yes |
| aws_s3 | yes | yes |
| azure_logs_ingestion | — | yes |
| azure_monitor_logs | yes | yes |
| elasticsearch | yes | yes |
| file | yes | yes |
| kafka | yes | yes |
| loki | yes | yes |
| console | yes | yes |
| prometheus | yes | yes |
| gcp | yes | yes |
| splunk_hec | yes | yes |
| http | yes | yes |
| socket | yes | yes |
| opentelemetry | yes | yes |

Components not in `ocp-logging` exist in the source tree but are not compiled into the Red Hat binary.

## Patched Dependencies

The fork carries local patches for two crates via `[patch.crates-io]` in `Cargo.toml`:

### `patch/openssl/`

Forked from the `openssl` crate. Two purposes:

1. **FIPS compliance** (LOG-2552): Vector uses OpenSSL instead of upstream's Ring for all TLS operations, enabling FIPS-validated cryptography.
2. **TLS security profiles** (LOG-3398): Enforces OpenShift's `TLSSecurityProfile` settings (cipher suites, TLS versions) at the Vector level, so cluster administrators control the security posture globally.

### `patch/hyper/`

Forked from the `hyper` crate. Contains HTTP behavior modifications for reliability in OpenShift environments.

### Other patches (v0.47.0-rh only)

- `tokio-util` — vectordotdev fork for framed-read-continue-on-error behavior
- `nix` — vectordotdev fork for memfd gnu/musl support

Both were dropped in v0.54.0-rh (fixed upstream).

## Build System

### Dockerfiles

| File | Base | Rust | Purpose |
|------|------|------|---------|
| `Dockerfile` | `ubi9/ubi` | rustup | Local/developer builds |
| `Dockerfile.art` | `ubi9/ubi-minimal` | RPM `rust-toolset` | ART production image (shipped to customers). Uses `make build-offline` for air-gapped builds |
| `Dockerfile.in` | `ubi9/ubi-minimal` | RPM | OSBS/Brew template. Has `## EXCLUDE BEGIN ##` markers for OSBS processing. Uses `REMOTE_SOURCE` for Cachito-prefetched dependencies |
| `Dockerfile.unit` | `ubi9/ubi` | rustup | CI unit test runner. Selective COPY for build cache efficiency |

### Build constraints

- **`lld` linker**: Configured in `.cargo/config.toml`. GNU ld runs out of memory during linking.
- **`protoc` bundled**: Pre-built binaries in `thirdparty/protoc/` (architecture-aware). No network download during build.
- **`cargo-nextest` and `cargo-deny` bundled**: In `thirdparty/` for CI reproducibility.
- **Single-threaded tests**: `--test-threads 1` in the Makefile `test` target to limit memory usage.

### CI/CD

No in-repo CI pipelines. Builds happen in Red Hat's internal ART/OSBS infrastructure:
- `Dockerfile.art` → production image via ART (Automated Release Tooling)
- `Dockerfile.in` → processed by OSBS (OpenShift Build Service) / Brew

GitHub Actions on the dev branches are inherited from upstream and mostly not Red Hat-specific.

## Key Design Decisions

### OpenSSL over Ring (LOG-2552)

**Decision:** Replace Ring with OpenSSL for all cryptographic operations.
**Why:** FIPS compliance requires a validated crypto library. Ring is not FIPS-validated. OpenSSL is.
**Trade-off:** Adds the `patch/openssl/` maintenance burden and prevents trivial upstream merges of TLS-related code.

### TLS Security Profile Enforcement (LOG-3398)

**Decision:** All TLS connections respect OpenShift's `TLSSecurityProfile` (cipher suites, min/max TLS version).
**Why:** OpenShift cluster administrators expect uniform TLS policy enforcement across all components.
**Depends on:** The OpenSSL patch above — upstream's Ring-based TLS does not support security profile configuration.

### `detect_exceptions` Transform (LOG-6155)

**Decision:** A Red Hat-specific transform for multiline exception detection (Java stack traces, Python tracebacks, etc.) with configurable `message_key`.
**Why:** Upstream's multiline handling is less flexible. OpenShift workloads heavily use Java, and stack traces must be grouped into single log events.

## Differences Between Version Branches

| Aspect | v0.47.0-rh | v0.54.0-rh |
|--------|-----------|-----------|
| Upstream base | Vector v0.47.0 | Vector v0.54.0 |
| Rust edition | 2021 | 2024 |
| MSRV (rust-toolchain.toml) | 1.85 | 1.92 |
| `azure_logs_ingestion` sink | — | enabled |
| `tokio-util` / `nix` patches | present | removed (fixed upstream) |
| GCP Workload Identity Federation | — | LOG-9171 |
| TLS curve configuration | — | LOG-8968 |

## Upstream Sync Strategy

1. **Version-based merges:** Each branch is based on a specific upstream tag (e.g., v0.47.0). Upstream changes are incorporated by creating a new branch from the next upstream tag and replaying Red Hat patches.
2. **Patch minimization:** Features are contributed upstream when possible. Once accepted, the fork patch is simplified or removed on the next branch.
3. **Cherry-picks:** Critical fixes from upstream are cherry-picked into active branches between version merges.
4. **Conflict zones:** `Cargo.toml` (feature definitions, patch section) and TLS-related code are the primary conflict areas during merges.
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
AGENTS.md
67 changes: 67 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Contributing — Red Hat Vector Fork

This guide covers what is specific to contributing to this fork. For generic Vector development workflow, coding standards, and testing, see the [upstream CONTRIBUTING.md](https://github.com/vectordotdev/vector/blob/master/CONTRIBUTING.md).

## Branching Strategy

| Branch | Purpose |
|--------|---------|
| `rh-main` | Documentation only — **never commit code here** |
| `v0.47.0-rh` | Stable development — OpenShift 6.2–6.5 |
| `v0.54.0-rh` | Next-gen development — future releases |

## Submitting Changes

1. Check out the target version branch: `git checkout v0.47.0-rh`
2. Create a feature branch: `git checkout -b LOG-XXXX-short-description`
3. Make changes, test with `make test`
4. Push to your fork and open a PR against the version branch (not `rh-main`)
5. Include the Jira ticket ID (LOG-XXXX) in the PR title and description

If a fix applies to multiple version branches, submit a separate PR for each.

## Commit Messages

Use the Jira ticket ID and conventional commit format:

```
fix(file-source): handle symlink rotation during log collection

Fixes LOG-7506
```

PR titles follow upstream's [conventional commits](https://www.conventionalcommits.org) format:
```
feat(scope): description
fix(scope): description
chore(scope): description
```

## Code Review

Reviews follow the [OWNERS](OWNERS) file:

- **Approvers:** jcantrill, alanconway, xperimental
- **Reviewers:** jcantrill, Clee2691, vparfonov, cahartma

PRs require at least one approval. Significant changes require two approvers.

## Red Hat-Specific Code

When adding patches that diverge from upstream:

- Reference the Jira ticket in code comments explaining why
- If the feature could be upstreamed, note it in the PR description
- Update [ARCHITECTURE.md](ARCHITECTURE.md) if the change affects design decisions or the `ocp-logging` feature scope

## Contributing Back to Upstream

We minimize fork divergence by upstreaming changes when possible:

1. Implement and test the fix in this fork first
2. If the fix is generic (not OpenShift-specific), propose it to [upstream Vector](https://github.com/vectordotdev/vector)
3. Once accepted upstream, simplify or remove the fork patch on the next upstream merge

## Filing Issues

File issues at [Red Hat JIRA](https://issues.redhat.com) under the **LOG** project. Include [must-gather](https://github.com/openshift/cluster-logging-operator/tree/master/must-gather) output for debugging.
29 changes: 26 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,33 @@
# Vector
# Vector — Red Hat OpenShift Logging

This repo is a fork of [vector](https://github.com/vectordotdev/vector) and contains patches carried by Red Hat OpenShift Logging. This is a log collector and forwarder that resides on each OpenShift node to gather application and node logs. Please refer to the [cluster-logging-operator](https://github.com/openshift/cluster-logging-operator) for details regarding the operator that deploys and configures this image. This image is intended to be run in conjunction with the configuration and `run.sh` files provided by the operator. Experiences with the image outside that context may vary.
This is a fork of [Vector](https://github.com/vectordotdev/vector) maintained for Red Hat OpenShift Logging. Vector is a high-performance observability data pipeline written in Rust. This fork carries patches for OpenShift integration, FIPS-compliant TLS (OpenSSL), and reliability improvements.

The `rh-main` branch is empty except for this file. The branches used by various Red Hat releases are summarized here:
Vector runs as a DaemonSet on each OpenShift node, collecting application and infrastructure logs. It is deployed and configured by the [cluster-logging-operator](https://github.com/openshift/cluster-logging-operator). This image is intended to run with the configuration and `run.sh` files provided by the operator.

## Documentation

| Document | Purpose |
|----------|---------|
| [AGENTS.md](AGENTS.md) | AI agent and developer quick reference — fork-specific context, build commands, patched crates |
| [CONTRIBUTING.md](CONTRIBUTING.md) | How to submit changes — branching, PRs, code review, Jira conventions |
| [ARCHITECTURE.md](ARCHITECTURE.md) | Fork architecture — deployment model, `ocp-logging` feature scope, build system, design decisions |
| [Upstream AGENTS.md](https://github.com/vectordotdev/vector/blob/master/AGENTS.md) | Generic Vector development commands, project structure, Rust conventions |

## Quick Start (building from source)

```bash
git checkout v0.47.0-rh # or v0.54.0-rh
make build # builds with --features ocp-logging
make test # runs tests with --test-threads 1
```

## Release Branches

The `rh-main` branch contains only documentation. All development happens on version-specific branches:

| Release | Branch | Vector Version | Status |
|---------|-------------|----------------|--------|
| next | v0.54.0-rh | v0.54.0 | Development |
| 6.5 | v0.47.0-rh | v0.47.0 | Current |
| 6.4 | v0.47.0-rh | v0.47.0 | Current |
| 6.3 | v0.47.0-rh | v0.47.0 | EOL |
Expand All @@ -30,6 +52,7 @@ This project varies from the upstream with the following features:
| LOG-6789 | [Resolve error when using AWS credentials file authentication](https://github.com/ViaQ/vector/pull/197) | 6.2 | [Accepted](https://github.com/vectordotdev/vector/pull/22831) |
| LOG-7013 | [Update cloudwatch logs max event size to match new AWS limit](https://github.com/ViaQ/vector/pull/201) | 6.2 | [Accepted](https://github.com/vectordotdev/vector/pull/22886) |
| LOG-7417 | [Fix loki event timestamp out of range panic ](https://github.com/ViaQ/vector/pull/217) | 0.37.1-rh | [Cherry-pick](https://github.com/vectordotdev/vector/pull/20780) |

## Issues

Any issues can be filed at [Red Hat JIRA](https://issues.redhat.com). Please
Expand Down