Replace fluentd with fluent-bit in the operator

[hjiawei]

Description

New feature (Calico Enterprise): the LogCollector controller now deploys fluent-bit (calico-fluent-bit in calico-system) in place of fluentd, completing the log-collector migration on the operator side.

• Resource identity migration: namespace tigera-fluentd → calico-system; DaemonSet/ServiceAccount fluentd-node → calico-fluent-bit; TLS secret → calico-fluent-bit-tls; metrics port 9081 → 2020 (fluent-bit's built-in HTTP server).
• Configuration is rendered in fluent-bit's YAML schema into per-OS ConfigMaps (calico-fluent-bit-conf/-windows), subPath-mounted on Linux and directory-mounted on Windows, started with -c. The render loads the Go plugins via plugins_file, defines parsers inline, applies the record-transform Lua filter, and inlines user-provided fluent-bit YAML filter lists. A rendered-config hash annotation rolls the pods on config-only changes.
• Tail inputs use the producing components' real log paths, with SQLite offset DBs and filesystem buffering under /var/log/calico/calico-fluent-bit; a pos-migrator init container (Linux and Windows) seeds offsets from the legacy fluentd .pos files and pre-creates the tailed directories. Windows tails the same log types the fluentd Windows variant shipped.
• The linseed output matches only Linseed-bound tags, authenticates with mTLS + the pod's ServiceAccount token, and retries without limit against bounded filesystem storage. S3/Splunk/Syslog outputs mirror fluentd's per-type fan-out (standard AWS credential env vars, endpoint scheme honored, syslog ships the whole record as JSON via a per-output Lua processor with TLS properly enabled).
• NonClusterHost renders the :9880 HTTP input with client-certificate verification; the input Service is cleaned up when the resource is removed.
• eks-log-forwarder runs the fluent-bit image with a rendered in_eks → linseed pipeline and health probes (no startup init container; the input plugin resolves its resume point from Linseed).
• Probes hit :2020/api/v1/health; the ServiceMonitor scrapes plain HTTP (fluent-bit's monitoring server has no TLS) with access restricted by the component NetworkPolicy, and legacy fluentd monitors are deleted.
• The LogCollector controller no longer creates/owns the calico-system namespace (deleting the LogCollector must not garbage-collect it); the deprecated fluentdDaemonSet override is honored as an alias of the new calicoFluentBitDaemonSet field (with container-name translation); legacy tigera-fluentd resources — the namespace last — are cleaned up idempotently.

Testing: render, controller, and monitor unit suites updated/extended (ConfigMap-content assertions replacing the env-var assertions); the rendered configuration was validated against the real fluent-bit binary; the full migration was validated end-to-end on a test cluster — all log types flowing to Linseed/Elasticsearch, fluentd resources fully removed, tail-offset handover without re-shipping, NonClusterHost ingestion with client-certificate enforcement, and EKS/Windows render shapes verified.

Release Note

The LogCollector now deploys fluent-bit (calico-fluent-bit in calico-system) in place of fluentd, with operator-rendered configuration and automatic migration of fluentd tail positions.

For PR author

• Tests for change.
• If changing pkg/apis/, run make gen-files
• If changing versions, run make gen-versions

For PR reviewers

A note for code reviewers - all pull requests must have the following:

• Milestone set according to targeted release.
• Appropriate labels:
  • kind/bug if this is a bugfix.
  • kind/enhancement if this is a a new feature.
  • enterprise if this PR applies to Calico Enterprise only.

[Copilot]

Pull request overview

This PR completes the operator-side migration of the LogCollector from fluentd to fluent-bit (Calico Enterprise), including updated resource identities (namespace/workload names), rendered configuration shape, and updated policy/monitoring expectations across the rendering and controllers.

Changes:

• Replace fluentd-specific rendering/controller wiring with fluent-bit equivalents (names, ports, RBAC, ServiceMonitor behavior).
• Update expected Calico policies/selectors to reflect fluent-bit running in calico-system and using port 2020.
• Extend the LogCollector CRD/API to add calicoFluentBitDaemonSet and treat the deprecated fluentdDaemonSet as an alias during migration.

Reviewed changes

Copilot reviewed 47 out of 48 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
pkg/render/tiers/tiers_test.go	Updates tier rendering tests to stop expecting a separate LogCollector namespace.
pkg/render/testutils/expected_policies/node_local_dns_ipv6.json	Removes legacy tigera-fluentd namespace from node-local-dns selector expectations.
pkg/render/testutils/expected_policies/node_local_dns_ipv4.json	Removes legacy tigera-fluentd namespace from node-local-dns selector expectations.
pkg/render/testutils/expected_policies/node_local_dns_dual.json	Removes legacy tigera-fluentd namespace from node-local-dns selector expectations.
pkg/render/testutils/expected_policies/linseed.json	Updates Linseed ingress policy sources from fluentd labels/namespace to fluent-bit in calico-system.
pkg/render/testutils/expected_policies/linseed_ocp.json	Same as above for OpenShift policy expectations.
pkg/render/testutils/expected_policies/linseed_ocp_dpi_enabled.json	Same as above for OpenShift + DPI-enabled scenario.
pkg/render/testutils/expected_policies/linseed_dpi_enabled.json	Same as above for DPI-enabled scenario.
pkg/render/testutils/expected_policies/guardian.json	Updates Guardian policy expectations to allow fluent-bit instead of fluentd.
pkg/render/testutils/expected_policies/guardian_ocp.json	Same as above for OpenShift policy expectations.
pkg/render/testutils/expected_policies/fluentbit_unmanaged.json	Updates fluent-bit “allow metrics” policy expectations (name/namespace/selector/port).
pkg/render/testutils/expected_policies/fluentbit_unmanaged_ocp.json	Same as above for OpenShift.
pkg/render/testutils/expected_policies/fluentbit_managed.json	Updates managed-cluster policy expectations for fluent-bit metrics access.
pkg/render/testutils/expected_policies/es-gateway.json	Updates ES gateway ingress policy sources from fluentd to fluent-bit.
pkg/render/testutils/expected_policies/es-gateway_ocp.json	Same as above for OpenShift.
pkg/render/testutils/expected_policies/dns.json	Removes legacy tigera-fluentd namespace from DNS policy selector expectations.
pkg/render/testutils/expected_policies/dns_ocp.json	Same as above for OpenShift DNS policy expectations.
pkg/render/nonclusterhost/nonclusterhost.go	Adds Linseed RBAC permissions for DNS logs in non-cluster-host flow.
pkg/render/nonclusterhost/nonclusterhost_test.go	Updates RBAC test expectations to include dnslogs.
pkg/render/monitor/monitor.go	Renames fluentd monitoring constants to fluent-bit and switches ServiceMonitor to plain HTTP fluent-bit metrics endpoint.
pkg/render/monitor/monitor_test.go	Updates monitor rendering tests for renamed ServiceMonitor and the removal of TLS/scheme relabeling.
pkg/render/manager.go	Updates manager-rendered NetworkPolicy destination service name and cluster-wide namespace list (includes a problematic entry noted in comments).
pkg/render/manager_test.go	Updates manager tests to expect fluent-bit service names.
pkg/render/logstorage/linseed/linseed.go	Updates Linseed policy source entity rule reference to fluent-bit.
pkg/render/logstorage/esgateway/esgateway.go	Updates ES gateway policy source entity rule reference to fluent-bit.
pkg/render/intrusion_detection.go	Updates comments to reflect fluent-bit consuming IDS logs.
pkg/render/guardian.go	Updates Guardian policy source entity rule reference to fluent-bit.
pkg/render/fluentd.go	Removes the fluentd renderer implementation.
pkg/render/fluentd_test.go	Removes fluentd rendering unit tests.
pkg/render/common/elasticsearch/service.go	Renames isFluentd flag to isFluentBit and updates related commentary for managed-cluster Linseed routing.
pkg/render/applicationlayer/applicationlayer.go	Updates comment about WAF logs being consumed by fluent-bit.
pkg/imports/crds/operator/operator.tigera.io_logcollectors.yaml	Adds calicoFluentBitDaemonSet and updates fluentdDaemonSet docs/enums for the migration (some descriptions still say “Fluentd”, noted in comments).
pkg/controller/tiers/tiers_controller.go	Stops treating LogCollector as its own namespace when computing tier namespace lists.
pkg/controller/monitor/prometheus.go	Watches fluent-bit ServiceMonitor name instead of fluentd.
pkg/controller/monitor/monitor_controller.go	Switches monitored TLS secret from fluentd to fluent-bit.
pkg/controller/monitor/monitor_controller_test.go	Updates controller tests to reference fluent-bit ServiceMonitor and secrets.
pkg/controller/logstorage/secrets/secret_controller.go	Updates upstream cert collection references from fluentd TLS secret to fluent-bit TLS secret.
pkg/controller/logstorage/secrets/secret_controller_test.go	Updates tests to create/check fluent-bit TLS secret scenarios.
pkg/controller/logcollector/logcollector_controller.go	Rewires controller from fluentd to fluent-bit rendering (secrets, filters, services, namespace ownership behavior).
pkg/controller/logcollector/logcollector_controller_test.go	Updates controller tests for fluent-bit DaemonSet naming, config-driven outputs, and non-creation of calico-system.
pkg/components/enterprise.go	Renames enterprise component definitions from fluentd to fluent-bit (linux/windows).
hack/gen-versions/enterprise.go.tpl	Updates version generation template keys/structs for fluent-bit components.
config/enterprise_versions.yml	Replaces fluentd component entries with fluent-bit equivalents.
api/v1/zz_generated.deepcopy.go	Adds deepcopy support for CalicoFluentBitDaemonSet on LogCollectorSpec.
api/v1/logcollector_types.go	Adds calicoFluentBitDaemonSet field and documents fluentdDaemonSet as deprecated alias.
api/v1/fluentd_daemonset_types.go	Updates override container/initContainer enums to fluent-bit naming (backward-compat concerns noted in comments).

Files not reviewed (1)

• api/v1/zz_generated.deepcopy.go: Generated file

[Copilot]

Pull request overview

Copilot reviewed 49 out of 50 changed files in this pull request and generated 1 comment.

Files not reviewed (1)

• api/v1/zz_generated.deepcopy.go: Generated file

Comments suppressed due to low confidence (2)

pkg/controller/monitor/monitor_controller.go:156

• Monitor no longer scrapes fluent-bit over mTLS (serviceMonitorFluentBit is plain HTTP), so the controller shouldn't watch or require the fluent-bit TLS secret. Keeping it here creates an unnecessary dependency on the LogCollector controller and can degrade Monitor reconciliation when LogCollector isn't installed (secret not found).

	for _, secret := range []string{
		certificatemanagement.CASecretName,
		esmetrics.ElasticsearchMetricsServerTLSSecret,
		monitor.PrometheusServerTLSSecretName,
		render.FluentBitTLSSecretName,
		render.NodePrometheusTLSServerSecret,
		kubecontrollers.KubeControllerPrometheusTLSSecret,
		render.EKSLogForwarderTLSSecretName,
	} {

pkg/controller/monitor/monitor_controller.go:345

• Since fluent-bit metrics scraping is now plain HTTP, Monitor's trusted bundle no longer needs the fluent-bit certificate. Including it makes reconciliation fail with NotFound if LogCollector hasn't created the secret yet (or isn't installed).

	for _, certificateName := range []string{
		esmetrics.ElasticsearchMetricsServerTLSSecret,
		render.FluentBitTLSSecretName,
		render.NodePrometheusTLSServerSecret,
		render.CalicoAPIServerTLSSecretName,
		kubecontrollers.KubeControllerPrometheusTLSSecret,
	} {

[Brian-McM]

This is a breaking change since the container name has changed from fluentd to fluentbit.

[hjiawei]

Right — the container rename would have broken existing fluentdDaemonSet overrides in two ways: CRD validation (already addressed by keeping both names in the enum: calico-fluent-bit;fluentd, and the legacy init-container name likewise) and override-to-container matching. The latter now goes through the shared containerNameAliases mechanism in pkg/render/common/components with fluentd → calico-fluent-bit and tigera-fluentd-prometheus-tls-key-cert-provisioner → calico-fluent-bit-tls-key-cert-provisioner, replacing the bespoke translation this branch had. Added render tests covering a legacy fluentdDaemonSet override that uses the fluentd-era names (resources land on the renamed containers) and the precedence rule when both fields are set.

The rename of the workload/container itself (kubectl logs -c fluentd, dashboards, etc.) remains an intentional, documented breaking change of the migration — it's the API contract (fluentdDaemonSet overrides) that stays compatible for the deprecation window.