Skip to content

[WIP] Add Kafka event sourcing for Go functions#3950

Open
aliok wants to merge 4 commits into
knative:mainfrom
aliok:kafka-cloudevents-source
Open

[WIP] Add Kafka event sourcing for Go functions#3950
aliok wants to merge 4 commits into
knative:mainfrom
aliok:kafka-cloudevents-source

Conversation

@aliok

@aliok aliok commented Jul 16, 2026

Copy link
Copy Markdown
Member

WIP until knative-extensions/func-go#185 is merged

Summary

  • Add run.kafka config to func.yaml (brokers, topic, consumerGroup)
    instead of a separate invoke: kafka type
  • Deployer and runner automatically set FUNC_TRANSPORT=kafka plus
    KAFKA_BROKERS, KAFKA_TOPIC, KAFKA_CONSUMER_GROUP env vars from the
    config — user never sets these manually
  • Scaffolding main.go branches at runtime between kafka.Start() and
    ce.Start() based on FUNC_TRANSPORT
  • Add Kafka consumption docs to CloudEvents template READMEs
  • Add fmt.Println(e) to instanced CloudEvents template to match static
    template behavior

How it works

Users add a run.kafka section to their existing CloudEvents function:

invoke: cloudevent
run:
  kafka:
    brokers: "my-cluster-kafka-bootstrap.kafka:9092"
    topic: "my-topic"
    consumerGroup: "my-function-group"

No code changes — the same Handle(event.Event) method receives Kafka
messages automatically converted to CloudEvents by the runtime.

Dependencies

Limitations

  • Go only

Supersedes #3923

Closes #3940

@knative-prow knative-prow Bot requested review from dsimansk and jrangelramos July 16, 2026 11:59
@knative-prow

knative-prow Bot commented Jul 16, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: aliok
Once this PR has been reviewed and has the lgtm label, please assign dprotaso for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@knative-prow knative-prow Bot added the size/L 🤖 PR changes 100-499 lines, ignoring generated files. label Jul 16, 2026
@aliok aliok changed the title Add Kafka event sourcing for Go functions [WIP] Add Kafka event sourcing for Go functions Jul 16, 2026
@knative-prow knative-prow Bot added the do-not-merge/work-in-progress 🤖 PR should not merge because it is a work in progress. label Jul 16, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces Kafka-backed event consumption for Go CloudEvents functions by adding a run.kafka configuration block (brokers/topic/consumerGroup) and wiring deploy/run paths to select Kafka transport at runtime via environment variables.

Changes:

  • Adds run.kafka to the function model, validation, and func.yaml JSON schema.
  • Injects FUNC_TRANSPORT=kafka plus KAFKA_* env vars from run.kafka in the runner and Kubernetes/Knative deployers.
  • Updates Go CloudEvents templates/scaffolding and READMEs to support/describe Kafka consumption, and aligns instanced template logging with static template behavior.

Reviewed changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
templates/go/scaffolding/static-cloudevents/main.go Switches runtime start between CloudEvents HTTP and Kafka based on FUNC_TRANSPORT.
templates/go/scaffolding/instanced-cloudevents/main.go Same transport switch for instanced CloudEvents functions.
templates/go/cloudevents/README.md Adds Kafka consumption documentation for the Go CloudEvents template.
templates/go/cloudevents/function.go Adds event echo output to match static template behavior.
templates/go/.static-cloudevents/README.md Adds Kafka consumption documentation for the static Go CloudEvents template.
schema/func_yaml-schema.json Extends schema with run.kafka configuration (KafkaConfig).
pkg/scaffolding/signatures.go Minor formatting/comment tweak to signature map.
pkg/scaffolding/signatures_test.go Asserts kafka invoke/template remains unknown in scaffolding signature selection.
pkg/knative/deployer.go Appends Kafka env vars when generating/updating Knative services.
pkg/k8s/deployer.go Appends Kafka env vars during Kubernetes deployment generation.
pkg/k8s/deployer_test.go Adds unit tests for Kafka env var injection helper.
pkg/functions/runner.go Adds Kafka env var injection for local/host runner jobs.
pkg/functions/runner_test.go Adds unit tests validating runner env injection behavior.
pkg/functions/function.go Adds KafkaConfig, validation rules, and RunSpec.Kafka field.
pkg/functions/function_test.go Adds YAML round-trip and omit-empty tests for Run.Kafka.
docs/reference/func_create.md Updates generated CLI reference output to include a Go kafka template.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +23 to +27
## Consuming from Kafka

This function can also consume messages from Apache Kafka. Kafka messages are
automatically converted to CloudEvents and delivered to the same `Handle`
method -- no code changes required.
Comment on lines 32 to 35
go cloudevents
go http
go kafka
node cloudevents
Comment thread pkg/k8s/deployer.go
Comment on lines +702 to +713
func AppendKafkaEnvs(envVars []corev1.EnvVar, kafka *fn.KafkaConfig) []corev1.EnvVar {
if kafka == nil || kafka.Brokers == "" || kafka.Topic == "" || kafka.ConsumerGroup == "" {
return envVars
}
envVars = append(envVars,
corev1.EnvVar{Name: "FUNC_TRANSPORT", Value: "kafka"},
corev1.EnvVar{Name: "KAFKA_BROKERS", Value: kafka.Brokers},
corev1.EnvVar{Name: "KAFKA_TOPIC", Value: kafka.Topic},
corev1.EnvVar{Name: "KAFKA_CONSUMER_GROUP", Value: kafka.ConsumerGroup},
)
return envVars
}
Comment on lines 7 to 9
ce "knative.dev/func-go/cloudevents"
"knative.dev/func-go/kafka"

Comment on lines 7 to 9
ce "knative.dev/func-go/cloudevents"
"knative.dev/func-go/kafka"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/work-in-progress 🤖 PR should not merge because it is a work in progress. size/L 🤖 PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Kafka functions: initial implementation (invoke type, templates, scaffolding)

2 participants