[WIP] Add Kafka event sourcing for Go functions#3950
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: aliok The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Pull request overview
This PR introduces Kafka-backed event consumption for Go CloudEvents functions by adding a run.kafka configuration block (brokers/topic/consumerGroup) and wiring deploy/run paths to select Kafka transport at runtime via environment variables.
Changes:
- Adds
run.kafkato the function model, validation, andfunc.yamlJSON schema. - Injects
FUNC_TRANSPORT=kafkaplusKAFKA_*env vars fromrun.kafkain the runner and Kubernetes/Knative deployers. - Updates Go CloudEvents templates/scaffolding and READMEs to support/describe Kafka consumption, and aligns instanced template logging with static template behavior.
Reviewed changes
Copilot reviewed 16 out of 17 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| templates/go/scaffolding/static-cloudevents/main.go | Switches runtime start between CloudEvents HTTP and Kafka based on FUNC_TRANSPORT. |
| templates/go/scaffolding/instanced-cloudevents/main.go | Same transport switch for instanced CloudEvents functions. |
| templates/go/cloudevents/README.md | Adds Kafka consumption documentation for the Go CloudEvents template. |
| templates/go/cloudevents/function.go | Adds event echo output to match static template behavior. |
| templates/go/.static-cloudevents/README.md | Adds Kafka consumption documentation for the static Go CloudEvents template. |
| schema/func_yaml-schema.json | Extends schema with run.kafka configuration (KafkaConfig). |
| pkg/scaffolding/signatures.go | Minor formatting/comment tweak to signature map. |
| pkg/scaffolding/signatures_test.go | Asserts kafka invoke/template remains unknown in scaffolding signature selection. |
| pkg/knative/deployer.go | Appends Kafka env vars when generating/updating Knative services. |
| pkg/k8s/deployer.go | Appends Kafka env vars during Kubernetes deployment generation. |
| pkg/k8s/deployer_test.go | Adds unit tests for Kafka env var injection helper. |
| pkg/functions/runner.go | Adds Kafka env var injection for local/host runner jobs. |
| pkg/functions/runner_test.go | Adds unit tests validating runner env injection behavior. |
| pkg/functions/function.go | Adds KafkaConfig, validation rules, and RunSpec.Kafka field. |
| pkg/functions/function_test.go | Adds YAML round-trip and omit-empty tests for Run.Kafka. |
| docs/reference/func_create.md | Updates generated CLI reference output to include a Go kafka template. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ## Consuming from Kafka | ||
|
|
||
| This function can also consume messages from Apache Kafka. Kafka messages are | ||
| automatically converted to CloudEvents and delivered to the same `Handle` | ||
| method -- no code changes required. |
| go cloudevents | ||
| go http | ||
| go kafka | ||
| node cloudevents |
| func AppendKafkaEnvs(envVars []corev1.EnvVar, kafka *fn.KafkaConfig) []corev1.EnvVar { | ||
| if kafka == nil || kafka.Brokers == "" || kafka.Topic == "" || kafka.ConsumerGroup == "" { | ||
| return envVars | ||
| } | ||
| envVars = append(envVars, | ||
| corev1.EnvVar{Name: "FUNC_TRANSPORT", Value: "kafka"}, | ||
| corev1.EnvVar{Name: "KAFKA_BROKERS", Value: kafka.Brokers}, | ||
| corev1.EnvVar{Name: "KAFKA_TOPIC", Value: kafka.Topic}, | ||
| corev1.EnvVar{Name: "KAFKA_CONSUMER_GROUP", Value: kafka.ConsumerGroup}, | ||
| ) | ||
| return envVars | ||
| } |
| ce "knative.dev/func-go/cloudevents" | ||
| "knative.dev/func-go/kafka" | ||
|
|
| ce "knative.dev/func-go/cloudevents" | ||
| "knative.dev/func-go/kafka" | ||
|
|
WIP until knative-extensions/func-go#185 is merged
Summary
run.kafkaconfig tofunc.yaml(brokers,topic,consumerGroup)instead of a separate
invoke: kafkatypeFUNC_TRANSPORT=kafkaplusKAFKA_BROKERS,KAFKA_TOPIC,KAFKA_CONSUMER_GROUPenv vars from theconfig — user never sets these manually
main.gobranches at runtime betweenkafka.Start()andce.Start()based onFUNC_TRANSPORTfmt.Println(e)to instanced CloudEvents template to match statictemplate behavior
How it works
Users add a
run.kafkasection to their existing CloudEvents function:No code changes — the same
Handle(event.Event)method receives Kafkamessages automatically converted to CloudEvents by the runtime.
Dependencies
Limitations
Supersedes #3923
Closes #3940