Skip to content

Consolidate Node Peer Annotations #1914

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

Consolidate Node Peer Annotations #1914

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
328b6f5
Consolidate node peer annotations to address #1393
catherinetcai Sep 9, 2025
a079c7c
Re-organize structure of bgpPeerConfigs into separate packages
catherinetcai Oct 11, 2025
08610c4
Break out bgp peer config into separate package, add tests
catherinetcai Oct 22, 2025
2b86c85
Update markdown docs
catherinetcai Oct 22, 2025
6adf22e
Update docs changes
catherinetcai Oct 28, 2025
dba861d
Fix extraneous commas and quotes in README. Implement Stringer interf…
catherinetcai Nov 2, 2025
ff1e971
Update stringifying configs
catherinetcai Nov 10, 2025
0b7b33c
Add b64 type test
catherinetcai Nov 11, 2025
052c3d5
Fix port and localip defaults logic, add more test cases
catherinetcai Nov 18, 2025
e2d2568
Fixes according to linter
catherinetcai Nov 22, 2025
850179b
Cleanup with another round of go mod tidy and some minor fixes from r…
catherinetcai Nov 22, 2025
dbed2c1
Revert some formatting changes to files that there weren't any real c…
catherinetcai Nov 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 61 additions & 18 deletions docs/bgp.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ pod IPs, service IPs, etc.).

This is the default mode. All nodes in the clusters form iBGP peering relationships with rest of the nodes forming a
full node-to-node mesh. Each node advertise the pod CIDR allocated to the nodes with its peers (the rest of the nodes in
the cluster). There is no configuration required in this mode. All the nodes in the cluster are associated with the
the cluster). There is no configuration required in this mode. All the nodes in the cluster are associated with the
private ASN 64512 implicitly (which can be configured with `--cluster-asn` flag) and users are transparent to use of
iBGP. This mode is suitable in public cloud environments or small cluster deployments.

Expand All @@ -30,7 +30,7 @@ kubectl annotate node <kube-node> "kube-router.io/node.asn=64512"

Only nodes within same ASN form full mesh. Two nodes with different ASNs never get peered.

### Route-Reflector setup Without Full Mesh
### Route-Reflector setup Without Full Mesh

This model supports the common scheme of using a Route Reflector Server node to concentrate peering from client peers.
This has the big advantage of not needing full mesh, and will scale better. In this mode kube-router expects each node
Expand Down Expand Up @@ -75,11 +75,45 @@ For example:

### Node Specific External BGP Peers

Alternatively, each node can be configured with one or more node specific BGP peers. Information regarding node specific
BGP peer is read from node API object annotations:
Each node can be configured with one or more node specific BGP peers using the `kube-router.io/peers` node annotation.
Previously, these settings were configured using individual `kube-router.io/peer.*` annotations.
While these individual annotations are still supported, they're now deprecated and
will be removed in a future release.

#### Using Consolidated Annotation

The `kube-router.io/peers` annotation accepts peer configurations in YAML format with the following fields:

- `remoteip` (required): The IP address of the peer
- `remoteasn` (required): The ASN of the peer
- `localip` (optional): Local IP address to use for this peer connection
- `password` (optional): Base64 encoded password for BGP authentication
- `port` (optional): BGP port (defaults to 179 if not specified)

```shell
kubectl annotate node <kube-node> \
kube-router.io/peers="$(cat <<'EOF'
- remoteip: 192.168.1.99
remoteasn: 65000
password: U2VjdXJlUGFzc3dvcmQK
- remoteip: 192.168.1.100
remoteasn: 65000
password: U2VjdXJlUGFzc3dvcmQK
EOF
)"
```

#### Using Individual Annotations (Deprecated)

> **NOTE:** The individual peer annotations listed below are deprecated in favor of the consolidated `kube-router.io/peers`
> annotation. They are maintained for backward compatibility but will be removed in a future release.

Node-specific BGP peer configs can also be set via individual node API object annotations:

- `kube-router.io/peer.ips`
- `kube-router.io/peer.asns`
- `kube-router.io/peer.passwords`
- `kube-router.io/peer.localips`

For example, users can annotate node object with below commands:

Expand All @@ -106,26 +140,23 @@ kubectl annotate node <kube-node> "kube-router.io/path-prepend.repeat-n=5"

### BGP Peer Local IP configuration

In some setups it might be desirable to set a local IP address used for connecting external BGP peers. This can be
accomplished on nodes with annotations:
In some setups it might be desirable to set a local IP address used for connecting external BGP peers.

- `kube-router.io/peer.localips`

If set, this must be a list with a local IP address for each peer, or left empty to use nodeIP.
When using the `kube-router.io/peers` annotation, specify the `localip` field for each peer as shown in the
[Node Specific External BGP Peers](#node-specific-external-bgp-peers) section above.

Example:
When using individual annotations, you can specify the local IP address using `kube-router.io/peer.localips`:

```shell
kubectl annotate node <kube-node> "kube-router.io/peer.localips=10.1.1.1,10.1.1.2"
```

This will instruct kube-router to use IP `10.1.1.1` for first BGP peer as a local address, and use `10.1.1.2`for the
second.
If set, this must be a list with a local IP address for each peer, or left empty to use nodeIP.

### BGP Peer Password Authentication

The examples above have assumed there is no password authentication with BGP peer routers. If you need to use a password
for peering, you can use the `--peer-router-passwords` command-line option, the `kube-router.io/peer.passwords` node
If you need to use a password for peering with BGP peer routers, you can configure it using the `kube-router.io/peers`
annotation, the `--peer-router-passwords` command-line option, the deprecated `kube-router.io/peer.passwords` node
annotation, or the `--peer-router-passwords-file` command-line option.

#### Base64 Encoding Passwords
Expand All @@ -142,7 +173,15 @@ U2VjdXJlUGFzc3dvcmQ=

#### Password Configuration Examples

In this CLI flag example the first router (192.168.1.99) uses a password, while the second (192.168.1.100) does not.
**Using the consolidated annotation (recommended):**

When using the `kube-router.io/peers` annotation, specify the `password` field with a base64 encoded password for each
peer that requires authentication. See the
[Node Specific External BGP Peers](#node-specific-external-bgp-peers) section for an example.

**Using CLI flags:**

In this example the first router (192.168.1.99) uses a password, while the second (192.168.1.100) does not:

```sh
--peer-router-ips="192.168.1.99,192.168.1.100"
Expand All @@ -152,14 +191,18 @@ In this CLI flag example the first router (192.168.1.99) uses a password, while

Note the comma indicating the end of the first password.

Here's the same example but configured as node annotations:
**Using individual annotations (deprecated):**

Here's the same example but configured with individual node annotations:

```shell
kubectl annotate node <kube-node> "kube-router.io/peer.ips=192.168.1.99,192.168.1.100"
kubectl annotate node <kube-node> "kube-router.io/peer.asns=65000,65000"
kubectl annotate node <kube-node> "kube-router.io/peer.passwords=U2VjdXJlUGFzc3dvcmQK,"
```

**Using a password file:**

Finally, to include peer passwords as a file you would run kube-router with the following option:

```shell
Expand All @@ -168,8 +211,8 @@ Finally, to include peer passwords as a file you would run kube-router with the
--peer-router-passwords-file="/etc/kube-router/bgp-passwords.conf"
```

The password file, closely follows the syntax of the command-line and node annotation options.
Here, the first peer IP (192.168.1.99) would be configured with a password, while the second would not.
The password file closely follows the syntax of the command-line and node annotation options.
Here, the first peer IP (192.168.1.99) would be configured with a password, while the second would not:

```sh
U2VjdXJlUGFzc3dvcmQK,
Expand Down
137 changes: 67 additions & 70 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,89 +1,81 @@
module github.com/cloudnativelabs/kube-router/v2
Copy link
Collaborator

@aauren aauren Nov 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like the rebase here, favored your older library versions instead of the ones that were updated by dependabot. We should fix this up to favor the upgraded library versions (as well as Go 1.25).


require (
github.com/aws/aws-sdk-go-v2 v1.40.0
github.com/aws/aws-sdk-go-v2/config v1.31.17
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.14
github.com/aws/aws-sdk-go-v2/service/ec2 v1.274.0
github.com/aws/smithy-go v1.23.2
github.com/aws/aws-sdk-go-v2 v1.38.3
github.com/aws/aws-sdk-go-v2/config v1.31.6
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.6
github.com/aws/aws-sdk-go-v2/service/ec2 v1.249.0
github.com/aws/smithy-go v1.23.0
github.com/ccoveille/go-safecast/v2 v2.0.0
github.com/coreos/go-iptables v0.8.0
github.com/docker/docker v28.5.2+incompatible
github.com/docker/docker v28.4.0+incompatible
github.com/goccy/go-yaml v1.18.0
github.com/google/go-cmp v0.7.0
github.com/hashicorp/go-version v1.7.0
github.com/moby/ipvs v1.1.0
github.com/onsi/ginkgo v1.16.5
github.com/onsi/gomega v1.38.2
github.com/osrg/gobgp/v3 v3.37.0
github.com/prometheus/client_golang v1.23.2
github.com/prometheus/client_golang v1.23.0
github.com/spf13/pflag v1.0.10
github.com/stretchr/testify v1.11.1
github.com/vishvananda/netlink v1.3.1
github.com/vishvananda/netns v0.0.5
golang.org/x/net v0.46.0
golang.org/x/sys v0.38.0
google.golang.org/grpc v1.76.0
google.golang.org/protobuf v1.36.10
k8s.io/api v0.34.2
k8s.io/apimachinery v0.34.2
k8s.io/client-go v0.34.2
k8s.io/cri-api v0.34.2
golang.org/x/net v0.43.0
golang.org/x/sys v0.35.0
google.golang.org/grpc v1.75.0
google.golang.org/protobuf v1.36.8
k8s.io/api v0.34.0
k8s.io/apimachinery v0.34.0
k8s.io/client-go v0.34.0
k8s.io/cri-api v0.34.0
k8s.io/klog/v2 v2.130.1
k8s.io/utils v0.0.0-20250820121507-0af2bda4dd1d
k8s.io/utils v0.0.0-20250604170112-4c0f3b243397
sigs.k8s.io/yaml v1.6.0
)

require (
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.18.21 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.14 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.14 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.14 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.30.1 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.5 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.39.1 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.18.10 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.6 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.6 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.1 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.6 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.29.1 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.34.2 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.38.2 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/containerd/errdefs v1.0.0 // indirect
github.com/containerd/errdefs/pkg v0.3.0 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da // indirect
github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 // indirect
github.com/distribution/reference v0.6.0 // indirect
github.com/docker/go-connections v0.6.0 // indirect
github.com/docker/go-connections v0.5.0 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/eapache/channels v1.1.0 // indirect
github.com/eapache/queue v1.1.0 // indirect
github.com/emicklei/go-restful/v3 v3.13.0 // indirect
github.com/emicklei/go-restful/v3 v3.12.2 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/fsnotify/fsnotify v1.9.0 // indirect
github.com/fsnotify/fsnotify v1.7.0 // indirect
github.com/fxamacker/cbor/v2 v2.9.0 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-openapi/jsonpointer v0.22.0 // indirect
github.com/go-openapi/jsonreference v0.21.1 // indirect
github.com/go-openapi/swag v0.24.1 // indirect
github.com/go-openapi/swag/cmdutils v0.24.0 // indirect
github.com/go-openapi/swag/conv v0.24.0 // indirect
github.com/go-openapi/swag/fileutils v0.24.0 // indirect
github.com/go-openapi/swag/jsonname v0.24.0 // indirect
github.com/go-openapi/swag/jsonutils v0.24.0 // indirect
github.com/go-openapi/swag/loading v0.24.0 // indirect
github.com/go-openapi/swag/mangling v0.24.0 // indirect
github.com/go-openapi/swag/netutils v0.24.0 // indirect
github.com/go-openapi/swag/stringutils v0.24.0 // indirect
github.com/go-openapi/swag/typeutils v0.24.0 // indirect
github.com/go-openapi/swag/yamlutils v0.24.0 // indirect
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
github.com/go-openapi/jsonpointer v0.21.0 // indirect
github.com/go-openapi/jsonreference v0.21.0 // indirect
github.com/go-openapi/swag v0.23.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/google/gnostic-models v0.7.0 // indirect
github.com/google/go-cmp v0.7.0 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/k-sone/critbitgo v1.4.0 // indirect
github.com/mailru/easyjson v0.9.1 // indirect
github.com/magiconair/properties v1.8.7 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/sys/atomicwriter v0.1.0 // indirect
github.com/moby/term v0.0.0-20221205130635-1aeaba878587 // indirect
Expand All @@ -93,45 +85,50 @@ require (
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/nxadm/tail v1.4.8 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
github.com/opencontainers/image-spec v1.1.0 // indirect
github.com/pelletier/go-toml/v2 v2.2.2 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.66.1 // indirect
github.com/prometheus/procfs v0.17.0 // indirect
github.com/sagikazarmark/locafero v0.12.0 // indirect
github.com/prometheus/common v0.65.0 // indirect
github.com/prometheus/procfs v0.16.1 // indirect
github.com/sagikazarmark/locafero v0.6.0 // indirect
github.com/sagikazarmark/slog-shim v0.1.0 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/cast v1.10.0 // indirect
github.com/spf13/viper v1.21.0 // indirect
github.com/sourcegraph/conc v0.3.0 // indirect
github.com/spf13/afero v1.11.0 // indirect
github.com/spf13/cast v1.6.0 // indirect
github.com/spf13/viper v1.19.0 // indirect
github.com/stretchr/objx v0.5.2 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/x448/float16 v0.8.4 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0 // indirect
go.opentelemetry.io/otel v1.38.0 // indirect
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.53.0 // indirect
go.opentelemetry.io/otel v1.37.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.24.0 // indirect
go.opentelemetry.io/otel/metric v1.38.0 // indirect
go.opentelemetry.io/otel/trace v1.38.0 // indirect
go.yaml.in/yaml/v2 v2.4.3 // indirect
go.opentelemetry.io/otel/metric v1.37.0 // indirect
go.opentelemetry.io/otel/trace v1.37.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.yaml.in/yaml/v2 v2.4.2 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/oauth2 v0.31.0 // indirect
golang.org/x/term v0.36.0 // indirect
golang.org/x/text v0.30.0 // indirect
golang.org/x/time v0.13.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250908214217-97024824d090 // indirect
gopkg.in/evanphx/json-patch.v4 v4.13.0 // indirect
golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 // indirect
golang.org/x/oauth2 v0.30.0 // indirect
golang.org/x/term v0.34.0 // indirect
golang.org/x/text v0.28.0 // indirect
golang.org/x/time v0.9.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250707201910-8d1bb00bc6a7 // indirect
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
gotest.tools/v3 v3.4.0 // indirect
k8s.io/kube-openapi v0.0.0-20250910181357-589584f1c912 // indirect
sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 // indirect
k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b // indirect
sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 // indirect
sigs.k8s.io/randfill v1.0.0 // indirect
sigs.k8s.io/structured-merge-diff/v6 v6.3.0 // indirect
)

go 1.25.0
go 1.24.0

toolchain go1.25.1
toolchain go1.24.1
Loading