Skip to content

Commit 224f3c6

Browse files
jwttyjwtty
authored
test: use clusterloader2 framework for large scale testing (#102)
* test: use clusterloader2 framework for large scale testing Signed-off-by: Wantong Jiang <wantjian@microsoft.com> * fix comments Signed-off-by: Wantong Jiang <wantjian@microsoft.com> --------- Signed-off-by: Wantong Jiang <wantjian@microsoft.com>
1 parent 385c310 commit 224f3c6

File tree

8 files changed

+304
-1
lines changed

8 files changed

+304
-1
lines changed

‎.github/workflows/markdown-lint.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ jobs:
1111
runs-on: ubuntu-latest
1212
steps:
1313
- uses: actions/checkout@v4
14-
- uses: gaurav-nelson/github-action-markdown-link-check@v1
14+
- uses: tcort/github-action-markdown-link-check@v1
1515
with:
1616
# this will only show errors in the output
1717
use-quiet-mode: 'yes'

‎hack/cl2/README.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
This folder contains a list of [clusterloader2](https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/docs/GETTING_STARTED.md) configuration manifests for large scale testing.
2+
3+
## Prerequisites
4+
5+
Follow [clusterloader2 GETTING STARTED](https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/docs/GETTING_STARTED.md) to clone the `perf-tests` repository.
6+
7+
## Execute Tests
8+
9+
Run `clusterloader2` under `perf-tests/clusterloader2` directory:
10+
```bash
11+
go run cmd/clusterloader.go --testconfig=<path to the test yaml> --provider=local --kubeconfig=<path to the hub cluster kubeconfig> --v=2 --enable-exec-service=false
12+
```
13+
We need to set `--enable-exec-service=false` to prevent creating agnostic deployment on the hub cluster as hub
14+
cluster does not allow pod creation.
15+
16+
## Cleanup Resources After Tests
17+
18+
By default, clusterloader2 automatically deletes all the generated namespaces after the tests.
19+
In addition, we also want to delete all the CRPs created. To facilitate cleanup process, run the simple script
20+
provided in this folder:
21+
```
22+
export KUBECONFIG=<your path to the kubeconfig file>
23+
./cleanup.sh
24+
```

‎hack/cl2/busy_cluster.yaml‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
name: busy_cluster_test
2+
3+
{{$duration := "30m"}}
4+
{{$count := 1}}
5+
{{$namespaceCount := 1000}}
6+
7+
namespace:
8+
number: {{$namespaceCount}}
9+
prefix: busy-cluster-test-ns
10+
11+
tuningSets:
12+
- name: Uniform10qps
13+
qpsLoad:
14+
qps: 10
15+
- name: SteppedLoad
16+
steppedLoad:
17+
burstSize: 20
18+
stepDelay: 30s
19+
20+
steps:
21+
- name: Create CRPs
22+
phases:
23+
- replicasPerNamespace: {{$namespaceCount}}
24+
tuningSet: Uniform10qps
25+
objectBundle:
26+
- basename: test-crp
27+
objectTemplatePath: "manifests/test-crp.yaml"
28+
- name: Wait for CRPs to be Ready
29+
measurements:
30+
- Identifier: WaitForGenericK8sObjects
31+
Method: WaitForGenericK8sObjects
32+
Params:
33+
objectGroup: placement.kubernetes-fleet.io
34+
objectVersion: v1beta1
35+
objectResource: clusterresourceplacements
36+
timeout: {{$duration}}
37+
successfulConditions:
38+
- ClusterResourcePlacementAvailable=True
39+
failedConditions:
40+
- ClusterResourcePlacementAvailable=False
41+
minDesiredObjectCount: {{$namespaceCount}}
42+
maxFailedObjectCount: 0
43+
- module:
44+
path: /modules/configmaps.yaml
45+
params:
46+
namespaceCount: {{$namespaceCount}}
47+
configmapCount: {{$count}}
48+
- module:
49+
path: /modules/configmaps.yaml
50+
params:
51+
namespaceCount: {{$namespaceCount}}
52+
configmapCount: {{MultiplyInt $count 2}}
53+
- module:
54+
path: /modules/configmaps.yaml
55+
params:
56+
namespaceCount: {{$namespaceCount}}
57+
configmapCount: {{MultiplyInt $count 3}}
58+
- module:
59+
path: /modules/configmaps.yaml
60+
params:
61+
namespaceCount: {{$namespaceCount}}
62+
configmapCount: {{MultiplyInt $count 4}}
63+
- module:
64+
path: /modules/configmaps.yaml
65+
params:
66+
namespaceCount: {{$namespaceCount}}
67+
configmapCount: {{MultiplyInt $count 5}}
68+
- module:
69+
path: /modules/configmaps.yaml
70+
params:
71+
namespaceCount: {{$namespaceCount}}
72+
configmapCount: {{MultiplyInt $count 6}}
73+
- module:
74+
path: /modules/configmaps.yaml
75+
params:
76+
namespaceCount: {{$namespaceCount}}
77+
configmapCount: {{MultiplyInt $count 7}}
78+
- module:
79+
path: /modules/configmaps.yaml
80+
params:
81+
namespaceCount: {{$namespaceCount}}
82+
configmapCount: {{MultiplyInt $count 8}}
83+
- module:
84+
path: /modules/configmaps.yaml
85+
params:
86+
namespaceCount: {{$namespaceCount}}
87+
configmapCount: {{MultiplyInt $count 9}}
88+
- module:
89+
path: /modules/configmaps.yaml
90+
params:
91+
namespaceCount: {{$namespaceCount}}
92+
configmapCount: {{MultiplyInt $count 10}}
93+
- name: Wait for CRPs to be Ready
94+
measurements:
95+
- Identifier: WaitForGenericK8sObjects
96+
Method: WaitForGenericK8sObjects
97+
Params:
98+
objectGroup: placement.kubernetes-fleet.io
99+
objectVersion: v1beta1
100+
objectResource: clusterresourceplacements
101+
timeout: {{$duration}}
102+
successfulConditions:
103+
- ClusterResourcePlacementAvailable=True
104+
failedConditions:
105+
- ClusterResourcePlacementAvailable=False
106+
minDesiredObjectCount: {{$namespaceCount}}
107+
maxFailedObjectCount: 0

‎hack/cl2/busy_cluster_burst.yaml‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
name: busy_cluster_burst_test
2+
3+
{{$duration := "10m"}}
4+
{{$count := 1000}}
5+
6+
namespace:
7+
number: 2
8+
prefix: busy-cluster-burst-test-ns
9+
10+
tuningSets:
11+
- name: Uniform100qps
12+
qpsLoad:
13+
qps: 100
14+
15+
steps:
16+
- name: Create CRPs
17+
phases:
18+
- replicasPerNamespace: 2
19+
tuningSet: Uniform100qps
20+
objectBundle:
21+
- basename: test-crp
22+
objectTemplatePath: "manifests/test-crp.yaml"
23+
- name: Create ConfigMaps
24+
phases:
25+
- namespaceRange:
26+
min: 1
27+
max: 2
28+
replicasPerNamespace: {{$count}}
29+
tuningSet: Uniform100qps
30+
objectBundle:
31+
- basename: test-configmap
32+
objectTemplatePath: "manifests/test-configmap.yaml"
33+
- name: Wait for CRPs to be Ready
34+
measurements:
35+
- Identifier: WaitForGenericK8sObjects
36+
Method: WaitForGenericK8sObjects
37+
Params:
38+
objectGroup: placement.kubernetes-fleet.io
39+
objectVersion: v1beta1
40+
objectResource: clusterresourceplacements
41+
timeout: {{$duration}}
42+
successfulConditions:
43+
- ClusterResourcePlacementAvailable=True
44+
failedConditions:
45+
- ClusterResourcePlacementAvailable=False
46+
minDesiredObjectCount: 2
47+
maxFailedObjectCount: 0
48+
- name: Delete ConfigMaps
49+
phases:
50+
- namespaceRange:
51+
min: 1
52+
max: 2
53+
replicasPerNamespace: 0
54+
tuningSet: Uniform100qps
55+
objectBundle:
56+
- basename: test-configmap
57+
objectTemplatePath: "manifests/test-configmap.yaml"
58+
- name: Wait for CRPs to be Ready
59+
measurements:
60+
- Identifier: WaitForGenericK8sObjects
61+
Method: WaitForGenericK8sObjects
62+
Params:
63+
objectGroup: placement.kubernetes-fleet.io
64+
objectVersion: v1beta1
65+
objectResource: clusterresourceplacements
66+
timeout: {{$duration}}
67+
successfulConditions:
68+
- ClusterResourcePlacementAvailable=True
69+
failedConditions:
70+
- ClusterResourcePlacementAvailable=False
71+
minDesiredObjectCount: 2
72+
maxFailedObjectCount: 0
73+
- name: Create ConfigMaps
74+
phases:
75+
- namespaceRange:
76+
min: 1
77+
max: 2
78+
replicasPerNamespace: {{$count}}
79+
tuningSet: Uniform100qps
80+
objectBundle:
81+
- basename: test-configmap
82+
objectTemplatePath: "manifests/test-configmap.yaml"
83+
- name: Wait for CRPs to be Ready
84+
measurements:
85+
- Identifier: WaitForGenericK8sObjects
86+
Method: WaitForGenericK8sObjects
87+
Params:
88+
objectGroup: placement.kubernetes-fleet.io
89+
objectVersion: v1beta1
90+
objectResource: clusterresourceplacements
91+
timeout: {{$duration}}
92+
successfulConditions:
93+
- ClusterResourcePlacementAvailable=True
94+
failedConditions:
95+
- ClusterResourcePlacementAvailable=False
96+
minDesiredObjectCount: 2
97+
maxFailedObjectCount: 0
98+
- name: Delete ConfigMaps
99+
phases:
100+
- namespaceRange:
101+
min: 1
102+
max: 2
103+
replicasPerNamespace: 0
104+
tuningSet: Uniform100qps
105+
objectBundle:
106+
- basename: test-configmap
107+
objectTemplatePath: "manifests/test-configmap.yaml"
108+
- name: Wait for CRPs to be Ready
109+
measurements:
110+
- Identifier: WaitForGenericK8sObjects
111+
Method: WaitForGenericK8sObjects
112+
Params:
113+
objectGroup: placement.kubernetes-fleet.io
114+
objectVersion: v1beta1
115+
objectResource: clusterresourceplacements
116+
timeout: {{$duration}}
117+
successfulConditions:
118+
- ClusterResourcePlacementAvailable=True
119+
failedConditions:
120+
- ClusterResourcePlacementAvailable=False
121+
minDesiredObjectCount: 2
122+
maxFailedObjectCount: 0

‎hack/cl2/cleanup.sh‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
#!/usr/bin/env bash
2+
3+
set -o errexit
4+
set -o nounset
5+
set -o pipefail
6+
7+
: "${KUBECONFIG:?Environment variable KUBECONFIG must be set}"
8+
9+
echo "Deleting all CRPs generated during the load test from the hub cluster..."
10+
kubectl delete crp -l test=cl2-test

‎hack/cl2/manifests/test-configmap.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Test configmap with 1k payload
2+
apiVersion: v1
3+
kind: ConfigMap
4+
metadata:
5+
name: {{.Name}}
6+
data:
7+
random: O0RKEYYCLRvp03c8gUPGINapuUPUpJrL1WQQmP5UOngJJsbiPF876mELH1MyD1bw2FKrr4IuOHzjl0hiL7SvO46HRCKsEHAVFhelhTzwdR9CI5A9jjthpNR4t812R15MyumLrqMAhXv4ynykUa7bt1XtzsIhYjARSKthtPvN9Hbge26PX7lhQjsT1vm42bw7HHEfRVc2P0IwZluchc6aqJfhpJcNQGSjN0GjeuzkNP7V7exWkZqtAt7JnZneq3X72XMxrdQB3sdbK3WEYdsajUAyBO0Eznn8Ab8FSfwrvKUCbCQ8C2UophqThWwzAiMw1y5BQjT3EaIqeLPWDhrqkbMHzSbpcdXLHv8Ine5594QTZZYm8II0PZd3HdpoXjzeAAix6w1c4hXFwtQejti3ZDqZNDPrEUuepRradonBRcZcFccLTGiK6BUNNzZMC3yP5DNR2yWE6P30NdeBMb7mkAFhAVUp46KZUghF60Jpu0HSmEHs0gA0dBR2gyyRsWFjT03IGxixZ2sCuOTCEqdvatTVKi6FBjigRNnxZXyL9pzyyXhTRrdcts72lvKsbFp1n0rQmQUItfKVOXC1bdfGUPaoRA4jQXdUAVdJadYfTG12ETgxXGuKc44lkTFGMKuMA4RRZYZRZGJubW18rhfnleiiOFjPL83NJ8g81xFuqf4TWhXdgwhPwe2CvIeF2lfdmyer10Y8GC4Yl7LAFp5GshpmjeDZcuGj31NankEMZLY0znRW7M551yy2TCNHzsDaYtnPR4611wNu3nVQIN8iAXmPnnMev2bDfh8l5A14CUtONGno4DQVs4IyTmpwN1oyOxsXPaL8m2r9FtUzn7JCdepirNfm0ImVUDvlR96v6axjjGvoVWzQtNDQWdnJC6x1KKs9mDuERexSwGc4F56Iec5Q2a2QB9EFxdEOSDKFjGq1az7PHOf8uqOurzFvG6dylwQnKN5oy0PFW8wgxyJO6hsUDfrslO3zrh9gtcOIt9tHL3xo70bF8hPkTinn8xRs

‎hack/cl2/manifests/test-crp.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
apiVersion: placement.kubernetes-fleet.io/v1beta1
2+
kind: ClusterResourcePlacement
3+
metadata:
4+
name: {{.Name}}
5+
labels:
6+
test: cl2-test
7+
spec:
8+
resourceSelectors:
9+
- group: ""
10+
kind: Namespace
11+
name: busy-cluster-test-ns-{{AddInt .Index 1}}
12+
version: v1
13+
strategy:
14+
type: RollingUpdate
15+
rollingUpdate:
16+
maxUnavailable: 25%
17+
maxSurge: 25%
18+
unavailablePeriodSeconds: 60
19+
revisionHistoryLimit: 15

‎hack/cl2/modules/configmaps.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{{$namespaces := .namespaceCount}}
2+
{{$count := .configmapCount}}
3+
4+
steps:
5+
- name: Create ConfigMaps
6+
phases:
7+
- namespaceRange:
8+
min: 1
9+
max: {{$namespaces}}
10+
replicasPerNamespace: {{$count}}
11+
tuningSet: SteppedLoad
12+
objectBundle:
13+
- basename: test-configmap
14+
objectTemplatePath: "manifests/test-configmap.yaml"

0 commit comments

Comments
 (0)