Skip to content
This repository was archived by the owner on Apr 13, 2020. It is now read-only.

Commit 0e77d21

Browse files
edaenaedaena
authored
Document service introspection walkthrough (#104)
1 parent 95a501a commit 0e77d21

File tree

5 files changed

+191
-116
lines changed

5 files changed

+191
-116
lines changed

‎docs/images/cors-menu.png‎

6.86 KB
Loading

‎docs/images/cors-settings.png‎

43.8 KB
Loading

‎docs/images/service_introspection.png‎

44.4 KB
Loading

‎docs/service-introspection-onboarding.md‎

Lines changed: 185 additions & 111 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,47 @@
1-
# Onboard a Bedrock project to use Service Introspection
1+
# Service Introspection: Getting Started
22

3-
If you have already followed the steps
4-
[here](https://github.com/microsoft/bedrock/tree/master/gitops) to setup the
5-
pipelines for a GitOps workflow in Bedrock, you may update your pipelines to
6-
send data to the Spektate storage, which will help you run the introspection
7-
tool on your services.
3+
Service Introspection shows information about a
4+
[Bedrock GitOps workflow](https://github.com/microsoft/bedrock/tree/master/gitops).
85

9-
## Pre-Requisites
6+
Service introspection is used via the `spk deployment` commands. More
7+
information about the commands is available in the command reference
8+
[here](https://github.com/CatalystCode/spk/blob/master/docs/service-introspection.md).
109

11-
Service introspection tool needs an Azure storage account to store information
12-
about your pipelines and services.
10+
The following diagram shows the main components of service introspection.
11+
![spk service introspection diagram](./images/service_introspection.png)
1312

14-
If you don't already have an Azure storage account you would like to use, use
15-
the `spk deployment onboard` command which will create a storage account in your
16-
subscription.
13+
To use service introspection you first need to make sure you have the following
14+
pre-requisites.
1715

18-
You may also create this storage account manually. You will need to have the
19-
following properties of this storage before proceeding:
16+
## Components
17+
18+
1. GitOps pipelines workflow in Bedrock. To setup the workflow, follow
19+
[these](https://github.com/microsoft/bedrock/tree/master/gitops)
20+
instructions.
21+
2. [Service introspection storage in Azure](#service-introspection-storage). See
22+
below for instructions on how to create one.
23+
3. [Pipelines configuration](#pipelines-configuration)
24+
25+
### Service introspection storage
26+
27+
Service introspection tool needs a database to store the information about your
28+
pipelines, builds and deployments. Currently, service introspection supports
29+
storage in the form of an Azure Storage table. Follow the steps below to create
30+
it or use an existing one.
31+
32+
#### 1.Create an Azure storage account:
33+
34+
**Option 1:**
35+
36+
Use the
37+
[`spk deployment onboard`](https://github.com/CatalystCode/spk/blob/master/docs/service-introspection.md#onboard)
38+
command.
39+
40+
**Option 2:**
41+
42+
Create the account manually or use an existing storage account. You will need to
43+
have the following properties of this storage before proceeding as they are
44+
required to configure:
2045

2146
- Name of the storage account
2247
- Access key to this storage account
@@ -25,101 +50,150 @@ following properties of this storage before proceeding:
2550
Once you have a storage account with a table, you may proceed to start updating
2651
the pipelines to send data to Spektate storage.
2752

28-
## Update the pipelines to send data to storage
29-
30-
1. Create a variable group with the following variables, which will be used by
31-
the tasks in each of the pipelines to access the storage.
32-
33-
- `ACCOUNT_KEY`: Set this to the access key for your storage account
34-
- `ACCOUNT_NAME`: Set this to the name of your storage account
35-
- `PARTITION_KEY`: This field can be a distinguishing key that recognizea
36-
your source repository in the storage, for eg. in this example, we're using
37-
the name of the source repository `hello-bedrock`
38-
- `TABLE_NAME`: Set this to the name of the table in your storage account
39-
that you prefer to use
40-
41-
![](./images/variable_group.png)
42-
43-
Make sure that you update the pipelines in the following steps to include
44-
this variable group, such as below:
45-
46-
```yaml
47-
variables:
48-
- group: <your-variable-group-name>
49-
```
50-
51-
2. To your CI pipeline that runs from the source repository to build the docker
52-
image, copy and paste the following task which will update the database for
53-
every build that runs from the source repository to show up in Spektate.
54-
55-
```yaml
56-
- bash: |
57-
curl $SCRIPT > script.sh
58-
chmod +x ./script.sh
59-
tag_name="hello-spektate-$(Build.SourceBranchName)-$(Build.BuildId)"
60-
commitId=$(Build.SourceVersion)
61-
commitId=$(echo "${commitId:0:7}")
62-
./script.sh $(ACCOUNT_NAME) $(ACCOUNT_KEY) $(TABLE_NAME) $(PARTITION_KEY) p1 $(Build.BuildId) imageTag $tag_name commitId $commitId service $(Build.Repository.Name)
63-
displayName: Update manifest pipeline details in CJ db
64-
env:
65-
SCRIPT: https://raw.githubusercontent.com/catalystcode/spk/master/scripts/update_introspection.sh
66-
```
67-
68-
Make sure the variable `tag_name` is set to the tag name for the image being
69-
built in your docker step.
70-
71-
**Note**: The earlier in the pipeline you add this task, the earlier it will
72-
send data to Spektate. Adding it before the crucial steps is recommended
73-
since it will capture details about failures if the important steps fail.
74-
75-
3. To your CD release pipeline (ACR to HLD), add the following lines of code to
76-
the end of your last release task (make sure this is not a separate task in
77-
the process):
78-
79-
```yaml
53+
**Note:** The Azure storage account is needed to store information about your
54+
pipelines and services that is displayed by service introspection.
55+
56+
#### 2. Create a table. Follow these
57+
58+
[instructions](https://docs.microsoft.com/en-us/azure/storage/tables/table-storage-quickstart-portal).
59+
60+
#### 3. Storage account CORS settings
61+
62+
Configure the CORS settings for the storage account to allow requests from the
63+
service introspection dasbhoard.
64+
65+
1. Go to the [Azure portal](https://portal.azure.com)
66+
2. Search for the name of your storage account
67+
3. Click the CORS options on the menu on the left side:
68+
69+
![cors menu option](./images/cors-menu.png)
70+
71+
Add the following settings under **Table Service**:
72+
![cors settings](./images/cors-settings.png)
73+
74+
**Note:** If you are running the service introspection spk dashboard in a port
75+
other than `4040`, add that entry in the settings instead.
76+
77+
### Pipelines Configuration
78+
79+
The Bedrock GitOps pipelines need to be configured to start sending data to
80+
`spk` service introspection. This is done by adding a script snippet in each
81+
`azure-pipelines.yml` configuration.
82+
83+
#### 1. Configure a variable group
84+
85+
To send data from Azure pipelines to the service introspection storage created
86+
previously a variable group needs to be configured in Azure DevOps (where the
87+
pipelines are).
88+
89+
To configure the variable group run:
90+
91+
```
92+
spk variable-group create
93+
```
94+
95+
#### 2. CI pipeline configuration
96+
97+
The CI pipeline runs from the source repository to build a docker image.
98+
99+
Paste the following task in its corresponding `azure-pipelines.yml`:
100+
101+
```yaml
102+
- bash: |
103+
curl $SCRIPT > script.sh
104+
chmod +x ./script.sh
105+
tag_name="hello-spektate-$(Build.SourceBranchName)-$(Build.BuildId)"
106+
commitId=$(Build.SourceVersion)
107+
commitId=$(echo "${commitId:0:7}")
108+
./script.sh $(ACCOUNT_NAME) $(ACCOUNT_KEY) $(TABLE_NAME) $(PARTITION_KEY) p1 $(Build.BuildId) imageTag $tag_name commitId $commitId service $(Build.Repository.Name)
109+
displayName: Update manifest pipeline details in CJ db
110+
env:
111+
SCRIPT: https://raw.githubusercontent.com/catalystcode/spk/master/scripts/update_introspection.sh
112+
```
113+
114+
This task will update the service introspection storage table for every build
115+
that runs from the source repository. This information will be available for use
116+
by service introspection.
117+
118+
##### Note:
119+
120+
- Make sure the variable `tag_name` is set to the tag name for the image being
121+
built in your docker step.
122+
123+
- Add the task before the crucial steps in your pipeline. This will capture
124+
details about failures if the important steps fail.
125+
126+
#### 3. CD release pipeline (ACR to HLD) configuration
127+
128+
The CD release pipeline updates the docker image number in the HLD.
129+
130+
Paste the following task in its corresponding `azure-pipelines.yml`:
131+
132+
```yaml
133+
latest_commit=$(git rev-parse --short HEAD)
134+
echo "latest_commit=$latest_commit"
135+
136+
# Download update storage script
137+
curl https://raw.githubusercontent.com/catalystcode/spk/master/scripts/update_introspection.sh > script.sh
138+
chmod +x script.sh
139+
140+
./script.sh $(ACCOUNT_NAME) $(ACCOUNT_KEY) $(TABLE_NAME) $(PARTITION_KEY) imageTag $(Build.BuildId) p2 $(Release.ReleaseId) hldCommitId $latest_commit env $(Release.EnvironmentName)
141+
```
142+
143+
This task is the same as the one from step 1 but instead passes the information
144+
that corresponds to the CD release pipeline.
145+
146+
#### 4. HLD manifest pipeline configuration
147+
148+
The HLD manifest pipeline builds the HLD using `fabrikate` and generates
149+
resource manifests that are then placed in the resource manifest repository.
150+
151+
Paste the following task in the `azure-pipelines.yml` file **before** the
152+
`fabrikate` steps:
153+
154+
```yaml
155+
- bash: |
156+
curl $SCRIPT > script.sh
157+
chmod +x ./script.sh
158+
commitId=$(Build.SourceVersion)
159+
commitId=$(echo "${commitId:0:7}")
160+
./script.sh $(ACCOUNT_NAME) $(ACCOUNT_KEY) $(TABLE_NAME) $(PARTITION_KEY) hldCommitId $commitId p3 $(Build.BuildId)
161+
displayName: Update manifest pipeline details in CJ db
162+
env:
163+
SCRIPT: https://raw.githubusercontent.com/catalystcode/spk/master/scripts/update_introspection.sh
164+
```
165+
166+
Paste the following task after the `fabrikate` step:
167+
168+
```yaml
169+
- script: |
170+
cd "$HOME"/hello-bedrock-manifest
171+
curl $SCRIPT > script.sh
172+
chmod +x ./script.sh
80173
latest_commit=$(git rev-parse --short HEAD)
81-
echo "latest_commit=$latest_commit"
82-
83-
# Download update storage script
84-
curl https://raw.githubusercontent.com/catalystcode/spk/master/scripts/update_introspection.sh > script.sh
85-
chmod +x script.sh
86-
87-
./script.sh $(ACCOUNT_NAME) $(ACCOUNT_KEY) $(TABLE_NAME) $(PARTITION_KEY) imageTag $(Build.BuildId) p2 $(Release.ReleaseId) hldCommitId $latest_commit env $(Release.EnvironmentName)
88-
```
89-
90-
4. To the HLD to manifest pipeline, we will need to add two tasks, one that
91-
updates the storage with the pipeline Id and another with an update for the
92-
commit Id that was made into the manifest repo. The reason these two are
93-
currently separate steps is to track more information about failures (if they
94-
were to happen). For the first step, before the fabrikate steps, add the step
95-
below:
96-
97-
```yaml
98-
- bash: |
99-
curl $SCRIPT > script.sh
100-
chmod +x ./script.sh
101-
commitId=$(Build.SourceVersion)
102-
commitId=$(echo "${commitId:0:7}")
103-
./script.sh $(ACCOUNT_NAME) $(ACCOUNT_KEY) $(TABLE_NAME) $(PARTITION_KEY) hldCommitId $commitId p3 $(Build.BuildId)
104-
displayName: Update manifest pipeline details in CJ db
105-
env:
106-
SCRIPT: https://raw.githubusercontent.com/catalystcode/spk/master/scripts/update_introspection.sh
107-
```
108-
109-
For the step to update manifest commit Id:
110-
111-
```yaml
112-
- script: |
113-
cd "$HOME"/hello-bedrock-manifest
114-
curl $SCRIPT > script.sh
115-
chmod +x ./script.sh
116-
latest_commit=$(git rev-parse --short HEAD)
117-
./script.sh $(ACCOUNT_NAME) $(ACCOUNT_KEY) $(TABLE_NAME) $(PARTITION_KEY) p3 $(Build.BuildId) manifestCommitId $latest_commit
118-
displayName: Update commit id in database
119-
env:
120-
SCRIPT: https://raw.githubusercontent.com/catalystcode/spk/master/scripts/update_introspection.sh
121-
```
122-
123-
5. Kick off a full deployment from the source to docker pipeline, and you should
124-
see some entries coming into the database for each subsequent deployment
125-
after the tasks have been added!
174+
./script.sh $(ACCOUNT_NAME) $(ACCOUNT_KEY) $(TABLE_NAME) $(PARTITION_KEY) p3 $(Build.BuildId) manifestCommitId $latest_commit
175+
displayName: Update commit id in database
176+
env:
177+
SCRIPT: https://raw.githubusercontent.com/catalystcode/spk/master/scripts/update_introspection.sh
178+
```
179+
180+
This task will update the `manifestCommitId`.
181+
182+
## Getting started
183+
184+
After completing the steps in this guide, you should be able to:
185+
186+
- Fill out the `azure_devops` and `introspection` settings in
187+
[`spk-config.yaml`](https://github.com/CatalystCode/spk/blob/master/spk-config.yaml)
188+
so that you can use service introspection. More information about `spk` config
189+
can be found on the [main page](https://github.com/catalystcode/spk).
190+
191+
- Validate and verify the `spk-config.yaml` settings and the service
192+
introspection storage using
193+
[`spk deployment validate`](https://github.com/CatalystCode/spk/blob/master/docs/service-introspection.md#validate)
194+
195+
- Get information about your deployment using
196+
[`spk deployment get`](https://github.com/CatalystCode/spk/blob/master/docs/service-introspection.md#get)
197+
198+
- Launch the dashboard to visualize the data using
199+
[`spk deployment dashboard`](https://github.com/CatalystCode/spk/blob/master/docs/service-introspection.md#dashboard)

‎docs/service-introspection.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,11 @@ Service Introspection shows information about Bedrock deployments:
66
- Time the service was changed or errored
77
- Deployment state of the service is
88

9-
**Important**: This tool will not be useful to you until you go through the
10-
steps to onboard your pipelines to start using Spektate. There's an onboarding
11-
guide located [here](./service-introspection-onboarding.md). If you've already
12-
onboarded the pipelines to start sending data to Spektate storage, you may skip
13-
the onboarding guide and read about how to use this tool below.
9+
**Important:**
10+
11+
To use service introspection, begin with the steps on
12+
[Service Introspection: Getting Started](./service-introspection-onboarding.md).
13+
This will walk you through setting up a Bedrock GitOps pipeline workflow.
1414

1515
Usage:
1616

@@ -24,6 +24,7 @@ Commands:
2424
- [get](#get)
2525
- [onboard](#onboard)
2626
- [dashboard](#dashboard)
27+
- [variable-group](variable-group.md)
2728

2829
Global options:
2930

0 commit comments

Comments
 (0)