Update READMEs and Event Catalog Docs

Author: William Chrisp

.gitignore

@@ -258,3 +258,7 @@ Thumbs.db
 # CDKTF Specifics
 requirements.txt
 all_files/
+api_key_rotation/
+!api_key_rotation/main.py
+imports/
+cdktf.out

Makefile

@@ -16,19 +16,19 @@ docs-run: docs-build
 
 # Test Commands
 unittest:
-	pipenv run pytest -m 'not integration' tests/
+	pipenv run pytest -m 'not integration' tests/src tests/publisher
 
 integration-test:
-	pipenv run pytest -m 'integration' tests/
+	pipenv run pytest -m 'integration' tests/src tests/publisher
 
 test:
-	pipenv run pytest --cov=src --cov=publisher --cov-fail-under=81 --cov-report term-missing tests/
+	pipenv run pytest --cov=src --cov=publisher --cov-fail-under=81 --cov-report term-missing tests/src tests/publisher
 
 watch:
-	ptw -- -m 'not integration' tests/
+	ptw -- -m 'not integration' tests/src tests/publisher
 
 e2e:
-	pipenv run behave
+	cd tests/; pipenv run behave
 
 # Infrastructure Commands
 build-python:

README.md

@@ -1,5 +1,5 @@
-<h1 align="center">Flight Controller on AWS Cloud Platform</h1>
-<p align="center">This repository contains the code necessary for Flight Controller on AWS.</p>
+<h1 align="center">Flight Controller</h1>
+<p align="center">This repository contains the code necessary for Flight Controller</p>
 
 ## What is Flight Controller?
 
@@ -9,33 +9,36 @@ The intent is to make it trivial to add new measures, allowing teams to be data
 
 The approach to scaling a landing zone on AWS is [elaborated here](https://aws.amazon.com/blogs/mt/flight-controller-by-contino-a-solution-built-on-aws-control-tower/).  
 
-## Architecture
+## Repository Structure
 
-![Flight Controller Architecture](images/flight_controller.png)
+- [Flight Controller Event Catalog](https://legendary-spoon-f82c1640.pages.github.io): `docs/`
+  - [README.md](docs/README.md)
+- Flight Controller Infrastructure Terraform CDK Code - `infrastructure/`
+  - [README.md](infrastructure/README.md)
+- Flight Controller Custom Publisher Code - `publisher/`
+  - [README.md](publisher/README.md)
+- Flight Controller Code - `src/`
+  - [README.md](src/README.md)
+- Flight Controller Tests - `tests/`
+  - [README.md](tests/README.md)
 
 ## Development
 
 ### Prerequisites
 
 - Python version 3.10
 - Pipenv (version 2022.10.12 proven working)
-- Terraform version 1.2.7
-- Docker installed
+- Terraform CDK version 0.15.5
+- Docker
 
-- Terraform CDK
-    - NVM -> npm and node https://github.com/nvm-sh/nvm#usage 
-    - Terraform 
-    - Yarn -> npm install --global yarn
-    - TFCDK -> npm install --global cdktf-cli@latest -- WARNING required NODE 16.13
-
-Optional
+#### Optional
 
 - For macOS get graphviz for the diagrams to work
-    - brew install graphviz  
-- Granted https://granted.dev/
-    - brew tap common-fate/granted
-    - brew install granted
-    - granted registry add https://github.com/contino/apac-scaffolding-granted-registry.git/granted.yml
+  - brew install graphviz  
+- [Granted](https://granted.dev/)
+  - brew tap common-fate/granted
+  - brew install granted
+  - granted registry add https://github.com/contino/apac-scaffolding-granted-registry.git/granted.yml
 
 ### Start of the day
 
@@ -45,7 +48,7 @@ Optional
 
 `make local`
 
-### Make Commands
+### General Make Commands
 
 - `make local` setup local environment and install dependencies.
 - `make docs-run` install, build and run a dev version of the docs.
@@ -55,114 +58,38 @@ Optional
 - `make destroy` destroys the both infra & Grafana with TF CDK.
 
 Testing is split into four commands:
+
 - `make unittest` runs all the unit tests (i.e. tests that are not [marked as integration](https://docs.pytest.org/en/7.1.x/example/markers.html)).
 - `make integration-test` run all the integration tests.
 - `make test` runs all the tests and reports on coverage. 
 - `make e2e` runs the end to end BDD tests using [behave](https://github.com/behave/behave).
+- `make watch` runs all the unit tests on file change. Allowing the test code while making live changes.
 
+### Merging changes
 
-### Testing from console
-
-After you deploy your infrastructure you can interact with the eventbridge on AWS by sending different messages by running the following from the 'infra_aws_cdk' folder:
-
-- Send Project requested event
-    - `aws events put-events --entries file://./tests/events/test-eventR.json` 
-- Send Project assigned event
-    - `aws events put-events --entries file://./tests/events/test-eventA.json` 
-- Send Project created event 
-    - `aws events put-events --entries file://./tests/events/test-eventC.json`
-
+At the current time there are no branch protections. However, as the build process creates a commit for every build, to keep the git history clean, please `rebase/squash` your commits before pushing. You can do this by running `git fetch origin main && git rebase -i origin/main`, `edit`ing the first commit, and applying `fixup` to all following commits.
 
 ### Code Structure
 
 The code is structured in the [Clean Architecture](https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html) pattern.
 
 ![Clean Architecture](images/CleanArchitecture.jpeg)
 
-The current layers are:
-
-1. `Entities`, which contains domain objects
-2. `Usecases`, which orchestrate domain objects
-3. `Adapters`, which convert events and data into known types
-4. `Drivers`, which interact with data storage
-5. `Entrypoints`, which handle the event from AWS, retrieve and store data through drivers and call adapters to perform the needed business logic
-
 The core rule of Clean Architecture, is that a layer can only depend on the layers that have come before it. E.g. code in the `usecases` layer, may depend on `entities`, but cannot depend on `adapters` or `drivers`.
 
 When developing, it is simplest to start at the first layer and work down ending up with the entrypoint. This forces you to focus on the domain objects first before considering external services.
 
-### Deploying a change
-
-- Assume the correct AWS account/refresh credentials
-- Synth, plan and deploy both infra & Grafana Dashboard.
-    - `make build`
-    - `make plan`
-    - `make deploy`
-
-### Managing Grafana API Keys
-
-While writing this, Grafana API Keys are valid for maximum 30 days only. 
-Hopefully, Amazon will address this limitation in the future - but in the meantime, this simple pattern can be used to automatically rotate an API key every 29 days and store it for use in AWS Secrets Manager.
-
-![Grafana](images/manage_grafana_api_key.png)
-
-
-The solution is made up of two components:
-
-1. AWS Secret is created with a rotation lifecycle policy that will trigger a Lambda function every 29 days
-2. AWS Lambda Function that will create a new API key in Amazon Managed Grafana and update the AWS Secret with the new key
-
-
-Python code handling the rotation is stored  at `src/entrypoint/grafana_lambda.py`. The code expects two input variables to retrieve secret values from AWS Secrets Manager viz. `grafana_api_key_name` and `grafana_workspace_id`. As the name suggest these values are secret names being used as filters. 
-
-
-### Merging changes
-
-At the current time there are no branch protections. However, as the build process creates a commit for every build, to keep the git history clean, please `rebase/squash` your commits before pushing. You can do this by running `git fetch origin main && git rebase -i origin/main`, `edit`ing the first commit, and applying `fixup` to all following commits.
-
-# How to create a new metric?
-
-1. First of all, you need to define a SLO. In this example we will work with 2 SLO's. These SLO's are related to creation of a "project" at a client. 
-    
-    - "time taken between requesting a project and assigning the request to be executed"
-    - "time taken between requesting a project and finally creating the project"
-
-2. Next you need to understand the "process" for "creating a project". This includes actors, systems, and automation that contributes to creation of a "project" from start to end. From the process diagram we need to identify the events that are associated each component of the "process". "Event" in the process should have minimum the following fields
-    
-    - `aggregate_id`: this is id to all events that is part of an instantioantion of a process. For example, when one requests, assigns, and creates a project all the events will have the same correlation id.
-    - `event_type`: this is the type of event, this can be any string representing what happened.
-    - `event_version`: this is the version of the event.
-    - `time`: time when the event happened
-    - (Optional) `payload`: Any other detail one needs to add to the event for additional needs.
-
-3. Map the event to entities by creating a python file for entities under code/src/entities. See for example the projects.py for entities. Entities come in two form:
-    
-    - Event Entites: corresponding to the events of the domain
-    - Metric Entities: corresponding to the SLO's you have defined in step 1
-
-![See diagram](images/processDiagram.png)
-
-4. Calculation of metrics are handled in the usescases. Insert a new use case under code/src/usecases that calculates the metric you are after. You can see the example how the project creation metrics are calculated in the projects.py.
-
-5. Controller is where you covert external message to "entity". It delegates the metric handling to "usecases" above. you can find the controller at code/src/adapters folder
-
-6. Once you have finised the entites, usecases, and adaptors it is time to build the external layers. First one is the driver layer where events are stored. Drivers are implemented based on the choice of cloud environment. you can see two examples one being the bigquery and the other is DynamoDB+TimesStream. these dirvers interact with the actual DB that stores the entities and metrics to be used. This layer also contains the UI where the metrics are shown.
-
-7. One last code is needed to provide an entry point to the system. This is the part where platform message comes in and every part of the system is orchestrated. 
-
-![See diagram](images/UMLdiagram.png)
-
-
-### Manage Grafana Dashboards
-
-TF CDK is used to build and modify Grafana dashboard and the panels that goes inside dashboard. Dashboard configuration is  magnaged via `dashboard.json` located within the `infrastructure` folder.
-
-
 # Roadmap
 
 - [ ] Update integration and automation tests
 - [ ] Create Dockerized lambda code for product creation metric
 
-# Questions
+# Frequently Asked Questions
+
+- How do I create a new metric?
+  - [README.md](src/README.md#how-to-create-a-new-metric)
+- How do I update the Grafana Panels?
+  - TF CDK is used to build and modify Grafana dashboard and the panels that goes inside dashboard. Dashboard configuration is  managed via `dashboard.json` located within the `infrastructure` folder. Export the new configuration from grafana after making your changes, now update the dashboard.json with the updated configuration.
 
-- How can I use the timestream with the hosted grafana?
+- How do I update the Event Docs?
+  - [README.md](docs/README.md)

docs/README.md

@@ -1 +1,27 @@
-# My Event Catalog
+<h1 align="center">Flight Controller - Event Catalog</h1>
+<p align="center">Documentation for Flight Controller's Events</p>
+
+## Link
+
+[Flight Controller Event Catalog](https://legendary-spoon-f82c1640.pages.github.io)
+
+## Documentation Structure
+
+- Event Domains (EG. Account, Backup, Guardrail): `domains/`
+- Domain Events (EG. AccountCreated, AccountRequested): `domains/${DOMAIN}/events/`
+- Event Services (Account Vending Machine, Flight Controller): `services/`
+
+## Development
+
+Making changes to the Event Catalog Documentation is pretty straight forward. You will need to add and update the relevant files within the 3 folders listed above. See others for examples.  
+More info in the official Event Catalog [documentation](https://www.eventcatalog.dev/docs/introduction).
+
+### Running Locally
+
+ `make docs_run`. This can be useful to confirm your changes.
+
+### Make Commands
+
+- `make docs_install` (npm install) Will install all NPM requirements
+- `make docs_build` (npm install & npm run build) Will build the Event Catalog and output to the folder `docs/out`. This is used to confirm you changes are valid.
+- `make docs_run` (npm install & npm run build & npm run dev) Will run a local dev instance of the site, so you can see you changes.

docs/domains/Account/events/AccountCreated/examples/aws.json

@@ -0,0 +1,10 @@
+{
+  "Source": "contino.custom",
+  "DetailType": "Account Created",
+  "Detail": {
+          "event_type": "account_created",
+          "aggregate_id": "123456789",
+          "time": "1679368469"
+      },
+  "EventBusName": "main_lambda_bus_cdktf"
+}

docs/domains/Account/events/AccountCreated/examples/example.json

@@ -9,4 +9,8 @@ owners:
     - josharmi
 ---
 
+## <u>Details</u>
+
+Flight Controller looks at the last pertinent account requested event and calculates the time from it being requested to created.
+
 <NodeGraph title="Consumer / Producer Diagram" />

docs/domains/Account/events/AccountCreatedLeadTime[Metric]/index.md

@@ -0,0 +1,10 @@
+{
+  "Source": "contino.custom",
+  "DetailType": "Account Deleted",
+  "Detail": {
+          "event_type": "account_deleted",
+          "aggregate_id": "123456789",
+          "time": "1679368469"
+      },
+  "EventBusName": "main_lambda_bus_cdktf"
+}

docs/domains/Account/events/AccountDeleted/examples/aws.json

@@ -0,0 +1,10 @@
+{
+  "Source": "contino.custom",
+  "DetailType": "Account Requested",
+  "Detail": {
+          "event_type": "account_requested",
+          "aggregate_id": "123456789",
+          "time": "1679368469"
+      },
+  "EventBusName": "main_lambda_bus_cdktf"
+}