Document initial setup procedure

Author: QuLogic

README.md

@@ -31,12 +31,12 @@ the server with consistent settings.
 Setup
 -----
 
-Before you can run our ansible playbooks, you need to meet the following
+Before you can run our Ansible playbooks, you need to meet the following
 prerequisites:
 
 * Create a DigitalOcean API token, and pass it to the inventory generator by
   setting the `DO_API_TOKEN` environment variable.
-* Set the vault decryption password of the ansible vaulted file with our
+* Set the vault decryption password of the Ansible vaulted file with our
   secrets. This may be done by setting the `ANSIBLE_VAULT_PASSWORD_FILE`
   environment variable to point to a file containing the password.
 * Download all the collections the playbooks depend on with the following
@@ -55,3 +55,94 @@ There is currently only one playbook:
 
 * `matplotlib.org.yml`, for the main matplotlib.org hosting. This playbook
   operates on droplets with the `website` tag in DigitalOcean.
+
+Provisioning a new server
+=========================
+
+Naming
+------
+
+We follow a simplified version of the naming scheme on [this blog
+post](https://mnx.io/blog/a-proper-server-naming-scheme/):
+
+* Servers are named `<prefix>.matplotlib.org` in A records.
+* Servers get a functional CNAME alias (e.g., `web01.matplotlib.org`).
+* matplotlib.org is a CNAME to the functional CNAME of a server.
+
+We use [planets in our Solar System](https://namingschemes.com/Solar_System)
+for the name prefix. When creating a new server, pick the next one in the list.
+
+Initial setup
+-------------
+
+The summary of the initial setup is:
+
+1. Create the droplet with monitoring and relevant SSH keys.
+2. Assign new droplet to the matplotlib.org project and the Web firewall.
+3. Grab the SSH host fingerprints.
+4. Reboot.
+
+We currently use a simple $5 droplet from DigitalOcean. You can create one from
+the control panel, or using the `doctl` utility. Be sure to enable monitoring,
+and add the `website` tag and relevant SSH keys to the droplet. An example of
+using `doctl` is the following:
+
+```
+doctl compute droplet create \
+    --image fedora-35-x64 \
+    --region tor1 \
+    --size s-1vcpu-1gb \
+    --ssh-keys <key-id>,<key-id> \
+    --tag-name website \
+    --enable-monitoring \
+    venus.matplotlib.org
+```
+
+Note, you will have to use `doctl compute ssh-key list` to get the IDs of the
+relevant SSH keys saved on DigitalOcean, and substitute them above. Save the ID
+of the new droplet from the output, e.g., in:
+
+```
+ID           Name       Public IPv4    Private IPv4    Public IPv6    Memory    VCPUs    Disk    Region    Image            VPC UUID    Status    Tags       Features                    Volumes
+294098687    mpl.org                                                  1024      1        25      tor1      Fedora 35 x64                new       website    monitoring,droplet_agent
+```
+
+the droplet ID is 294098687.
+
+
+You should also assign the new droplet to the `matplotlib.org` project and the
+`Web` firewall:
+
+```
+doctl projects list
+# Get ID of the matplotlib.org project from the output.
+doctl projects resources assign <project-id> --resource=do:droplet:<droplet-id>
+
+
+doctl compute firewall list
+# Get ID of the Web firewall from the output.
+doctl compute firewall add-droplets <firewall-id> --droplet-ids <droplet-id>
+```
+
+Then, to ensure you are connecting to the expected server, you should grab the
+SSH host keys via the DigitalOcean Droplet Console:
+
+```
+for f in /etc/ssh/ssh_host_*_key; do
+  ssh-keygen -l -f $f;
+done
+```
+
+Note down the outputs to verify later, e.g.,
+
+```
+# Use these for comparison when connecting yourself.
+1024 SHA256:ExviVyBRoNKsZpgmIfBaejh1ElOpJ/9fC+ki2Fn5Xj4 root@venus.matplotlib.org (DSA)
+256 SHA256:hLA7ePr0D4AgiC21IXowtbpcUNnTGgpPB7NOYepQtxg root@venus.matplotlib.org (ECDSA)
+256 SHA256:MggFZQbZ7wID1Se2EmOwAm8AaJeA97L8sD8DhSrKy1g root@venus.matplotlib.org (ED25519)
+3072 SHA256:MCkDgfbn0sMTCtvAtfD0HmGJV3LVTjpUj6IcfWRHRQo root@venus.matplotlib.org (RSA)
+```
+
+Finally, you should reboot the droplet. This is due to a bug in cloud-init on
+DigitalOcean, which generates a new machine ID after startup, causing system
+logs to be seem invisible.