build: update Helm chart (#101)

* Edit Helm Chart

- Add resources
- Add a key to change the storage mode
  - if RWO, the prowes pods will be deployed on the same nodes

* Add initResources

* Fix min requests

* Update mongodb and rabbitMQ

* Fix secrets

* Update Chart.yaml version

* Fix

* Add NOTES.txt

* Fix NOTES.txt

Author: trispera

deployment/Chart.yaml

@@ -2,5 +2,5 @@ apiVersion: v2
 name: prowes
 description: A proWES Helm chart for Kubernetes
 type: application
-version: 0.1.0
-appVersion: 1.16.0
+version: 2.0.0
+appVersion: 2.0.0

deployment/README.md

@@ -3,8 +3,6 @@
 - [Kubernetes deployment for proWES](#kubernetes-deployment-for-wes)
     - [Usage](#usage)
         - [Updates](#updates)
-        - [Using with an external MongoDB](#using-with-an-external-mongodb)
-        - [Setting up RabbitMQ for testing on OpenShift](#setting-up-rabbitmq-for-testing-on-openshift)
     - [Technical details](#technical-details)
         - [MongoDB](#mongodb)
         - [RabbitMQ](#rabbitmq)
@@ -53,95 +51,46 @@ Create a Kubernetes Secret from the `.netrc` file:
 kubectl create secret generic netrc --from-file .netrc
 ```
 
-After this you can deploy proWES using `kubectl`:
+You need to edit the `values.yaml` file to specify your `applicationDomain` and the `clusterType`
 
-```bash
-cd deployment/common/wes
-ls wes-* | xargs -L1 kubectl create -f
-```
-
-Once proWES is deployed, you can expose it with the YAML files found under
-`deployment/ingress`. Which file to use depends on your cluster and how ingress
-is configured there.
-
-Creating an OpenShift Route:
+After this you can deploy proWES using `helm`:
 
 ```bash
-cd deployment/ingress
-oc create -f wes-route.yaml
+helm install prowes . -f values.yaml
 ```
 
 ### Updates
 
-If you make changes to any of the Deployments, you can update them with
-`kubectl`. For example, this is how you would update the Celery worker Deployment:
+If you want to edit any of the Deployments, you can update them with
+`helm` and the `values.yaml` file. Once edited, you can run this command:
 
 ```bash
-kubectl replace -f wes-celery-deployment.yaml
+helm upgrade prowes . -f values.yaml
 ```
 
-The OpenShift specific objects need to be updated using the `oc` tool instead.
-Also, if you update the Route you must use the `--force` flag. This removes and
-recreates the Route.
-
 If you want to point to a different FTP server or change the login credentials
 for the current FTP server, you can update the `.netrc` secret like so:
 
 ```bash
 kubectl create secret generic netrc --from-file .netrc --dry-run -o yaml | kubectl apply -f -
 ```
 
-If you want to update the configuration, you can update the ConfigMap or use a
-different ConfigMap with the same name. The Deployments expect to find the
-`app_config.yaml` ConfigMap with the name `wes-config`. You can update the
-ConfigMap like so:
-
-```bash
-kubectl replace -f wes-configmap.yaml
-```
-
-### Using with an external MongoDB
-
-In certain situations, you may want to run an external MongoDB. There is an
-example headless service file for this case under `deployment/mongodb`.
-
-Make a copy of the example Service:
-
-```bash
-cd deployment/mongodb
-cp mongodb-service-external.yaml.example mongodb-service-external.yaml
-```
-
-Edit the `mongodb-service-external.yaml` file and under `externalName` add your
-external MongoDB's host name. After this, create the Service:
-
-```bash
-kubectl create -f mongodb-service-external.yaml
-```
+## Technical details
 
-It is assumed that the external MongoDB already has a database called
-`prowes-db` created and a user with read-write access available. Next you
-will need to configure the MongoDB user and password in a secret (replace
-`<username>` and `<password>`):
+### MongoDB
 
-```bash
-kubectl create secret generic mongodb --from-literal=database-user=<username> --from-literal=database-password=<password>
-```
+The MongoDB database is deployed using:
 
-Depending on your MongoDB provider, the port may differ from 27017. In this
-case, you will need to modify the Deployment files accordingly to use the
-correct port (replace `<my external mongodb port>`):
+- `templates/mongodb/mongodb-deployment.yaml`
 
-```bash
-cd deployment/common
-find . -name *.yaml | xargs -L1 sed -i -e 's/27017/<my external mongodb port>/g'
-```
+### RabbitMQ
 
-After this you can deploy proWES (see above).
+The message broker RabbitMQ that allows the app to communicate with the worker
+is deployed using:
 
-## Technical details
+- `templates/rabbitmq/rabbitmq-deployment.yaml`
 
-### ApplicaWES
+### WES
 
 proWES consists of five deployments: a Flask server and a Celery worker. These
 are deployed using:
@@ -150,22 +99,14 @@ are deployed using:
 - `templates/prowes/celery-deployment.yaml`
 
 These deployments depend on setting up a shared ReadWriteMany volume between
-Flask and Celery (`wes-volume.yaml`) and a shared ConfigMap
 (`wes-configmap.yaml`).
 
-### MongoDB
 
-The MongoDB database is deployed using:
-
-- `templates/mongodb/mongodb-deployment.yaml`
-
-### RabbitMQ
+## Destroy
 
-The message broker RabbitMQ that allows the app to communicate with the worker
-is deployed using:
+Simply run:
 
-- `templates/rabbitmq/rabbitmq-deployment.yaml`
+```bash
+helm uninstall prowes
+```
 
-The disadvantage with this template is that it only works with OpenShift because
-it uses objects that are only available in OpenShift. For plain Kubernetes a
-different approach needs to be used.

deployment/templates/NOTES.txt

@@ -0,0 +1,11 @@
+Elixir Cloud proWES is being deployed!
+
+Once deployed:
+
+  1. Access the API via https://{{ .Values.prowes.appName }}.{{ .Values.applicationDomain }}/ga4gh/wes/v1/
+
+     To test the connection, you can run:
+
+       curl -X GET "https://{{ .Values.prowes.appName }}.{{ .Values.applicationDomain }}/ga4gh/wes/v1/service-info" -H "Accept: application/json"
+
+  2. Access the Swagger UI via https://{{ .Values.prowes.appName }}.{{ .Values.applicationDomain }}/ga4gh/wes/v1/ui

deployment/templates/flower/flower-deployment.yaml

@@ -19,3 +19,4 @@ spec:
         command: ['flower']
         args: ['--broker=amqp://guest:guest@rabbitmq:5672//', '--port=5555', '--basic_auth={{ .Values.flower.basicAuth }}']
         name: flower
+        resources: {{- toYaml .Values.flower.resources | nindent 10 }}

deployment/templates/mongodb/mongo-init-script.yaml

@@ -0,0 +1,41 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: mongo-init-script 
+data:
+  init-script.js: |
+    db = db.getSiblingDB('runStore');
+    dbproWES = db.getSiblingDB('{{ tpl .Values.mongodb.secret.databaseName . }}')
+
+    dbproWES.createUser({
+      user: "{{ tpl .Values.mongodb.secret.databaseUser . }}",
+      pwd: "{{ tpl .Values.mongodb.secret.databasePassword . }}",
+      roles: [
+        {
+          role: "readWrite",
+          db: "{{ tpl .Values.mongodb.secret.databaseName . }}"
+        }
+      ]
+    });
+
+    // Create the 'runs' and 'service_info' collections
+    // Database configuration from https://github.com/elixir-cloud-aai/proWES/blob/59b53a4bdd06dd83ff41754b27d97309073ed237/pro_wes/config.yaml#L30
+    db.createCollection('runs');
+    db.runs.createIndex(
+      { run_id: 1, task_id: 1 },
+      { unique: true, sparse: true }
+    );
+    db.createCollection('service_info');
+    db.service_info.createIndex(
+      { id: 1 }
+    );
+
+    dbproWES.createCollection('runs');
+    dbproWES.runs.createIndex(
+      { run_id: 1, task_id: 1 },
+      { unique: true, sparse: true }
+    );
+    dbproWES.createCollection('service_info');
+    dbproWES.service_info.createIndex(
+      { id: 1}
+    );

deployment/templates/mongodb/mongodb-deployment.yaml

@@ -16,25 +16,30 @@ spec:
     spec:
       containers:
         - env:
-          - name: MONGODB_USER
+          - name: MONGO_INITDB_ROOT_USERNAME
             valueFrom:
               secretKeyRef:
-                key: database-user
+                key: databaseRootUsername
                 name: {{ .Values.mongodb.appName }}
-          - name: MONGODB_PASSWORD
+          - name: MONGO_INITDB_ROOT_PASSWORD
             valueFrom:
               secretKeyRef:
-                key: database-password
+                key: databaseRootPassword
                 name: {{ .Values.mongodb.appName }}
-          - name: MONGODB_ADMIN_PASSWORD
+          - name: MONGO_INITDB_DATABASE
             valueFrom:
               secretKeyRef:
-                key: database-admin-password
+                key: databaseName
                 name: {{ .Values.mongodb.appName }}
-          - name: MONGODB_DATABASE
+          - name: MONGO_APP_USERNAME
             valueFrom:
               secretKeyRef:
-                key: database-name
+                key: databaseUser
+                name: {{ .Values.mongodb.appName }}
+          - name: MONGO_APP_PASSWORD
+            valueFrom:
+              secretKeyRef:
+                key: databasePassword
                 name: {{ .Values.mongodb.appName }}
           image: {{ .Values.mongodb.image }}
           imagePullPolicy: IfNotPresent
@@ -57,20 +62,23 @@ spec:
                 - '-i'
                 - '-c'
                 - >-
-                  mongo 127.0.0.1:27017/$MONGODB_DATABASE -u $MONGODB_USER -p
-                  $MONGODB_PASSWORD --eval="quit()"
+                  mongosh --host 127.0.0.1:27017 -u $MONGO_INITDB_ROOT_USERNAME -p $MONGO_INITDB_ROOT_PASSWORD --authenticationDatabase admin $MONGO_INITDB_DATABASE --eval="quit()"
             failureThreshold: 3
-            initialDelaySeconds: 3
+            initialDelaySeconds: 30
             periodSeconds: 10
             successThreshold: 1
-            timeoutSeconds: 1
-          resources:
-            limits:
-              memory: 512Mi
+            timeoutSeconds: 50
+          resources: {{- toYaml .Values.mongodb.resources | nindent 12 }}
           volumeMounts:
-            - mountPath: /var/lib/mongodb/data
+            - mountPath: /data/db
               name: mongodb-data
+            - name: init-script
+              mountPath: /docker-entrypoint-initdb.d/init-script.js
+              subPath: init-script.js
       volumes:
         - name: mongodb-data
           persistentVolumeClaim:
             claimName: {{ .Values.mongodb.appName }}-volume
+        - name: init-script
+          configMap:
+            name: mongo-init-script

deployment/templates/mongodb/mongodb-pvc.yaml

@@ -4,7 +4,7 @@ metadata:
   name: {{ .Values.mongodb.appName }}-volume
 spec:
   accessModes:
-    - ReadWriteMany
+    - {{ .Values.storageAccessMode }}
   resources:
     requests:
       storage: {{ .Values.mongodb.volumeSize }}

deployment/templates/mongodb/mongodb-secret.yaml

@@ -4,7 +4,7 @@ type: Opaque
 metadata:
   name: {{ .Values.mongodb.appName }}
 data:
-  database-admin-password: {{ .Values.mongodb.databaseAdminPassword  | b64enc }}
-  database-name: {{ .Values.mongodb.databaseName  | b64enc }}
-  database-password: {{ .Values.mongodb.databasePassword  | b64enc }}
-  database-user: {{ .Values.mongodb.databaseUser  | b64enc }}
+  {{- range $key, $val := .Values.mongodb.secret }}
+    "{{ $key }}": "{{ tpl $val $ | b64enc }}"
+  {{- end }}
+

deployment/templates/prowes/celery-deployment.yaml

@@ -16,6 +16,7 @@ spec:
         image: busybox
         command: [ 'mkdir' ]
         args: [ '-p', '/data/db', '/data/output', '/data/tmp' ]
+        resources: {{- toYaml .Values.celeryWorker.initResources | nindent 10 }}
         volumeMounts:
         - mountPath: /data
           name: prowes-volume
@@ -34,29 +35,23 @@ spec:
         - name: MONGO_USERNAME
           valueFrom:
             secretKeyRef:
-              key: database-user
+              key: databaseUser
               name: {{ .Values.mongodb.appName }}
         - name: MONGO_PASSWORD
           valueFrom:
             secretKeyRef:
-              key: database-password
+              key: databasePassword
               name: {{ .Values.mongodb.appName }}
         - name: MONGO_DBNAME
           valueFrom:
             secretKeyRef:
-              key: database-name
+              key: databaseName
               name: {{ .Values.mongodb.appName }}
         - name: RABBIT_HOST
           value: {{ .Values.rabbitmq.appName }}
         - name: RABBIT_PORT
           value: "5672"
-        resources:
-          requests:
-            memory: "512Mi"
-            cpu: "300m"
-          limits:
-            memory: "8Gi"
-            cpu: "1"
+        resources: {{- toYaml .Values.celeryWorker.resources | nindent 10 }}
         volumeMounts:
         - mountPath: /data
           name: prowes-volume