diff --git a/README.md b/README.md index d446993..ec3ecb2 100644 --- a/README.md +++ b/README.md @@ -1 +1,10 @@ -This repository contains the buildbot environment for server and worker hosts. +# Buildbot setup for the DataCompressor + +This repository contains the buildbot environment used for automated builds of the [Data Compressor](https://github.com/CenterForSecureEnergyInformatics/data-compressor) + +Subfolder contents: +- [buildbot-buildmaster](buildbot-buildmaster): the buildbot server configuration. +- [raspberry-pi-workers](raspberry-pi-workers): buildbot worker for some variants of the Raspberry Pi. +- [windows-worker](windows-worker): buildbot worker for Windows 10 pro + +Please refer to the README files in the subfolders for setup instructions and information. diff --git a/buildbot-buildmaster/README.md b/buildbot-buildmaster/README.md new file mode 100644 index 0000000..dd6bd54 --- /dev/null +++ b/buildbot-buildmaster/README.md @@ -0,0 +1,32 @@ +# buildbot-buildmaster +This project contains dockerized services, serving a buildbot buildmaster to the public WEB over TLS. +It is intended to be used on a Linux server. +Currently, it runs under Debian Buster. +## Setup +1. Clone this repository to your server. +2. Install docker and docker-compose. +3. Create a dedicated, unprivileged user for this project +4. Add the user to the docker group +5. Clone this repository to your server. You only need the folder containing this README. + - Optional: create a branch for your secrets (git checkout -b secrets) +6. Follow the setup instructions in the subfolders. Start with traefik and then buildbot. + +This project comes mostly preconfigured. + +What you have to do - an overview (refer to the README files in the subfolders): +- Changing passwords / specifying credentials (see [buildbot/README.md](buildbot/README.md)) +- Providing a certificate for TLS connections ([traefik/certs](traefik/certs)) +- Adjusting the WEB URL ([buildbot/docker-compose.yml](buildbot/docker-compose.yml)) +- Creating a persistent data directory (see [buildbot/README.md](buildbot/README.md)) +- Adjusting file permissions (traefik and buildbot secrets) + +## Usage + ``` + cd traefik + docker-compose up -d + cd ../buildbot + docker-compose up -d + ``` + +You might have to restart traefik again, when workers can't connect to the buildmaster. +Typically the raspberry-pis have this problem. diff --git a/buildbot-buildmaster/buildbot/README.md b/buildbot-buildmaster/buildbot/README.md new file mode 100644 index 0000000..aa611ed --- /dev/null +++ b/buildbot-buildmaster/buildbot/README.md @@ -0,0 +1,102 @@ +# buildbot-buildmaster +This directory contains the configuration of the buildbot buildmaster. + +The buildmaster listens to changes on the following repositories: +- https://github.com/CenterForSecureEnergyInformatics/data-compressor (pull requests and branch master) +- https://github.com/CenterForSecureEnergyInformatics/data-compressor-tests (branch master) + +When changes are detected (or the force button in the web ui is pressed), the project is built and tested. +## Usage +This project is dockerized and uses docker-compose. +The file [docker-compose.yml](docker-compose.yml) tells docker-compose what to do, so you have to change into the directory containing the file, before executing any of these commands! +### Starting +- `docker-compose up -d` + +or to restart +- `docker-compose restart` + +Note: If the Raspberry Pis can't connect, simply go to `../traefik` and run docker-compose restart from there, too. +### Stopping +- docker-compose down +### Updating +Run the following steps in this order: +``` +docker-compose down +docker-compose pull +docker-compose build +docker image prune +docker-compose up -d +``` +### Debugging +To view logs in realtime, run +`docker-compose logs -f` + +Exit with `CRTL+C` +## How it works +Buildbot-workers are defined in [master.cfg](master.cfg). +The workers are connected via port 9989 to the buildmaster. +Workers on different platforms are implemented in this setup: +- Windows 10. Unfortunately, tls seems not to work on Windows with buildbot, so the VM running the buildmaster has to share a subnet with the Windows-VM (which it does in our setup). +- Raspberri Pi (1, 2b and 3b+), dockerized. They use a TLS connection handled by the reverse proxy traefik. +- Linux, dockerized. See [multiarch_dockerfile](multiarch_dockerfile) and [docker-compose.yml](docker-compose.yml) for the definition. It runs on the same host as the buildmaster. + +Buildfactories define which steps are to be executed on a build. +For each platform/architecture a scheduler is defined in this configuration. + +Builders are assigned to jobs (defined by factories). +Finally, schedulers trigger builds on the actual workers. + +For a detailed overview, please refer to the [Buildbot Manual](http://docs.buildbot.net/latest/manual/introduction.html). + +## Setup / Configuration +The file [master.cfg](master.cfg) is the main configuration file of buildbot, [docker-compose.yml](docker-compose.yml) handles all services involved. +### master.cfg +Everything is pre-configured in this setup and does not require many changes. +Please refer to [Buildbot Manual](http://docs.buildbot.net/latest/manual/introduction.html) for details, as a detailed explanation is beyond the scope of this README. +#### Credentials for the WEB UI +To force builds, you need to be logged in. +In [master.cfg](master.cfg), fill in your e-mail address(es) under `util.RolesFromEmails(admins=["you@email.provider"])` +Make sure, that: +- Your credentials are filled in [secrets/htpasswd](secrets/htpasswd) (only clear text, so keep permissions (600) in mind). +- The e-mail address is from a contributor in the github-project. +#### Workers +DO NOT CHANGE [master.cfg](master.cfg) IN THIS STEP! + +All workers are pre-configured. +Each of them has a name (do not change!) and a password (please change!), specified in .env files: +- [../windows/windows.env](../windows/windows.env) +- The .env files in the subfolders of [../rasperry-pi/](../raspberry-pi) +- [multiarch.env](multiarch.env) + +Make sure to set these passwords on the corresponding workers as well! +On the workers, the files are located in the same folders and files to make this step easier. + +#### GitHubPullRequestPoller +Important Note: for the GithubPullrequestPoller to work, the owner and repository name (NOT the URL) have to be provided. +In this configuration, the following are used: +- `compressorRepoName = 'data-compressor'` +- `compressorRepoOwner = 'CenterForSecureEnergyInformatics'` + +### docker-compose.yml +The following services are specified here: +- buildbot-buildmaster: the buildmaster and the webinterface itself +- db: a database for the buildmaster +- worker: a buildbot worker running linux (used to crosscompile), defined in multiarch_dockerfile + +#### Persistent Data Storage +You have to create a directory for persistent storage for the database service. +- `mkdir -p /data/buildbot/db` + +If you are unhappy with this location, you can specify another one in docker-compose.yml. +To do so, modify the volume of the service "db" accordingly. + +#### WEB URL +If you aren't running this service under `mendel.fh-salzburg.ac.at`, you have to replace all occurences in [docker-compose.yml](docker-compose.yml) with a different URL. +You'll find them in the labels of the service buildbot-buildmaster. +### Subnet for non-TLS communication +Windows workers currently cannot connect to the buildmaster via TLS. +For this reason, the VM hosting the buildmaster and the Windows 10 VM running the worker share a subnet to ensure secure communication. +You have to provide the IP of the Subnet in [docker-compose.yml](docker-compose.yml). +To do so, replace `` with your the server's IP in that subnet. +### db.env +Specify a database password. diff --git a/buildbot-buildmaster/buildbot/db.env b/buildbot-buildmaster/buildbot/db.env new file mode 100644 index 0000000..9b512ca --- /dev/null +++ b/buildbot-buildmaster/buildbot/db.env @@ -0,0 +1,6 @@ +# database parameters are shared between containers +POSTGRES_PASSWORD=changeme! +POSTGRES_USER=buildbot +POSTGRES_DB=buildbot +# in master.cfg, this variable is str.format()ed with the environment variables +BUILDBOT_DB_URL=postgresql+psycopg2://{POSTGRES_USER}:{POSTGRES_PASSWORD}@db/{POSTGRES_DB} diff --git a/buildbot-buildmaster/buildbot/docker-compose.yml b/buildbot-buildmaster/buildbot/docker-compose.yml new file mode 100644 index 0000000..ee16f6a --- /dev/null +++ b/buildbot-buildmaster/buildbot/docker-compose.yml @@ -0,0 +1,82 @@ +version: '3' +services: + buildbot-buildmaster: + image: buildbot/buildbot-master:master + restart: unless-stopped + env_file: + - ./db.env + - ./multiarch.env + - ../windows/windows.env + - ../raspberry-pi/armv6/armv6.env + - ../raspberry-pi/armv7/armv7.env + - ../raspberry-pi/armv8/armv8.env + - ../raspberry-pi/arm64v8/arm64v8.env + environment: + - BUILDBOT_CONFIG_DIR=config + - BUILDBOT_WORKER_PORT=9989 + - BUILDBOT_WEB_URL=https://mendel.fh-salzburg.ac.at/ + - BUILDBOT_WEB_PORT=tcp:port=80 + - BUILDBOT_TITLE=datacompressor testing + - BUILDBOT_TITLE_URL=https://github.com/CenterForSecureEnergyInformatics + volumes: + - ./secrets/:/var/lib/buildbot/secrets + - ./master.cfg:/var/lib/buildbot/master.cfg + - ../traefik/dump/mendel.fh-salzburg.ac.at/:/var/lib/buildbot/certificate.pem:ro + - ../traefik/dump/mendel.fh-salzburg.ac.at/:/var/lib/buildbot/privateky.pem:ro + networks: + - proxy + - buildbot + ports: + - :9989:9989 + labels: + - "traefik.enable=true" + - "traefik.docker.network=proxy" + + - "traefik.http.routers.buildbot.rule=Host(`mendel.fh-salzburg.ac.at`)" + - "traefik.http.routers.buildbot.entrypoints=http" + - "traefik.http.routers.buildbot.middlewares=redirect-to-https@file" + + - "traefik.http.routers.buildbot-secure.rule=Host(`mendel.fh-salzburg.ac.at`)" + - "traefik.http.routers.buildbot-secure.entrypoints=https" + - "traefik.http.routers.buildbot-secure.service=buildbot" + - "traefik.http.routers.buildbot-secure.tls=true" + - "traefik.http.routers.buildbot-secure.tls.certresolver=http" + - "traefik.http.routers.buildbot-secure.middlewares=hsts-header@file" + + - "traefik.http.services.buildbot.loadbalancer.server.port=80" + + - "traefik.tcp.routers.buildbot.rule=HostSNI(`mendel.fh-salzburg.ac.at`)" + - "traefik.tcp.routers.buildbot.entrypoints=buildbot" + - "traefik.tcp.routers.buildbot.service=buildbot" + - "traefik.tcp.routers.buildbot.tls=true" + + - "traefik.tcp.services.buildbot.loadbalancer.server.port=9989" + db: + env_file: + - db.env + image: "postgres:9.4" + restart: unless-stopped + volumes: + - /data/buildbot/db:/var/lib/postgresql/data + networks: + - buildbot + + worker: + build: + context: . + dockerfile: multiarch_dockerfile + restart: unless-stopped + env_file: + - ./multiarch.env + environment: + BUILDMASTER: buildbot-buildmaster + BUILDMASTER_PORT: 9989 + networks: + - buildbot + +networks: + proxy: + external: true + buildbot: + external: false + diff --git a/buildbot-buildmaster/buildbot/master.cfg b/buildbot-buildmaster/buildbot/master.cfg new file mode 100644 index 0000000..0aedf65 --- /dev/null +++ b/buildbot-buildmaster/buildbot/master.cfg @@ -0,0 +1,302 @@ +# -*- python -*- +# ex: set filetype=python: + +import os + +from buildbot.plugins import * +# This is a sample buildmaster config file. It must be installed as +# 'master.cfg' in your buildmaster's base directory. + +# This is the dictionary that the buildmaster pays attention to. We also use +# a shorter alias to save typing. +c = BuildmasterConfig = {} + +secretsdir = os.environ.get("SECRETSDIR", "secrets") +c['secretsProviders'] = [secrets.SecretInAFile(dirname=secretsdir)] + +####### WORKERS + +# The 'workers' list defines the set of recognized workers. Each element is +# a Worker object, specifying a unique worker name and password. The same +# worker name and password must be configured on the worker. +multiarch_worker = os.environ.get("MULTIARCH_NAME",'multiarch') +armv6_worker = os.environ.get("ARMV6_NAME",'rpi_armv6') +armv7_worker = os.environ.get("ARMV7_NAME",'rpi_armv7') +armv8_worker = os.environ.get("ARMV8_NAME",'rpi_armv8') +arm64v8_worker = os.environ.get("ARM64V8_NAME",'rpi_arm64v8') +windows_worker = os.environ.get("WINDOWS_NAME",'windows') +c['workers'] = [ + worker.Worker(multiarch_worker, os.environ.get("MULTIARCH_PASS",'pass')), + worker.Worker(armv6_worker, os.environ.get("ARMV6_PASS",'pass')), + worker.Worker(armv7_worker, os.environ.get("ARMV7_PASS",'pass')), + worker.Worker(armv8_worker, os.environ.get("ARMV8_PASS",'pass')), + worker.Worker(arm64v8_worker, os.environ.get("ARM64V8_PASS",'pass')), + worker.Worker(windows_worker, os.environ.get("WINDOWS_PASS",'pass')), + ] + +if 'BUILDBOT_MQ_URL' in os.environ: + c['mq'] = { + 'type' : 'wamp', + 'router_url': os.environ['BUILDBOT_MQ_URL'], + 'realm': os.environ.get('BUILDBOT_MQ_REALM', 'buildbot').decode('utf-8'), + 'debug' : 'BUILDBOT_MQ_DEBUG' in os.environ, + 'debug_websockets' : 'BUILDBOT_MQ_DEBUG' in os.environ, + 'debug_lowlevel' : 'BUILDBOT_MQ_DEBUG' in os.environ, + } +# 'protocols' contains information about protocols which master will use for +# communicating with workers. You must define at least 'port' option that workers +# could connect to your master with this protocol. +# 'port' must match the value configured into the workers (with their +# --master option) +c['protocols'] = {'pb': {'port': os.environ.get("BUILDBOT_WORKER_PORT", 9989)}} + +####### CHANGESOURCES + +# the 'change_source' setting tells the buildmaster how it should find out +# about source code changes. Here we point to the buildbot clone of pyflakes. + +testRepoName = "data-compressor-tests" +testRepoOwner = 'CenterForSecureEnergyInformatics' +testRepo = "https://github.com/" + testRepoOwner + "/" + testRepoName + ".git" + +compressorRepoName = 'data-compressor' +compressorRepoOwner = 'CenterForSecureEnergyInformatics' +compressorRepo = "https://github.com/" + compressorRepoOwner + "/" + compressorRepoName + ".git" + +#compressorRepo = 'https://github.com/CenterForSecureEnergyInformatics/data-compressor.git' +token = open(secretsdir + "/githubToken").read().strip() + +c['change_source'] = [] +c['change_source'].append(changes.GitPoller( + compressorRepo, + workdir='gitpoller-workdir', + branches=['master'], + pollInterval=30)) +c['change_source'].append(changes.GitHubPullrequestPoller( + owner=compressorRepoOwner, + repo=compressorRepoName, + pollInterval=30, + token=token)) + +####### SCHEDULERS + +# Configure the Schedulers, which decide how to react to incoming changes. In this +# case, just kick off a 'runtests' build + +# first, we define the builders: +x86_64 = "x86_64" +i386 = "i386" +armv6_native = "armv6_native" +armv7_native = "armv7_native" +armv8_native = "armv8_native" +arm64v8_native = "arm64v8_native" +arm = "arm" +win_x64 = "win_x64" +win32 = "win32" +# ------ + +c['schedulers'] = [] +c['schedulers'].append(schedulers.AnyBranchScheduler( + name="anyBranch", + treeStableTimer=None, + builderNames=[ x86_64, i386, armv6_native , armv7_native , armv8_native , arm64v8_native , arm , win_x64 , win32 ])) +c['schedulers'].append(schedulers.ForceScheduler( + name="force", + builderNames=[ x86_64, i386, armv6_native , armv7_native , armv8_native , arm64v8_native , arm , win_x64 , win32 ])) + +####### FACTORIES +testWorkdirName = testRepoName +testdir = "../" + testWorkdirName + "/tests/" + +desc = "building and testing" +descDone = "build and test" + +windowsShell = "C:\\Program Files\\Git\\bin\\bash.exe" + +factoryLinux_x86_64 = util.BuildFactory() +factoryLinux_x86_64.addSteps([ + steps.Git(name="git - tests", repourl=testRepo, workdir=testWorkdirName, mode='full', alwaysUseLatest='true'), + steps.Git(name="git - data-compressor", repourl=compressorRepo, workdir='data-compressor', mode='full', alwaysUseLatest='true'), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "checkBits.sh", "x86_64"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "buildAndTest.sh", "x86_64", "8"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "buildAndTest.sh", "x86_64", "16"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "buildAndTest.sh", "x86_64", "32"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "buildAndTest.sh", "x86_64", "64"]), +]) + +factoryLinux_i386 = util.BuildFactory() +factoryLinux_i386.addSteps([ + steps.Git(name="git - tests", repourl=testRepo, workdir=testWorkdirName, mode='full', alwaysUseLatest='true'), + steps.Git(name="git - data-compressor", repourl=compressorRepo, workdir='data-compressor', mode='full', alwaysUseLatest='true'), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "checkBits.sh", "i386"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "buildAndTest.sh", "i386", "8"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "buildAndTest.sh", "i386", "16"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "buildAndTest.sh", "i386", "32"]), +]) + +factoryLinux_armhf = util.BuildFactory() +factoryLinux_armhf.addSteps([ + steps.Git(name="git - tests", repourl=testRepo, workdir=testWorkdirName, mode='full', alwaysUseLatest='true'), + steps.Git(name="git - data-compressor", repourl=compressorRepo, workdir='data-compressor', mode='full', alwaysUseLatest='true'), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "checkBits.sh", "armhf"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "buildAndTest.sh", "armhf", "8"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "buildAndTest.sh", "armhf", "16"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "buildAndTest.sh", "armhf", "32"]), +]) + +factory_arm= util.BuildFactory() +factory_arm.addSteps([ + steps.Git(name="git - tests", repourl=testRepo, workdir=testWorkdirName, mode='full', alwaysUseLatest='true'), + steps.Git(name="git - data-compressor", repourl=compressorRepo, workdir='data-compressor', mode='full', alwaysUseLatest='true'), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "checkBits.sh", "arm"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "buildAndTest.sh", "arm", "8"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "buildAndTest.sh", "arm", "16"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "buildAndTest.sh", "arm", "32"]), +]) + +factory_aarch64= util.BuildFactory() +factory_aarch64.addSteps([ + steps.Git(name="git - tests", repourl=testRepo, workdir=testWorkdirName, mode='full', alwaysUseLatest='true'), + steps.Git(name="git - data-compressor", repourl=compressorRepo, workdir='data-compressor', mode='full', alwaysUseLatest='true'), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "checkBits.sh", "arm"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "buildAndTest.sh", "arm", "8"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "buildAndTest.sh", "arm", "16"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "buildAndTest.sh", "arm", "32"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[testdir + "buildAndTest.sh", "arm", "64"]), +]) + +factoryWin_x64 = util.BuildFactory() +factoryWin_x64.addSteps([ + steps.Git(name="git - tests", repourl=testRepo, workdir=testWorkdirName, mode='full', alwaysUseLatest='true'), + steps.Git(name="git - data-compressor", repourl=compressorRepo, workdir='data-compressor', mode='full', alwaysUseLatest='true'), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[windowsShell, testdir + "checkBits.sh", "x64"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[windowsShell, testdir + "buildAndTest.sh", "x64", "8"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[windowsShell, testdir + "buildAndTest.sh", "x64", "16"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[windowsShell, testdir + "buildAndTest.sh", "x64", "32"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[windowsShell, testdir + "buildAndTest.sh", "x64", "64"]), +]) + +factoryWin32 = util.BuildFactory() +factoryWin32.addSteps([ + steps.Git(name="git - tests", repourl=testRepo, workdir=testWorkdirName, mode='full', alwaysUseLatest='true'), + steps.Git(name="git - data-compressor", repourl=compressorRepo, workdir='data-compressor', mode='full', alwaysUseLatest='true'), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[windowsShell, testdir + "checkBits.sh", "win32"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[windowsShell, testdir + "buildAndTest.sh", "win32", "8"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[windowsShell, testdir + "buildAndTest.sh", "win32", "16"]), + steps.ShellCommand(description=desc, descriptionDone=descDone, command=[windowsShell, testdir + "buildAndTest.sh", "win32", "32"]), +]) + +####### BUILDERS + +# The 'builders' list defines the Builders, which tell Buildbot how to perform a build: +# what steps, and which workers can execute them. Note that any particular build will +# only take place on one worker. + +c['builders'] = [] + +c['builders'].append( + util.BuilderConfig(name=x86_64, + workernames=[multiarch_worker], + factory=factoryLinux_x86_64)) + +c['builders'].append( + util.BuilderConfig(name=i386, + workernames=[multiarch_worker], + factory=factoryLinux_i386)) + +c['builders'].append( + util.BuilderConfig(name=arm, + workernames=[multiarch_worker], + factory=factoryLinux_armhf)) + +c['builders'].append( + util.BuilderConfig(name=armv6_native, + workernames=[armv6_worker], + factory=factory_arm)) + +c['builders'].append( + util.BuilderConfig(name=armv7_native, + workernames=[armv7_worker], + factory=factory_arm)) + +c['builders'].append( + util.BuilderConfig(name=armv8_native, + workernames=[armv8_worker], + factory=factory_arm)) + +c['builders'].append( + util.BuilderConfig(name=arm64v8_native, + workernames=[arm64v8_worker], + factory=factory_aarch64)) + +c['builders'].append( + util.BuilderConfig(name=win_x64, + workernames=[windows_worker], + factory=factoryWin_x64)) + +c['builders'].append( + util.BuilderConfig(name=win32, + workernames=[windows_worker], + factory=factoryWin32)) + +################ + +gs = reporters.GitHubStatusPush(token=util.Secret("githubToken"), + context=util.Interpolate("buildbot/%(prop:buildername)s"), + startDescription='Build started.', + endDescription='Build end', + verbose=True) + +c['services'] = [] +#c['services'].append(gs) + +####### STATUS TARGETS + +# 'status' is a list of Status Targets. The results of each build will be +# pushed to these targets. buildbot/status/*.py has a variety to choose from, +# like IRC bots. + +#c['status'] = [] + +####### PROJECT IDENTITY + +# the 'title' string will appear at the top of this buildbot installation's +# home pages (linked to the 'titleURL'). + +c['title'] = os.environ.get("BUILDBOT_TITLE", "buildbot") +c['titleURL'] = os.environ.get("BUILDBOT_TITLE_URL", "https://github.com/CenterForSecureEnergyInformatics/data-compressor") + +# the 'buildbotURL' string should point to the location where the buildbot's +# internal web server is visible. This typically uses the port number set in +# the 'www' entry below, but with an externally-visible host name which the +# buildbot cannot figure out without some help. + +c['buildbotURL'] = os.environ.get("BUILDBOT_WEB_URL", "http://localhost:8010/") + +# minimalistic config to activate new web UI +c['www'] = dict(port=os.environ.get("BUILDBOT_WEB_PORT", 8010), + plugins=dict(waterfall_view={}, console_view={}, grid_view={})) + + +c['www']['avatar_methods'] = [] + +authz = util.Authz( + allowRules = [ + util.AnyControlEndpointMatcher(role="admins") + ], + roleMatchers = [ + util.RolesFromEmails(admins=["you@email.provider"]) + ] +) +auth = util.HTPasswdAuth(secretsdir + "/htpasswd") +c['www']['auth'] = auth +c['www']['authz'] = authz + +####### DB URL + +c['db'] = { + # This specifies what database buildbot uses to store its state. You can leave + # this at its default for all but the largest installations. + 'db_url' : os.environ.get("BUILDBOT_DB_URL", "sqlite://").format(**os.environ), +} +c['buildbotNetUsageData'] = None diff --git a/buildbot-buildmaster/buildbot/multiarch.env b/buildbot-buildmaster/buildbot/multiarch.env new file mode 100644 index 0000000..f4c8024 --- /dev/null +++ b/buildbot-buildmaster/buildbot/multiarch.env @@ -0,0 +1,2 @@ +MULTIARCH_NAME=multiarch +MULTIARCH_PASS=changeme! diff --git a/buildbot-buildmaster/buildbot/multiarch_dockerfile b/buildbot-buildmaster/buildbot/multiarch_dockerfile new file mode 100644 index 0000000..9bf79c5 --- /dev/null +++ b/buildbot-buildmaster/buildbot/multiarch_dockerfile @@ -0,0 +1,41 @@ +FROM ubuntu:bionic + +# hints related to apt and multiple architectures: +# https://stackoverflow.com/questions/37706635/in-docker-apt-get-install-fails-with-failed-to-fetch-http-archive-ubuntu-com +# https://wiki.ubuntuusers.de/sources.list/ +# https://wiki.debian.org/QemuUserEmulation +# https://wiki.debian.org/Multiarch/HOWTO + +RUN mv /etc/apt/sources.list /etc/apt/sources.list.old + +RUN printf \ +"deb [arch=amd64,i386] http://de.archive.ubuntu.com/ubuntu bionic main restricted universe multiverse\n\ +deb [arch=amd64,i386] http://de.archive.ubuntu.com/ubuntu bionic-updates main restricted universe multiverse\n\ +deb [arch=amd64,i386] http://de.archive.ubuntu.com/ubuntu bionic-security main restricted universe multiverse\n\ +deb [arch=amd64,i386] http://de.archive.ubuntu.com/ubuntu bionic-backports main restricted universe multiverse\n\ +deb [arch=armhf] http://ports.ubuntu.com/ubuntu-ports/ bionic main universe restricted multiverse\n\ +deb [arch=armhf] http://ports.ubuntu.com/ubuntu-ports/ bionic-security main universe restricted multiverse\n\ +deb [arch=armhf] http://ports.ubuntu.com/ubuntu-ports/ bionic-updates main universe restricted multiverse\n\ +deb [arch=armhf] http://ports.ubuntu.com/ubuntu-ports/ bionic-backports main universe restricted multiverse\n"\ +> /etc/apt/sources.list + + +RUN apt-get update && \ + apt-get install -y git && \ + dpkg --add-architecture i386 && \ + dpkg --add-architecture armhf && \ + apt-get update && \ + apt-get install -y build-essential gcc-multilib mingw-w64 wine64 wine32 qemu binfmt-support qemu-user-static && \ + apt-get install -y crossbuild-essential-armhf libc6:armhf && \ + apt-get install -y python-dev python-pip + +RUN pip install buildbot-worker + +RUN groupadd -r buildbot && \ + useradd -r -g buildbot buildbot && \ + mkdir /worker && \ + chown buildbot:buildbot /worker +# Install your build-dependencies here ... +USER buildbot +WORKDIR /worker +ENTRYPOINT buildbot-worker create-worker . $BUILDMASTER $MULTIARCH_NAME $MULTIARCH_PASS && unset BUILDMASTER MULTIARCH_NAME MULTIARCH_PASS BUILDMASTER_PORT && buildbot-worker start --nodaemon diff --git a/buildbot-buildmaster/buildbot/secrets/README.md b/buildbot-buildmaster/buildbot/secrets/README.md new file mode 100644 index 0000000..6c7e032 --- /dev/null +++ b/buildbot-buildmaster/buildbot/secrets/README.md @@ -0,0 +1,18 @@ +# Secrets +This is the place for secrets. +Be sure to set the correct file permissions! (chmod 600) +## htpasswd file +[htpasswd](htpasswd) contains the login credentials for the users defined in [../master.cfg](../master.cfg) +They are defined in the following form: + +`contributor@email.provider:cleartext-password` + +Note, that: +- Passwords have to be set in clear text +- Users have to be specified in [../master.cfg](../master.cfg) +- The user has to be a contributor in the github-project +## github access token +[githubToken](githubToken) contains the access token used by buildbot to communicate with Github. +You can define it in your Github account. + + diff --git a/buildbot-buildmaster/buildbot/secrets/githubToken b/buildbot-buildmaster/buildbot/secrets/githubToken new file mode 100644 index 0000000..af6823c --- /dev/null +++ b/buildbot-buildmaster/buildbot/secrets/githubToken @@ -0,0 +1 @@ +YourTokenHere diff --git a/buildbot-buildmaster/buildbot/secrets/htpasswd b/buildbot-buildmaster/buildbot/secrets/htpasswd new file mode 100644 index 0000000..9cf5bf9 --- /dev/null +++ b/buildbot-buildmaster/buildbot/secrets/htpasswd @@ -0,0 +1,2 @@ +contributor@email.provider:cleartext-password +another.contributor@email.provider:cleartext-password diff --git a/buildbot-buildmaster/raspberry-pi/README.md b/buildbot-buildmaster/raspberry-pi/README.md new file mode 100644 index 0000000..111311e --- /dev/null +++ b/buildbot-buildmaster/raspberry-pi/README.md @@ -0,0 +1,6 @@ +This folder contains the credentials of the buildbot-workers (running on Raspberry Pis) on the buildbot buildmaster. + +The credentials (builder name and password) have to be placed in the .env files located in the subfolders. +Each subfolder represents an architecture, e.g. armv6. + +The .env files are required by the buildbot-buildmaster (see master.cfg). diff --git a/buildbot-buildmaster/raspberry-pi/arm64v8/arm64v8.env b/buildbot-buildmaster/raspberry-pi/arm64v8/arm64v8.env new file mode 100644 index 0000000..a32f40e --- /dev/null +++ b/buildbot-buildmaster/raspberry-pi/arm64v8/arm64v8.env @@ -0,0 +1,2 @@ +ARM64V8_PASS=changeme! +ARM64V8_NAME=rpi3b_plus_64 diff --git a/buildbot-buildmaster/raspberry-pi/armv6/armv6.env b/buildbot-buildmaster/raspberry-pi/armv6/armv6.env new file mode 100644 index 0000000..6dfc6d3 --- /dev/null +++ b/buildbot-buildmaster/raspberry-pi/armv6/armv6.env @@ -0,0 +1,2 @@ +ARMV6_PASS=changeme! +ARMV6_NAME=rpi1 diff --git a/buildbot-buildmaster/raspberry-pi/armv7/armv7.env b/buildbot-buildmaster/raspberry-pi/armv7/armv7.env new file mode 100644 index 0000000..fbcad3f --- /dev/null +++ b/buildbot-buildmaster/raspberry-pi/armv7/armv7.env @@ -0,0 +1,2 @@ +ARMV7_PASS=changeme! +ARMV7_NAME=rpi2b diff --git a/buildbot-buildmaster/raspberry-pi/armv8/armv8.env b/buildbot-buildmaster/raspberry-pi/armv8/armv8.env new file mode 100644 index 0000000..ab24618 --- /dev/null +++ b/buildbot-buildmaster/raspberry-pi/armv8/armv8.env @@ -0,0 +1,2 @@ +ARMV8_PASS=changeme! +ARMV8_NAME=rpi3b_plus_32 diff --git a/buildbot-buildmaster/traefik/README.md b/buildbot-buildmaster/traefik/README.md new file mode 100644 index 0000000..8aa5d1a --- /dev/null +++ b/buildbot-buildmaster/traefik/README.md @@ -0,0 +1,64 @@ +# Reverse Proxy (traefik) +This folder contains the configuration for the reverse proxy in front of the buildbot buildmaster. +It is based on the traefik project: https://containo.us/traefik/ + +The proxy handles TLS connections and automatically redirects from port 80 to port 443. +Furthermore, it listens on port 9989 and provides a TLS connection for buildbot-workers to the buildmaster. + +The services behind traefik reside in private networks, only the ports needed are exposed to the internet. + +Traefik needs access to the docker-socket: `var/run/docker.sock`. +Since this could result in vulnerabilities, the socket is made available over a socket-proxy (image: `tecnativa/docker-socket-proxy`), which runs as a service next to traefik. + +The automatically obtained certs are extracted from [acme.json](acme.json) by the dockerized service `cert-dumper` (image: `ldez/traefik-certs-dumper:v2.7.0`) and stored in the folder [dynamic](dynamic). +## Usage +This project is dockerized and uses docker-compose. +The file docker-compose.yml tells docker-compose what to do, so you have to change into the directory containing the file, before executing any of these commands! +### Starting +- `docker-compose up -d` + +or to restart +- `docker-compose restart` + +Note: If the Raspberry Pis can't connect to the buildmaster, simply run `docker-compose` restart. +### Stopping +- `docker-compose down` +### Updating +Run the following steps in this order: +``` +docker-compose down +docker-compose pull +docker-compose build +docker image prune +docker-compose up -d +``` +### Debugging +To view logs in realtime, run +- `docker-compose logs -f` + +Exit with `CTRL+C` + +## Setup / Configuration +Traefik has a static configuration ([docker-compose.yml](docker-compose.yml) and [traefik.yml](traefik.yml)) and a dynamic configuration ([acme.json](acme.json) and [dynamic](dynamic)). +Trafik handles certificate renewals automatically. +The certificates are stored in [acme.json](acme.json). +The service `cert-dumper` extracts and stores the certificates in the folder [dump](dump). + +### Steps: + +You have to change the file permissions for [acme.json](acme.json) and [dynamic](dynamic). +Todo so, run: +- `chmod 600 acme.json` +- `chmod 600 acme.json` + +For the `cert-dumper` to be able to create files under a username running the services, you have to add the following to your `~/.bashrc`: + +``` +export COMPOSE_UID=$(id -u) +export COMPOSE_GID=$(id -g) +``` + +### Configuration of services behind traefik: +The services handled by traefik are configured via labels in the corresponding docker-compose.yml files, in our case [../buildbot/docker-compose.yml](../buildbot/docker-compose.yml). +Please refer to [../buildbot/README.md](../buildbot/README.md) for setup instructions. + diff --git a/buildbot-buildmaster/traefik/acme.json b/buildbot-buildmaster/traefik/acme.json new file mode 100644 index 0000000..e69de29 diff --git a/buildbot-buildmaster/traefik/docker-compose.yml b/buildbot-buildmaster/traefik/docker-compose.yml new file mode 100644 index 0000000..6449e3e --- /dev/null +++ b/buildbot-buildmaster/traefik/docker-compose.yml @@ -0,0 +1,52 @@ +version: '3' + +services: + socket-proxy: + image: tecnativa/docker-socket-proxy + container_name: socket-proxy + restart: unless-stopped + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + environment: + CONTAINERS: 1 + networks: + - socketproxy + cert-dumper: + image: ldez/traefik-certs-dumper:v2.7.0 + restart: unless-stopped + container_name: cert-dumper + # add the following to your ~/.bashrc: + # export COMPOSE_UID=$(id -u) + # export COMPOSE_GID=$(id -g) + # OR add IDs directly to .env + user: ${COMPOSE_UID:?Please export COMPOSE_UID; see comments in docker-compose.yml}:${COMPOSE_GID:?Please export COMPOSE_UID; see comments in docker-compose.yml} + working_dir: $HOME + entrypoint: "/usr/bin/traefik-certs-dumper file --source $HOME/acme.json --dest $HOME/dump --domain-subdir --crt-ext=.pem --key-ext=.pem --version v2 --watch" + network_mode: "none" + volumes: + - ./acme.json:$HOME/acme.json:ro + - ./dump:$HOME/dump + + traefik: + image: traefik:v2.0 + container_name: traefik + restart: unless-stopped + security_opt: + - no-new-privileges:true + networks: + - proxy + - socketproxy + ports: + - 80:80 + - 443:443 + - 9989:9989 + volumes: + - /etc/localtime:/etc/localtime:ro + - ./traefik.yml:/traefik.yml:ro + - ./dynamic/:/etc/traefik/conf/:ro + - ./acme.json:/acme.json +networks: + socketproxy: + external: true + proxy: + external: true diff --git a/buildbot-buildmaster/traefik/dynamic/redirect-to-https.yml b/buildbot-buildmaster/traefik/dynamic/redirect-to-https.yml new file mode 100644 index 0000000..81000b3 --- /dev/null +++ b/buildbot-buildmaster/traefik/dynamic/redirect-to-https.yml @@ -0,0 +1,7 @@ +http: + middlewares: + redirect-to-https: + redirectScheme: + scheme: https + permanent: true + diff --git a/buildbot-buildmaster/traefik/dynamic/tls.yml b/buildbot-buildmaster/traefik/dynamic/tls.yml new file mode 100644 index 0000000..5c3da65 --- /dev/null +++ b/buildbot-buildmaster/traefik/dynamic/tls.yml @@ -0,0 +1,23 @@ +# tls options. derived from toml-file generated by: +# https://ssl-config.mozilla.org/#server=traefik&version=2&config=intermediate&guideline=5.4 + +http: + middlewares: + hsts-header: + headers: + customResponseHeaders: + Strict-Transport-Security: "max-age=63072000" + stsPreload: true +tls: + options: + default: + minVersion: "VersionTLS12" + sniStrict: true + cipherSuites: + - "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384" + - "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384" + - "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256" + - "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256" + - "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305" + - "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305" + diff --git a/buildbot-buildmaster/traefik/traefik.yml b/buildbot-buildmaster/traefik/traefik.yml new file mode 100644 index 0000000..b284a82 --- /dev/null +++ b/buildbot-buildmaster/traefik/traefik.yml @@ -0,0 +1,24 @@ +entryPoints: + http: + address: ":80" + https: + address: ":443" + buildbot: + address: ":9989" + +providers: + docker: + endpoint: "tcp://socket-proxy:2375" + watch: true + exposedByDefault: false + + file: + directory: "/etc/traefik/conf/" + +certificatesResolvers: + http: + acme: + email: admin@email.address + storage: acme.json + httpChallenge: + entryPoint: http diff --git a/buildbot-buildmaster/windows/README.md b/buildbot-buildmaster/windows/README.md new file mode 100644 index 0000000..e298ec7 --- /dev/null +++ b/buildbot-buildmaster/windows/README.md @@ -0,0 +1,2 @@ +Place the credentials (worker name and password) of the buildbot-worker running on windows in the .env file. +This is required by the buildbot-buildmaster (see master.cfg). diff --git a/buildbot-buildmaster/windows/windows.env b/buildbot-buildmaster/windows/windows.env new file mode 100644 index 0000000..e2068e2 --- /dev/null +++ b/buildbot-buildmaster/windows/windows.env @@ -0,0 +1,2 @@ +WINDOWS_NAME=winworker +WINDOWS_PASS=changeme! diff --git a/raspberry-pi-workers/README.md b/raspberry-pi-workers/README.md new file mode 100644 index 0000000..b564121 --- /dev/null +++ b/raspberry-pi-workers/README.md @@ -0,0 +1,19 @@ +# Raspberry Pi workers +Dockerized buildbot-workers for various versions of the Raspberry Pi. + +## Prerequisites +- A physical Rasperry Pi + - Run `uname -m` to determine the architecture +- You have to have docker and docker-compose installed. +- Add your unprivileged user to the docker group + +## Setup +1. Clone / copy this repository to the Raspberry Pi + - (Optional) Create a branch for your configuration including secrets (`git checkout -b secrets`) +2. Modify URL and port of the buildmaster in [buildmaster.env](buildmaster.env) +4. `cd` into the folder fitting your model (run `uname -m` if you are unsure) + - Modify the passphrase and worker name in the `.env` file + - Make sure that the buildmaster is configured with the same values +5. Run `docker-compose up -d` + +Note that docker-compose restarts the service automatically after reboots. diff --git a/raspberry-pi-workers/arm64v8/arm64v8.env b/raspberry-pi-workers/arm64v8/arm64v8.env new file mode 100644 index 0000000..a32f40e --- /dev/null +++ b/raspberry-pi-workers/arm64v8/arm64v8.env @@ -0,0 +1,2 @@ +ARM64V8_PASS=changeme! +ARM64V8_NAME=rpi3b_plus_64 diff --git a/raspberry-pi-workers/arm64v8/arm64v8_dockerfile b/raspberry-pi-workers/arm64v8/arm64v8_dockerfile new file mode 100644 index 0000000..fe82137 --- /dev/null +++ b/raspberry-pi-workers/arm64v8/arm64v8_dockerfile @@ -0,0 +1,10 @@ +FROM arm64v8/alpine + +RUN apk add --update --no-cache libc-dev gcc make g++ openssh git file grep py3-setuptools python3-dev libffi libffi-dev openssl openssl-dev vim +RUN pip3 install --upgrade pip buildbot-worker pyopenssl service_identity +RUN addgroup -S buildbot && adduser -S buildbot -G buildbot +RUN mkdir /worker && chown buildbot:buildbot /worker +RUN apk add --update --no-cache bash +USER buildbot +WORKDIR /worker +ENTRYPOINT buildbot-worker create-worker --use-tls . $BUILDMASTER $ARM64V8_NAME $ARM64V8_PASS && unset BUILDMASTER ARM64V8_NAME ARM64V8_PASS BUILDMASTER_PORT && buildbot-worker start --nodaemon diff --git a/raspberry-pi-workers/arm64v8/docker-compose.yml b/raspberry-pi-workers/arm64v8/docker-compose.yml new file mode 100644 index 0000000..4e92a33 --- /dev/null +++ b/raspberry-pi-workers/arm64v8/docker-compose.yml @@ -0,0 +1,10 @@ +version: '3' +services: + worker: + build: + context: . + dockerfile: arm64v8_dockerfile + restart: unless-stopped + env_file: + - ../buildmaster.env + - ./arm64v8.env diff --git a/raspberry-pi-workers/armv6/armv6.env b/raspberry-pi-workers/armv6/armv6.env new file mode 100644 index 0000000..6dfc6d3 --- /dev/null +++ b/raspberry-pi-workers/armv6/armv6.env @@ -0,0 +1,2 @@ +ARMV6_PASS=changeme! +ARMV6_NAME=rpi1 diff --git a/raspberry-pi-workers/armv6/armv6_dockerfile b/raspberry-pi-workers/armv6/armv6_dockerfile new file mode 100644 index 0000000..63a5f8b --- /dev/null +++ b/raspberry-pi-workers/armv6/armv6_dockerfile @@ -0,0 +1,10 @@ +FROM arm32v6/bash + +RUN apk add --update --no-cache libc-dev gcc openssh git file py3-setuptools python3-dev libffi libffi-dev openssl openssl-dev vim +RUN pip3 install --upgrade pip buildbot-worker pyopenssl service_identity +RUN addgroup -S buildbot && adduser -S buildbot -G buildbot +RUN mkdir /worker && chown buildbot:buildbot /worker +RUN apk add --update --no-cache make g++ +USER buildbot +WORKDIR /worker +ENTRYPOINT buildbot-worker create-worker --use-tls . $BUILDMASTER $ARMV6_NAME $ARMV6_PASS && unset BUILDMASTER ARMV6_NAME ARMV6_PASS BUILDMASTER_PORT && buildbot-worker start --nodaemon diff --git a/raspberry-pi-workers/armv6/docker-compose.yml b/raspberry-pi-workers/armv6/docker-compose.yml new file mode 100644 index 0000000..8287dec --- /dev/null +++ b/raspberry-pi-workers/armv6/docker-compose.yml @@ -0,0 +1,10 @@ +version: '3' +services: + worker: + build: + context: . + dockerfile: armv6_dockerfile + restart: unless-stopped + env_file: + - ./armv6.env + - ../buildmaster.env diff --git a/raspberry-pi-workers/armv7/armv7.env b/raspberry-pi-workers/armv7/armv7.env new file mode 100644 index 0000000..fbcad3f --- /dev/null +++ b/raspberry-pi-workers/armv7/armv7.env @@ -0,0 +1,2 @@ +ARMV7_PASS=changeme! +ARMV7_NAME=rpi2b diff --git a/raspberry-pi-workers/armv7/armv7_dockerfile b/raspberry-pi-workers/armv7/armv7_dockerfile new file mode 100644 index 0000000..edbd39f --- /dev/null +++ b/raspberry-pi-workers/armv7/armv7_dockerfile @@ -0,0 +1,10 @@ +FROM arm32v7/bash + +RUN apk add --update --no-cache libc-dev gcc openssh git file grep py3-setuptools python3-dev libffi libffi-dev openssl openssl-dev vim +RUN pip3 install --upgrade pip buildbot-worker pyopenssl service_identity +RUN addgroup -S buildbot && adduser -S buildbot -G buildbot +RUN mkdir /worker && chown buildbot:buildbot /worker +RUN apk add --update --no-cache make g++ +USER buildbot +WORKDIR /worker +ENTRYPOINT buildbot-worker create-worker --maxdelay=30 --maxretries=999 --keepalive=60 --use-tls . $BUILDMASTER $ARMV7_NAME $ARMV7_PASS && unset BUILDMASTER ARMV7_NAME ARMV7_PASS BUILDMASTER_PORT && buildbot-worker start --nodaemon diff --git a/raspberry-pi-workers/armv7/docker-compose.yml b/raspberry-pi-workers/armv7/docker-compose.yml new file mode 100644 index 0000000..67939b7 --- /dev/null +++ b/raspberry-pi-workers/armv7/docker-compose.yml @@ -0,0 +1,10 @@ +version: '3' +services: + worker: + build: + context: . + dockerfile: armv7_dockerfile + restart: unless-stopped + env_file: + - ../buildmaster.env + - ./armv7.env diff --git a/raspberry-pi-workers/armv8/armv8.env b/raspberry-pi-workers/armv8/armv8.env new file mode 100644 index 0000000..ab24618 --- /dev/null +++ b/raspberry-pi-workers/armv8/armv8.env @@ -0,0 +1,2 @@ +ARMV8_PASS=changeme! +ARMV8_NAME=rpi3b_plus_32 diff --git a/raspberry-pi-workers/armv8/armv8_dockerfile b/raspberry-pi-workers/armv8/armv8_dockerfile new file mode 100644 index 0000000..b0f3aea --- /dev/null +++ b/raspberry-pi-workers/armv8/armv8_dockerfile @@ -0,0 +1,10 @@ +FROM arm32v7/bash + +RUN apk add --update --no-cache libc-dev gcc openssh git file grep py3-setuptools python3-dev libffi libffi-dev openssl openssl-dev vim +RUN pip3 install --upgrade pip buildbot-worker pyopenssl service_identity +RUN addgroup -S buildbot && adduser -S buildbot -G buildbot +RUN mkdir /worker && chown buildbot:buildbot /worker +RUN apk add --update --no-cache make g++ +USER buildbot +WORKDIR /worker +ENTRYPOINT buildbot-worker create-worker --use-tls . $BUILDMASTER $ARMV8_NAME $ARMV8_PASS && unset BUILDMASTER ARMV8_NAME ARMV8_PASS BUILDMASTER_PORT && buildbot-worker --verbose start --nodaemon diff --git a/raspberry-pi-workers/armv8/docker-compose.yml b/raspberry-pi-workers/armv8/docker-compose.yml new file mode 100644 index 0000000..e8b2652 --- /dev/null +++ b/raspberry-pi-workers/armv8/docker-compose.yml @@ -0,0 +1,10 @@ +version: '3' +services: + worker: + build: + context: . + dockerfile: armv8_dockerfile + restart: unless-stopped + env_file: + - ../buildmaster.env + - ./armv8.env diff --git a/raspberry-pi-workers/buildmaster.env b/raspberry-pi-workers/buildmaster.env new file mode 100644 index 0000000..fae223a --- /dev/null +++ b/raspberry-pi-workers/buildmaster.env @@ -0,0 +1,2 @@ +BUILDMASTER=your.buildbot.buildmaster +BUILDMASTER_PORT=9989 diff --git a/windows-worker/README.md b/windows-worker/README.md new file mode 100644 index 0000000..c75a23b --- /dev/null +++ b/windows-worker/README.md @@ -0,0 +1,23 @@ +# Windows-worker +Buildbot-worker on Windows 10 pro + +## Prerequisites +- A working Windows 10 pro setup. + - Set a password for your user +- Git including Git Bash. Download the current version here: https://git-scm.com/download/win +- Microsoft C++ Build Toolsi 2019. Download: https://visualstudio.microsoft.com/de/downloads/ +- Python. Currently verison 3.8.2 is in use. Download: https://www.python.org/downloads/ +## Warning +Some problem prevents a tls connection between the buildbot-worker on windows and the reverse proxy (traefik) in front of the buildmaster. +Because of this, tls is deactivated! +Make sure to establish the connection between worker and buildmaster in a trusted environment only. +Currently, the buildmaster and the windows-vm share a subnet for this purpose. +## Setup +1. Clone / copy this repository to your windows host + * (Optional) Create a branch for your configuration including secrets (git checkout -b secrets) +2. Modify URL and port of the buildmaster in [buildmaster.env](buildmaster.env) +3. Modify the passphrase and worker name in [windows.env](windows.env) + * Make sure that the buildmaster is configured with the same values +4. Change the username and password in the file [setupPythonEnv.py](setupPythonEnv.py) +4. Run Git-Bash as Administrator and navigate to this directory + * Run `./setupPythonEnv.py` diff --git a/windows-worker/buildbot-worker-start.cmd b/windows-worker/buildbot-worker-start.cmd new file mode 100644 index 0000000..15dd340 --- /dev/null +++ b/windows-worker/buildbot-worker-start.cmd @@ -0,0 +1,2 @@ +call sandbox\Scripts\activate.bat +buildbot-worker start --nodaemon diff --git a/windows-worker/buildmaster.env b/windows-worker/buildmaster.env new file mode 100644 index 0000000..75d595a --- /dev/null +++ b/windows-worker/buildmaster.env @@ -0,0 +1,2 @@ +BUILDMASTER=your.buildbot.buildmaster +BUILDMASTER_PORT=9988 diff --git a/windows-worker/setupPythonEnv.sh b/windows-worker/setupPythonEnv.sh new file mode 100755 index 0000000..7f04257 --- /dev/null +++ b/windows-worker/setupPythonEnv.sh @@ -0,0 +1,46 @@ +#!/bin/bash + +source windows.env +source buildmaster.env +function isadmin() +{ + # only for windows. + net session > /dev/null 2>&1 + if [ $? -eq 0 ]; then echo "admin" + else echo "user"; fi +} + +kernel="$(uname -s)" + +if [[ $kernel == MINGW64* ]]; then + DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd -W )" + + if [[ $(isadmin) == admin ]]; then + python -m venv sandbox + source sandbox/Scripts/activate + pip install --upgrade --trusted-host pypi.org pip incremental pywin32 buildbot[bundle] pyopenssl service_identity + # this is why we ned to be admin: + python sandbox/Scripts/pywin32_postinstall.py -install + + buildbot-worker create-worker . "${BUILDMASTER:?}" "${WINDOWS_NAME:?}" "${WINDOWS_PASS:?}" + + # service setup: + # download and unzip nssm + curl https://nssm.cc/ci/nssm-2.24-101-g897c7ad.zip -o nssm.zip + archivepath=$(zipinfo -1 nssm.zip | grep exe | grep 64) + unzip -p nssm.zip $archivepath > nssm.exe + + #install service + ./nssm.exe install $servicename $DIR/buildbot-worker-start.cmd + + #modify username (keep ".\\"!) and password + ./nssm.exe set $servicename ObjectName '.\\invalid-username' 'invalid-password' + ./nssm.exe start $servicename + else + echo "you need to run this terminal as administrator" + exit 1 + fi +else + echo "Error. This script is intended for windows (mingw64)" + exit 1 +fi diff --git a/windows-worker/windows.env b/windows-worker/windows.env new file mode 100644 index 0000000..e2068e2 --- /dev/null +++ b/windows-worker/windows.env @@ -0,0 +1,2 @@ +WINDOWS_NAME=winworker +WINDOWS_PASS=changeme!