Developing Services (Old)

From Dark Peak
Jump to: navigation, search

Services are packaged as docker containers, using docker-compose (previously fig) to define relationships between the containers that make up a service.

Gain Co-Admin

Mention your idea for a service on the email list and get at least one person to volunteer to help admin the service once its running, and ideally help you develop it.

Then update the Services Roadmap to show that what you're working on.

Requirements

Before developing a service, you'll need to install the following:

Getting the code

git clone ssh://darkpeak@git.darkpeak.org/darkpeak-services.git

Starting a service

Inside the service's directory in the darkpeak-services repository, run the following:

docker-compose up

This will fetch the underlying images and build the containers required for the service. It will then start the containers and bind the appropriate port numbers on your host machine (look in docker-compose.yml for port mappings).

Anatomy of a service

myservice/
    docker-compose.yml
    mywebserver/
        Dockerfile
        nginx.conf
        public/
            index.html
            style.css
    mydatabase/
        Dockerfile
    scripts/
        updates_available.sh
        backup.sh
        restore.sh

The above gives you some idea of the parts that might make up a Dark Peak service. The docker-compose.yml file describes how to start the mywebserver and mydatabase containers and link them together. It also describes which ports to expose on the host system. The scripts directory contains shell scripts to manage the service. The scripts talk to a docker daemon in order to handle common tasks such as checking for package updates and performing backups.

Container guidelines

Try to follow the conventions used in other Dockerfiles where possible. Having a more homogeneous system makes administration and maintenance tasks easier.

  • Use of the debian:wheezy base image is preferred
  • Rebuilding a container without using the cache should produce a fully up-to-date service with no out of date packages. This can usually be achieved by adding RUN apt-get update && apt-get upgrade -y to your Dockerfile
  • Do not mount host volumes in the docker-compose.yml config, descibe volumes inside the Dockerfiles using the VOLUME keyword and mount them in other containers using volumes_from in your docker-compose.yml.
  • You should be able to deploy the service knowing only the URI for the docker daemon, no knowledge of the underlying remote machine should be required.

Subdomains, URLs and ports

Traffic is routed to services through HAProxy (which is, itself, a service). By looking in the docker-compose.yml file for HAProxy you can see the port numbers used by other services, each of which get a block of 10 ports by default (eg, 6010, 6020, 6030...). This allows for services which expose multiple ports to assign them consecutively and make them a little easier to remember during development.

When you start the HAProxy container it listens by default on ports 7080, 7443 and 7022 on the host machine for ports 80, 443 and 22 respectively. In order to route the traffic from those ports there is a config file in haproxy/haproxy.cfg which usually matches based on subdomain. When developing a service you should claim a port range and add a rule here to route traffic then test it out by requesting a subdomain from one of the 7*** ports on your host system.

In order to setup a local wildcard domain for development on Linux you can use dnsmasq. The following will setup any domain ending in .dev to point to 127.0.0.1 on Debian (and probably Ubuntu).

$ sudo apt-get install dnsmasq

Edit /etc/dnsmasq.conf and add "address=/dev/127.0.0.1"

$ sudo service dnsmasq restart

Edit /etc/dhcp/dhclient.conf and add "prepend domain-name-servers 127.0.0.1;".

$ sudo service network-manager restart

Now you can test HAProxy routes using https://subdomain.localhost.dev:port style urls (since the HAProxy rules are set to only match the first part of the domain).

Scripts

Each service must provide scripts to help with maintenance. Currently, they are as follows:

  • scripts/backup.sh DIRNAME - Backup a service. Accepts one argument, the directory on the host machine to put the backup files
  • scripts/restore.sh DIRNAME - Restore a backup. Accepts one argument, the directory on the host where the backup files reside
  • scripts/check_updates.sh - Checks running containers for available package updates - prints nothing if no updates available, prints one package per line otherwise - rebuilding the service's containers with --no-cache should make this return an empty result again (see #Container guidelines)

Deployment to a VM

You can obtain a vagrant project describing the base system from:

git clone git://git.darkpeak.org/~caolan/darkpeak-host.git

This can then be provisioned and started using:

vagrant up

After it has started you can setup an SSH tunnel to access the docker daemon within using the ssh_tunnel.sh script. It should then be possible to start services on the VM using the docker-compose command as before:

DOCKER_HOST="tcp://localhost:4243" docker-compose up -d

Some services have a modified yaml file for deployment to a vagrant VM and you may have to specify that on the command line too:

DOCKER_HOST="tcp://localhost:4243" docker-compose -f docker-compose.vagrant.yml up -d

You can also inspect logs and start/stop services on the VM as usual by using the DOCKER_HOST variable. This works with both docker and docker-compose commands.

In order to test out subdomain routing (like in the #Subdomains, URLs and ports section) you may want to add another entry to your dnsmasq.conf, eg "address=/darkpeak.dev/192.168.33.10".

User Authentication

A service can authenticate Dark Peak users by communicating with the central LDAP directory.

The LDAP directory currently supports user authentication via an anonymous bind. Configure your service to talk to the directory at one of these URLs:

ldap://id.darkpeak.org:389/  # Unencrypted
ldaps://id.darkpeak.org:636/ # Over TLS (recommended)

The "search base" to use for Dark Peak user accounts is:

cn=users,cn=accounts,dc=darkpeak,dc=org

With a search filter of:

(uid=%u)

Where "%u" is the username supplied by the user.