Home Development of Websites Docker compose and merging projects with mixer

Docker compose and merging projects with mixer

by admin

One of the issues you face when setting up a developer environment using Docker and Docker-compose is how to combine several different projects together. Provided, of course, that all the projects have docker-compose.yml file.
Docker compose and merging projects with mixer
There may be several reasons why it becomes necessary to do this :

  • Developing the low-connected components of a huge system. Where each project can essentially be a separate standalone application
  • Connecting individual components for testing. Bringing out mock -services and tests into separate containers with their own linking and interaction logic
  • External to the project, systems which nevertheless ‘live’ in a docker environment

Content

  • Resources
  • Problem

    Actual combination of N files into one and running them as a complete application was the main problem. What were the secondary problems that prevented just merging the existing files one by one?

    • Container name conflicts with entire configuration tree update
    • Conflicts of ports forwarded to host machine
    • Redefining service properties
    • Allow relative paths
    • Removal of unnecessary or duplicated services

    Solution

    Thus having spent a couple of weekends in mind it came to pass to write a simple python preprocessor that will do all the dirty chore of merging docker-compose files.
    As a consequence, the result is a small (600 lines of code) script which solves all the problems I once had.
    However, as a bonus, in addition to the problems mentioned above, the script helped me solve another interesting problem. Namely, after making some small conclusions, I managed to run several instances of the application in parallel with one docker-compose file.

    Examples of usage

    A few steps that allow you to quickly use this solution

    Installation

    You need to download a binary file which is packed python module
    Installing

    sudo wget https://github.com/paunin/docker-compose-mixer/blob/master/dist/dc-mixer?raw=true -O /usr/local/bin/dc-mixersudo chmod +x /usr/local/bin/dc-mixer

    Configuration

    Configuration content docker-compose-mixer.yml file is essentially a small config file that describes exactly how two or more projects will start together.
    Let’s say we have two projects project A and project B c by two docker-compose.yml files.
    Projects, being different versions of the system, are an application with a set of services : java-application, redis, rabbitmq, mail , project project A (new version) also contains mysql service.
    project A

    $ cat ../proj_a/docker-compose.ymlsources:build: images/sourcesvolumes:- .:/var/application.hostapplication:build: images/javadockerfile: application.ymldns:- 8.8.8.8- 9.9.9.9hostname: application.hostworking_dir: /var/application.hostcgroup_parent: m-executor-abcdlinks:- redis:redis- rabbitmq- mail- mysql:dbvolumes_from:- sourcescommand: "/start.sh"env_file:- ./env_files/application.env- ./env_files/rabbit.envenvironment:- DB_DRIVER: mysql- DB_PORT: 3306ports:- "80:80" #http- "1098:1098" #jmxexternal_links:- redis_1- project_db_1:mysqlextra_hosts:- "somehost:162.242.195.82"labels:com.example.description: "Accounting webapp"com.example.department: "Finance"redis:labels:- "com.example.description=Accounting webapp"- "com.example.department=Finance"extends:file: ../redis.ymlservice: redisbaseexpose:ports:- "6379:6379"- "127.0.0.1:6370:6370"log_driver: "syslog"log_opt:syslog-address: "tcp://192.168.0.42:123"net: "bridge"rabbitmq:build: images/rabbitmqports:- "15672:15672"volumes_from:- sourcescommand: /start.shenv_file:- ./env_files/rabbit.envmail:build: images/mailhostname: maildomainname: application.hostexpose:- 25- 143ports:- "25:25"- "143:143"volumes:- ./images/mail/spamassassin:/tmp/spamassassin/- ./images/mail/postfix:/tmp/postfix/- ./images/mail/mail:/tmp/mail/mysql:build: images/mysqlports:- 3602

    project B

    $ cat ./proj_b/docker-compose.ymlsources:build: images/sourcesvolumes:- .:/var/application.hostapplication:build: images/javadockerfile: application.ymldns:- 8.8.8.8- 9.9.9.9hostname: application.hostworking_dir: /var/application.hostcgroup_parent: m-executor-abcdlinks:- redis:redis- rabbitmq- mailvolumes_from:- sourcescommand: "/start.sh"env_file:- ./env_files/application.env- ./env_files/rabbit.envenvironment:- VARIABLE: valueports:- "80:80" #http- "1098:1098" #jmxexternal_links:- redis_1- project_db_1:mysqlextra_hosts:- "somehost:162.242.195.82"labels:com.example.description: "Accounting webapp"com.example.department: "Finance"redis:labels:- "com.example.description=Accounting webapp"- "com.example.department=Finance"extends:file: ../redis.ymlservice: redisbaseexpose:ports:- "6379:6379"- "127.0.0.1:6370:6373"log_driver: "syslog"log_opt:syslog-address: "tcp://192.168.0.42:123"net: "bridge"rabbitmq:build: images/rabbitmqports:- "15672:15672"volumes_from:- sourcescommand: /start.shenv_file:- ./env_files/rabbit.envmail:build: images/mailhostname: maildomainname: application.hostexpose:- 25- 143ports:- "25:25"- "143:143"volumes:- ./images/mail/spamassassin:/tmp/spamassassin/- ./images/mail/postfix:/tmp/postfix/- ./images/mail/mail:/tmp/mail/

    The basic task is to combine two environments into one using only one rabbitmq service. In addition, you need to create a new service pgsql and connect it to the first project (replacing mysql)
    The configuration for merging files must be in the file /docker-compose-mixer.yml (option -i, --input-file can define a different name for the file)
    docker-compose-mixer.yml

    $ cat ./docker-compose-mixer.ymlincludes:proja: ../proj_a/docker-compose.ymlprojb: ./proj_b/docker-compose.ymlignores:- projbrabbitmqoverrides:projbapplication:links:- projaredis:redis- projarabbitmq:rabbitmq- projamail:mailprojaapplication:links:- projaredis:redis- projarabbitmq:rabbitmq- projamail:mail- pgsql:dbenvironment:- DB_DRIVER: pgsql- DB_PORT: 5432master_services:pgsql:image: pgsql:latestexpose:- 5432ports:5432:5432

    Launch

    Command ` dc-mixer -v ` starts the preprocessor in verbal mode and saves the result to a file. The startup options help control the behavior.
    Launch options

    $ dc-mixer --helpCompile docker-compose from several docker-compose.yml filesUsage:dc-mixer [options]Options:-h, --help Print help information-i, --input-file Input file (default `docker-compose-mixer.yml` in current directory)-o, --output-file Output file (default `docker-compose.yml` in current directory)-h, --help Print help information-v, --verbose Enable verbose modeFor more information read documentation: https://github.com/paunin/docker-compose-mixer

    Result

    By default, the result of file merge will be saved in the file /docker-compose.yml (option -o, --output-file can define a different name for the file)
    Result of the script

    $ dc-mixer -v -o docker-compose.ymlDEBUG:root:Start compiling compose file...DEBUG:root:Input file: /Users/paunin/Sites/dc-mixer.local/examples/example2/proj/docker-compose-mixer.yml; output file: docker-compose.ymlDEBUG:root:Mixer config is below:{'overrides': {'projbapplication': {'links': ['projaredis:redis', 'projarabbitmq:rabbitmq', 'projamail:mail']}, 'projaapplication': {'environment': {'DB_DRIVER': 'pgsql', 'DB_PORT': 5432}, 'links': ['projaredis:redis', 'projarabbitmq:rabbitmq', 'projamail:mail', 'pgsql:db']}}, 'master_services': {'pgsql': {'image': 'pgsql:latest', 'expose': [5432], 'ports': '5432:5432'}}, 'ignores': ['projbrabbitmq'], 'includes': {'projb': './proj_b/docker-compose.yml', 'proja': '../proj_a/docker-compose.yml'}}DEBUG:root:Creating scope for file: /Users/paunin/Sites/dc-mixer.local/examples/example2/proj/proj_b/docker-compose.yml and prefix: projbDEBUG:root:Creating scope for file: /Users/paunin/Sites/dc-mixer.local/examples/example2/proj_a/docker-compose.yml and prefix: projaDEBUG:root:Resolving services namesDEBUG:root:Resolving services paths withDEBUG:root:Resolving services portsDEBUG:root:Redefined ports:{'projaapplication': {80: 81, 1098: 1099}, 'projaredis': {6370: 6371, 6379: 6380}, 'projamail': {25: 26, 143: 144}}DEBUG:root:Result scope is:{'projasources': {'build': '../proj_a/images/sources', 'volumes': ['./../proj_a:/var/application.host']}, 'projaredis': {'log_opt': {'syslog-address': 'tcp://192.168.0.42:123'}, 'log_driver': 'syslog', 'expose': None, 'labels': ['com.example.description=Accounting webapp', 'com.example.department=Finance'], 'extends': {'service': 'redisbase', 'file': '../redis.yml'}, 'net': 'bridge', 'ports': ['6380:6379', '127.0.0.1:6371:6370']}, 'projbmail': {'domainname': 'application.host', 'expose': [25, 143], 'hostname': 'mail', 'build': 'proj_b/images/mail', 'volumes': ['./proj_b/images/mail/spamassassin:/tmp/spamassassin/', './proj_b/images/mail/postfix:/tmp/postfix/', './proj_b/images/mail/mail:/tmp/mail/'], 'ports': ['25:25', '143:143']}, 'projamail': {'domainname': 'application.host', 'expose': [25, 143], 'hostname': 'mail', 'build': '../proj_a/images/mail', 'volumes': ['./../proj_a/images/mail/spamassassin:/tmp/spamassassin/', './../proj_a/images/mail/postfix:/tmp/postfix/', './../proj_a/images/mail/mail:/tmp/mail/'], 'ports': ['26:25', '144:143']}, 'pgsql': {'image': 'pgsql:latest', 'expose': [5432], 'ports': '5432:5432'}, 'projbapplication': {'hostname': 'application.host', 'links': ['projaredis:redis', 'projarabbitmq:rabbitmq', 'projamail:mail'], 'cgroup_parent': 'm-executor-abcd', 'labels': {'com.example.description': 'Accounting webapp', 'com.example.department': 'Finance'}, 'extra_hosts': ['somehost:162.242.195.82'], 'environment': [{'VARIABLE': 'value'}], 'working_dir': '/var/application.host', 'command': '/start.sh', 'build': 'proj_b/images/java', 'dns': ['8.8.8.8', '9.9.9.9'], 'volumes_from': ['projbsources'], 'env_file': ['proj_b/env_files/application.env', 'proj_b/env_files/rabbit.env'], 'dockerfile': 'application.yml', 'ports': ['80:80', '1098:1098'], 'external_links': ['redis_1', 'project_db_1:mysql']}, 'projaapplication': {'hostname': 'application.host', 'links': ['projaredis:redis', 'projarabbitmq:rabbitmq', 'projamail:mail', 'pgsql:db'], 'cgroup_parent': 'm-executor-abcd', 'labels': {'com.example.description': 'Accounting webapp', 'com.example.department': 'Finance'}, 'extra_hosts': ['somehost:162.242.195.82'], 'environment': {'DB_DRIVER': 'pgsql', 'DB_PORT': 5432}, 'working_dir': '/var/application.host', 'command': '/start.sh', 'build': '../proj_a/images/java', 'dns': ['8.8.8.8', '9.9.9.9'], 'volumes_from': ['projasources'], 'env_file': ['../proj_a/env_files/application.env', '../proj_a/env_files/rabbit.env'], 'dockerfile': 'application.yml', 'ports': ['81:80', '1099:1098'], 'external_links': ['redis_1', 'project_db_1:mysql']}, 'projamysql': {'build': '../proj_a/images/mysql', 'ports': []}, 'projarabbitmq': {'volumes_from': ['projasources'], 'env_file': ['../proj_a/env_files/rabbit.env'], 'command': '/start.sh', 'build': '../proj_a/images/rabbitmq', 'ports': ['15672:15672']}, 'projbredis': {'log_opt': {'syslog-address': 'tcp://192.168.0.42:123'}, 'log_driver': 'syslog', 'expose': None, 'labels': ['com.example.description=Accounting webapp', 'com.example.department=Finance'], 'extends': {'service': 'redisbase', 'file': 'redis.yml'}, 'net': 'bridge', 'ports': ['6379:6379', '127.0.0.1:6370:6373']}, 'projbsources': {'build': 'proj_b/images/sources', 'volumes': ['./proj_b:/var/application.host']}}DEBUG:root:Save result scope in the file "docker-compose.yml"

    Resulting docker-compose.yml

    pgsql:expose:- 5432image: "pgsql:latest"ports: "5432:5432"projaapplication:build: ../proj_a/images/javacgroup_parent: m-executor-abcdcommand: /start.shdns:- "8.8.8.8"- "9.9.9.9"dockerfile: application.ymlenv_file:- ../proj_a/env_files/application.env- ../proj_a/env_files/rabbit.envenvironment:DB_DRIVER: pgsqlDB_PORT: 5432external_links:- redis_1- "project_db_1:mysql"extra_hosts:- "somehost:162.242.195.82"hostname: application.hostlabels:com.example.department: Financecom.example.description: "Accounting webapp"links:- "projaredis:redis"- "projarabbitmq:rabbitmq"- "projamail:mail"- "pgsql:db"ports:- "81:80"- "1099:1098"volumes_from:- projasourcesworking_dir: /var/application.hostprojamail:build: ../proj_a/images/maildomainname: application.hostexpose:- 25- 143hostname: mailports:- "26:25"- "144:143"volumes:- "./../proj_a/images/mail/spamassassin:/tmp/spamassassin/"- "./../proj_a/images/mail/postfix:/tmp/postfix/"- "./../proj_a/images/mail/mail:/tmp/mail/"projamysql:build: ../proj_a/images/mysqlports: []projarabbitmq:build: ../proj_a/images/rabbitmqcommand: /start.shenv_file:- ../proj_a/env_files/rabbit.envports:- "15672:15672"volumes_from:- projasourcesprojaredis:expose: ~extends:file: ../redis.ymlservice: redisbaselabels:- "com.example.description=Accounting webapp"- com.example.department=Financelog_driver: sysloglog_opt:syslog-address: "tcp://192.168.0.42:123"net: bridgeports:- "6380:6379"- "127.0.0.1:6371:6370"projasources:build: ../proj_a/images/sourcesvolumes:- "./../proj_a:/var/application.host"projbapplication:build: proj_b/images/javacgroup_parent: m-executor-abcdcommand: /start.shdns:- "8.8.8.8"- "9.9.9.9"dockerfile: application.ymlenv_file:- proj_b/env_files/application.env- proj_b/env_files/rabbit.envenvironment:-VARIABLE: valueexternal_links:- redis_1- "project_db_1:mysql"extra_hosts:- "somehost:162.242.195.82"hostname: application.hostlabels:com.example.department: Financecom.example.description: "Accounting webapp"links:- "projaredis:redis"- "projarabbitmq:rabbitmq"- "projamail:mail"ports:- "80:80"- "1098:1098"volumes_from:- projbsourcesworking_dir: /var/application.hostprojbmail:build: proj_b/images/maildomainname: application.hostexpose:- 25- 143hostname: mailports:- "25:25"- "143:143"volumes:- "./proj_b/images/mail/spamassassin:/tmp/spamassassin/"- "./proj_b/images/mail/postfix:/tmp/postfix/"- "./proj_b/images/mail/mail:/tmp/mail/"projbredis:expose: ~extends:file: redis.ymlservice: redisbaselabels:- "com.example.description=Accounting webapp"- com.example.department=Financelog_driver: sysloglog_opt:syslog-address: "tcp://192.168.0.42:123"net: bridgeports:- "6379:6379"- "127.0.0.1:6370:6373"projbsources:build: proj_b/images/sourcesvolumes:- "./proj_b:/var/application.host"

    I would like to draw attention to the debugger output section ` DEBUG:root:Redefined ports ` which gives information about the ports that were automatically changed.
    The source code of the example can be found here

    Resources

    You may also like