GitlabCI build failed, docker.sock: write: broken pipe

docker, gitlab-ci, gitlab-ci-runner, rancher

Solution

It's a known bug with latest versions of gitlab-ci-multi-runner (starting at `v1.5.0`).

You can use version `1.4.2` until the bug is properly fixed. Here is your `docker run` command using the latest working version:

$ docker run -d --name gitlab-runner --restart always \
-v ~/docker/gitlab-runner/config:/etc/gitlab-runner \
-v /var/run/docker.sock:/var/run/docker.sock \
gitlab/gitlab-runner:v1.4.2

An issue already exists on Gitlab CI Multi Runner repository: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1617

Edit: Please use the `v1.4.2` instead of `v1.4.3`. The `v1.4.3` is wrongly tagged on Docker Hub:

$ docker run --rm -it gitlab/gitlab-runner:v1.4.3 -v
Version:      1.5.2
Git revision: 76fdacd
Git branch:   1-5-stable
GO version:   go1.6.3
Built:        Wed, 24 Aug 2016 13:35:10 +0000
OS/Arch:      linux/amd64

Problem

I was trying to implement a new CI workflow using GitlabCI but now I can't understand the issue. I followed this doc and I choose the Docker socket binding method. Everything is running on the same machine, a RancherOS : ``` $ uname -a Linux xxx 4.2.8-ckt4-rancher #1 SMP Tue Apr 5 16:12:00 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 57801c725226 gitlab/gitlab-runner:latest "/usr/bin/dumb-init /" 14 hours ago Up 14 hours gitlab-runner 50e3a9dabe77 gitlab/gitlab-ce "/assets/wrapper" 17 hours ago Up 17 hours 80/tcp, 443/tcp, 0.0.0.0:1022->22/tcp gitlab ed2b6d9a8ea7 jwilder/nginx-proxy "/app/docker-entrypoi" 6 days ago Up 3 days 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp proxy ``` Run command of the runner's container ``` $ docker run -d --name gitlab-runner --restart always \ -v ~/docker/gitlab-runner/config:/etc/gitlab-runner \ -v /var/run/docker.sock:/var/run/docker.sock \ gitlab/gitlab-runner:latest ``` Runner's config ``` $ sudo cat ~/docker/gitlab-runner/config/config.toml concurrent = 1 check_interval = 0 [[runners]] name = "ITrancher, docker based runner 4" url = "https://git.xxx.com" token = "xxx" executor = "docker" [runners.docker] tls_verify = false image = "docker:latest" privileged = false disable_cache = false volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"] [runners.cache] [runners.kubernetes] host = "" cert_file = "" key_file = "" ca_file = "" image = "" namespace = "" privileged = false cpus = "" memory = "" service_cpus = "" service_memory = "" ``` Here is my .gitlab-ci.yml file ``` image: docker:latest before_script: - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.xxx.com stages: - build build: stage: build script: - docker build --pull -f Dockerfile-php -t registry.xxx.com/dev/kayako-php:latest . - docker push registry.xxx.com/dev/kayako-php:latest - docker build --pull -f Dockerfile-nginx -t registry.xxx.com/dev/kayako-nginx:latest . - docker push registry.xxx.com/dev/kayako-nginx:latest ``` And this is what I get into GitlabCI when the build is hooked ``` Running with gitlab-ci-multi-runner 1.6.1 (c52ad4f) Using Docker executor with image docker:latest ... ERROR: Failed to create container volume for /builds/dev write unix @->/var/run/docker.sock: write: broken pipe ERROR: Preparation failed: write unix @->/var/run/docker.sock: write: broken pipe Will be retried in 3s ... Using Docker executor with image docker:latest ... ERROR: Failed to create container volume for /builds/dev write unix @->/var/run/docker.sock: write: broken pipe ERROR: Preparation failed: write unix @->/var/run/docker.sock: write: broken pipe Will be retried in 3s ... Using Docker executor with image docker:latest ... ERROR: Failed to create container volume for /builds/dev write unix @->/var/run/docker.sock: write: broken pipe ERROR: Preparation failed: write unix @->/var/run/docker.sock: write: broken pipe Will be retried in 3s ... ERROR: Build failed (system failure): write unix @->/var/run/docker.sock: write: broken pipe ``` And this is the output when I try to run the build locally with `--debug` flag inside my gitlab-runner container. ``` # gitlab-runner --debug exec docker build Runtime platform arch=amd64 os=linux revision=c52ad4f version=1.6.1 Running with gitlab-ci-multi-runner 1.6.1 (c52ad4f) Shell configuration: environment: [] dockercommand: - sh - -c - "if [ -x /usr/local/bin/bash ]; then\n\texec /usr/local/bin/bash \nelif [ -x /usr/bin/bash ]; then\n\texec /usr/bin/bash \nelif [ -x /bin/bash ]; then\n\texec /bin/bash \nelif [ -x /usr/local/bin/sh ]; then\n\texec /usr/local/bin/sh \nelif [ -x /usr/bin/sh ]; then\n\texec /usr/bin/sh \nelif [ -x /bin/sh ]; then\n\texec /bin/sh \nelse\n\techo shell not found\n\texit 1\nfi\n\n" command: bash arguments: [] passfile: false extension: "" build=1 project=1 Using Docker executor with image docker:latest ... Applying docker.Client transport fix: &{false 0xc820306810 <nil> 0xc820238320 unix:///var/run/docker.sock 0xc820232780 0xc820340ba0 [1 18] [] [] <nil>} host=unix:///var/run/docker.sock Creating build volume... build=1 project=1 Looking for prebuilt image gitlab-runner-prebuilt-x86_64:c52ad4f ... build=1 project=1 Loading prebuilt image... build=1 project=1 Closed all idle connections for docker.Client: &{false 0xc820306900 <nil> 0x39d9ee0 unix:///var/run/docker.sock 0xc820232780 0xc820340ba0 [1 18] [1 22] [1 18] 0xc820306930} ERROR: Preparation failed: write unix @->/var/run/docker.sock: write: broken pipe Will be retried in 3s ... Shell configuration: environment: [] dockercommand: - sh - -c - "if [ -x /usr/local/bin/bash ]; then\n\texec /usr/local/bin/bash \nelif [ -x /usr/bin/bash ]; then\n\texec /usr/bin/bash \nelif [ -x /bin/bash ]; then\n\texec /bin/bash \nelif [ -x /usr/local/bin/sh ]; then\n\texec /usr/local/bin/sh \nelif [ -x /usr/bin/sh ]; then\n\texec /usr/bin/sh \nelif [ -x /bin/sh ]; then\n\texec /bin/sh \nelse\n\techo shell not found\n\texit 1\nfi\n\n" command: bash arguments: [] passfile: false extension: "" build=1 project=1 Using Docker executor with image docker:latest ... Applying docker.Client transport fix: &{false 0xc820306900 <nil> 0x39d9ee0 unix:///var/run/docker.sock 0xc820232780 0xc820340ba0 [1 18] [1 22] [1 18] 0xc820306930} host=unix:///var/run/docker.sock Creating build volume... build=1 project=1 Looking for prebuilt image gitlab-runner-prebuilt-x86_64:c52ad4f ... build=1 project=1 Loading prebuilt image... build=1 project=1 Closed all idle connections for docker.Client: &{false 0xc8203072f0 <nil> 0x39d9ee0 unix:///var/run/docker.sock 0xc820232780 0xc820340ba0 [1 18] [1 22] [1 18] 0xc820306930} ERROR: Preparation failed: write unix @->/var/run/docker.sock: write: broken pipe Will be retried in 3s ... Shell configuration: environment: [] dockercommand: - sh - -c - "if [ -x /usr/local/bin/bash ]; then\n\texec /usr/local/bin/bash \nelif [ -x /usr/bin/bash ]; then\n\texec /usr/bin/bash \nelif [ -x /bin/bash ]; then\n\texec /bin/bash \nelif [ -x /usr/local/bin/sh ]; then\n\texec /usr/local/bin/sh \nelif [ -x /usr/bin/sh ]; then\n\texec /usr/bin/sh \nelif [ -x /bin/sh ]; then\n\texec /bin/sh \nelse\n\techo shell not found\n\texit 1\nfi\n\n" command: bash arguments: [] passfile: false extension: "" build=1 project=1 Using Docker executor with image docker:latest ... Applying docker.Client transport fix: &{false 0xc8203072f0 <nil> 0x39d9ee0 unix:///var/run/docker.sock 0xc820232780 0xc820340ba0 [1 18] [1 22] [1 18] 0xc820306930} host=unix:///var/run/docker.sock Creating build volume... build=1 project=1 Looking for prebuilt image gitlab-runner-prebuilt-x86_64:c52ad4f ... build=1 project=1 Loading prebuilt image... build=1 project=1 Closed all idle connections for docker.Client: &{false 0xc820307a10 <nil> 0x39d9ee0 unix:///var/run/docker.sock 0xc820232780 0xc820340ba0 [1 18] [1 22] [1 18] 0xc820306930} ERROR: Preparation failed: write unix @->/var/run/docker.sock: write: broken pipe Will be retried in 3s ... ERROR: Build failed (system failure): write unix @->/var/run/docker.sock: write: broken pipe FATAL: write unix @->/var/run/docker.sock: write: broken pipe ``` But my docker.sock binding seems to be ok : ``` # echo -e "GET /containers/json HTTP/1.0\r\n" | nc -U /var/run/docker.sock HTTP/1.0 200 OK Content-Type: application/json Server: Docker/1.10.3 (linux) Date: Wed, 19 Oct 2016 10:22:58 GMT [{json is ok}] ``` Anyone has an idea ?

Original source