The platforms we plan to run on our cloud are generally web-based, and each listening on their own unique TCP port. When a container in a swarm exposes a port, then connecting to any swarm member on that port will result in your request being forwarded to the appropriate host running the container. (Docker calls this the swarm "routing mesh")
When running docker stack deploy I receive the following error. “unsupported Compose file version: 3.2” What version of docker are you running?
Mmm, I think that might be a typo.
Try starting your traefik.yml file with:
version: "3"
instead of
version: "3.2"
3.1 throws unsupported port features. I got it working by doing it manually. On to the next step. Another issue I had is you had to create the swarm before you can add a service.
Right, there was a bit of an impossible sequence of tasks there! I’ve updated the sequence to make the sequence more linear 
Looks like letsencrypt has wildcard certs now. Could simplify things here.
When Traefik supports them too, yes 
Any Idea why this would happen?
I’m running Ubuntu 16.04 server
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
31fr0uie802t traefik_traefik.l5xg1og2o5xvvwvkyjfobgowr traefik:latest ceph-osd2 Ready Rejected less than a second ago "invalid mount config for type…"
zb5zy6l89pb3 traefik_traefik.6aoffkpohmlywxsd4i0pxnq0l traefik:latest ceph-osd3 Ready Rejected 2 seconds ago "invalid mount config for type…"
u9ysaf6k06a2 traefik_traefik.r6j74o9ccwlxmsc7kcgbyb10g traefik:latest ceph-osd1 Running Starting less than a second ago
yws52v7o9juo traefik_traefik.l5xg1og2o5xvvwvkyjfobgowr traefik:latest ceph-osd2 Shutdown Rejected 56 seconds ago "invalid mount config for type…"
zrn9t81l51gk traefik_traefik.6aoffkpohmlywxsd4i0pxnq0l traefik:latest ceph-osd3 Shutdown Rejected about a minute ago "invalid mount config for type…"
z3js6q8t8c34 traefik_traefik.l5xg1og2o5xvvwvkyjfobgowr traefik:latest ceph-osd2 Shutdown Rejected about a minute ago "invalid mount config for type…"
ymzh3evjocyq \_ traefik_traefik.l5xg1og2o5xvvwvkyjfobgowr traefik:latest ceph-osd2 Shutdown Rejected 3 minutes ago "invalid mount config for type…"
zc67mny0phk1 traefik_traefik.r6j74o9ccwlxmsc7kcgbyb10g traefik:latest ceph-osd1 Shutdown Failed 4 minutes ago "starting container failed: co…"
xzfx2oc4xa6t \_ traefik_traefik.r6j74o9ccwlxmsc7kcgbyb10g traefik:latest ceph-osd1 Shutdown Failed 5 minutes ago "starting container failed: co…"
Check that traefik.toml and acme.json exist? If acme.json doesn’t exist, create it, by running:
touch acme.json
chmod 600 acme.json
D
yes they are created and I verified again just to make sure.
The only time I’ve seen that error has been when one of the 3 volume mounts (example below) are invalid:
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /var/data/config/traefik/traefik.toml:/traefik.toml:ro
- /var/data/config/traefik/acme.json:/acme.json
verified paths, they are vaild and even changed the permissions to wide open just to make sure it wasn’t a permissions thing. What else are you thing that could be wrong that I could check? Played around with the versions of the docker-compose file that didn’t help either.
@ggilley is your ubuntu 16.04 traefik working ? I read earlier somewhere that you were running your setup on ubuntu.
On CentOS, you need to permit access to /var/run/docker.sock. Maybe there’s an equivalent permissions issue on Ubuntu?
ok let me check that out and ill report back, ill be sure to buy you a few beers after this is all over
what i will do is spin up a centos7 vm and run the traefik portion to compare and see if selinux is causing this.
Centos7 error
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
ny3ole833rrdammjrtatep3d5 traefik_traefik.6tv5ulxzqmt2u5tqnv2o11kli traefik:latest@sha256:96fe09a867c29dfd5ecf240c955ae104dc7d3422bcc3d42b74a0d0fcc5a2377f CentOS Ready Ready 1 second ago
vfjurvtj5h0ydaqq5l6iao8tx \_ traefik_traefik.6tv5ulxzqmt2u5tqnv2o11kli traefik:latest@sha256:96fe09a867c29dfd5ecf240c955ae104dc7d3422bcc3d42b74a0d0fcc5a2377f CentOS Shutdown Failed 16 seconds ago "starting container failed: container
fe61bb52954f07e606d6e67e4d69fb0bb5defde193676fde9ad54c5b3da0bd00: endpoint join on GW Network failed: driver failed programming external connectivity on endpoint gateway_fe61bb52954f (12914e1af13f0a579efb85263524913f66e5ed1cb8a22b7da352cfc402054151): Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use"
@Playboy713226 yes, my traefik is running on ubuntu 16.04. Looking at my config, I can’t find anything special I needed to do. I remember searching around for a way to test the configuration to make sure it was working. I do run docker as root.
@ggilley Hmmmmmm, I didn’t do anything special to mine other than follow a different guide to get my ceph up and running. I run docker as root as well. Let me ask this, can someone show me a working output? That way I know what right looks like and I can try to troubleshoot from there. I verified the paths were present, I even created the paths on the other 2 nodes just to rule out that as an issue.
ok so here is what I did so far…which i should have checked in the first place…i stopped apache from running, that stopped my port 80 bind issue. Here is my output from that also see issue below. Any ideas I am still currently troubleshooting the issue.
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
mw201pgb7ire traefik_traefik.r6j74o9ccwlxmsc7kcgbyb10g traefik:latest ceph-osd1 Ready Assigned less than a second ago
umeiispusonw traefik_traefik.l5xg1og2o5xvvwvkyjfobgowr traefik:latest ceph-osd2 Running Running 4 seconds ago *:443->443/tcp,*:80->80/tcp
zyusp5ve8gq8 traefik_traefik.r6j74o9ccwlxmsc7kcgbyb10g traefik:latest ceph-osd1 Shutdown Failed 1 second ago "task: non-zero exit (2)"
tkphnhul41xb traefik_traefik.6aoffkpohmlywxsd4i0pxnq0l traefik:latest ceph-osd3 Running Running 13 seconds ago *:443->443/tcp,*:80->80/tcp
zzjnmib81mn1 traefik_traefik.r6j74o9ccwlxmsc7kcgbyb10g traefik:latest ceph-osd1 Shutdown Failed 3 minutes ago "task: non-zero exit (2)"
zj6nqt11k9l5 traefik_traefik.l5xg1og2o5xvvwvkyjfobgowr traefik:latest ceph-osd2 Shutdown Failed 12 minutes ago "task: non-zero exit (2)"
zfwed2xjq0pk traefik_traefik.6aoffkpohmlywxsd4i0pxnq0l traefik:latest ceph-osd3 Shutdown Failed 14 minutes ago "task: non-zero exit (2)"
zyxsuw83pw67 traefik_traefik.r6j74o9ccwlxmsc7kcgbyb10g traefik:latest ceph-osd1 Shutdown Failed 26 minutes ago "starting container failed: co…"
zz6yi5pe0bhe traefik_traefik.6aoffkpohmlywxsd4i0pxnq0l traefik:latest ceph-osd3 Shutdown Failed 28 minutes ago "starting container failed: co…"
zzriq00ghauc \_ traefik_traefik.6aoffkpohmlywxsd4i0pxnq0l traefik:latest ceph-osd3 Shutdown Failed 28 minutes ago "starting container failed: co…"
zy8yu4pldjja traefik_traefik.r6j74o9ccwlxmsc7kcgbyb10g traefik:latest ceph-osd1 Shutdown Failed 26 minutes ago "starting container failed: co…"
zzyc5dw6b232 traefik_traefik.l5xg1og2o5xvvwvkyjfobgowr traefik:latest ceph-osd2 Shutdown Failed 27 minutes ago "starting container failed: co…"
zzu8ooogkay0 \_ traefik_traefik.l5xg1og2o5xvvwvkyjfobgowr traefik:latest ceph-osd2 Shutdown Failed 27 minutes ago "starting container failed: co…"
zzzh8iemd94e \_ traefik_traefik.l5xg1og2o5xvvwvkyjfobgowr traefik:latest ceph-osd2 Shutdown Failed 27 minutes ago "starting container failed: co…"
zylgh51uhhe5 traefik_traefik.6aoffkpohmlywxsd4i0pxnq0l traefik:latest ceph-osd3 Shutdown Failed 28 minutes ago "starting container failed: co…"
What’s the output of
docker service logs traefik_traefik
?
April 12 ok so here is what Idid so far…which i should have checked in the first
place…i stopped apache from running, that stopped my
port 80 bind issue. Here is my output from that also see
issue below. Any ideas I am still currently
troubleshooting the issue.
> ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
> mw201pgb7ire traefik_traefik.r6j74o9ccwlxmsc7kcgbyb10g traefik:latest ceph-osd1 Ready Assigned less than a second ago
> umeiispusonw traefik_traefik.l5xg1og2o5xvvwvkyjfobgowr traefik:latest ceph-osd2 Running Running 4 seconds ago *:443->443/tcp,*:80->80/tcp
> zyusp5ve8gq8 traefik_traefik.r6j74o9ccwlxmsc7kcgbyb10g traefik:latest ceph-osd1 Shutdown Failed 1 second ago "task: non-zero exit (2)"
> tkphnhul41xb traefik_traefik.6aoffkpohmlywxsd4i0pxnq0l traefik:latest ceph-osd3 Running Running 13 seconds ago *:443->443/tcp,*:80->80/tcp
> zzjnmib81mn1 traefik_traefik.r6j74o9ccwlxmsc7kcgbyb10g traefik:latest ceph-osd1 Shutdown Failed 3 minutes ago "task: non-zero exit (2)"
> zj6nqt11k9l5 traefik_traefik.l5xg1og2o5xvvwvkyjfobgowr traefik:latest ceph-osd2 Shutdown Failed 12 minutes ago "task: non-zero exit (2)"
> zfwed2xjq0pk traefik_traefik.6aoffkpohmlywxsd4i0pxnq0l traefik:latest ceph-osd3 Shutdown Failed 14 minutes ago "task: non-zero exit (2)"
> zyxsuw83pw67 traefik_traefik.r6j74o9ccwlxmsc7kcgbyb10g traefik:latest ceph-osd1 Shutdown Failed 26 minutes ago "starting container failed: co…"
> zz6yi5pe0bhe traefik_traefik.6aoffkpohmlywxsd4i0pxnq0l traefik:latest ceph-osd3 Shutdown Failed 28 minutes ago "starting container failed: co…"
> zzriq00ghauc \_ traefik_traefik.6aoffkpohmlywxsd4i0pxnq0l traefik:latest ceph-osd3 Shutdown Failed 28 minutes ago "starting container failed: co…"
> zy8yu4pldjja traefik_traefik.r6j74o9ccwlxmsc7kcgbyb10g traefik:latest ceph-osd1 Shutdown Failed 26 minutes ago "starting container failed: co…"
> zzyc5dw6b232 traefik_traefik.l5xg1og2o5xvvwvkyjfobgowr traefik:latest ceph-osd2 Shutdown Failed 27 minutes ago "starting container failed: co…"
> zzu8ooogkay0 \_ traefik_traefik.l5xg1og2o5xvvwvkyjfobgowr traefik:latest ceph-osd2 Shutdown Failed 27 minutes ago "starting container failed: co…"
> zzzh8iemd94e \_ traefik_traefik.l5xg1og2o5xvvwvkyjfobgowr traefik:latest ceph-osd2 Shutdown Failed 27 minutes ago "starting container failed: co…"
> zylgh51uhhe5 traefik_traefik.6aoffkpohmlywxsd4i0pxnq0l traefik:latest ceph-osd3 Shutdown Failed 28 minutes ago "starting container failed: co…"
Kinda hard to read like this i hope this helps.
traefik_traefik.0.k156miipb60z@ceph-osd1 | time=“2018-04-13T00:59:43Z” level=info msg=“Using TOML configuration file //traefik.toml”
traefik_traefik.0.k156miipb60z@ceph-osd1 | time=“2018-04-13T00:59:43Z” level=warning msg=“web provider configuration is deprecated, you should use these options : api, rest provider, ping and metrics”
traefik_traefik.0.k156miipb60z@ceph-osd1 | time=“2018-04-13T00:59:43Z” level=warning msg=“ACME.OnDemand is deprecated”
traefik_traefik.0.k156miipb60z@ceph-osd1 | time=“2018-04-13T00:59:43Z” level=info msg=“Traefik version v1.5.4 built on 2018-03-15_01:35:21PM”
traefik_traefik.0.k156miipb60z@ceph-osd1 | time=“2018-04-13T00:59:43Z” level=info msg="
traefik_traefik.0.k156miipb60z@ceph-osd1 | Stats collection is disabled.
traefik_traefik.0.k156miipb60z@ceph-osd1 | Help us improve Traefik by turning this feature on
traefik_traefik.0.k156miipb60z@ceph-osd1 | More details on: https://docs.traefik.io/basics/#collected-data
traefik_traefik.0.k156miipb60z@ceph-osd1 | "
traefik_traefik.0.k156miipb60z@ceph-osd1 | time=“2018-04-13T00:59:43Z” level=debug msg=“Global configuration loaded {“LifeCycle”:{“RequestAcceptGraceTimeout”:0,“GraceTimeOut”:10000000000},“GraceTimeOut”:0,“Debug”:false,“CheckNewVersion”:true,“SendAnonymousUsage”:false,“AccessLogsFile”:”“,“AccessLog”:null,“TraefikLogsFile”:”“,“TraefikLog”:null,“LogLevel”:“DEBUG”,“EntryPoints”:{“http”:{“Network”:”“,“Address”:”:80",“TLS”:null,“Redirect”:{“entryPoint”:“https”},“Auth”:null,“WhitelistSourceRange”:null,“Compress”:false,“ProxyProtocol”:null,“ForwardedHeaders”:{“Insecure”:true,“TrustedIPs”:null}},“https”:{“Network”:“”,“Address”:“:443”,“TLS”:{“MinVersion”:“”,“CipherSuites”:null,“Certificates”:null,“ClientCAFiles”:null,“ClientCA”:{“Files”:null,“Optional”:false}},“Redirect”:null,“Auth”:null,“WhitelistSourceRange”:null,“Compress”:false,“ProxyProtocol”:null,“ForwardedHeaders”:{“Insecure”:true,“TrustedIPs”:null}},“traefik”:{“Network”:“”,“Address”:“:8080”,“TLS”:null,“Redirect”:null,“Auth”:null,“WhitelistSourceRange”:null,“Compress”:false,“ProxyProtocol”:null,“ForwardedHeaders”:{“Insecure”:true,“TrustedIPs”:null}}},“Cluster”:null,“Constraints”:[],“ACME”:{“Email”:“\u003cyour LetsEncrypt email address\u003e”,“Domains”:[{“Main”:“ralphaeljohnson.com”,“SANs”:null}],“Storage”:“acme.json”,“StorageFile”:“”,“OnDemand”:true,“OnHostRule”:true,“CAServer”:“”,“EntryPoint”:“https”,“DNSChallenge”:null,“HTTPChallenge”:null,“DNSProvider”:“”,“DelayDontCheckDNS”:0,“ACMELogging”:true,“TLSConfig”:null},“DefaultEntryPoints”:[“http”,“https”],“ProvidersThrottleDuration”:2000000000,“MaxIdleConnsPerHost”:200,“IdleTimeout”:0,“InsecureSkipVerify”:false,“RootCAs”:null,“Retry”:null,“HealthCheck”:{“Interval”:30000000000},“RespondingTimeouts”:null,“ForwardingTimeouts”:null,“Web”:{“Address”:“:8080”,“CertFile”:“”,“KeyFile”:“”,“ReadOnly”:false,“Statistics”:null,“Metrics”:null,“Path”:“/”,“Auth”:null,“Debug”:false},“Docker”:{“Watch”:true,“Filename”:“”,“Constraints”:null,“Trace”:false,“DebugLogGeneratedTemplate”:false,“Endpoint”:“unix:///var/run/docker.sock”,“Domain”:“ralphaeljohnson.com”,“TLS”:null,“ExposedByDefault”:true,“UseBindPortIP”:false,“SwarmMode”:true},“File”:null,“Marathon”:null,“Consul”:null,“ConsulCatalog”:null,“Etcd”:null,“Zookeeper”:null,“Boltdb”:null,“Kubernetes”:null,“Mesos”:null,“Eureka”:null,“ECS”:null,“Rancher”:null,“DynamoDB”:null,“ServiceFabric”:null,“Rest”:null,“API”:{“EntryPoint”:“traefik”,“Dashboard”:true,“Debug”:false,“CurrentConfigurations”:null,“Statistics”:null},“Metrics”:null,“Ping”:{“EntryPoint”:“traefik”}}"
traefik_traefik.0.k156miipb60z@ceph-osd1 | time=“2018-04-13T00:59:43Z” level=info msg=“Preparing server http &{Network: Address::80 TLS: Redirect:0xc42018de30 Auth: WhitelistSourceRange:[] Compress:false ProxyProtocol: ForwardedHeaders:0xc4205dd120} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s”
traefik_traefik.0.k156miipb60z@ceph-osd1 | time=“2018-04-13T00:59:43Z” level=info msg=“Preparing server https &{Network: Address::443 TLS:0xc4206cf600 Redirect: Auth: WhitelistSourceRange:[] Compress:false ProxyProtocol: ForwardedHeaders:0xc4205dd140} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s”
traefik_traefik.0.k156miipb60z@ceph-osd1 | time=“2018-04-13T00:59:43Z” level=info msg=“Starting server on :80”
traefik_traefik.0.pc6vf4c0yvnh@ceph-osd3 | time=“2018-04-13T00:59:47Z” level=info msg=“Using TOML configuration file //traefik.toml”
traefik_traefik.0.pc6vf4c0yvnh@ceph-osd3 | time=“2018-04-13T00:59:47Z” level=warning msg=“web provider configuration is deprecated, you should use these options : api, rest provider, ping and metrics”
traefik_traefik.0.pc6vf4c0yvnh@ceph-osd3 | time=“2018-04-13T00:59:47Z” level=warning msg=“ACME.OnDemand is deprecated”
traefik_traefik.0.pc6vf4c0yvnh@ceph-osd3 | time=“2018-04-13T00:59:47Z” level=info msg=“Traefik version v1.5.4 built on 2018-03-15_01:35:21PM”
traefik_traefik.0.pc6vf4c0yvnh@ceph-osd3 | time=“2018-04-13T00:59:47Z” level=info msg="
traefik_traefik.0.pc6vf4c0yvnh@ceph-osd3 | Stats collection is disabled.
traefik_traefik.0.pc6vf4c0yvnh@ceph-osd3 | Help us improve Traefik by turning this feature on
traefik_traefik.0.pc6vf4c0yvnh@ceph-osd3 | More details on: https://docs.traefik.io/basics/#collected-data
traefik_traefik.0.pc6vf4c0yvnh@ceph-osd3 | "
traefik_traefik.0.pc6vf4c0yvnh@ceph-osd3 | time=“2018-04-13T00:59:47Z” level=debug msg=“Global configuration loaded {“LifeCycle”:{“RequestAcceptGraceTimeout”:0,“GraceTimeOut”:10000000000},“GraceTimeOut”:0,“Debug”:false,“CheckNewVersion”:true,“SendAnonymousUsage”:false,“AccessLogsFile”:”“,“AccessLog”:null,“TraefikLogsFile”:”“,“TraefikLog”:null,“LogLevel”:“DEBUG”,“EntryPoints”:{“http”:{“Network”:”“,“Address”:”:80",“TLS”:null,“Redirect”:{“entryPoint”:“https”},“Auth”:null,“WhitelistSourceRange”:null,“Compress”:false,“ProxyProtocol”:null,“ForwardedHeaders”:{“Insecure”:true,“TrustedIPs”:null}},“https”:{“Network”:“”,“Address”:“:443”,“TLS”:{“MinVersion”:“”,“CipherSuites”:null,“Certificates”:null,“ClientCAFiles”:null,“ClientCA”:{“Files”:null,“Optional”:false}},“Redirect”:null,“Auth”:null,“WhitelistSourceRange”:null,“Compress”:false,“ProxyProtocol”:null,“ForwardedHeaders”:{“Insecure”:true,“TrustedIPs”:null}},“traefik”:{“Network”:“”,“Address”:“:8080”,“TLS”:null,“Redirect”:null,“Auth”:null,“WhitelistSourceRange”:null,“Compress”:false,“ProxyProtocol”:null,“ForwardedHeaders”:{“Insecure”:true,“TrustedIPs”:null}}},“Cluster”:null,“Constraints”:[],“ACME”:{“Email”:“\u003cyour LetsEncrypt email address\u003e”,“Domains”:[{“Main”:“ralphaeljohnson.com”,“SANs”:null}],“Storage”:“acme.json”,“StorageFile”:“”,“OnDemand”:true,“OnHostRule”:true,“CAServer”:“”,“EntryPoint”:“https”,“DNSChallenge”:null,“HTTPChallenge”:null,“DNSProvider”:“”,“DelayDontCheckDNS”:0,“ACMELogging”:true,“TLSConfig”:null},“DefaultEntryPoints”:[“http”,“https”],“ProvidersThrottleDuration”:2000000000,“MaxIdleConnsPerHost”:200,“IdleTimeout”:0,“InsecureSkipVerify”:false,“RootCAs”:null,“Retry”:null,“HealthCheck”:{“Interval”:30000000000},“RespondingTimeouts”:null,“ForwardingTimeouts”:null,“Web”:{“Address”:“:8080”,“CertFile”:“”,“KeyFile”:“”,“ReadOnly”:false,“Statistics”:null,“Metrics”:null,“Path”:“/”,“Auth”:null,“Debug”:false},“Docker”:{“Watch”:true,“Filename”:“”,“Constraints”:null,“Trace”:false,“DebugLogGeneratedTemplate”:false,“Endpoint”:“unix:///var/run/docker.sock”,“Domain”:“ralphaeljohnson.com”,“TLS”:null,“ExposedByDefault”:true,“UseBindPortIP”:false,“SwarmMode”:true},“File”:null,“Marathon”:null,“Consul”:null,“ConsulCatalog”:null,“Etcd”:null,“Zookeeper”:null,“Boltdb”:null,“Kubernetes”:null,“Mesos”:null,“Eureka”:null,“ECS”:null,“Rancher”:null,“DynamoDB”:null,“ServiceFabric”:null,“Rest”:null,“API”:{“EntryPoint”:“traefik”,“Dashboard”:true,“Debug”:false,“CurrentConfigurations”:null,“Statistics”:null},“Metrics”:null,“Ping”:{“EntryPoint”:“traefik”}}"
traefik_traefik.0.pc6vf4c0yvnh@ceph-osd3 | time=“2018-04-13T00:59:47Z” level=info msg=“Preparing server http &{Network: Address::80 TLS: Redirect:0xc4202e1710 Auth: WhitelistSourceRange:[] Compress:false ProxyProtocol: ForwardedHeaders:0xc420392a00} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s”
traefik_traefik.0.pc6vf4c0yvnh@ceph-osd3 | time=“2018-04-13T00:59:47Z” level=info msg=“Preparing server https &{Network: Address::443 TLS:0xc42061f400 Redirect: Auth: WhitelistSourceRange:[] Compress:false ProxyProtocol: ForwardedHeaders:0xc420392a20} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s”
traefik_traefik.0.pc6vf4c0yvnh@ceph-osd3 | time=“2018-04-13T00:59:47Z” level=info msg=“Starting server on :80”
Well, that’s enough to tell us that traefik is running, so doesn’t really explain why docker stack ps keeps showing that it exited…