While Docker Swarm is great for keeping containers running (and restarting those that fail), it does nothing for persistent storage. This means if you actually want your containers to keep any data persistent across restarts (hint: you do!), you need to provide shared storage to every docker node.
Thank you for the great guide!
Is this configuration expandable? Is it possible to add nodes and change redundancy configuration on the live cluster?
And what was the problem with GlusterFS? There’s an anchor to explanation, but it doesn’t exist. Perhaps, something has changed since 2019?
Sort of. Ceph is happy to extend storage space by adding OSD, but changing the redundancy configuration is usually not as easy. You could (for example) change your failure domain from host to rack, for example, but once you’ve created a pool of 3 replicas, you’re stuck with it. Any other configuration will require a new pool, and a long and painful (ask me how I know!) migration…
Kia ora! Do you have any experience with utilising Docker Volume plugins/drivers that connect to Ceph RBDs rather than using a CephFS mounts that are then bind mounted to the containers? Potentially its unnecessary faff, but I wondered whether it would provide better performance and reliability, particularly with database applications (particularly those that run on sqlite).
I’m currently running GlusterFS and it is awful (so much so that I don’t run most database workloads off the shared storage), so just wanted to check whether also moving away from bind mounts (as I used today with Gluster) would also be helpful.
Only noticed this message now, sorry for the delayed response!
I’ve got no direct experience using RBD with Docker Volumes, but I can relay that in Kubernetes, there’s a massive performance improvement to be gained by using RBD instead of CephFS. We switched the ElfHosted Plex volumes to RBD for this reason - it sucks a bit to loose the RWX nature of the volume, but performance for millions of little files / SQLite is much better 
Yeah, I’m mostly thinking about it for the config / database storage volumes of a container, so good to hear you’ve experienced signicant performance benefits in a similar situation.
My problem now is finding a definitive way to connect at the docker volume level to Ceph in RBD mode, as so far it has proven difficult. It’s like all the docker storage plugins are abandonware because Docker themselves have abandoned any serious support for anything.
You might not like my answer here, but it rhymes with “Goobernetfleas”
Hahaha 
it feels like such a massive jump…
yeah- im starin down the barrel of a learning curve myself 
- first containers- docker then swarm now k8s? its karma for doin tech architecture n not getting to play with the toys ourselves before now- lucky we have the smartest best friends here to help us along the way! so grateful!
Hey there - I’m running Ceph on my three node cluster, based on how you have things set up. However, I run into a problem when I reboot a node that has anything like Portainer on it. I’ll drain the node, demote the node, then reboot, but the Portainer *.db files will become corrupt. From my searching, it seems like this is due to trying to store the data on Ceph storage. Anybody else run into this?
Hello!
I used this tutorial to install Ceph on a 4 node raspberry pi cluster and its great! I just have one question:
I have been using Grafana Enterprise to view data from my Influxdb2 server and I had to shut it down because this set up uses its own Grafana instance. At first I was going to just connect this Grafana instance to my influx database but this is using Grafana OSS v6 which does not support Influxdb2 
Is there a way to upgrade an individual container in this set up? I attempted to point the cephadm file to the latest Grafana enterprise image before running the bootstrap but no luck 
Any help would be greatly appreciated!
I’ve read your recipe on Ceph for Docker containers, but still do have one question, which apparently has not been addressed anywhere: I’d like to add Gamesha NFS to the Ceph cluster I’ve created with your kind help, so that clients like PCs attached to my network can store files there. But: I can’t find an instruction on how to add it to the Ceph installation, which itself lives in a Docker container. The Ceph dashboard itself does feature a menu point named NFS, but only says that I’ve gotta install Gamesha first.
What can I do here? Thank you.
I’d probably look into deploying a lightweight container and expose the NFS share(s) through that with the Ceph cluster being the backing storage. There might be a little bit of latency added but it shouldn’t be too bad. Keeps the individual jobs separated nice and cleanly IMO.
Well yeah, this would be my final solution if there was nothing “cleaner”. In the Ceph web interface, hhere is even a menu point named NFS, but I only get the message that I must install Ganesha NFS first.
I was able to connect to my filesystem on windows using ceph-dokan and NSSM. It wasn’t clean and I had to turn off secure boot so I’m still looking into other alternatives but for the meantime it’s working.
For other Debian SBC on my network I just mounted it with ceph-common.
Did you mount the Ceph cluster on the client(s) via an NFS share?
No,
For the Debian clients I installed ceph-common and mounted as a ceph volume just like in this tutorial.
For the windows clients I used ceph-dokan to mount the volume as a drive. I followed the ceph documentation here: Mount CephFS on Windows — Ceph Documentation
So technically speaking I never made a network share, I just installed ceph on each machine that needed access. The only other thing I did differently from this tutorial is I installed v19 instead of v15 by changing the curl to the one listed in the ceph documentation.
curl --silent --remote-name --location https://download.ceph.com/rpm-19.1.0/el9/noarch/cephadm
I tried the newest version first, but I had some issues most likely from trying to run all of this on a RPi 4b with 4gb of RAM.
Wow! You installed Ceph directly on the clients. But isn’t this procedure more complicated and error-prone than giving the clients “just” an NFS share, which in turn accesses a Ceph cluster?