[Archived] Shared Storage (Ceph) Jewel

Comments Cookbook
41

Thanks, I’m really happy that I finally have HA storage solution ready to be added to docker (somehow). The only issue I’ve found and is and it’s a worrying one, is if you use CephFS which in a lot of ways would make things much easier for all the docker managers/workers as its one filesystem shared. This seems to run perfectly as long as all nodes are online (i.e. healthy cluster), however as soon as you switch one off, I found that if you touched a file or even echoed a few chars to a file, the whole ssh session/machine locked up. If you connected to another session you would see the load on the box go completely nuts (high) and this doesn’t recover until the node is back online. As soon as it comes back, the session in the other window finishes. Dead weird!

Anyway was upsetting me thinking that ceph may not be what I am looking for as I must make sure I can have at least one physical node offline, that I decided to try out RBD. Now after finally getting it to work due to some silly bug (fix ile mention in a sec) I found that it doesn’t suffer this problem, in fact, there was a slight/brief mini lock of about a second or two and then recovered even with the node being offline still. You could then write again and it was instant and stayed instant while node offline. As soon as you brought the other node back, there was no interruption. This makes me very happy :slight_smile:

Anyway, thank you for explaining about the PG’s, that still gives me a headake mind you as the calc says I should use 256 which is exactly what you have in your guide, however, this refused to work period. The only way I could get the MDS online was to lower it until it finally accepted it. So I am still a bit confused by it all but in the morning I will add another storage node and see if this resolves the issue as I would like to get it to 256 which is what it should be from my understanding.

BTW if I get the PG numbers wrong, whats the worst that can happen? im guessing lost data so I have to get perfect but still how when it refuses it lol.

Anyway, as promised, if you try and mount with rbd and you get:

[ 4756.633338] rbd: image foo: image uses unsupported features: 0x38

Update your /etc/ceph.conf file with the following extra line:

rbd default features = 3

The original source of this was from here:

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-February/016554.html

Lastly, before I head to bed as its just gone 2:30 AM, do you know how I can share rbd for docker. What I mean is, I need persistent data over the docker swarm and I am not sure if you can mount the same rbd on another host at the same time? I am highly guessing the answer is no. So if this is the case, what workaround/solution could you suggest as cephFS is really not playing ball?

Oh and another last tip: if you didn’t know there is an awesome dashboard for ceph and it’s already there! its disabled by default and can be easily enabled with:

ceph mgr module enable dashboard

Then connect on port 7000 by default. Enjoy for those who don’t know.

Thanks in advance and have a great evening.

42

Mmm… I wonder whether this is due to the fact that you can mount MDS using either using kernel or userspace driver (http://docs.ceph.com/docs/master/start/quick-cephfs/) - would you be willing to try the user-space (FUSE) driver? I agree that while having a working RBD is good, it doesn’t solve your problem re high availability to your data. (No, you can’t mount an RBD volume from more than one host)

D

43

I think the worst that’ll happen is some inefficiency re your storage. Especially at the small scale we’re at here (3/4 OSDs), PG tuning is probably nominal. Once you scale up to (for example) 50 OSDs, having the correct PG numbers would be the difference between requesting a 1GB file, and having Ceph pulling 100 chunks of data from 50 separate disks, vs 10 chunks of data from 5 separate disks (so a 10-fold difference in throughput)

D

44

This isn’t a problem, I will have a play around later today. I am in no rush to get this all online properly, I want to make sure all the necessary testing is done so I can be assured that data is safe.

One thing I am looking into is, how do you set the warning levels for raw space used? I have used this calc:

http://florian.ca/ceph-calculator/

But they don’t tell you how to set it which is a bit annoying.

I will let you know hopefully today if it’s any better. If you have any other ideas, do drop me a quick note, thank you.

With regards to the PG’s, thanks for explaining and putting my mind at rest. I highly doubt I ever go over 5 OSD’s.

45

In my Day Job™, I use the check_osd plugin from check_ceph_df plugin from GitHub - ceph/ceph-nagios-plugins: Nagios plugins for Ceph to monitor my Ceph cluster’s free space (we have multiple pools with multiple replica levels).

I have a half-baked recipe in mind to add a monitoring stack to the cookbook, but I haven’t figured out all the pieces of the puzzle yet. Monitoring the health (and available space) of Ceph should definately be part of that, when it’s eventually ready :slight_smile:

46

Thank you very much. I will add this later to my nagios.

47

2017-09-28 14:15:18.259983 7fc56c6de700 0 – 192.168.2.11:0/1017529 >> 192.168.2.11:6789/0 pipe(0x7fc568067010 sd=4 :44144 s=1 pgs=0 cs=0 l=1 c=0x7fc56805da10).connect protocol feature mismatch, my 83ffffffffffff < peer 481dff8eea4fffb missing 400000000000000

I am getting this error. What version of ceph daemon should I be running? Tag? The latest is 12.2

48

Take a look at this comment from Sep - I think it may be the same issue?

49

mismatch error went away by running ceph osd crush tunables hammer inside ceph-mon. Now I am stuck on mounting the drive.

  warnings.warn(DEPRECATION_WARNING)
2018-01-18 21:52:19  /entrypoint.sh: SUCCESS
starting osd.0 at - osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
2018-01-18 21:52:20.143303 7fd4f26f6e00 -1 osd.0 0 log_to_monitors {default=true}
2018-01-18 21:52:21.181689 7fd4d984b700 -1 osd.0 0 waiting for initial osdmap

[root@atomic2 ~]# MYHOST=hostname -s
[root@atomic2 ~]# echo -e "

Mount cephfs volume \n

$MYHOST:6789:/ /var/data/ ceph
name=dockerswarm
,secret=AQBjF2FaIimkDRAAV40gEfcjI2kp4UyXXiJ66A==
,noatime,_netdev,context=system_u:object_r:svirt_sandbox_file_t:s0
0 2" >> /etc/fstab
[root@atomic2 ~]# mount -a
mount: mount atomic2:6789:/ on /var/data failed: Connection timed out

Another question? Is this correct or should it be mgr at the bottom instead of mgs?
docker run -d --net=host
–privileged=true
–pid=host
-v /etc/ceph:/etc/ceph
-v /var/lib/ceph/:/var/lib/ceph/
–name=“ceph-mgr”
–restart=always
ceph/daemon mgs

50

Mm, nice catch. Yes, it should be mgr at the bottom, and not mgs - I’ll fix that. Re your mounting problem, is the mds container running on the host atomic2 ?

If you run netstat -na | grep 6789, is the port open?

51

another piece that might help us resolve this issue.

[root@atomic1 ~]# ceph osd pool set cephfs_data size 2
Error ENOENT: unrecognized pool ‘cephfs_data’
[root@atomic1 ~]# ceph osd pool set cephfs_metadata size 2
Error ENOENT: unrecognized pool ‘cephfs_metadata’

52

All 3 atomic instances.

image

53

It doesn’t looks as if the mds containers are running properly. (Should they be restarting?)

This is the netstat -na output from one of my atomic hosts:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
<snip>

So it’s the first column with the IP:port that indicates your host is listening on that port, not the second (that’s the target host for any active connections)

What’s the output from docker logs ceph-mds ?

54

[root@atomic1 ~]# docker logs ceph-mds
2018-01-19 00:42:43 /entrypoint.sh: static: does not generate config
HEALTH_OK
2018-01-19 00:42:44 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:42:44 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:42:44 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:42:44 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:42:44 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:42:46 /entrypoint.sh: static: does not generate config
2018-01-19 00:42:46 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:42:46 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:42:46 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:42:46 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:42:46 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:42:49 /entrypoint.sh: static: does not generate config
2018-01-19 00:42:49 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:42:49 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:42:49 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:42:49 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:42:49 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:42:52 /entrypoint.sh: static: does not generate config
2018-01-19 00:42:52 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:42:52 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:42:52 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:42:52 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:42:52 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:42:55 /entrypoint.sh: static: does not generate config
2018-01-19 00:42:55 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:42:55 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:42:55 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:42:55 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:42:55 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:43:00 /entrypoint.sh: static: does not generate config
2018-01-19 00:43:00 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:43:00 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:43:00 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:43:00 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:43:00 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:43:05 /entrypoint.sh: static: does not generate config
2018-01-19 00:43:05 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:43:05 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:43:05 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:43:05 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:43:05 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:43:14 /entrypoint.sh: static: does not generate config
2018-01-19 00:43:14 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:43:14 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:43:14 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:43:14 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:43:14 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:43:30 /entrypoint.sh: static: does not generate config
2018-01-19 00:43:30 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:43:30 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:43:30 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:43:30 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:43:30 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:43:57 /entrypoint.sh: static: does not generate config
2018-01-19 00:43:57 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:43:57 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:43:57 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:43:57 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:43:57 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:44:51 /entrypoint.sh: static: does not generate config
2018-01-19 00:44:51 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:44:51 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:44:51 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:44:51 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:44:51 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:46:36 /entrypoint.sh: static: does not generate config
2018-01-19 00:46:36 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:46:36 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:46:36 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:46:36 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:46:36 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:50:03 /entrypoint.sh: static: does not generate config
2018-01-19 00:50:03 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:50:03 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:50:03 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:50:03 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:50:03 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:57:00 /entrypoint.sh: static: does not generate config
2018-01-19 00:57:00 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:57:00 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:57:00 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:57:00 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:57:00 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:57:14 /entrypoint.sh: static: does not generate config
2018-01-19 00:57:14 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:57:14 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:57:14 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:57:14 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:57:14 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:57:17 /entrypoint.sh: static: does not generate config
2018-01-19 00:57:17 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:57:17 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:57:17 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:57:17 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:57:17 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:57:20 /entrypoint.sh: static: does not generate config
2018-01-19 00:57:20 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:57:20 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:57:20 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:57:20 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:57:20 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:57:23 /entrypoint.sh: static: does not generate config
2018-01-19 00:57:23 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:57:23 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:57:23 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:57:23 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:57:23 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:57:27 /entrypoint.sh: static: does not generate config
2018-01-19 00:57:27 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:57:27 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:57:27 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:57:27 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:57:27 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:57:32 /entrypoint.sh: static: does not generate config
2018-01-19 00:57:32 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:57:32 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:57:32 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:57:32 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:57:32 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:57:41 /entrypoint.sh: static: does not generate config
2018-01-19 00:57:41 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:57:41 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:57:41 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:57:41 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:57:41 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:57:57 /entrypoint.sh: static: does not generate config
2018-01-19 00:57:57 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:57:57 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:57:57 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:57:57 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:57:57 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:58:25 /entrypoint.sh: static: does not generate config
2018-01-19 00:58:25 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:58:25 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:58:25 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:58:25 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:58:25 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 00:59:19 /entrypoint.sh: static: does not generate config
2018-01-19 00:59:19 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 00:59:19 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 00:59:19 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 00:59:19 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 00:59:19 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 01:01:03 /entrypoint.sh: static: does not generate config
2018-01-19 01:01:03 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 01:01:03 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 01:01:03 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 01:01:03 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 01:01:03 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 01:04:31 /entrypoint.sh: static: does not generate config
2018-01-19 01:04:31 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 01:04:31 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 01:04:31 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 01:04:31 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 01:04:31 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 01:11:23 /entrypoint.sh: static: does not generate config
2018-01-19 01:11:23 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 01:11:23 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 01:11:23 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 01:11:23 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 01:11:23 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 01:25:05 /entrypoint.sh: static: does not generate config
2018-01-19 01:25:05 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 01:25:05 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 01:25:05 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 01:25:05 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 01:25:05 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 01:52:25 /entrypoint.sh: static: does not generate config
2018-01-19 01:52:25 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 01:52:26 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 01:52:26 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 01:52:26 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 01:52:26 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 02:47:05 /entrypoint.sh: static: does not generate config
2018-01-19 02:47:05 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 02:47:05 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 02:47:05 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 02:47:05 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 02:47:05 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 04:36:21 /entrypoint.sh: static: does not generate config
2018-01-19 04:36:21 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 04:36:21 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 04:36:21 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 04:36:21 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 04:36:21 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
2018-01-19 08:14:50 /entrypoint.sh: static: does not generate config
2018-01-19 08:14:50 /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2018-01-19 08:14:50 /entrypoint.sh: static: make it available with the help of your configuration management system.
2018-01-19 08:14:50 /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2018-01-19 08:14:50 /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2018-01-19 08:14:50 /entrypoint.sh: static: if you’re interested, please visit: GitHub - ceph/ceph-ansible: Ansible playbooks to deploy Ceph, the distributed filesystem.
Error ERANGE: pg_num 256 size 3 would mean 768 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)
[root@atomic1 ~]#

55

image

56

ATOMIC1 server
added ‘mon max pg per osd = 10000’ ceph.conf

image

ATOMIC2 Server
docker logs -f ceph-mds
image

57

I got it, I think. I little change from your guide. I did the OSD container second with the following command.

docker run -d --net=host
–privileged=true
–pid=host
-v /etc/ceph:/etc/ceph
-v /var/lib/ceph/:/var/lib/ceph/
-v /dev/:/dev/
-e OSD_FORCE_ZAP=1
-e OSD_DEVICE=/dev/sdb
-e OSD_TYPE=disk
–name=“ceph-osd”
–restart=always
ceph/daemon osd_ceph_disk

There is also a mistake in your mount command. It’s missing a space between zeros at the end.

MYHOST=hostname -s
echo -e "

Mount cephfs volume \n

$MYHOST:6789:/ /var/data/ ceph
name=dockerswarm
,secret=
,noatime,_netdev,context=system_u:object_r:svirt_sandbox_file_t:s0
0 2" >> /etc/fstab

58

You’re right, it looks like newer versions of the contain use a different argument to ceph/daemon to initialize the OSD. You’re right about the typo too :wink:

I’ve updated the recipe accordingly, thanks!

59

I have ceph up and running under Ubuntu along with rancher and home assistant but I’m have an issue with one host. If host1 goes offline the rest of the host drop the ceph mount and freak out till it comes back online. You can’t even run ceph-s on any other host. This just happens to host1.

60

Mmm. how many mons do you have, and what’s the output from ceph -s ?