Proxmox 5 beta OSD trouble...

Vassilis Kipouros said:
root@pve1:~# journalctl --since "2017-03-27 18:00:00" --until "2017-03-27 18:10:00"
-- Logs begin at Mon 2017-03-27 17:33:58 EEST, end at Mon 2017-03-27 18:30:06 EEST. --
Mar 27 18:02:44 pve1 pvedaemon[1929]: <root@pam> starting task UPID:pve1:00003A7B:0002A3D0:58D92994:vncshell::root@pam:
Mar 27 18:02:44 pve1 pvedaemon[14971]: starting vnc proxy UPID:pve1:00003A7B:0002A3D0:58D92994:vncshell::root@pam:
Mar 27 18:02:44 pve1 pvedaemon[14971]: launch command: /usr/bin/vncterm -rfbport 5900 -timeout 10 -authpath /nodes/pve1 -perm Sys.Console -notls -listen localhost -c /bin/login -f root
Mar 27 18:02:45 pve1 login[14973]: pam_unix(login:session): session opened for user root by (uid=0)
Mar 27 18:02:45 pve1 systemd[1]: Created slice User Slice of root.
Mar 27 18:02:45 pve1 systemd[1]: Starting User Manager for UID 0...
Mar 27 18:02:45 pve1 systemd-logind[1117]: New session 4 of user root.
Mar 27 18:02:45 pve1 systemd[14974]: pam_unix(systemd-user:session): session opened for user root by (uid=0)
Mar 27 18:02:45 pve1 systemd[1]: Started Session 4 of user root.
Mar 27 18:02:45 pve1 systemd[14974]: Listening on GnuPG cryptographic agent and passphrase cache.
Mar 27 18:02:45 pve1 systemd[14974]: Reached target Timers.
Mar 27 18:02:45 pve1 systemd[14974]: Reached target Paths.
Mar 27 18:02:45 pve1 systemd[14974]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Mar 27 18:02:45 pve1 systemd[14974]: Listening on GnuPG network certificate management daemon.
Mar 27 18:02:45 pve1 systemd[14974]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Mar 27 18:02:45 pve1 systemd[14974]: Listening on GnuPG cryptographic agent (access for web browsers).
Mar 27 18:02:45 pve1 systemd[14974]: Starting D-Bus User Message Bus Socket.
Mar 27 18:02:45 pve1 systemd[14974]: Listening on D-Bus User Message Bus Socket.
Mar 27 18:02:45 pve1 systemd[14974]: Reached target Sockets.
Mar 27 18:02:45 pve1 systemd[14974]: Reached target Basic System.
Mar 27 18:02:45 pve1 systemd[14974]: Reached target Default.
Mar 27 18:02:45 pve1 systemd[14974]: Startup finished in 31ms.
Mar 27 18:02:45 pve1 systemd[1]: Started User Manager for UID 0.
Mar 27 18:02:45 pve1 login[14996]: ROOT LOGIN on '/dev/pts/4'
Mar 27 18:02:55 pve1 kernel: Alternate GPT is invalid, using primary GPT.
Mar 27 18:02:55 pve1 kernel: sdg: sdg1 sdg2
Mar 27 18:02:55 pve1 kernel: Alternate GPT is invalid, using primary GPT.
Mar 27 18:02:55 pve1 kernel: sdg: sdg1 sdg2
Mar 27 18:02:57 pve1 kernel: sdg:
Mar 27 18:03:13 pve1 pvedaemon[1929]: <root@pam> starting task UPID:pve1:00003BDE:0002AF0A:58D929B1:cephcreateosd:sdg:root@pam:
Mar 27 18:03:13 pve1 kernel: Alternate GPT is invalid, using primary GPT.
Mar 27 18:03:13 pve1 kernel: sdg:
Mar 27 18:03:16 pve1 kernel: sdg:
Mar 27 18:03:16 pve1 kernel: sdg:
Mar 27 18:03:17 pve1 kernel: sdg: sdg2
Mar 27 18:03:18 pve1 systemd[1]: Starting Ceph disk activation: /dev/sdg2...
Mar 27 18:03:18 pve1 kernel: sdg: sdg2
Mar 27 18:03:18 pve1 kernel: sdg: sdg2
Mar 27 18:03:18 pve1 sh[15419]: main_trigger: main_trigger: Namespace(cluster='ceph', dev='/dev/sdg2', dmcrypt=None, dmcrypt_key_dir='/etc/ceph/dmcrypt-keys', func=<function main_trigger at 0x7fb394b16398>,
Mar 27 18:03:18 pve1 sh[15419]: command: Running command: /sbin/init --version
Mar 27 18:03:18 pve1 sh[15419]: command_check_call: Running command: /bin/chown ceph:ceph /dev/sdg2
Mar 27 18:03:18 pve1 sh[15419]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:18 pve1 sh[15419]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:18 pve1 sh[15419]: main_trigger: trigger /dev/sdg2 parttype 45b0969e-9b03-4f30-b4c6-b4b80ceff106 uuid 4925f1a4-ad1f-408c-b61f-b42266b2177c
Mar 27 18:03:18 pve1 sh[15419]: command: Running command: /usr/sbin/ceph-disk --verbose activate-journal /dev/sdg2
Mar 27 18:03:19 pve1 systemd[1]: Stopped Ceph disk activation: /dev/sdg2.
Mar 27 18:03:19 pve1 systemd[1]: Starting Ceph disk activation: /dev/sdg2...
Mar 27 18:03:19 pve1 sh[15419]: main_trigger:
Mar 27 18:03:19 pve1 sh[15419]: main_trigger: get_dm_uuid: get_dm_uuid /dev/sdg2 uuid path is /sys/dev/block/8:98/dm/uuid
Mar 27 18:03:19 pve1 sh[15419]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:19 pve1 sh[15419]: command: Running command: /usr/bin/ceph-osd --get-device-fsid /dev/sdg2
Mar 27 18:03:19 pve1 sh[15419]: get_space_osd_uuid: Journal /dev/sdg2 has OSD UUID 00000000-0000-0000-0000-000000000000
Mar 27 18:03:19 pve1 sh[15419]: main_activate_space: activate: OSD device not present, not starting, yet
Mar 27 18:03:19 pve1 systemd[1]: Stopped Ceph disk activation: /dev/sdg2.
Mar 27 18:03:19 pve1 systemd[1]: Starting Ceph disk activation: /dev/sdg2...
Mar 27 18:03:19 pve1 sh[15475]: main_trigger: main_trigger: Namespace(cluster='ceph', dev='/dev/sdg2', dmcrypt=None, dmcrypt_key_dir='/etc/ceph/dmcrypt-keys', func=<function main_trigger at 0x7f71b371e398>,
Mar 27 18:03:19 pve1 sh[15475]: command: Running command: /sbin/init --version
Mar 27 18:03:19 pve1 sh[15475]: command_check_call: Running command: /bin/chown ceph:ceph /dev/sdg2
Mar 27 18:03:19 pve1 sh[15475]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:19 pve1 sh[15475]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:19 pve1 sh[15475]: main_trigger: trigger /dev/sdg2 parttype 45b0969e-9b03-4f30-b4c6-b4b80ceff106 uuid 4925f1a4-ad1f-408c-b61f-b42266b2177c
Mar 27 18:03:19 pve1 sh[15475]: command: Running command: /usr/sbin/ceph-disk --verbose activate-journal /dev/sdg2
Mar 27 18:03:19 pve1 sh[15475]: main_trigger:
Mar 27 18:03:19 pve1 sh[15475]: main_trigger: get_dm_uuid: get_dm_uuid /dev/sdg2 uuid path is /sys/dev/block/8:98/dm/uuid
Mar 27 18:03:19 pve1 sh[15475]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:19 pve1 sh[15475]: command: Running command: /usr/bin/ceph-osd --get-device-fsid /dev/sdg2
Mar 27 18:03:19 pve1 sh[15475]: get_space_osd_uuid: Journal /dev/sdg2 has OSD UUID 00000000-0000-0000-0000-000000000000
Mar 27 18:03:19 pve1 sh[15475]: main_activate_space: activate: OSD device not present, not starting, yet
Mar 27 18:03:20 pve1 kernel: sdg: sdg2
Mar 27 18:03:20 pve1 sh[15492]: main_trigger: main_trigger: Namespace(cluster='ceph', dev='/dev/sdg2', dmcrypt=None, dmcrypt_key_dir='/etc/ceph/dmcrypt-keys', func=<function main_trigger at 0x7fecbf24d398>,
Mar 27 18:03:20 pve1 sh[15492]: command: Running command: /sbin/init --version
Mar 27 18:03:20 pve1 sh[15492]: command_check_call: Running command: /bin/chown ceph:ceph /dev/sdg2
Mar 27 18:03:20 pve1 sh[15492]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:20 pve1 sh[15492]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:20 pve1 sh[15492]: main_trigger: trigger /dev/sdg2 parttype 45b0969e-9b03-4f30-b4c6-b4b80ceff106 uuid 4925f1a4-ad1f-408c-b61f-b42266b2177c
Mar 27 18:03:20 pve1 sh[15492]: command: Running command: /usr/sbin/ceph-disk --verbose activate-journal /dev/sdg2
Mar 27 18:03:20 pve1 systemd[1]: Stopped Ceph disk activation: /dev/sdg2.
Mar 27 18:03:20 pve1 systemd[1]: Starting Ceph disk activation: /dev/sdg2...
Mar 27 18:03:20 pve1 sh[15492]: main_trigger:
Mar 27 18:03:20 pve1 sh[15492]: main_trigger: get_dm_uuid: get_dm_uuid /dev/sdg2 uuid path is /sys/dev/block/8:98/dm/uuid
Mar 27 18:03:20 pve1 sh[15492]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:20 pve1 sh[15492]: command: Running command: /usr/bin/ceph-osd --get-device-fsid /dev/sdg2
Mar 27 18:03:20 pve1 sh[15492]: get_space_osd_uuid: Journal /dev/sdg2 has OSD UUID 00000000-0000-0000-0000-000000000000
Mar 27 18:03:20 pve1 sh[15492]: main_activate_space: activate: OSD device not present, not starting, yet
Mar 27 18:03:20 pve1 kernel: sdg: sdg2
Mar 27 18:03:20 pve1 kernel: sdg: sdg1 sdg2
Mar 27 18:03:20 pve1 sh[15530]: main_trigger: main_trigger: Namespace(cluster='ceph', dev='/dev/sdg2', dmcrypt=None, dmcrypt_key_dir='/etc/ceph/dmcrypt-keys', func=<function main_trigger at 0x7f408dcf7398>,
Mar 27 18:03:20 pve1 sh[15530]: command: Running command: /sbin/init --version
Mar 27 18:03:20 pve1 sh[15530]: command_check_call: Running command: /bin/chown ceph:ceph /dev/sdg2
Mar 27 18:03:20 pve1 sh[15530]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:20 pve1 sh[15530]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:20 pve1 sh[15530]: main_trigger: trigger /dev/sdg2 parttype 45b0969e-9b03-4f30-b4c6-b4b80ceff106 uuid 4925f1a4-ad1f-408c-b61f-b42266b2177c
Mar 27 18:03:20 pve1 sh[15530]: command: Running command: /usr/sbin/ceph-disk --verbose activate-journal /dev/sdg2
Mar 27 18:03:21 pve1 systemd[1]: Stopped Ceph disk activation: /dev/sdg2.
Mar 27 18:03:21 pve1 systemd[1]: Starting Ceph disk activation: /dev/sdg2...
Mar 27 18:03:21 pve1 sh[15530]: main_trigger:
Mar 27 18:03:21 pve1 sh[15530]: main_trigger: get_dm_uuid: get_dm_uuid /dev/sdg2 uuid path is /sys/dev/block/8:98/dm/uuid
Mar 27 18:03:21 pve1 sh[15530]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:21 pve1 sh[15530]: command: Running command: /usr/bin/ceph-osd --get-device-fsid /dev/sdg2
Mar 27 18:03:21 pve1 sh[15530]: get_space_osd_uuid: Journal /dev/sdg2 has OSD UUID 00000000-0000-0000-0000-000000000000
Mar 27 18:03:21 pve1 sh[15530]: main_activate_space: activate: OSD device not present, not starting, yet
Mar 27 18:03:21 pve1 systemd[1]: Stopped Ceph disk activation: /dev/sdg2.
Mar 27 18:03:21 pve1 systemd[1]: ceph-disk@dev-sdg2.service: Start request repeated too quickly.
Mar 27 18:03:21 pve1 systemd[1]: Failed to start Ceph disk activation: /dev/sdg2.
Mar 27 18:03:21 pve1 systemd[1]: ceph-disk@dev-sdg2.service: Unit entered failed state.
Mar 27 18:03:21 pve1 systemd[1]: ceph-disk@dev-sdg2.service: Failed with result 'start-limit-hit'.
Mar 27 18:03:21 pve1 sh[15593]: main_trigger: main_trigger: Namespace(cluster='ceph', dev='/dev/sdg2', dmcrypt=None, dmcrypt_key_dir='/etc/ceph/dmcrypt-keys', func=<function main_trigger at 0x7f6ba7770398>,
Mar 27 18:03:21 pve1 sh[15593]: command: Running command: /sbin/init --version
Mar 27 18:03:21 pve1 sh[15593]: command_check_call: Running command: /bin/chown ceph:ceph /dev/sdg2
Mar 27 18:03:21 pve1 sh[15593]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:21 pve1 sh[15593]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:21 pve1 sh[15593]: main_trigger: trigger /dev/sdg2 parttype 45b0969e-9b03-4f30-b4c6-b4b80ceff106 uuid 4925f1a4-ad1f-408c-b61f-b42266b2177c
Mar 27 18:03:21 pve1 sh[15593]: command: Running command: /usr/sbin/ceph-disk --verbose activate-journal /dev/sdg2
Mar 27 18:03:21 pve1 sh[15593]: main_trigger:
Mar 27 18:03:21 pve1 sh[15593]: main_trigger: get_dm_uuid: get_dm_uuid /dev/sdg2 uuid path is /sys/dev/block/8:98/dm/uuid
Mar 27 18:03:21 pve1 sh[15593]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:21 pve1 sh[15593]: command: Running command: /usr/bin/ceph-osd --get-device-fsid /dev/sdg2
Mar 27 18:03:21 pve1 sh[15593]: get_space_osd_uuid: Journal /dev/sdg2 has OSD UUID 00000000-0000-0000-0000-000000000000
Mar 27 18:03:21 pve1 sh[15593]: main_activate_space: activate: OSD device not present, not starting, yet
Mar 27 18:03:22 pve1 kernel: sdg: sdg1 sdg2
Mar 27 18:03:22 pve1 systemd[1]: ceph-disk@dev-sdg2.service: Start request repeated too quickly.
Mar 27 18:03:22 pve1 systemd[1]: Failed to start Ceph disk activation: /dev/sdg2.
Mar 27 18:03:22 pve1 systemd[1]: ceph-disk@dev-sdg2.service: Failed with result 'start-limit-hit'.
Mar 27 18:03:27 pve1 kernel: XFS (sdg1): Mounting V5 Filesystem
Mar 27 18:03:27 pve1 kernel: XFS (sdg1): Ending clean mount
Mar 27 18:03:28 pve1 kernel: XFS (sdg1): Unmounting Filesystem
Mar 27 18:03:28 pve1 kernel: sdg: sdg1 sdg2
Mar 27 18:03:28 pve1 kernel: sdg: sdg1 sdg2
Mar 27 18:03:28 pve1 systemd[1]: Starting Ceph disk activation: /dev/sdg2...
Mar 27 18:03:28 pve1 sh[15781]: main_trigger: main_trigger: Namespace(cluster='ceph', dev='/dev/sdg2', dmcrypt=None, dmcrypt_key_dir='/etc/ceph/dmcrypt-keys', func=<function main_trigger at 0x7fe9020b8398>,
Mar 27 18:03:28 pve1 sh[15781]: command: Running command: /sbin/init --version
Mar 27 18:03:28 pve1 sh[15781]: command_check_call: Running command: /bin/chown ceph:ceph /dev/sdg2
Mar 27 18:03:28 pve1 sh[15781]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:28 pve1 sh[15781]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:28 pve1 sh[15781]: main_trigger: trigger /dev/sdg2 parttype 45b0969e-9b03-4f30-b4c6-b4b80ceff106 uuid 4925f1a4-ad1f-408c-b61f-b42266b2177c
Mar 27 18:03:28 pve1 sh[15781]: command: Running command: /usr/sbin/ceph-disk --verbose activate-journal /dev/sdg2
Mar 27 18:03:29 pve1 systemd[1]: Stopped Ceph disk activation: /dev/sdg2.
Mar 27 18:03:29 pve1 systemd[1]: Starting Ceph disk activation: /dev/sdg2...
Mar 27 18:03:29 pve1 sh[15781]: main_trigger:
Mar 27 18:03:29 pve1 sh[15781]: main_trigger: get_dm_uuid: get_dm_uuid /dev/sdg2 uuid path is /sys/dev/block/8:98/dm/uuid
Mar 27 18:03:29 pve1 sh[15781]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:29 pve1 sh[15781]: command: Running command: /usr/bin/ceph-osd --get-device-fsid /dev/sdg2
Mar 27 18:03:29 pve1 sh[15781]: get_space_osd_uuid: Journal /dev/sdg2 has OSD UUID 00000000-0000-0000-0000-000000000000
Mar 27 18:03:29 pve1 sh[15781]: main_activate_space: activate: OSD device not present, not starting, yet
Mar 27 18:03:29 pve1 kernel: sdg: sdg1 sdg2
Mar 27 18:03:29 pve1 sh[15803]: main_trigger: main_trigger: Namespace(cluster='ceph', dev='/dev/sdg2', dmcrypt=None, dmcrypt_key_dir='/etc/ceph/dmcrypt-keys', func=<function main_trigger at 0x7f85ccd1a398>,
Mar 27 18:03:29 pve1 sh[15803]: command: Running command: /sbin/init --version
Mar 27 18:03:29 pve1 sh[15803]: command_check_call: Running command: /bin/chown ceph:ceph /dev/sdg2
Mar 27 18:03:29 pve1 sh[15803]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:29 pve1 sh[15803]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:29 pve1 sh[15803]: main_trigger: trigger /dev/sdg2 parttype 45b0969e-9b03-4f30-b4c6-b4b80ceff106 uuid 4925f1a4-ad1f-408c-b61f-b42266b2177c
Mar 27 18:03:29 pve1 sh[15803]: command: Running command: /usr/sbin/ceph-disk --verbose activate-journal /dev/sdg2
Mar 27 18:03:29 pve1 systemd[1]: Stopped Ceph disk activation: /dev/sdg2.
Mar 27 18:03:29 pve1 systemd[1]: Starting Ceph disk activation: /dev/sdg2...
Mar 27 18:03:30 pve1 sh[15803]: main_trigger:
Mar 27 18:03:30 pve1 sh[15803]: main_trigger: get_dm_uuid: get_dm_uuid /dev/sdg2 uuid path is /sys/dev/block/8:98/dm/uuid
Mar 27 18:03:30 pve1 sh[15803]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:30 pve1 sh[15803]: command: Running command: /usr/bin/ceph-osd --get-device-fsid /dev/sdg2
Mar 27 18:03:30 pve1 sh[15803]: get_space_osd_uuid: Journal /dev/sdg2 has OSD UUID 00000000-0000-0000-0000-000000000000
Mar 27 18:03:30 pve1 sh[15803]: main_activate_space: activate: OSD device not present, not starting, yet
Mar 27 18:03:30 pve1 pvedaemon[1929]: <root@pam> end task UPID:pve1:00003BDE:0002AF0A:58D929B1:cephcreateosd:sdg:root@pam: OK
Mar 27 18:03:30 pve1 sh[15834]: main_trigger: main_trigger: Namespace(cluster='ceph', dev='/dev/sdg2', dmcrypt=None, dmcrypt_key_dir='/etc/ceph/dmcrypt-keys', func=<function main_trigger at 0x7f625e117398>,
Mar 27 18:03:30 pve1 sh[15834]: command: Running command: /sbin/init --version
Mar 27 18:03:30 pve1 sh[15834]: command_check_call: Running command: /bin/chown ceph:ceph /dev/sdg2
Mar 27 18:03:30 pve1 sh[15834]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:30 pve1 sh[15834]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:30 pve1 sh[15834]: main_trigger: trigger /dev/sdg2 parttype 45b0969e-9b03-4f30-b4c6-b4b80ceff106 uuid 4925f1a4-ad1f-408c-b61f-b42266b2177c
Mar 27 18:03:30 pve1 sh[15834]: command: Running command: /usr/sbin/ceph-disk --verbose activate-journal /dev/sdg2
Mar 27 18:03:30 pve1 sh[15834]: main_trigger:
Mar 27 18:03:30 pve1 sh[15834]: main_trigger: get_dm_uuid: get_dm_uuid /dev/sdg2 uuid path is /sys/dev/block/8:98/dm/uuid
Mar 27 18:03:30 pve1 sh[15834]: command: Running command: /sbin/blkid -o udev -p /dev/sdg2
Mar 27 18:03:30 pve1 sh[15834]: command: Running command: /usr/bin/ceph-osd --get-device-fsid /dev/sdg2
Mar 27 18:03:30 pve1 sh[15834]: get_space_osd_uuid: Journal /dev/sdg2 has OSD UUID 00000000-0000-0000-0000-000000000000
Mar 27 18:03:30 pve1 sh[15834]: main_activate_space: activate: OSD device not present, not starting, yet
Mar 27 18:03:30 pve1 systemd[1]: Started Ceph disk activation: /dev/sdg2.
Mar 27 18:04:05 pve1 smartd[1109]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 69 to 68
Mar 27 18:04:05 pve1 smartd[1109]: Device: /dev/sdc [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 117 to 118
Mar 27 18:04:05 pve1 smartd[1109]: Device: /dev/sdd [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 153 to 150
Mar 27 18:04:06 pve1 smartd[1109]: Device: /dev/sde [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 116 to 115
Mar 27 18:04:06 pve1 smartd[1109]: Device: /dev/sdf [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 58
Mar 27 18:04:06 pve1 smartd[1109]: Device: /dev/sdf [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 42
Mar 27 18:07:39 pve1 pveproxy[2843]: worker 2845 finished
Mar 27 18:07:39 pve1 pveproxy[2843]: starting 1 worker(s)
Mar 27 18:07:39 pve1 pveproxy[2843]: worker 17382 started
Mar 27 18:07:43 pve1 pveproxy[17381]: got inotify poll request in wrong process - disabling inotify
Mar 27 18:09:57 pve1 pvedaemon[1931]: <root@pam> successful auth for user 'root@pam'
root@pve1:~#
Click to expand...

sounds to me like ceph only "finds" the journal partition, but not the actual OSD. what does "blkid /dev/sdg" say? what about "udevadm info /dev/sdg" , "udevadm info /dev/sdg1" and "udevadm info /dev/sdg2"?
 
Vassilis Kipouros said:
Also out of the blue I have an osd partial full, which is probably a false alarm...

How can one osd be partial full when all have the same weight???

View attachment 5022
Click to expand...

ceph cannot always balance the data totally equally, so slightly unequal utilization is to be expected. your cluster is already nearing the recommended maximum fullness in general (if OSDs fail now, you'll be in trouble because there is not really enough space for rebalancing)
 
fabian said:
sounds to me like ceph only "finds" the journal partition, but not the actual OSD. what does "blkid /dev/sdg" say? what about "udevadm info /dev/sdg" , "udevadm info /dev/sdg1" and "udevadm info /dev/sdg2"?
Click to expand...

root@pve1:~# blkid /dev/sdg
/dev/sdg: TYPE="zfs_member" PTUUID="6a689459-9ee2-4ebf-b8be-be7169e3ce72" PTTYPE="gpt"

root@pve1:~# udevadm info /dev/sdg
P: /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:04:00.0/ata7/host6/target6:0:0/6:0:0:0/block/sdg
N: sdg
S: disk/by-id/ata-ST31000333AS_9TE0FLBQ
S: disk/by-id/wwn-0x5000c500105f3c9a
S: disk/by-path/pci-0000:04:00.0-ata-1
E: DEVLINKS=/dev/disk/by-id/wwn-0x5000c500105f3c9a /dev/disk/by-id/ata-ST31000333AS_9TE0FLBQ /dev/disk/by-path/pci-0000:04:00.0-ata-1
E: DEVNAME=/dev/sdg
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:04:00.0/ata7/host6/target6:0:0/6:0:0:0/block/sdg
E: DEVTYPE=disk
E: ID_ATA=1
E: ID_ATA_DOWNLOAD_MICROCODE=1
E: ID_ATA_FEATURE_SET_HPA=1
E: ID_ATA_FEATURE_SET_HPA_ENABLED=1
E: ID_ATA_FEATURE_SET_PM=1
E: ID_ATA_FEATURE_SET_PM_ENABLED=1
E: ID_ATA_FEATURE_SET_SECURITY=1
E: ID_ATA_FEATURE_SET_SECURITY_ENABLED=0
E: ID_ATA_FEATURE_SET_SECURITY_ENHANCED_ERASE_UNIT_MIN=174
E: ID_ATA_FEATURE_SET_SECURITY_ERASE_UNIT_MIN=174
E: ID_ATA_FEATURE_SET_SECURITY_FROZEN=1
E: ID_ATA_FEATURE_SET_SMART=1
E: ID_ATA_FEATURE_SET_SMART_ENABLED=1
E: ID_ATA_ROTATION_RATE_RPM=7200
E: ID_ATA_SATA=1
E: ID_ATA_SATA_SIGNAL_RATE_GEN1=1
E: ID_ATA_SATA_SIGNAL_RATE_GEN2=1
E: ID_ATA_WRITE_CACHE=1
E: ID_ATA_WRITE_CACHE_ENABLED=1
E: ID_BUS=ata
E: ID_FS_TYPE=zfs_member
E: ID_FS_USAGE=filesystem
E: ID_FS_VERSION=15
E: ID_MODEL=ST31000333AS
E: ID_MODEL_ENC=ST31000333AS\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
E: ID_PART_TABLE_TYPE=gpt
E: ID_PART_TABLE_UUID=6a689459-9ee2-4ebf-b8be-be7169e3ce72
E: ID_PATH=pci-0000:04:00.0-ata-1
E: ID_PATH_TAG=pci-0000_04_00_0-ata-1
E: ID_REVISION=SD1B
E: ID_SERIAL=ST31000333AS_9TE0FLBQ
E: ID_SERIAL_SHORT=9TE0FLBQ
E: ID_TYPE=disk
E: ID_WWN=0x5000c500105f3c9a
E: ID_WWN_WITH_EXTENSION=0x5000c500105f3c9a
E: MAJOR=8
E: MINOR=96
E: SUBSYSTEM=block
E: TAGS=:systemd:
E: USEC_INITIALIZED=2267524

root@pve1:~# udevadm info /dev/sdg1
P: /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:04:00.0/ata7/host6/target6:0:0/6:0:0:0/block/sdg/sdg1
N: sdg1
S: disk/by-id/ata-ST31000333AS_9TE0FLBQ-part1
S: disk/by-id/wwn-0x5000c500105f3c9a-part1
S: disk/by-path/pci-0000:04:00.0-ata-1-part1
E: DEVLINKS=/dev/disk/by-path/pci-0000:04:00.0-ata-1-part1 /dev/disk/by-id/wwn-0x5000c500105f3c9a-part1 /dev/disk/by-id/ata-ST31000333AS_9TE0FLBQ-part1
E: DEVNAME=/dev/sdg1
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:04:00.0/ata7/host6/target6:0:0/6:0:0:0/block/sdg/sdg1
E: DEVTYPE=partition
E: ID_ATA=1
E: ID_ATA_DOWNLOAD_MICROCODE=1
E: ID_ATA_FEATURE_SET_HPA=1
E: ID_ATA_FEATURE_SET_HPA_ENABLED=1
E: ID_ATA_FEATURE_SET_PM=1
E: ID_ATA_FEATURE_SET_PM_ENABLED=1
E: ID_ATA_FEATURE_SET_SECURITY=1
E: ID_ATA_FEATURE_SET_SECURITY_ENABLED=0
E: ID_ATA_FEATURE_SET_SECURITY_ENHANCED_ERASE_UNIT_MIN=174
E: ID_ATA_FEATURE_SET_SECURITY_ERASE_UNIT_MIN=174
E: ID_ATA_FEATURE_SET_SECURITY_FROZEN=1
E: ID_ATA_FEATURE_SET_SMART=1
E: ID_ATA_FEATURE_SET_SMART_ENABLED=1
E: ID_ATA_ROTATION_RATE_RPM=7200
E: ID_ATA_SATA=1
E: ID_ATA_SATA_SIGNAL_RATE_GEN1=1
E: ID_ATA_SATA_SIGNAL_RATE_GEN2=1
E: ID_ATA_WRITE_CACHE=1
E: ID_ATA_WRITE_CACHE_ENABLED=1
E: ID_BUS=ata
E: ID_FS_TYPE=zfs_member
E: ID_FS_USAGE=filesystem
E: ID_FS_VERSION=15
E: ID_MODEL=ST31000333AS
E: ID_MODEL_ENC=ST31000333AS\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
E: ID_PART_TABLE_TYPE=gpt
E: ID_PART_TABLE_UUID=6a689459-9ee2-4ebf-b8be-be7169e3ce72
E: ID_PATH=pci-0000:04:00.0-ata-1
E: ID_PATH_TAG=pci-0000_04_00_0-ata-1
E: ID_REVISION=SD1B
E: ID_SERIAL=ST31000333AS_9TE0FLBQ
E: ID_SERIAL_SHORT=9TE0FLBQ
E: ID_TYPE=disk
E: ID_WWN=0x5000c500105f3c9a
E: ID_WWN_WITH_EXTENSION=0x5000c500105f3c9a
E: MAJOR=8
E: MINOR=97
E: PARTN=1
E: PARTNAME=ceph data
E: SUBSYSTEM=block
E: TAGS=:systemd:
E: USEC_INITIALIZED=1775166861

root@pve1:~# udevadm info /dev/sdg2
P: /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:04:00.0/ata7/host6/target6:0:0/6:0:0:0/block/sdg/sdg2
N: sdg2
S: disk/by-id/ata-ST31000333AS_9TE0FLBQ-part2
S: disk/by-id/wwn-0x5000c500105f3c9a-part2
S: disk/by-partlabel/ceph\x20journal
S: disk/by-parttypeuuid/45b0969e-9b03-4f30-b4c6-b4b80ceff106.4925f1a4-ad1f-408c-b61f-b42266b2177c
S: disk/by-partuuid/4925f1a4-ad1f-408c-b61f-b42266b2177c
S: disk/by-path/pci-0000:04:00.0-ata-1-part2
E: DEVLINKS=/dev/disk/by-partlabel/ceph\x20journal /dev/disk/by-partuuid/4925f1a4-ad1f-408c-b61f-b42266b2177c /dev/disk/by-parttypeuuid/45b0969e-9b03-4f30-b4c6-b4b80ceff106.4925f1a4-ad1f-408c-b61f-b42266b2177c /dev/disk/by-path/pci-0000:04:00.0-ata-1-part2 /dev/disk/by-id/ata-ST31000333AS_9TE0FLBQ-part2 /dev/disk/by-id/wwn-0x5000c500105f3c9a-part2
E: DEVNAME=/dev/sdg2
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:04:00.0/ata7/host6/target6:0:0/6:0:0:0/block/sdg/sdg2
E: DEVTYPE=partition
E: ID_ATA=1
E: ID_ATA_DOWNLOAD_MICROCODE=1
E: ID_ATA_FEATURE_SET_HPA=1
E: ID_ATA_FEATURE_SET_HPA_ENABLED=1
E: ID_ATA_FEATURE_SET_PM=1
E: ID_ATA_FEATURE_SET_PM_ENABLED=1
E: ID_ATA_FEATURE_SET_SECURITY=1
E: ID_ATA_FEATURE_SET_SECURITY_ENABLED=0
E: ID_ATA_FEATURE_SET_SECURITY_ENHANCED_ERASE_UNIT_MIN=174
E: ID_ATA_FEATURE_SET_SECURITY_ERASE_UNIT_MIN=174
E: ID_ATA_FEATURE_SET_SECURITY_FROZEN=1
E: ID_ATA_FEATURE_SET_SMART=1
E: ID_ATA_FEATURE_SET_SMART_ENABLED=1
E: ID_ATA_ROTATION_RATE_RPM=7200
E: ID_ATA_SATA=1
E: ID_ATA_SATA_SIGNAL_RATE_GEN1=1
E: ID_ATA_SATA_SIGNAL_RATE_GEN2=1
E: ID_ATA_WRITE_CACHE=1
E: ID_ATA_WRITE_CACHE_ENABLED=1
E: ID_BUS=ata
E: ID_FS_TYPE=zfs_member
E: ID_FS_USAGE=filesystem
E: ID_FS_VERSION=15
E: ID_MODEL=ST31000333AS
E: ID_MODEL_ENC=ST31000333AS\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
E: ID_PART_ENTRY_DISK=8:96
E: ID_PART_ENTRY_NAME=ceph\x20journal
E: ID_PART_ENTRY_NUMBER=2
E: ID_PART_ENTRY_OFFSET=2048
E: ID_PART_ENTRY_SCHEME=gpt
E: ID_PART_ENTRY_SIZE=10485760
E: ID_PART_ENTRY_TYPE=45b0969e-9b03-4f30-b4c6-b4b80ceff106
E: ID_PART_ENTRY_UUID=4925f1a4-ad1f-408c-b61f-b42266b2177c
E: ID_PART_TABLE_TYPE=gpt
E: ID_PART_TABLE_UUID=6a689459-9ee2-4ebf-b8be-be7169e3ce72
E: ID_PATH=pci-0000:04:00.0-ata-1
E: ID_PATH_TAG=pci-0000_04_00_0-ata-1
E: ID_REVISION=SD1B
E: ID_SERIAL=ST31000333AS_9TE0FLBQ
E: ID_SERIAL_SHORT=9TE0FLBQ
E: ID_TYPE=disk
E: ID_WWN=0x5000c500105f3c9a
E: ID_WWN_WITH_EXTENSION=0x5000c500105f3c9a
E: MAJOR=8
E: MINOR=98
E: PARTN=2
E: PARTNAME=ceph journal
E: SUBSYSTEM=block
E: TAGS=:systemd:
E: USEC_INITIALIZED=1775172511

root@pve1:~#
 
I recommend wiping that disk a bit more than just with "sgdisk -Z" and see if that helps. e.g., you could try "zpool labelclear /dev/sdg" "zpool labelclear /dev/sdg1" "zpool labelclear /dev/sdg2", followed by "sgdisk -Z /dev/sdg".
 
It worked!!! Thank you so much!! Finally the 18th OSD is in!

I tried wipping the disk with dd the other day but it didn't work... You zpool commands saved me...

Now the only last glitch is the clock skew... How do I fix that?

upload_2017-3-28_16-13-25.png
 
Before you used v5 how did your servers use to sync after a few minutes?

This is exactly what NTP is and does.
 
Let's try not to insult one another...
I know what NTP is and does & have manually set it up and used in other cases.
In my Proxmox 4.4 cluster (before the upgrade to 5.0)
time sync was working fine without manually setting up NTP.
All hosts had same settings in their Time tab and nothing more.
 
Vassilis Kipouros said:
Let's try not to insult one another...
I know what NTP is and does & have manually set it up and used in other cases.
In my Proxmox 4.4 cluster (before the upgrade to 5.0)
time sync was working fine without manually setting up NTP.
All hosts had same settings in their Time tab and nothing more.
Click to expand...

PVE uses systemd-timesyncd by default, which uses NTP, but is not as accurate as the standard NTP client. there have been reports about systemd-timesyncd not syncing good enough for Ceph on PVE 4.x as well - see https://bugzilla.proxmox.com/show_bug.cgi?id=998 . maybe it's time to re-investigate that issue for 5.x?
 
After installing ntp on all three nodes with the default configuration,
nodes 1 and 2 work ok, but node 3 still refuses to sync.

root@pve3:~# ntpstat
unsynchronised
polling server every 64 s
root@pve3:~# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
0.debian.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.001
1.debian.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.001
2.debian.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.001
3.debian.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.001
62.1.45.120 (xa 151.236.222.81 3 u 66 128 37 14.539 89362.4 42552.4
skevosd.arx.gr 193.93.167.241 3 u 59 64 37 22.040 42489.2 27809.3
cache.asda.gr 193.93.167.239 2 u 7 64 77 14.580 49352.5 28256.0
aquarius.altecs 193.67.79.202 2 u 33 128 177 15.029 26717.6 42831.1
sucker.mfa.gr 193.93.167.239 2 u 41 64 177 13.657 25113.6 43037.4
skevosc.arx.gr 193.93.167.241 3 u 18 64 37 22.255 58113.7 19733.5
root@pve3:~#


Any tips?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back