Proxmox 5 beta OSD trouble...

Discussion in 'Proxmox VE: Installation and configuration' started by Vassilis Kipouros, Mar 23, 2017.

  1. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,390
    Likes Received:
    523
    sounds to me like ceph only "finds" the journal partition, but not the actual OSD. what does "blkid /dev/sdg" say? what about "udevadm info /dev/sdg" , "udevadm info /dev/sdg1" and "udevadm info /dev/sdg2"?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  2. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,390
    Likes Received:
    523
    ceph cannot always balance the data totally equally, so slightly unequal utilization is to be expected. your cluster is already nearing the recommended maximum fullness in general (if OSDs fail now, you'll be in trouble because there is not really enough space for rebalancing)
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. Vassilis Kipouros

    Joined:
    Nov 2, 2016
    Messages:
    46
    Likes Received:
    3
    root@pve1:~# blkid /dev/sdg
    /dev/sdg: TYPE="zfs_member" PTUUID="6a689459-9ee2-4ebf-b8be-be7169e3ce72" PTTYPE="gpt"

    root@pve1:~# udevadm info /dev/sdg
    P: /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:04:00.0/ata7/host6/target6:0:0/6:0:0:0/block/sdg
    N: sdg
    S: disk/by-id/ata-ST31000333AS_9TE0FLBQ
    S: disk/by-id/wwn-0x5000c500105f3c9a
    S: disk/by-path/pci-0000:04:00.0-ata-1
    E: DEVLINKS=/dev/disk/by-id/wwn-0x5000c500105f3c9a /dev/disk/by-id/ata-ST31000333AS_9TE0FLBQ /dev/disk/by-path/pci-0000:04:00.0-ata-1
    E: DEVNAME=/dev/sdg
    E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:04:00.0/ata7/host6/target6:0:0/6:0:0:0/block/sdg
    E: DEVTYPE=disk
    E: ID_ATA=1
    E: ID_ATA_DOWNLOAD_MICROCODE=1
    E: ID_ATA_FEATURE_SET_HPA=1
    E: ID_ATA_FEATURE_SET_HPA_ENABLED=1
    E: ID_ATA_FEATURE_SET_PM=1
    E: ID_ATA_FEATURE_SET_PM_ENABLED=1
    E: ID_ATA_FEATURE_SET_SECURITY=1
    E: ID_ATA_FEATURE_SET_SECURITY_ENABLED=0
    E: ID_ATA_FEATURE_SET_SECURITY_ENHANCED_ERASE_UNIT_MIN=174
    E: ID_ATA_FEATURE_SET_SECURITY_ERASE_UNIT_MIN=174
    E: ID_ATA_FEATURE_SET_SECURITY_FROZEN=1
    E: ID_ATA_FEATURE_SET_SMART=1
    E: ID_ATA_FEATURE_SET_SMART_ENABLED=1
    E: ID_ATA_ROTATION_RATE_RPM=7200
    E: ID_ATA_SATA=1
    E: ID_ATA_SATA_SIGNAL_RATE_GEN1=1
    E: ID_ATA_SATA_SIGNAL_RATE_GEN2=1
    E: ID_ATA_WRITE_CACHE=1
    E: ID_ATA_WRITE_CACHE_ENABLED=1
    E: ID_BUS=ata
    E: ID_FS_TYPE=zfs_member
    E: ID_FS_USAGE=filesystem
    E: ID_FS_VERSION=15
    E: ID_MODEL=ST31000333AS
    E: ID_MODEL_ENC=ST31000333AS\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
    E: ID_PART_TABLE_TYPE=gpt
    E: ID_PART_TABLE_UUID=6a689459-9ee2-4ebf-b8be-be7169e3ce72
    E: ID_PATH=pci-0000:04:00.0-ata-1
    E: ID_PATH_TAG=pci-0000_04_00_0-ata-1
    E: ID_REVISION=SD1B
    E: ID_SERIAL=ST31000333AS_9TE0FLBQ
    E: ID_SERIAL_SHORT=9TE0FLBQ
    E: ID_TYPE=disk
    E: ID_WWN=0x5000c500105f3c9a
    E: ID_WWN_WITH_EXTENSION=0x5000c500105f3c9a
    E: MAJOR=8
    E: MINOR=96
    E: SUBSYSTEM=block
    E: TAGS=:systemd:
    E: USEC_INITIALIZED=2267524

    root@pve1:~# udevadm info /dev/sdg1
    P: /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:04:00.0/ata7/host6/target6:0:0/6:0:0:0/block/sdg/sdg1
    N: sdg1
    S: disk/by-id/ata-ST31000333AS_9TE0FLBQ-part1
    S: disk/by-id/wwn-0x5000c500105f3c9a-part1
    S: disk/by-path/pci-0000:04:00.0-ata-1-part1
    E: DEVLINKS=/dev/disk/by-path/pci-0000:04:00.0-ata-1-part1 /dev/disk/by-id/wwn-0x5000c500105f3c9a-part1 /dev/disk/by-id/ata-ST31000333AS_9TE0FLBQ-part1
    E: DEVNAME=/dev/sdg1
    E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:04:00.0/ata7/host6/target6:0:0/6:0:0:0/block/sdg/sdg1
    E: DEVTYPE=partition
    E: ID_ATA=1
    E: ID_ATA_DOWNLOAD_MICROCODE=1
    E: ID_ATA_FEATURE_SET_HPA=1
    E: ID_ATA_FEATURE_SET_HPA_ENABLED=1
    E: ID_ATA_FEATURE_SET_PM=1
    E: ID_ATA_FEATURE_SET_PM_ENABLED=1
    E: ID_ATA_FEATURE_SET_SECURITY=1
    E: ID_ATA_FEATURE_SET_SECURITY_ENABLED=0
    E: ID_ATA_FEATURE_SET_SECURITY_ENHANCED_ERASE_UNIT_MIN=174
    E: ID_ATA_FEATURE_SET_SECURITY_ERASE_UNIT_MIN=174
    E: ID_ATA_FEATURE_SET_SECURITY_FROZEN=1
    E: ID_ATA_FEATURE_SET_SMART=1
    E: ID_ATA_FEATURE_SET_SMART_ENABLED=1
    E: ID_ATA_ROTATION_RATE_RPM=7200
    E: ID_ATA_SATA=1
    E: ID_ATA_SATA_SIGNAL_RATE_GEN1=1
    E: ID_ATA_SATA_SIGNAL_RATE_GEN2=1
    E: ID_ATA_WRITE_CACHE=1
    E: ID_ATA_WRITE_CACHE_ENABLED=1
    E: ID_BUS=ata
    E: ID_FS_TYPE=zfs_member
    E: ID_FS_USAGE=filesystem
    E: ID_FS_VERSION=15
    E: ID_MODEL=ST31000333AS
    E: ID_MODEL_ENC=ST31000333AS\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
    E: ID_PART_TABLE_TYPE=gpt
    E: ID_PART_TABLE_UUID=6a689459-9ee2-4ebf-b8be-be7169e3ce72
    E: ID_PATH=pci-0000:04:00.0-ata-1
    E: ID_PATH_TAG=pci-0000_04_00_0-ata-1
    E: ID_REVISION=SD1B
    E: ID_SERIAL=ST31000333AS_9TE0FLBQ
    E: ID_SERIAL_SHORT=9TE0FLBQ
    E: ID_TYPE=disk
    E: ID_WWN=0x5000c500105f3c9a
    E: ID_WWN_WITH_EXTENSION=0x5000c500105f3c9a
    E: MAJOR=8
    E: MINOR=97
    E: PARTN=1
    E: PARTNAME=ceph data
    E: SUBSYSTEM=block
    E: TAGS=:systemd:
    E: USEC_INITIALIZED=1775166861

    root@pve1:~# udevadm info /dev/sdg2
    P: /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:04:00.0/ata7/host6/target6:0:0/6:0:0:0/block/sdg/sdg2
    N: sdg2
    S: disk/by-id/ata-ST31000333AS_9TE0FLBQ-part2
    S: disk/by-id/wwn-0x5000c500105f3c9a-part2
    S: disk/by-partlabel/ceph\x20journal
    S: disk/by-parttypeuuid/45b0969e-9b03-4f30-b4c6-b4b80ceff106.4925f1a4-ad1f-408c-b61f-b42266b2177c
    S: disk/by-partuuid/4925f1a4-ad1f-408c-b61f-b42266b2177c
    S: disk/by-path/pci-0000:04:00.0-ata-1-part2
    E: DEVLINKS=/dev/disk/by-partlabel/ceph\x20journal /dev/disk/by-partuuid/4925f1a4-ad1f-408c-b61f-b42266b2177c /dev/disk/by-parttypeuuid/45b0969e-9b03-4f30-b4c6-b4b80ceff106.4925f1a4-ad1f-408c-b61f-b42266b2177c /dev/disk/by-path/pci-0000:04:00.0-ata-1-part2 /dev/disk/by-id/ata-ST31000333AS_9TE0FLBQ-part2 /dev/disk/by-id/wwn-0x5000c500105f3c9a-part2
    E: DEVNAME=/dev/sdg2
    E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:04:00.0/ata7/host6/target6:0:0/6:0:0:0/block/sdg/sdg2
    E: DEVTYPE=partition
    E: ID_ATA=1
    E: ID_ATA_DOWNLOAD_MICROCODE=1
    E: ID_ATA_FEATURE_SET_HPA=1
    E: ID_ATA_FEATURE_SET_HPA_ENABLED=1
    E: ID_ATA_FEATURE_SET_PM=1
    E: ID_ATA_FEATURE_SET_PM_ENABLED=1
    E: ID_ATA_FEATURE_SET_SECURITY=1
    E: ID_ATA_FEATURE_SET_SECURITY_ENABLED=0
    E: ID_ATA_FEATURE_SET_SECURITY_ENHANCED_ERASE_UNIT_MIN=174
    E: ID_ATA_FEATURE_SET_SECURITY_ERASE_UNIT_MIN=174
    E: ID_ATA_FEATURE_SET_SECURITY_FROZEN=1
    E: ID_ATA_FEATURE_SET_SMART=1
    E: ID_ATA_FEATURE_SET_SMART_ENABLED=1
    E: ID_ATA_ROTATION_RATE_RPM=7200
    E: ID_ATA_SATA=1
    E: ID_ATA_SATA_SIGNAL_RATE_GEN1=1
    E: ID_ATA_SATA_SIGNAL_RATE_GEN2=1
    E: ID_ATA_WRITE_CACHE=1
    E: ID_ATA_WRITE_CACHE_ENABLED=1
    E: ID_BUS=ata
    E: ID_FS_TYPE=zfs_member
    E: ID_FS_USAGE=filesystem
    E: ID_FS_VERSION=15
    E: ID_MODEL=ST31000333AS
    E: ID_MODEL_ENC=ST31000333AS\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
    E: ID_PART_ENTRY_DISK=8:96
    E: ID_PART_ENTRY_NAME=ceph\x20journal
    E: ID_PART_ENTRY_NUMBER=2
    E: ID_PART_ENTRY_OFFSET=2048
    E: ID_PART_ENTRY_SCHEME=gpt
    E: ID_PART_ENTRY_SIZE=10485760
    E: ID_PART_ENTRY_TYPE=45b0969e-9b03-4f30-b4c6-b4b80ceff106
    E: ID_PART_ENTRY_UUID=4925f1a4-ad1f-408c-b61f-b42266b2177c
    E: ID_PART_TABLE_TYPE=gpt
    E: ID_PART_TABLE_UUID=6a689459-9ee2-4ebf-b8be-be7169e3ce72
    E: ID_PATH=pci-0000:04:00.0-ata-1
    E: ID_PATH_TAG=pci-0000_04_00_0-ata-1
    E: ID_REVISION=SD1B
    E: ID_SERIAL=ST31000333AS_9TE0FLBQ
    E: ID_SERIAL_SHORT=9TE0FLBQ
    E: ID_TYPE=disk
    E: ID_WWN=0x5000c500105f3c9a
    E: ID_WWN_WITH_EXTENSION=0x5000c500105f3c9a
    E: MAJOR=8
    E: MINOR=98
    E: PARTN=2
    E: PARTNAME=ceph journal
    E: SUBSYSTEM=block
    E: TAGS=:systemd:
    E: USEC_INITIALIZED=1775172511

    root@pve1:~#
     
  4. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,390
    Likes Received:
    523
    I recommend wiping that disk a bit more than just with "sgdisk -Z" and see if that helps. e.g., you could try "zpool labelclear /dev/sdg" "zpool labelclear /dev/sdg1" "zpool labelclear /dev/sdg2", followed by "sgdisk -Z /dev/sdg".
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. Vassilis Kipouros

    Joined:
    Nov 2, 2016
    Messages:
    46
    Likes Received:
    3
    It worked!!! Thank you so much!! Finally the 18th OSD is in!

    I tried wipping the disk with dd the other day but it didn't work... You zpool commands saved me...

    Now the only last glitch is the clock skew... How do I fix that?

    upload_2017-3-28_16-13-25.png
     
  6. Ashley

    Ashley Member

    Joined:
    Jun 28, 2016
    Messages:
    267
    Likes Received:
    14
  7. Vassilis Kipouros

    Joined:
    Nov 2, 2016
    Messages:
    46
    Likes Received:
    3
  8. Ashley

    Ashley Member

    Joined:
    Jun 28, 2016
    Messages:
    267
    Likes Received:
    14
    Before you used v5 how did your servers use to sync after a few minutes?

    This is exactly what NTP is and does.
     
  9. Vassilis Kipouros

    Joined:
    Nov 2, 2016
    Messages:
    46
    Likes Received:
    3
    Let's try not to insult one another...
    I know what NTP is and does & have manually set it up and used in other cases.
    In my Proxmox 4.4 cluster (before the upgrade to 5.0)
    time sync was working fine without manually setting up NTP.
    All hosts had same settings in their Time tab and nothing more.
     
  10. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,390
    Likes Received:
    523
    PVE uses systemd-timesyncd by default, which uses NTP, but is not as accurate as the standard NTP client. there have been reports about systemd-timesyncd not syncing good enough for Ceph on PVE 4.x as well - see https://bugzilla.proxmox.com/show_bug.cgi?id=998 . maybe it's time to re-investigate that issue for 5.x?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  11. Vassilis Kipouros

    Joined:
    Nov 2, 2016
    Messages:
    46
    Likes Received:
    3
    After installing ntp on all three nodes with the default configuration,
    nodes 1 and 2 work ok, but node 3 still refuses to sync.

    root@pve3:~# ntpstat
    unsynchronised
    polling server every 64 s
    root@pve3:~# ntpq -p
    remote refid st t when poll reach delay offset jitter
    ==============================================================================
    0.debian.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.001
    1.debian.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.001
    2.debian.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.001
    3.debian.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.001
    62.1.45.120 (xa 151.236.222.81 3 u 66 128 37 14.539 89362.4 42552.4
    skevosd.arx.gr 193.93.167.241 3 u 59 64 37 22.040 42489.2 27809.3
    cache.asda.gr 193.93.167.239 2 u 7 64 77 14.580 49352.5 28256.0
    aquarius.altecs 193.67.79.202 2 u 33 128 177 15.029 26717.6 42831.1
    sucker.mfa.gr 193.93.167.239 2 u 41 64 177 13.657 25113.6 43037.4
    skevosc.arx.gr 193.93.167.241 3 u 18 64 37 22.255 58113.7 19733.5
    root@pve3:~#


    Any tips?
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice