Planning Proxmox VE 5.1: Ceph Luminous, Kernel 4.13, latest ZFS, LXC 2.1

godlike · Oct 13, 2017

DerDanilo said:
Proxmox should also have something similar to Heat/Terraform/ etc if it starts supporting cloud-init.

Suggestions
Why not use OpenStack to do real cloud computing? Proxmox is nice, even for bigger infrastructure projects. But for real cloud computing it's a different story. Have a look here. Mention my name when you get in contact with sales.

Cloudinit is not only for "cloud" computing. It is for automation of configuration of virtual machine and providing of metadata inside VM.
If you do it manually - you live in the past.
If Proxmox willing to be modern enough it should support Cloudinit(any way, link-local api or CD or whatever).
Proxmox already have good enough API and is supported by variety of tools like salt-cloud and so on.

CharlesErickT · Oct 13, 2017

Alright so I've upgraded from:

Code:

proxmox-ve: 5.0-23 (running kernel: 4.10.17-3-pve)
pve-manager: 5.0-32 (running version: 5.0-32/2560e073)
pve-kernel-4.10.17-2-pve: 4.10.17-20
pve-kernel-4.10.17-3-pve: 4.10.17-23
libpve-http-server-perl: 2.0-6
lvm2: 2.02.168-pve3
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-14
qemu-server: 5.0-15
pve-firmware: 2.0-2
libpve-common-perl: 5.0-18
libpve-guest-common-perl: 2.0-12
libpve-access-control: 5.0-6
libpve-storage-perl: 5.0-15
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.0-9
pve-qemu-kvm: 2.9.1-1
pve-container: 2.0-16
pve-firewall: 3.0-3
pve-ha-manager: 2.0-2
ksm-control-daemon: 1.2-2
glusterfs-client: 3.8.8-1
lxc-pve: 2.1.0-2
lxcfs: 2.0.7-pve4
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.6.5.11-pve18~bpo90

to the latest pvetest.

The upgrade went well and without a problem. After booting back up I had to

Code:

zpool upgrade rpool

to update my pool and now it's seems to be working fine so far.

I have a couple of questions:

1. Will I be able to upgrade to the official 5.1 release from pvetest ?
2. I was under the impression that ZFS 0.7.2 would bring a big difference in performance based on other forum thread from other people. It seems my performance is still not where it should be.

I'm using an LSI 9211 and SSDs for the ZIL as seen here:

Code:

root@dev-proxmox-1:~# zpool status
  pool: rpool
 state: ONLINE
  scan: scrub repaired 0B in 0h1m with 0 errors on Sun Oct  8 00:25:12 2017
config:

    NAME                                                     STATE     READ WRITE CKSUM
    rpool                                                    ONLINE       0     0     0
      mirror-0                                               ONLINE       0     0     0
        scsi-35000cca00063e45c-part2                         ONLINE       0     0     0
        scsi-35000c5000f2befa3-part2                         ONLINE       0     0     0
      mirror-1                                               ONLINE       0     0     0
        scsi-35000cca0001238d8-part2                         ONLINE       0     0     0
        scsi-35000cca00018c33c-part2                         ONLINE       0     0     0
      mirror-2                                               ONLINE       0     0     0
        scsi-35000cca000187790-part2                         ONLINE       0     0     0
        scsi-35000c50071a6ca17-part2                         ONLINE       0     0     0
    logs
      mirror-3                                               ONLINE       0     0     0
        ata-Samsung_SSD_850_EVO_250GB_S3PZNF1J823980F-part2  ONLINE       0     0     0
        ata-Samsung_SSD_850_EVO_250GB_S3PZNF1J824009F-part2  ONLINE       0     0     0
    cache
      ata-Samsung_SSD_850_EVO_250GB_S3PZNF1J823980F-part3    ONLINE       0     0     0
      ata-Samsung_SSD_850_EVO_250GB_S3PZNF1J824009F-part3    ONLINE       0     0     0

Here the output of fio where I get 3751 iops:

Code:

root@dev-proxmox-1:~# fio --name=synctest --ioengine=sync --iodepth=32 --rw=randwrite --bs=4k --size=1g .
synctest: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32
fio-2.16
Starting 1 process
synctest: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [w(1)] [98.6% done] [0KB/68592KB/0KB /s] [0/17.2K/0 iops] [eta 00m:01s]
synctest: (groupid=0, jobs=1): err= 0: pid=24556: Fri Oct 13 09:35:16 2017
  write: io=1024.0MB, bw=15008KB/s, iops=3751, runt= 69869msec
    clat (usec): min=9, max=109564, avg=260.29, stdev=563.51
     lat (usec): min=9, max=109565, avg=261.14, stdev=563.62
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   13], 10.00th=[   16], 20.00th=[   35],
     | 30.00th=[   44], 40.00th=[  151], 50.00th=[  205], 60.00th=[  346],
     | 70.00th=[  382], 80.00th=[  426], 90.00th=[  494], 95.00th=[  556],
     | 99.00th=[  796], 99.50th=[ 1160], 99.90th=[ 4080], 99.95th=[ 5984],
     | 99.99th=[12096]
    lat (usec) : 10=0.01%, 20=11.27%, 50=21.29%, 100=2.04%, 250=17.33%
    lat (usec) : 500=38.74%, 750=8.11%, 1000=0.59%
    lat (msec) : 2=0.37%, 4=0.15%, 10=0.09%, 20=0.01%, 50=0.01%
    lat (msec) : 250=0.01%
  cpu          : usr=2.41%, sys=88.51%, ctx=36971, majf=11, minf=281
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=262144/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=1024.0MB, aggrb=15007KB/s, minb=15007KB/s, maxb=15007KB/s, mint=69869msec, maxt=69869msec

All ZFS setting are mainly default except the import by id instead of /dev/sdX.

Using pgbench in an LXC container running postgres resulted in what seems like low tps:

Code:

postgres@dev-dbtst-1:/root$ pgbench -c 16 -j 8 -T 60 bench
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 1
query mode: simple
number of clients: 16
number of threads: 8
duration: 60 s
number of transactions actually processed: 23487
latency average = 40.907 ms
tps = 391.131788 (including connections establishing)
tps = 391.239537 (excluding connections establishing)

Anything I could tweak/check to increase performance ?

In the meantime I will benchmark using real world performance as some suggest because from what they say you can't really use benchmarking tool for ZFS

Thanks for your work !

Gerhard W. Recher · Oct 13, 2017

martin said:
We just uploaded a bunch of new packages to our pvetest repository (for testing only), including zfs 0.7.2 and pve-kernel-4.13.4-1-pve

waiting for feedback!

@martin thank for notifying !

ceph still not runs with rdma enabled

davidindra · Oct 14, 2017

Hi,
so I upgraded to ZFS 0.7.2 (added pvetest repo and launched apt-get upgrade) and now none of my VMs start (CTs aren't affected) with error that the disks are missing, because folder /dev/zvol just doesn't exist - the symlinks aren't there. /dev/zd* devices are visible. Some logs:
zfs list:

Code:

root@prox1:~# zfs list
NAME                           USED  AVAIL  REFER  MOUNTPOINT
rpool                          443G   456G    96K  /rpool
rpool-ssd                      106K   108G    19K  /dev/rpool-ssd
rpool/ROOT                    1.48G   456G    96K  /rpool/ROOT
rpool/ROOT/pve-1              1.48G   456G  1.48G  /
rpool/data                     433G   456G    96K  /rpool/data
rpool/data/subvol-100-disk-1   489M  7.52G   489M  /rpool/data/subvol-100-disk-1
rpool/data/subvol-102-disk-1   612M  7.41G   603M  /rpool/data/subvol-102-disk-1
rpool/data/subvol-108-disk-1  1.12G  6.95G  1.05G  /rpool/data/subvol-108-disk-1
rpool/data/subvol-109-disk-1   946M  17.1G   946M  /rpool/data/subvol-109-disk-1
rpool/data/vm-101-disk-1       121G   456G   105G  -
rpool/data/vm-101-disk-2      61.3G   456G  58.9G  -
rpool/data/vm-103-disk-1      4.08G   456G  4.07G  -
rpool/data/vm-104-disk-1      3.81G   456G  3.80G  -
rpool/data/vm-105-disk-1      21.7G   456G  21.7G  -
rpool/data/vm-106-disk-1      12.5G   456G  12.5G  -
rpool/data/vm-107-disk-1      19.8G   456G  19.8G  -
rpool/data/vm-107-disk-2      12.6G   456G  12.6G  -
rpool/data/vm-107-disk-3      69.9G   456G  69.9G  -
rpool/data/vm-110-disk-1       103G   456G   103G  -
rpool/data/vm-111-disk-1        64K   456G    64K  -
rpool/swap                    8.50G   458G  6.29G  -

zpool status:

Code:

root@prox1:~# zpool status pool: rpool state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(5) for details. scan: scrub repaired 0B in 11h13m with 0 errors on Sun Oct 8 11:37:29 2017 config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sda2 ONLINE 0 0 0 sdb2 ONLINE 0 0 0 errors: No known data errors

zpool upgrade -a:

Code:

root@prox1:~# zpool upgrade -a This system supports ZFS pool feature flags. cannot set property for 'rpool': invalid argument for this pool operation

Can you please somebody know, how to solve this problem?

Thank you really much for the responses.

jeffwadsworth · Oct 14, 2017

apt-get update && apt-get dist-upgrade

I don’t believe apt-get update is correct.

davidindra · Oct 14, 2017

jeffwadsworth said:
apt-get update && apt-get dist-upgrade

I don’t believe apt-get update is correct.

Man, you have totally saved my day! After dist-upgrade it's fully working again. Maybe some kernel incompatibility with new ZFS..?

Thanks a lot!

drdownload · Oct 14, 2017

Since 5.1 moves to 4.13: will the kernel/qemu support kvm-gt?

Chicken76 · Oct 16, 2017

CharlesErickT said:
After booting back up I had to

Code:

zpool upgrade rpool

to update my pool and now it's seems to be working fine so far.

What do you mean you 'had to' update the pool? Wasn't it running correctly without the pool upgrade?

riptide_wave · Oct 16, 2017

Just updated one of my boxes from Linux 4.10.17-3-pve #1 SMP PVE 4.10.17-23 to Linux 4.13.4-1-pve #1 SMP PVE 4.13.4-25 and sadly Infiniband is no longer working on this box. Below is the kernel panic reported in syslog:

Code:

Oct 16 13:17:50 C6100-1-N4 OpenSM[3770]: SM port is down
Oct 16 13:17:50 C6100-1-N4 kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
Oct 16 13:17:50 C6100-1-N4 kernel: IP: ib_free_recv_mad+0x44/0xa0 [ib_core]
Oct 16 13:17:50 C6100-1-N4 kernel: PGD 0
Oct 16 13:17:50 C6100-1-N4 kernel: P4D 0
Oct 16 13:17:50 C6100-1-N4 kernel:
Oct 16 13:17:50 C6100-1-N4 kernel: Oops: 0002 [#1] SMP
Oct 16 13:17:50 C6100-1-N4 kernel: Modules linked in: iptable_filter openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 softdog nf_nat_ipv4 nf_defrag_ipv6 nf_nat nf_conntrack libcrc32c nfnetlink_log nfnetlink ib_ipoib rdma_ucm ib_umad ib_uverbs bonding 8021q garp ipmi_ssif mrp intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc snd_pcm aesni_intel ast aes_x86_64 snd_timer crypto_simd ttm glue_helper snd cryptd dcdbas drm_kms_helper soundcore intel_cstate pcspkr drm joydev input_leds i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt ib_mthca lpc_ich ioatdma i5500_temp i7core_edac shpchp mac_hid ipmi_si ipmi_devintf ipmi_msghandler vhost_net vhost tap ib_iser rdma_cm iw_cm ib_cm ib_core sunrpc iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
Oct 16 13:17:50 C6100-1-N4 kernel:  ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs xor raid6_pq hid_generic usbmouse usbkbd usbhid hid igb(O) ahci dca mpt3sas raid_class ptp i2c_i801 libahci scsi_transport_sas pps_core
Oct 16 13:17:50 C6100-1-N4 kernel: CPU: 0 PID: 2833 Comm: kworker/0:1H Tainted: P          IO    4.13.4-1-pve #1
Oct 16 13:17:50 C6100-1-N4 kernel: Hardware name: Dell       XS23-TY3        /9CMP63, BIOS 1.71 09/17/2013
Oct 16 13:17:50 C6100-1-N4 kernel: Workqueue: ib-comp-wq ib_cq_poll_work [ib_core]
Oct 16 13:17:50 C6100-1-N4 kernel: task: ffffa069c6541600 task.stack: ffffb9a729054000
Oct 16 13:17:50 C6100-1-N4 kernel: RIP: 0010:ib_free_recv_mad+0x44/0xa0 [ib_core]
Oct 16 13:17:50 C6100-1-N4 kernel: RSP: 0018:ffffb9a729057d38 EFLAGS: 00010286
Oct 16 13:17:50 C6100-1-N4 kernel: RAX: ffffa069cb138a48 RBX: ffffa069cb138a10 RCX: 0000000000000000
Oct 16 13:17:50 C6100-1-N4 kernel: RDX: ffffb9a729057d38 RSI: 0000000000000000 RDI: ffffa069cb138a20
Oct 16 13:17:50 C6100-1-N4 kernel: RBP: ffffb9a729057d60 R08: ffffa072d2d49800 R09: ffffa069cb138ae0
Oct 16 13:17:50 C6100-1-N4 kernel: R10: ffffa069cb138ae0 R11: ffffa072b3994e00 R12: ffffb9a729057d38
Oct 16 13:17:50 C6100-1-N4 kernel: R13: ffffa069d1c90000 R14: 0000000000000000 R15: ffffa069d1c90880
Oct 16 13:17:50 C6100-1-N4 kernel: FS:  0000000000000000(0000) GS:ffffa069dba00000(0000) knlGS:0000000000000000
Oct 16 13:17:50 C6100-1-N4 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 16 13:17:50 C6100-1-N4 kernel: CR2: 0000000000000008 CR3: 00000011f51f2000 CR4: 00000000000006f0
Oct 16 13:17:50 C6100-1-N4 kernel: Call Trace:
Oct 16 13:17:50 C6100-1-N4 kernel:  ib_mad_recv_done+0x5cc/0xb50 [ib_core]
Oct 16 13:17:50 C6100-1-N4 kernel:  __ib_process_cq+0x5c/0xb0 [ib_core]
Oct 16 13:17:50 C6100-1-N4 kernel:  ib_cq_poll_work+0x20/0x60 [ib_core]
Oct 16 13:17:50 C6100-1-N4 kernel:  process_one_work+0x1e9/0x410
Oct 16 13:17:50 C6100-1-N4 kernel:  worker_thread+0x4b/0x410
Oct 16 13:17:50 C6100-1-N4 kernel:  kthread+0x109/0x140
Oct 16 13:17:50 C6100-1-N4 kernel:  ? process_one_work+0x410/0x410
Oct 16 13:17:50 C6100-1-N4 kernel:  ? kthread_create_on_node+0x70/0x70
Oct 16 13:17:50 C6100-1-N4 kernel:  ? SyS_exit_group+0x14/0x20
Oct 16 13:17:50 C6100-1-N4 kernel:  ret_from_fork+0x25/0x30
Oct 16 13:17:50 C6100-1-N4 kernel: Code: 28 00 00 00 48 89 45 e8 31 c0 4c 89 65 d8 48 8b 57 28 48 8d 47 28 4c 89 65 e0 48 39 d0 74 23 48 8b 77 28 48 8b 4f 30 48 8b 55 d8 <4c> 89 66 08 48 89 75 d8 48 89 11 48 89 4a 08 48 89 47 28 48 89
Oct 16 13:17:50 C6100-1-N4 kernel: RIP: ib_free_recv_mad+0x44/0xa0 [ib_core] RSP: ffffb9a729057d38
Oct 16 13:17:50 C6100-1-N4 kernel: CR2: 0000000000000008
Oct 16 13:17:50 C6100-1-N4 kernel: ---[ end trace 937ca6a9fe8de56f ]---

My hardware is as follows:
Dell C6100
2x Intel(R) Xeon(R) CPU X5650
Mellanox Technologies MT25208 [InfiniHost III Ex] (rev 20)

fabian · Oct 17, 2017

riptide_wave said:

Just updated one of my boxes from Linux 4.10.17-3-pve #1 SMP PVE 4.10.17-23 to Linux 4.13.4-1-pve #1 SMP PVE 4.13.4-25 and sadly Infiniband is no longer working on this box. Below is the kernel panic reported in syslog:

Code:

Oct 16 13:17:50 C6100-1-N4 OpenSM[3770]: SM port is down
Oct 16 13:17:50 C6100-1-N4 kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
Oct 16 13:17:50 C6100-1-N4 kernel: IP: ib_free_recv_mad+0x44/0xa0 [ib_core]
Oct 16 13:17:50 C6100-1-N4 kernel: PGD 0
Oct 16 13:17:50 C6100-1-N4 kernel: P4D 0
Oct 16 13:17:50 C6100-1-N4 kernel:
Oct 16 13:17:50 C6100-1-N4 kernel: Oops: 0002 [#1] SMP
Oct 16 13:17:50 C6100-1-N4 kernel: Modules linked in: iptable_filter openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 softdog nf_nat_ipv4 nf_defrag_ipv6 nf_nat nf_conntrack libcrc32c nfnetlink_log nfnetlink ib_ipoib rdma_ucm ib_umad ib_uverbs bonding 8021q garp ipmi_ssif mrp intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc snd_pcm aesni_intel ast aes_x86_64 snd_timer crypto_simd ttm glue_helper snd cryptd dcdbas drm_kms_helper soundcore intel_cstate pcspkr drm joydev input_leds i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt ib_mthca lpc_ich ioatdma i5500_temp i7core_edac shpchp mac_hid ipmi_si ipmi_devintf ipmi_msghandler vhost_net vhost tap ib_iser rdma_cm iw_cm ib_cm ib_core sunrpc iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
Oct 16 13:17:50 C6100-1-N4 kernel:  ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs xor raid6_pq hid_generic usbmouse usbkbd usbhid hid igb(O) ahci dca mpt3sas raid_class ptp i2c_i801 libahci scsi_transport_sas pps_core
Oct 16 13:17:50 C6100-1-N4 kernel: CPU: 0 PID: 2833 Comm: kworker/0:1H Tainted: P          IO    4.13.4-1-pve #1
Oct 16 13:17:50 C6100-1-N4 kernel: Hardware name: Dell       XS23-TY3        /9CMP63, BIOS 1.71 09/17/2013
Oct 16 13:17:50 C6100-1-N4 kernel: Workqueue: ib-comp-wq ib_cq_poll_work [ib_core]
Oct 16 13:17:50 C6100-1-N4 kernel: task: ffffa069c6541600 task.stack: ffffb9a729054000
Oct 16 13:17:50 C6100-1-N4 kernel: RIP: 0010:ib_free_recv_mad+0x44/0xa0 [ib_core]
Oct 16 13:17:50 C6100-1-N4 kernel: RSP: 0018:ffffb9a729057d38 EFLAGS: 00010286
Oct 16 13:17:50 C6100-1-N4 kernel: RAX: ffffa069cb138a48 RBX: ffffa069cb138a10 RCX: 0000000000000000
Oct 16 13:17:50 C6100-1-N4 kernel: RDX: ffffb9a729057d38 RSI: 0000000000000000 RDI: ffffa069cb138a20
Oct 16 13:17:50 C6100-1-N4 kernel: RBP: ffffb9a729057d60 R08: ffffa072d2d49800 R09: ffffa069cb138ae0
Oct 16 13:17:50 C6100-1-N4 kernel: R10: ffffa069cb138ae0 R11: ffffa072b3994e00 R12: ffffb9a729057d38
Oct 16 13:17:50 C6100-1-N4 kernel: R13: ffffa069d1c90000 R14: 0000000000000000 R15: ffffa069d1c90880
Oct 16 13:17:50 C6100-1-N4 kernel: FS:  0000000000000000(0000) GS:ffffa069dba00000(0000) knlGS:0000000000000000
Oct 16 13:17:50 C6100-1-N4 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 16 13:17:50 C6100-1-N4 kernel: CR2: 0000000000000008 CR3: 00000011f51f2000 CR4: 00000000000006f0
Oct 16 13:17:50 C6100-1-N4 kernel: Call Trace:
Oct 16 13:17:50 C6100-1-N4 kernel:  ib_mad_recv_done+0x5cc/0xb50 [ib_core]
Oct 16 13:17:50 C6100-1-N4 kernel:  __ib_process_cq+0x5c/0xb0 [ib_core]
Oct 16 13:17:50 C6100-1-N4 kernel:  ib_cq_poll_work+0x20/0x60 [ib_core]
Oct 16 13:17:50 C6100-1-N4 kernel:  process_one_work+0x1e9/0x410
Oct 16 13:17:50 C6100-1-N4 kernel:  worker_thread+0x4b/0x410
Oct 16 13:17:50 C6100-1-N4 kernel:  kthread+0x109/0x140
Oct 16 13:17:50 C6100-1-N4 kernel:  ? process_one_work+0x410/0x410
Oct 16 13:17:50 C6100-1-N4 kernel:  ? kthread_create_on_node+0x70/0x70
Oct 16 13:17:50 C6100-1-N4 kernel:  ? SyS_exit_group+0x14/0x20
Oct 16 13:17:50 C6100-1-N4 kernel:  ret_from_fork+0x25/0x30
Oct 16 13:17:50 C6100-1-N4 kernel: Code: 28 00 00 00 48 89 45 e8 31 c0 4c 89 65 d8 48 8b 57 28 48 8d 47 28 4c 89 65 e0 48 39 d0 74 23 48 8b 77 28 48 8b 4f 30 48 8b 55 d8 <4c> 89 66 08 48 89 75 d8 48 89 11 48 89 4a 08 48 89 47 28 48 89
Oct 16 13:17:50 C6100-1-N4 kernel: RIP: ib_free_recv_mad+0x44/0xa0 [ib_core] RSP: ffffb9a729057d38
Oct 16 13:17:50 C6100-1-N4 kernel: CR2: 0000000000000008
Oct 16 13:17:50 C6100-1-N4 kernel: ---[ end trace 937ca6a9fe8de56f ]---

My hardware is as follows:
Dell C6100
2x Intel(R) Xeon(R) CPU X5650
Mellanox Technologies MT25208 [InfiniHost III Ex] (rev 20)

please open up a new thread so we can follow up there!

suggested next steps would be
- test latest vanilla 4.13 kernel
- test latest vanilla 4.14 RC kernel
(e.g. from http://kernel.ubuntu.com/~kernel-ppa/mainline/ )

if neither of those work, testing vanilla (not latest) 4.13, 4.12 and 4.11 would be the next steps

CharlesErickT · Oct 17, 2017

Chicken76 said:
What do you mean you 'had to' update the pool? Wasn't it running correctly without the pool upgrade?

Yes it was booting and running correctly but giving me the following warning:

Code:

status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.

So to get all the new features from 0.7 I had to upgrade the pool

Chicken76 · Oct 18, 2017

Will Proxmox 5.1 include this fix for LXC?
https://github.com/lxc/lxcfs/pull/205

fabian · Oct 19, 2017

Chicken76 said:
Will Proxmox 5.1 include this fix for LXC?
https://github.com/lxc/lxcfs/pull/205

already contained in lxcfs 2.07-pve4 (since August

)

ivensiya · Oct 21, 2017

After upgrade from test repo:

error: no such device: ff7652jfegfhgef.
error: unknow filesystem.
entering rescue mode ...

zfs 0.7.x
pve 5
kernel 4.13

please help!

Rixton · Oct 22, 2017

Hi,
I also got the "zpool upgrade" issue, an apt-get update does however not resolve it for me
Got the same issue on both my

Code:

zpool status:
 
 pool: krikkit
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 24h48m with 0 errors on Mon Oct  9 01:12:02 2017
config:

    NAME                                          STATE     READ WRITE CKSUM
    krikkit                                       ONLINE       0     0     0
      raidz1-0                                    ONLINE       0     0     0
        ata-WDC_WD20EARS-00J2GB0_WD-WCAYY0190385  ONLINE       0     0     0
        ata-WDC_WD20EARS-00J2GB0_WD-WCAYY0148382  ONLINE       0     0     0
        ata-WDC_WD20EARS-00J2GB0_WD-WCAYY0171726  ONLINE       0     0     0
        ata-WDC_WD20EARS-00J2GB0_WD-WCAYY0156865  ONLINE       0     0     0

errors: No known data errors

  pool: milliways
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(5) for details.
  scan: scrub canceled on Sun Oct 22 09:26:46 2017
config:

    NAME                        STATE     READ WRITE CKSUM
    milliways                   ONLINE       0     0     0
      raidz1-0                  ONLINE       0     0     0
        sda                     ONLINE       0     0     0
        wwn-0x50014ee2629b7c29  ONLINE       0     0     0
        sdc                     ONLINE       0     0     0
        sdd                     ONLINE       0     0     0
        sdf                     ONLINE       0     0     0
        sdg                     ONLINE       0     0     0

errors: No known data errors

Code:

zpool upgrade milliways
This system supports ZFS pool feature flags.

cannot set property for 'milliways': invalid argument for this pool operation
root@milliways:/home/rixton#

Code:

zpool events
TIME                           CLASS
internal error: Bad file descriptor
Aborted
root@milliways:/home/rixton#

Also noticed that zed is running at 100% all the time, there is however no issues accessing anything on any of my drives

un1x0d · Oct 22, 2017

Hello.
Tell me, until the end of the month you plan to release a new release as announced?
We need to make a new cluster and would like to do it right away on the new version.
Thank you.

fabian · Oct 23, 2017

ivensiya said:
After upgrade from test repo:

error: no such device: ff7652jfegfhgef.
error: unknow filesystem.
entering rescue mode ...

zfs 0.7.x
pve 5
kernel 4.13

please help!

please open a new thread, and include the following output from the Grub rescue shell:

Code:

set
ls
ls $root/

fabian · Oct 23, 2017

Rixton said:

Hi,
I also got the "zpool upgrade" issue, an apt-get update does however not resolve it for me
Got the same issue on both my

Code:

zpool status:
 
 pool: krikkit
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 24h48m with 0 errors on Mon Oct  9 01:12:02 2017
config:

    NAME                                          STATE     READ WRITE CKSUM
    krikkit                                       ONLINE       0     0     0
      raidz1-0                                    ONLINE       0     0     0
        ata-WDC_WD20EARS-00J2GB0_WD-WCAYY0190385  ONLINE       0     0     0
        ata-WDC_WD20EARS-00J2GB0_WD-WCAYY0148382  ONLINE       0     0     0
        ata-WDC_WD20EARS-00J2GB0_WD-WCAYY0171726  ONLINE       0     0     0
        ata-WDC_WD20EARS-00J2GB0_WD-WCAYY0156865  ONLINE       0     0     0

errors: No known data errors

  pool: milliways
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(5) for details.
  scan: scrub canceled on Sun Oct 22 09:26:46 2017
config:

    NAME                        STATE     READ WRITE CKSUM
    milliways                   ONLINE       0     0     0
      raidz1-0                  ONLINE       0     0     0
        sda                     ONLINE       0     0     0
        wwn-0x50014ee2629b7c29  ONLINE       0     0     0
        sdc                     ONLINE       0     0     0
        sdd                     ONLINE       0     0     0
        sdf                     ONLINE       0     0     0
        sdg                     ONLINE       0     0     0

errors: No known data errors

Code:

zpool upgrade milliways
This system supports ZFS pool feature flags.

cannot set property for 'milliways': invalid argument for this pool operation
root@milliways:/home/rixton#

Code:

zpool events
TIME                           CLASS
internal error: Bad file descriptor
Aborted
root@milliways:/home/rixton#

Also noticed that zed is running at 100% all the time, there is however no issues accessing anything on any of my drives

that pools need a manual upgrade to enable (and/or activate) new features is intentional - you should only do it once you know you no longer need to be able to import that pool using an older ZFS version lacking those features. it is ZFS's variant of versioning, since there are three diverging open implementations matching based on a minimum version number is not really feasible, so each feature that affects compatibility gets a flag

could you check the log of ZED ("journalctl -b -u zfs-zed") and attempt to restart it ("systemctl restart zfs-zed")?

CharlesErickT · Oct 23, 2017

Just a general improvement recommendation but shouldn't the install create the pool by disk ID instead of /dev/sdX as per the ZOL best practice recommendation. As of now we need to select ZFS Raid 1 on install, change default to use the ID, reboot, add new disks, partition them and then add them to the pool to then convert to RAID 10 which could be all done by the install.

Rixton · Oct 23, 2017

fabian said:
that pools need a manual upgrade to enable (and/or activate) new features is intentional - you should only do it once you know you no longer need to be able to import that pool using an older ZFS version lacking those features. it is ZFS's variant of versioning, since there are three diverging open implementations matching based on a minimum version number is not really feasible, so each feature that affects compatibility gets a flag

could you check the log of ZED ("journalctl -b -u zfs-zed") and attempt to restart it ("systemctl restart zfs-zed")?

How can I update it when I get the error?
zpool upgrade milliways

Code:

This system supports ZFS pool feature flags.

cannot set property for 'milliways': invalid argument for this pool operation

Code:

Oct 23 19:01:38 milliways systemd[1]: Stopping ZFS Event Daemon (zed)...
Oct 23 19:01:38 milliways systemd[1]: Stopped ZFS Event Daemon (zed).
Oct 23 19:01:38 milliways systemd[1]: Started ZFS Event Daemon (zed).
Oct 23 19:01:38 milliways zed[12245]: ZFS Event Daemon 0.7.2-pve1~bpo90 (PID 12245)
Oct 23 19:01:38 milliways zed[12245]: Processing events since eid=0

(not allowed to posts any links, remove xx)
hxxps://image.ibb.co/mniLY6/Ska_rmavbild_2017_10_23_kl_19_02_05.png

zpool events

Code:

TIME                           CLASS
internal error: Bad file descriptor
Aborted

The whole issue seems like this one;
hxxps://github.com/zfsonlinux/zfs/issues/4720

Planning Proxmox VE 5.1: Ceph Luminous, Kernel 4.13, latest ZFS, LXC 2.1

Member

Member

Well-Known Member

New Member

Member

New Member

Member

Well-Known Member

Member

Proxmox Staff Member

Member

Well-Known Member

Proxmox Staff Member

Active Member

Active Member

Renowned Member

Proxmox Staff Member

Proxmox Staff Member

Member

Active Member