Sorry to activate this thread after 7 months ,but I'd like to know how's your experience so far with your setup. I am having the same hardware and about to set Proxmox up. Your and sb-jw's posts seem positive on this card and different from what I saw elsewhere.
New Proxmox box w/ ZFS - R730xd w/ PERC H730 Mini in “HBA Mode” - BIG NO NO?
- Thread starter KMPLSV
- Start date
I have acquired a Dell r830 with 4x E5-4650v4 2.20GHz processors with 56 total cores and 1TB ECC Registered ram for a sandbox to evaluate proxmox VE for a replacement to our existing ESXi environment.
I have set the controller to HBA with the cache disabled in the lifecycle controller. We are planning to using ZFS.
I will update with what happens as best I can.
Planning on installing Proxmox VE next week.
I have set the controller to HBA with the cache disabled in the lifecycle controller. We are planning to using ZFS.
I will update with what happens as best I can.
Planning on installing Proxmox VE next week.
I have Dell R730, with H730P Mini, I'll be using it in HBA with cache disabled. I'll let you guys know if anything goes wrong.
My changes over default config for H730P in BIOS (F2) > Device Settings:
1. Set Select Boot Device to None in Controller Management
2. Switched to HBA mode in Advanced Controller Management
2. Disabled Disk Cache for Non-RAID in Advanced Controller Properties
My changes over default config for H730P in BIOS (F2) > Device Settings:
1. Set Select Boot Device to None in Controller Management
2. Switched to HBA mode in Advanced Controller Management
2. Disabled Disk Cache for Non-RAID in Advanced Controller Properties
Last edited:
this is exactly my hardware, but I did not disable the cache. The zpool completely disapears after each reboot. Is this the cache? Do I need to disable it?
That's surprising and not expected, of course.
How did you create that pool? What gives "
zpool status" before reboot and "zpool import" after it?We are talking "write cache", right? It tells ZFS that data has been written to disk while it actually is not. Theoretically that's fine, as long as the Battery is okay. Personally I would recommend to disable it as ZFS has its very own caching system: the Adaptive Replacement Cache "ARC" and a five-second buffer in Ram for writing data = a Transaction Group ("TXG"), to be written to the ZFS-Intent-Log, "ZIL". For asynchronous writes this is highly optimized and works very well. For sync-writes the data has to be written to disk. And I do not like that the controller lies about this very fact.
You can just disable it to test the effect, right?
This is a pool with 8 600GB SAS 15k disks (which are leftover, I'd prefer higher capacity, it's just for backup)
On the other node the pool ist just "not there" after reboot
I tried a lot of force imports, pointed the zpool import to the disks. Disks are still sda to sdh. No changes.
Only thing is to recreate the pool, but it will be away again after reboot
sure I will do tomorrow, as the system is not usable like this at the moment. But I'm afraid, that it won't help, as the server had allready days of uptime. I can't imagine that the basic filesystem should not have been written allready
Christian
You may read
man zpool-import - there are some options to find a missing pool by searching for devices and also to import a "destroyed" pool (for example). Of course most of those commands are dangerous, but you don't have actual data on those drives yet, right?The drives are visible via
lsblk -f?Hi,
this zpool create above was a live command. So I recreated the pool.
after reboot:
this is totally weird. I'll check lifecycle controller tomorrow
this zpool create above was a live command. So I recreated the pool.
after reboot:
this is totally weird. I'll check lifecycle controller tomorrow
lets ignore the zpool for the moment. please post the output from
lsblk (do NOT grep anything)
ls -la /dev/disk/by-id/
lsblk (do NOT grep anything)
ls -la /dev/disk/by-id/
well, you have quite the loadout.
Before we begin, make sure you have multipath-boot-tools installed. it is critical for the proper ordering of devices during boot.
next, I assume the disks in question are the 600GB drives. assuming you can create a new zpool with striped mirrors:
zpool create mypool mirror wwn-0x5000c500cec1365f wwn-0x5000c500bd51b0ab mirror wwn-0x5000c5009a5928ef wwn-0x5000c500df2e7bfb mirror wwn-0x5000c500d2dfa42f wwn-0x5000c5007ebb7e37 mirror wwn-0x5000c500b7818a9f wwn-0x5000c500bd51ad03
one multipath-boot-tools is installed and the pool is created, reboot and see whats what.
Before we begin, make sure you have multipath-boot-tools installed. it is critical for the proper ordering of devices during boot.
next, I assume the disks in question are the 600GB drives. assuming you can create a new zpool with striped mirrors:
zpool create mypool mirror wwn-0x5000c500cec1365f wwn-0x5000c500bd51b0ab mirror wwn-0x5000c5009a5928ef wwn-0x5000c500df2e7bfb mirror wwn-0x5000c500d2dfa42f wwn-0x5000c5007ebb7e37 mirror wwn-0x5000c500b7818a9f wwn-0x5000c500bd51ad03
one multipath-boot-tools is installed and the pool is created, reboot and see whats what.
ok log below:
just for the protocol: I have 4 of these R730. All with the same hardware in it, all latest BIOS, latest Lifecycle/Drac, latest H330 Firmware, all in HBA Mode. 3 for Proxmox, 2 of them with this 600GB SAS 15k Dell drives. One with Proxmox Backup Server and 6 Seagate Barracuda SATA Disk (yes, roast me). Thist last one is also running ZFS and has no problems. So it must be something with these Dell 600GB Disks which aren't contemporary anyway imho. As the zpool is failling anyway all the time I will try some other disks too.
Ok, multipath-tools-boot killed my Proxmox boot environment....
There is a package comment to not install this package, if you're not booting from multipath. I did a rescue boot from CD and removed the package, rebooted again an everything is ok again - except for the zpool: There is no zpool anymore...
I'll try other disk but I can do that at the earliest on Wednesday. At the moment I think best I could do is to forget about this old Perc and to add another NVMe for CEPH, which would increase the osd count to 3 for each node and that would give me all the space I'd need without old hard drives.... This is planned anyway for next year
just for the protocol: I have 4 of these R730. All with the same hardware in it, all latest BIOS, latest Lifecycle/Drac, latest H330 Firmware, all in HBA Mode. 3 for Proxmox, 2 of them with this 600GB SAS 15k Dell drives. One with Proxmox Backup Server and 6 Seagate Barracuda SATA Disk (yes, roast me). Thist last one is also running ZFS and has no problems. So it must be something with these Dell 600GB Disks which aren't contemporary anyway imho. As the zpool is failling anyway all the time I will try some other disks too.
Ok, multipath-tools-boot killed my Proxmox boot environment....
There is a package comment to not install this package, if you're not booting from multipath. I did a rescue boot from CD and removed the package, rebooted again an everything is ok again - except for the zpool: There is no zpool anymore...
I'll try other disk but I can do that at the earliest on Wednesday. At the moment I think best I could do is to forget about this old Perc and to add another NVMe for CEPH, which would increase the osd count to 3 for each node and that would give me all the space I'd need without old hard drives.... This is planned anyway for next year