AMD EPYC 7401P 24-Core and SSDs for ZFS

DerDanilo

Renowned Member
Jan 21, 2017
441
100
63
CPU - AMD EPYC 7401P 24-Core
  • Did anybody test this CPU?
  • Is it supported by the kernel that is shipped with PVE 5.1?
We are thinking of getting of getting some of those nodes from Hetzner with very nice specs.

ZFS - VM Storage

The following DataCenter 1,92 TB SSDs are available. Any recommendation for a VM dedicated Raidz1 ZFS storage?

Other SSD models that may be used for additional storage (System will go on dedicated SW Raid1 SSDs).
Thanks in advance!
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
6,699
1,169
164
we did some tests with a dual socket Epyc 7351 system using PVE 5.1 - ran very stable and provided good performance!
 
  • Like
Reactions: Xahid

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
6,699
1,169
164
A full kernel package build takes about 5-6 minutes - but that is probably not the kind of stat you are looking for ;) maybe someone else can chime in with VM related results?
 

gag

Member
Jan 1, 2012
10
0
21
Ukraine
Hi!
I'm playing with the Hetzner Dedicated Root Server AX160-NVMe just now. I installed Proxmox VE 5.1 two days ago. Since then, the server has been spontaneously rebooted approximately 2 or 3 times. Nothing in system logs or ssh console. That's all I've received so far. Any questions?
 

DerDanilo

Renowned Member
Jan 21, 2017
441
100
63
I am very interessted what caused the reboots. Were they triggered by an event? CPU stall?
If there is nothing in the logs it might be that the hardware freezes and resets, so it can't even write to logs.

What filesystem do you use? What PVE repo?
 

gag

Member
Jan 1, 2012
10
0
21
Ukraine
I am very interessted what caused the reboots
I found that there was no SWAP partition(s) on my Debian OS, cosed by my fault (I'm wrote 'ART swap swap 4G', instead of 'PART...' in the installimage.conf file). I created swap partitions and turn them on. And surprisingly for me that server consumes swap memory with a good grace, even at low loads. I did not notice the similar behavior before for my PX91, PX121 and others /Intel based/. This is the first possible reason for the reboots and I continue to monitor the server.
What filesystem do you use? What PVE repo?
ext3 for /boot, ext4 - / (root), and LVM for VM's. This is a pretty "classic" setup. PVE 5.1 no-subscription.
I'm planning to got 3 Guest VMs: Windows 2008r2 RDP server for approx 50 desktops/users and two CentOS 7 VMs; One for PostgreSQL, another one - for licensing services.
Feel free to ask whatever you want.
 
Last edited:

DerDanilo

Renowned Member
Jan 21, 2017
441
100
63
Thanks for the quick response.

Regarding SWAP: Please set the following in your server and reboot: "echo 'vm.swapiness=10' >> /etc/sysctl.conf; sysctl -p"
We've had the same problem with overused SWAP for no obvious reason with our machines since the upgrade to PVE 5.x. After we changed this value the swap usage went down.

I recommend you use ZFS as storage for the VMs. Doing snapshots while VMs are running works faster and ZFS has some other nice features. We are planning on using the ZFS Snapshot sync feature to a backup node, to create backups every 15-30min on an additional 1GBit/s NIC. Hetzner installs those and even provides an additonal uplink for those (you need an addtional public IP of course).

It would be awesome if you could test the following:
( If you want I can do this tests if you provide me access to the server via Hetzner Robot server support access, not entire robot ! )

```
......
RAID 1
.....

PART /boot ext4 512M
PART lvm pve 48G

LV pve root / ext4 10G
LV pve root /var/logs ext4 5G
LV pve root /mnt ext4 500M
LV pve swap swap swap 32G
```
- Once the installimage is done, revert to the command line and create another partiotion on each disk/nvme and use "Linux filesystem" as format.
- Reboot
- Install PVE, make sure to install the zfs-utils (from PVE, NOT the package from Debian)
- Reboot
- With the remaining space you then use ZFS to create a pool:
```
zpool create -f -o ashift=12 zfsnvme1 mirror /dev/<empty partition on disk 1> /dev/<empty partition on disk 2>
zfs set compression=lz4 zfsnvme1
zfs create zfsnvme1/vm-data
zfs create zfsnvme1/sources

# Limit ZFS RAM usage to 16 GB
echo "options zfs zfs_arc_max=17179869184" >> /etc/modprobe.d/zfs.conf
# Set minimal usage to 4GB
echo "options zfs zfs_arc_min=4294967296" >> /etc/modprobe.d/zfs.conf
```
Proxmox Storage:
- Then add zfsnvme1/vm-data" as ZFS storage
- Add /zfsnvme1/sources as directory storage for ISO, Templates and whatever you want to put there except VM data.


Sorry for that many testing. But I did not yet get the approval to order one of those beauties to play arround with.
 

gag

Member
Jan 1, 2012
10
0
21
Ukraine
Thanks for advice! But unfortunately I can't perform all this tests, because we've run out of time. I need to move existing Windows RDP server along with two new VMs (database and licensing servers) in only two days. Until the previous PX91++ server cancellation date. Therefore, I can't use your /unfamiliar for me/ ZFS suggestion at the same time with the totally new AMD platform testing. I'm not ready for that yet, sorry! Although any other information about my current system, its stability and performance will be at your disposal.
 

DerDanilo

Renowned Member
Jan 21, 2017
441
100
63
No problem.

But I really recommend you to reinstall the node and create separate partitions for logs and mnt (in case you mount anything).
```
......
RAID 1
.....

PART /boot ext4 512M
PART lvm pve all

LV pve root / ext4 10G
LV pve root /var/logs ext4 5G
LV pve root /mnt ext4 500M
LV pve swap swap swap 32G
```

Then create thin-lvm space once the system is up and running. This is faster and better than using default plain lvm. ==> It also saves you disk space.
You can use the following script to backup your current config and just restore it once the new disk layout was done.
https://github.com/DerDanilo/proxmox-stuff

Can you do some speed tests with thin-lvm / ext 4 and your NVME disks?

cd into e.g. /tmp and run following commands
```
/usr/bin/time dd if=/dev/zero of=sometestfile bs=4M count=1000 oflag=direct; /usr/bin/time dd if=sometestfile of=/dev/null bs=4M count=1000 oflag=direct; rm sometestfile

/usr/bin/time dd if=/dev/zero of=sometestfile bs=128k count=100000 oflag=direct; /usr/bin/time dd if=sometestfile of=/dev/null bs=128k count=100000 oflag=direct; rm sometestfile

/usr/bin/time dd if=/dev/zero of=sometestfile bs=1G count=15 oflag=direct; /usr/bin/time dd if=sometestfile of=/dev/null bs=1G count=15 oflag=direct; rm sometestfile
```

If you get an error with oflag=direct please try oflag=sync.

Thanks!
 
Jul 22, 2017
30
0
11
21
Hi!
I'm playing with the Hetzner Dedicated Root Server AX160-NVMe just now. I installed Proxmox VE 5.1 two days ago. Since then, the server has been spontaneously rebooted approximately 2 or 3 times. Nothing in system logs or ssh console. That's all I've received so far. Any questions?

I have experienced random reboots since I deployed some load onto the server yesterday. I had around 3 random reboots in the last 24 hours. Any ideas? It was running fine for 4 days without any load.

Some weird issues I have been experiencing is, that swap usage is all the time 0% and RAM usage constantly going up.

I am using ZFS in RAID 1.
 
Last edited:

speedbird

Member
Nov 3, 2017
45
5
13
That was exactly my problem with ZFS as well though on a PX61 machine. Limited RAM usage but it went ahead, filled up all RAM and then just stalled. No recovery possible until the server was forcefully rebooted. Then decided to ditch ZFS as I can't have this uncontrolled behavior. Tested on Proxmox 5.1 at the time before the second ISO was released.
 

udo

Famous Member
Apr 22, 2009
5,918
180
83
Ahrensburg; Germany
That was exactly my problem with ZFS as well though on a PX61 machine. Limited RAM usage but it went ahead, filled up all RAM and then just stalled. No recovery possible until the server was forcefully rebooted. Then decided to ditch ZFS as I can't have this uncontrolled behavior. Tested on Proxmox 5.1 at the time before the second ISO was released.
Hi,
you know, that you can limit the zfs-memory (default half of all ram)?

This example is for an really low mem server:
Code:
cat /etc/modprobe.d/zfs.conf
# ZFS tuning for a proxmox machine that reserves 2GB for ZFS
#
# Don't let ZFS use less than 1GB and more than 2GB
options zfs zfs_arc_min=1073741824
options zfs zfs_arc_max=2147483648
depends on your system I would use more ram - like 4/6 or 8/12GB...

Udo
 
Jul 22, 2017
30
0
11
21
Hi,
you know, that you can limit the zfs-memory (default half of all ram)?

This example is for an really low mem server:
Code:
cat /etc/modprobe.d/zfs.conf
# ZFS tuning for a proxmox machine that reserves 2GB for ZFS
#
# Don't let ZFS use less than 1GB and more than 2GB
options zfs zfs_arc_min=1073741824
options zfs zfs_arc_max=2147483648
depends on your system I would use more ram - like 4/6 or 8/12GB...

Udo

This System has 128 GB of RAM and by default it seems to be limited to 64 GB as seen in the spoiler. Last time the System crashed it was at about 57 GB Memory Consumption, so I doubt memory consumption is the cause for the crash. There are no hints for anything happening before the crash, nothing in kern.log or messages.

root@epyc1:~# cat /proc/spl/kstat/zfs/arcstats
13 1 0x01 95 4560 2810498817 3383178020078
name type data
hits 4 908845
misses 4 936288
demand_data_hits 4 798631
demand_data_misses 4 845684
demand_metadata_hits 4 17365
demand_metadata_misses 4 38324
prefetch_data_hits 4 77676
prefetch_data_misses 4 50904
prefetch_metadata_hits 4 15173
prefetch_metadata_misses 4 1376
mru_hits 4 788774
mru_ghost_hits 4 0
mfu_hits 4 28574
mfu_ghost_hits 4 0
deleted 4 45
mutex_miss 4 0
evict_skip 4 1
evict_not_enough 4 0
evict_l2_cached 4 0
evict_l2_eligible 4 642560
evict_l2_ineligible 4 2048
evict_l2_skip 4 0
hash_elements 4 732935
hash_elements_max 4 733572
hash_collisions 4 68687
hash_chains 4 14849
hash_chain_max 4 3
p 4 33766153216
c 4 67532306432
c_min 4 4220769152
c_max 4 67532306432
size 4 5206345616
compressed_size 4 4762900480
uncompressed_size 4 6937823232
overhead_size 4 184642560
hdr_size 4 241655904
data_size 4 4692599808
metadata_size 4 254943232
dbuf_size 4 6576432
dnode_size 4 7758400
bonus_size 4 2811840
anon_size 4 9252864
anon_evictable_data 4 0
anon_evictable_metadata 4 0
mru_size 4 4635673088
mru_evictable_data 4 4306005504
mru_evictable_metadata 4 96033280
mru_ghost_size 4 0
mru_ghost_evictable_data 4 0
mru_ghost_evictable_metadata 4 0
mfu_size 4 302617088
mfu_evictable_data 4 244883968
mfu_evictable_metadata 4 48320000
mfu_ghost_size 4 0
mfu_ghost_evictable_data 4 0
mfu_ghost_evictable_metadata 4 0
l2_hits 4 0
l2_misses 4 0
l2_feeds 4 0
l2_rw_clash 4 0
l2_read_bytes 4 0
l2_write_bytes 4 0
l2_writes_sent 4 0
l2_writes_done 4 0
l2_writes_error 4 0
l2_writes_lock_retry 4 0
l2_evict_lock_retry 4 0
l2_evict_reading 4 0
l2_evict_l1cached 4 0
l2_free_on_write 4 0
l2_abort_lowmem 4 0
l2_cksum_bad 4 0
l2_io_error 4 0
l2_size 4 0
l2_asize 4 0
l2_hdr_size 4 0
memory_throttle_count 4 0
memory_direct_count 4 0
memory_indirect_count 4 0
memory_all_bytes 4 135064612864
memory_free_bytes 4 95257186304
memory_available_bytes 3 93146804224
arc_no_grow 4 0
arc_tempreserve 4 0
arc_loaned_bytes 4 0
arc_prune 4 0
arc_meta_used 4 513745808
arc_meta_limit 4 50649229824
arc_dnode_limit 4 5064922982
arc_meta_max 4 514542184
arc_meta_min 4 16777216
sync_wait_for_async 4 909
demand_hit_predictive_prefetch 4 15185
arc_need_free 4 0
arc_sys_free 4 2110384576

I have told Hetzner to replace the machine to exclude possible hardware failures.
 
Jul 22, 2017
30
0
11
21
That was exactly my problem with ZFS as well though on a PX61 machine. Limited RAM usage but it went ahead, filled up all RAM and then just stalled. No recovery possible until the server was forcefully rebooted. Then decided to ditch ZFS as I can't have this uncontrolled behavior. Tested on Proxmox 5.1 at the time before the second ISO was released.

My machine got automatically rebooted every time, no need for manual intervention.
 

speedbird

Member
Nov 3, 2017
45
5
13
Yes so did mine after a while but when I was watching it filling up the last few % then it suddenly hang (the UX / SSH) and I could either wait for like 1-3 hours for it to automatically reboot or reboot it forcefully. Either way, right after the system came back online, the RAM usage started growing again and didn't stop until the RAM was full again. This process repeated on loop for like 3-5 times a day. And that server wasn't even running anything big at all. Only VM on it was a minimal CentOS ISO sitting idle all day. Even without this VM it happened, it just took an hour longer for the RAM to fill up but it did. Deleting all machines didn't solve that. Reinstalling Proxmox vom scratch didn't solve it, buying subscription for enterprise repo and updating to the latest didn't solve it. No matter what I did, using ZFS resulted in an unusable system because you could watch it slowly dying all day long. I then reinstalled one again and chose LVM instead. This config now runs with 4 VMs in productive use for weeks without problems now, stable and logical ram usage, no spikes, no reboots, no nothing. If you compare all the problems with ZFS in general or in performance here in the forum with those who have problem with LVM or classical ext4 setups, one could come to the conclusion that ZoL isn't really worry-free and stable as it should be.
 
Jul 22, 2017
30
0
11
21
Yes so did mine after a while but when I was watching it filling up the last few % then it suddenly hang (the UX / SSH) and I could either wait for like 1-3 hours for it to automatically reboot or reboot it forcefully. Either way, right after the system came back online, the RAM usage started growing again and didn't stop until the RAM was full again. This process repeated on loop for like 3-5 times a day. And that server wasn't even running anything big at all. Only VM on it was a minimal CentOS ISO sitting idle all day. Even without this VM it happened, it just took an hour longer for the RAM to fill up but it did. Deleting all machines didn't solve that. Reinstalling Proxmox vom scratch didn't solve it, buying subscription for enterprise repo and updating to the latest didn't solve it. No matter what I did, using ZFS resulted in an unusable system because you could watch it slowly dying all day long. I then reinstalled one again and chose LVM instead. This config now runs with 4 VMs in productive use for weeks without problems now, stable and logical ram usage, no spikes, no reboots, no nothing. If you compare all the problems with ZFS in general or in performance here in the forum with those who have problem with LVM or classical ext4 setups, one could come to the conclusion that ZoL isn't really worry-free and stable as it should be.

What kind of DIsks are you using? I had EX41S-SSD Servers running with ZFS in RAID-1 with SSDs for 2 months without any ZFS related crashes.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!