Search results

  1. hepo

    MySQL performance issue on Proxmox with Ceph

    Hi community, needs some help with MySQL performance improvement. We have recently migrated MySQL DB (about 20GB) from bare metal to VM on Proxmox. Shortly after the DBA team started complaining about performance issues (slow queries and overall slowness). They have also ran variety of tests...
  2. hepo

    Random host reboot when moving VM disk from ceph to local-zfs

    Hi team and community, We have experienced PVE host reboot while moving VM disk from ceph to local-zfs. The local-zfs is 2 mirrored SSDs where PVE is installed. The disk move was triggered few times in attempt to troubleshoot potential ceph performance/latency issues. The host reboots are...
  3. hepo

    [SOLVED] Ceph (stretched cluster) performance troubleshooting

    Coming back on this as a follow-up... We were able to source Micron 7300 Pros 1.92TB NVMe's. Here are the results, same fio command was used: Jobs: 1 (f=1): [W(1)][100.0%][w=2230KiB/s][w=557 IOPS][eta 00m:00s] Samsung 970 EVO Plus 2TB Jobs: 1 (f=1): [W(1)][100.0%][w=274MiB/s][w=70.0k IOPS][eta...
  4. hepo

    Improve VM restore speed - ZFS datastore over NFS

    Good idea, will reach out to the HW vendor. Will also ask them to exchange the HBA330 as a next step. thanks
  5. hepo

    Improve VM restore speed - ZFS datastore over NFS

    I have done another test today - in short removed the RAID10 setup on the problematic server and re-created the pool in RAIDZ2 without cache drive. This is to ensure that I am comparing apples to apples. The results are posted in the link above. I have no idea why the R730 host delivers such...
  6. hepo

    Improve VM restore speed - ZFS datastore over NFS

    This starts to become TrueNAS performance comparison on two similar servers... I have prepared a document describing the setups and test - here (too big to upload to the forum) I welcome your review and comments! Any other tests I can perform? Thanks for your time
  7. hepo

    Improve VM restore speed - ZFS datastore over NFS

    First, result from the PVE host, this is where restore tests were done previously root@pve1:~# proxmox-backup-client benchmark SHA256 speed: 227.92 MB/s Compression speed: 369.36 MB/s Decompress speed: 1143.15 MB/s AES256/GCM speed: 2400.57 MB/s Verify speed: 319.00 MB/s...
  8. hepo

    Improve VM restore speed - ZFS datastore over NFS

    Thanks for the responses guys! Full disclosure on the setup: - we are virtualizing both TrueNAS and PBS on top of proxmox - the host is Dell R730xd with dual E5-2698 v4, 128GB RAM, HBA330 adapter where all the HDD's (6x12TB Toshiba MG07SCA12TE) are connected - The HBA card is pci passthrough'ed...
  9. hepo

    Fenced node emails, hundreds of them

    all except pve31 which is in 3rd DC and is only used for maintaining quorum. cfg file uploaded in the same location. I am happy to stop this here and would like to thank for the investigation. We are currently having performance problems with ceph (details here) and running most VMs on local...
  10. hepo

    Improve VM restore speed - ZFS datastore over NFS

    I have received zero responses - I am either unique or stupid o_O More testing happened... we added 500GB SSD as cache (L2ARC) vdev, no noticable impromevement. Moreover, here is the arcstat during restore Metadata misses are in place but insignificant, the prefetch misses are 100%. I am...
  11. hepo

    Fenced node emails, hundreds of them

    Nothing extraordinary there. Same link from my previous reply.
  12. hepo

    Fenced node emails, hundreds of them

    logs are too big to attach in the forum, hence uploaded here - link as far I see, pve12 is the one generating the events. Thanks for looking into this.
  13. hepo

    Fenced node emails, hundreds of them

    See attached file. Since I have purged the postfix queue I don't receive new emails. Yesterday when I reported this, there was nothing weird on the cluster, no VM migrations, just the email which were a lot. I am using this script to configure email -...
  14. hepo

    Fenced node emails, hundreds of them

    This appears to happen approximately every hour. pve12 appears to think it's the quorum master, but this is not correct. found 3000 messages in the postfix queue of pve12. also looks like the email server is throttling us hence may explain the "every hour phenomenon". purged the queue and...
  15. hepo

    Fenced node emails, hundreds of them

    We had an issue on one of our nodes today, it rebooted during VM migration (local-zfs volume)... this happens for the second time now. Since then I am getting hundreds emails like the following I see no HA issues, rebooted the host once again, not sure where to start... HELP
  16. hepo

    Update to 5.11.22-7-pve causes zfs issues

    Just posted the same issue on the PBS side of the forum... I would like to confirm that the new packaged fixed the "error" for me.
  17. hepo

    arcstat command now working - issue with latest patch??

    following this thread - https://forum.proxmox.com/threads/update-to-5-11-22-7-pve-causes-zfs-issues.99401/
  18. hepo

    arcstat command now working - issue with latest patch??

    root@pbs:~# arc_summary ------------------------------------------------------------------------ ZFS Subsystem Report Wed Nov 10 18:43:50 2021 Linux 5.11.22-7-pve 2.0.6-pve1 Machine: pbs (x86_64)...
  19. hepo

    arcstat command now working - issue with latest patch??

    I have access to 3 PBS servers and all of then create the same output. All 3 are vanilla installs with ZFS. Pushed a bunch of patches today and I am getting nervous. I was testing ZFS performance this morning and the command was definitely working! root@pbs:~# arcstat time read miss...
  20. hepo

    Improve VM restore speed - ZFS datastore over NFS

    And just to add for completeness... Our primary PBS server runs on dedicated hardware - 40cores, 64GB RAM, 40TB HDD ZFS pool. The ZFS cache have happily took a good 32+GB of the RAM and hence is performing really well when restores are concerned.