something is a bit off kilter with one of our pve hosts, where snapshots and outbound migrations are slow, but inbound migrations are normal. to elaborate, inbound migrations traverse the network at ~5.5 gbps; which is about the normal rate those cpu's do ssh tunnels. that would be fine. outbound migrations, however, are capped at around 700-800 mbps. doing two at a time brings it up slightly, but not double.
maybe someone can glance over the following and point me in the right direction:
pveversion output:
here is what a snapshot of a vm with 16 gb ram looks like. (besides taking forever):
this shows that the hard drives are writing at a decent rate:
hardware config:
intel xeon 3465x cpu
384 gb ddr5-4800 ram
nvme gen5 raid10 on zfs for vm storage/snapshot/migration destination.
is there anything else i can check that could be causing this?
maybe someone can glance over the following and point me in the right direction:
pveversion output:
Code:
pve-manager/8.4.19/a68fb383814bb1e6 (running kernel: 6.8.12-24-pve)here is what a snapshot of a vm with 16 gb ram looks like. (besides taking forever):
Code:
saving VM state and RAM using storage 'nvme5'
1.58 KiB in 0s
1.72 GiB in 1s
3.44 GiB in 2s
5.17 GiB in 3s
6.92 GiB in 4s
8.65 GiB in 5s
9.95 GiB in 6s
11.71 GiB in 7s
13.41 GiB in 8s
14.34 GiB in 9s
14.43 GiB in 10s
14.50 GiB in 11s
14.50 GiB in 12s
14.50 GiB in 13s
14.50 GiB in 14s
14.50 GiB in 15s
14.50 GiB in 16s
14.50 GiB in 17s
14.50 GiB in 18s
14.50 GiB in 19s
14.50 GiB in 20s
14.50 GiB in 21s
14.50 GiB in 22s
14.50 GiB in 23s
14.62 GiB in 24s
14.67 GiB in 25s
14.71 GiB in 26s
14.75 GiB in 27s
14.80 GiB in 28s
14.85 GiB in 29s
14.90 GiB in 30s
14.94 GiB in 31s
14.99 GiB in 32s
15.02 GiB in 33s
15.12 GiB in 34s
15.17 GiB in 35s
15.23 GiB in 36s
15.25 GiB in 37s
15.28 GiB in 38s
15.28 GiB in 39s
15.29 GiB in 40s
15.30 GiB in 41s
15.30 GiB in 42s
15.33 GiB in 43s
15.36 GiB in 44s
15.37 GiB in 45s
15.37 GiB in 46s
15.39 GiB in 47s
15.41 GiB in 48s
15.44 GiB in 49s
15.47 GiB in 50s
15.47 GiB in 51s
15.50 GiB in 52s
15.51 GiB in 53s
15.52 GiB in 54s
15.54 GiB in 55s
15.54 GiB in 56s
15.56 GiB in 57s
15.57 GiB in 58s
15.58 GiB in 59s
reducing reporting rate to every 10s
15.65 GiB in 1m 9s
15.85 GiB in 1m 20s
16.06 GiB in 1m 30s
16.20 GiB in 1m 40s
16.31 GiB in 1m 50s
16.51 GiB in 2m
16.59 GiB in 2m 10s
16.75 GiB in 2m 20s
16.91 GiB in 2m 30s
17.04 GiB in 2m 40s
17.15 GiB in 2m 50s
17.42 GiB in 3m
17.53 GiB in 3m 10s
17.68 GiB in 3m 21s
17.96 GiB in 3m 31s
18.12 GiB in 3m 41s
18.18 GiB in 3m 51s
18.33 GiB in 4m 1s
18.59 GiB in 4m 11s
18.76 GiB in 4m 21s
18.96 GiB in 4m 31s
19.11 GiB in 4m 41s
19.34 GiB in 4m 51s
19.52 GiB in 5m 1s
19.73 GiB in 5m 11s
19.92 GiB in 5m 22s
20.19 GiB in 5m 32s
20.38 GiB in 5m 42s
20.57 GiB in 5m 52s
20.88 GiB in 6m 2s
21.27 GiB in 6m 12s
21.52 GiB in 6m 22s
21.78 GiB in 6m 32s
21.98 GiB in 6m 42s
22.23 GiB in 6m 52s
22.53 GiB in 7m 2s
22.70 GiB in 7m 12s
22.98 GiB in 7m 22s
23.33 GiB in 7m 33s
23.53 GiB in 7m 43s
23.76 GiB in 7m 53s
23.97 GiB in 8m 3s
24.43 GiB in 8m 13s
25.01 GiB in 8m 23s
25.52 GiB in 8m 33s
26.09 GiB in 8m 43s
26.50 GiB in 8m 53s
26.76 GiB in 9m 3s
26.96 GiB in 9m 13s
27.04 GiB in 9m 23s
27.15 GiB in 9m 33s
27.55 GiB in 9m 43s
27.71 GiB in 9m 53s
28.03 GiB in 10m 3s
28.14 GiB in 10m 13s
28.42 GiB in 10m 23s
28.63 GiB in 10m 33s
28.74 GiB in 10m 44s
28.94 GiB in 10m 54s
29.03 GiB in 11m 4s
29.18 GiB in 11m 14s
29.43 GiB in 11m 24s
29.53 GiB in 11m 34s
29.73 GiB in 11m 44s
29.90 GiB in 11m 54s
30.03 GiB in 12m 4s
30.16 GiB in 12m 14s
30.30 GiB in 12m 24s
30.34 GiB in 12m 34s
30.42 GiB in 12m 44s
completed saving the VM state in 12m 54s, saved 32.44 GiB
snapshotting 'drive-scsi0' (nvme5:vm-150-disk-1)
snapshotting 'drive-efidisk0' (nvme5:vm-150-disk-0)
TASK OKthis shows that the hard drives are writing at a decent rate:
hardware config:
intel xeon 3465x cpu
384 gb ddr5-4800 ram
nvme gen5 raid10 on zfs for vm storage/snapshot/migration destination.
is there anything else i can check that could be causing this?
Last edited: