Search results

  1. spirit

    Ceph - VM with high IO wait

    please don't double post, https://forum.proxmox.com/threads/applying-pve-qemu-kvm-10-2-1-1-may-cause-extremely-high-%E2%80%9Ci-o-delay%E2%80%9D-and-extremely-high-%E2%80%9Ci-o-pressure-stalls%E2%80%9D-patches-in-the-test-repository.182186/page-2 the thread here is about Hi, please don't...
  2. spirit

    Ceph - VM with high IO wait

    the thing is that the host itself don't mount the rbd. It's done directly by each qemu process. the librbd is inside qemu, and this is the qemu process which is doing to connections to monitor && osd. (with kbrd it's different, as it's done by the host, but I don't have any experience with it...
  3. spirit

    EVPN SDN Multi-tenancy: Something similar to Inter-AS Option A?

    if you have a vyos router, the best way is to peer with evpn with other proxmox nodes. (1 peer by vrf. 1 proxmox each zone is a vrf). if your coke tenant is a layer2 behind vyos, vyos should announce their ip/mac inside the evpn zone.
  4. spirit

    Ceph - VM with high IO wait

    It's really done when the qemu process is starting when librbd is initialized. (like other tuning that you can do in ceph.conf). It could be possible to pass params to qemu commandline, but qemu rbd driver have limitations of parameters that you can pass to the command line, so the most clean...
  5. spirit

    [SOLVED] VXLAN ARP timeouts

    I 'm not sure that hook can work with live migration, as the arp entry need to be set after source vm stop (the arp entry need to be flushed from source horst) and before the resume of the target vm. This is a thing that we should implemented officially. (like we already do for mac address with...
  6. spirit

    I/O stall while backup to PBS is running and fdatasync failure after backup fails

    try to enable fleecing option in backup scheduling advanced option. when backup is running, the new writes are going first to the pbs storage and that can give you slowdown if the pbs is slow or have more lantency or have limited bandwidth. The fleecing option is creating a temporary disk on a...
  7. spirit

    Ceph - VM with high IO wait

    If can really confirm that memory allocator impact librbd latency. you can with a simple fio 4k randread or write. my last qemu patch from 2023 : https://lists.proxmox.com/pipermail/pve-devel/2023-May/056815.html default malloc: 60k iops 4k randread tcmalloc : 90k iops 4k randread (can be...
  8. spirit

    live migration: ram_save_setup failed: Input/output error

    for hardening, you can do something like DisableForwarding yes Match User root Address <<10.0.10.0/24,10.0.20.0/24>> DisableForwarding no AllowTcpForwarding yes X11Forwarding no AllowAgentForwarding no PermitTunnel no (replace network address by your proxmox management subnet)
  9. spirit

    Corosync link flapping with 3 nodes

    So, here, at 22:04, why is your link down ? node reboot ? if not, you have clearly a problem with nic , the cable, or the switch port[/spoiler]
  10. spirit

    Corosync link flapping with 3 nodes

    corosync mtu is always lower than real mtu. in my production, I have also 1397 pmtud with 1500 mtu on nic Feb 19 11:05:20 corosync[23618]: [KNET ] pmtud: PMTUD link change for host: 4 link: 0 from 469 to 1397
  11. spirit

    NAS iSCSI LUN and LVM-Shared on top - bad IO while having VM delete ongoing

    the current code is my $secure_delete_cmd = sub { my ($lvmpath) = @_; my $stepsize = $scfg->{'saferemove-stepsize'} // 32; $stepsize = $stepsize * 1024 * 1024; my $bdev = abs_path($lvmpath); my $sysdir = undef; if ($bdev && $bdev =~...
  12. spirit

    Ceph - VM with high IO wait

    Hi Tchaikov, I think that the user setup is hyperconverged with proxmox/ceph on the 3 nodes. So, I think that the rbd client is able to handle it without hook . could you confirm this ? (I have seen other proxmox users doing it in hyperconverged, and it seem to works fine) I would like to...
  13. spirit

    Corosync link flapping with 3 nodes

    it's look like to nic of the 3th is going down/up or flapping. nic driver bug ? maybe bad cable ? do you have any kernel log on the 3th node ? #dmesg ? maybe also try without bonding/lacp with 2 corosync links
  14. spirit

    Proxmox/Ceph - Disable OSD rebalancing

    I think what you want is to simply enable noout flag or set mon_osd_down_out_interval to 1hour instead 10min
  15. spirit

    ZFS mirror on 2x Crucial T705 (PCIe 5.0) causing txg_sync hangs under write load – no NVMe errors in dmesg

    10year old ceph blog, also apply for zfs https://ceph.io/en/news/blog/2014/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ don't expect any good performance with consumer...
  16. spirit

    Cluster retransmit issues

    About spanning-tree, you should really disable it on your physical switch port of for your proxmox nodes. a spanning tree convergence can happen on host reboot and broke the whole cluster for some second. you don't need change knet_mtu, it's auto-compute by corosync. it could be a bug with the...
  17. spirit

    Ceph - VM with high IO wait

    Strange that you also have high memory pressure "PSI some memory". do you have enable numa option on the vm ? you can also look at host numa stat # apt install numactl # numstat and look if you don't have a lot of "numa_miss" vs "numa_hit" on rbd side, you can also give a try to krbd vs...
  18. spirit

    Cluster retransmit issues

    can you send corosync log of each node ? (journalctl -u corosync). is the nic for corosync link dedicated ? or do you have vm,storage,backup,...running on it too ? (no bandwidth saturation ? ) no spanning tree on the network ? do you use bonding or not ?
  19. spirit

    Interest in VPP (Vector Packet Processing) as a dataplane option for Proxmox

    firewall=1 is create a linux bridge fwbr interface, so it's not usable here. Implementing the whole firewall code in vpp is another thing. (and currently proxmox code don't have any easy plugin code to implement different firewall)
  20. spirit

    How to disable the bridge level firewall in PVE 9.1.1

    So, why do you want to use pve-firewall ? (I mean, disable pve-firewall service or uncheck firewall checkbox on the vms) I'll try to look at proxmox-firewall code, but it shouldn't be needed