Search results

  1. D

    [SOLVED] Ceph - Schedule deep scrubs to prevent service degradation

    Updated commands to view scrubbing distribution: [root@kvm1 ~]# # By Day of week: [root@kvm1 ~]# for date in `ceph pg dump 2> /dev/null | grep active | awk '{print $23}' | cut -dT -f1`; do date +%A -d $date; done | sort | uniq -c; 30 Friday 79 Monday 101 Saturday 112 Sunday...
  2. D

    [SOLVED] Ceph - Schedule deep scrubs to prevent service degradation

    Hi Rob, Been upgrading all non-clustered PVE nodes and completed our first cluster upgrade which included moving from Nautilus to Octopus today. The 'ceph pg dump' output has changed yet again, although all that happened this time is that the space between the date and time has been replaced...
  3. D

    Cluster unavailable after adding about 15 Nodes

    Should it help you, or anyone else: Herewith an example of converting a host using the Linux bridge to OvS (Open vSwitch). Original Linux bridge: auto lo iface lo inet loopback auto bond0 iface bond0 inet manual slaves eth0,eth1 bond_miimon 100...
  4. D

    Proxmox Extremely slow recovery & Ceph

    Balance-rr round robins packet delivery which will 100% lead to packets arriving at the destination of of order. Not necessarily because the cables are different lengths, simply due to packets arriving on different NICs and being collected by separate threads running on differently loaded cores...
  5. D

    Cluster unavailable after adding about 15 Nodes

    vlan1 is untagged for VM traffic on 1st LACP which runs balance-slb, to avoid double processing packets to perfectly hash outgoing data on the slave interfaces. We run primarily virtual firewalls and routers on this cluster so packets per second with low latency is our priority. Intel based...
  6. D

    Cluster unavailable after adding about 15 Nodes

    Try changing your Corosync configuration to being a little less trigger happy. Herewith a working sample: [admin@kvm5a ~]# cat /etc/pve/corosync.conf logging { debug: off to_syslog: yes } nodelist { node { name: kvm5a nodeid: 1 quorum_votes: 1 ring0_addr: 10.254.1.2 }...
  7. D

    Proxmox Extremely slow recovery & Ceph

    Also, don't run balance-rr... Use a hashing based director where traffic for streams follow a consistent path...
  8. D

    VirtIO vs SCSI

    Yes and no... Writes are buffered but flushes are honoured so it would be transactionally safe. Only 'unsafe' ignores flushes...
  9. D

    Ceph RBD space reclamation

    Is the following a known bug in Ceph Nautilus v14.2.9? Running kernel RBD guest with partitions aligned to 1 MiB boundaries. If I create a 10 GiB file in a VM, delete it and then issue fstrim I get inconsistent feedback on the image space allocation: After having run 'dd if=/dev/urandom...
  10. D

    VMs remounting partition read-only and (Buffer) I/O errors since qemu 3.0

    I believe changes in kernel 5.4 now time out or return errors and may subsequently log problems whilst it went unnoticed previously. A relatively slow disk only sandbox cluster exhibits these errors occasionally with valid partition alignment but other systems have stopped logging these errors...
  11. D

    VMs remounting partition read-only and (Buffer) I/O errors since qemu 3.0

    We have also identified this as a problem since upgrading from kernel 5.3.18-3-pve to either 5.4.35-1-pve or 5.4.41-1-pve We operate 10 Ceph clusters and have the following setup on all of them: Ceph Nautilus 14.2.9 Kernel RBD with the following features enabled on each and every single image...
  12. D

    PVE 6.2 - CephFS - Problems mounting /var/lib/vz via fstab

    We really appreciate the flexibility Ceph provides and typically setup our clusters to use sparse RBD images with templates residing in a Ceph file system concurrently mounted on all our nodes. Since PVE 6.2 we are unable to mount CephFS via fstab as it says 'nonempty' is an unknown parameter...
  13. D

    [SOLVED] PVE 6.2 - Unable to start nested virtualisation guest

    Hi Wolfgang, Setting the CPU type as host and removing the '-args' line works perfectly. Can't migrate nested virtual guests anyway so those hoops I was jumping through were unnecessary. We restrict the CPU type for other guests to ensure they are able to migrate during maintenance. Many...
  14. D

    [SOLVED] PVE 6.2 - Unable to start nested virtualisation guest

    Hi Wolfgang, Herewith the output of the command: [admin@kvm1d ~]# qm showcmd 105 --pretty /usr/bin/kvm \ -id 105 \ -name pve-test \ -chardev 'socket,id=qmp,path=/var/run/qemu-server/105.qmp,server,nowait' \ -mon 'chardev=qmp,mode=control' \ -chardev...
  15. D

    [SOLVED] PVE 6.2 - Unable to start nested virtualisation guest

    Have a nested virtualisation PVE guest that has stopped working since upgrading to PVE 6.2 [admin@kvm1d ~]# cat /sys/module/kvm_intel/parameters/nested Y I temporarily remove the 'args' line from the VM configuration file, start the guest to record the 'cpu' parameters passed to the VM, shut...
  16. D

    Proxmox 5.0 and OVS with 'dot1q-tunnel'

    Hi Pieteras, You will most probably find that the following provides the wanted features: https://pve.proxmox.com/pve-docs/chapter-pvesdn.html
  17. D

    Proxmox 5.0 and OVS with 'dot1q-tunnel'

    libpve-common-perl 6.1 has a new format, herewith appropriate patches to provide dot1q-tunnel mode on the OvS bridge port that the virtual machine's network card attaches to: --- /usr/share/perl5/PVE/Network.pm.orig 2020-05-08 16:54:14.734230861 +0200 +++ /usr/share/perl5/PVE/Network.pm...
  18. D

    [SOLVED] Problem with new OvS commands in libpve-common-perl 6.1-1

    Hi, Updated libpve-common-perl package contains a restructured /usr/share/perl5/PVE/Network.pm script which has problems when a VM's network interface is tagged and trunked. Error message: () ovs-vsctl: "trunks" is not a valid integer or range can't add ovs port 'tap101i0' - command...
  19. D

    Any Proxmox Ceph Users Interested in helping test Benji

    Never mentioned that '/usr/local/bin/benji-backup.sh' is nothing more than a direct copy of the following: https://github.com/elemental-lf/benji/blob/master/scripts/ceph.sh