Search results

  1. J

    [SOLVED] Error code 11 when moving storage from zfspool to local: "no space left on device" but there is plenty of space

    Ah, I see. That, indeed, makes sense. Would making the volume bigger, say from 3 GB to 6 GB, and immediately try to move the storage from the zfspool pool to the local pool solve the issue? Also, were this to be a solution, would you like me to open a ticket on the bug tracker requesting for...
  2. J

    Debugging sporadic networking problems on LXC

    I just ran into this issue again. Yesterday night I upgraded the cluster to version 7.4-3 with the latest kernel and rebooted the nodes. Everything seems to be working fine. I just deleted the container with id 126 from my proxmox4 node and about 10 seconds later I executed my LXC provisioning...
  3. J

    [SOLVED] Error code 11 when moving storage from zfspool to local: "no space left on device" but there is plenty of space

    Thanks for your reply, Fabian. It does, indeed, compress fairly well, but not to the point where 400 GB of free space would not suffice. Do you see any value that gives a clue of what's happening in the output of the command you gave, @fabian? # zfs get all zfspool/subvol-106-disk-0 NAME...
  4. J

    [SOLVED] Error code 11 when moving storage from zfspool to local: "no space left on device" but there is plenty of space

    I would say that no, I don't have anything "special". I created the ZFS pool via the WebGUI, in mirror mode, with default values. As I said, I've seen it working about a year ago (probably 7.0 to 7.2, but don't take my word for it). It's not that big of a disk to be an issue. These are the LXC...
  5. J

    How to define levels of start order and delay for LXC

    Hey there! I have a 5-node cluster with a lot of LXC which basically fall into these categories: Common services (e.g., internal DNS) Database servers (e.g. PostgreSQL) Application servers (e.g. Python applications running with Gunicorn) Web servers (e.g. NGINX as reverse proxy for the...
  6. J

    [SOLVED] Error code 11 when moving storage from zfspool to local: "no space left on device" but there is plenty of space

    Indeed I can restore a backup of the LXC on proxmox5 if I select the zfspool storage of that node (same name, same setup, I did that on purpose). Should I open a bug at the bug tracker?
  7. J

    Debugging sporadic networking problems on LXC

    I have a 5-node cluster with LXC only. From time to time, some LXC loses connectivity. It's sporadic, sort of random even. But when it happens I am very lost at how to properly debug it to know the specific cause of it. This weekend happened again. Took the chance to upgrade the node from...
  8. J

    [SOLVED] Error code 11 when moving storage from zfspool to local: "no space left on device" but there is plenty of space

    I also tried restoring the LXC from the backup on Proxmox Backup Server (pbs1) without success. The error is the same. The node I was restoring it into has the same configuration as proxmox3, that is, a local pool and a zfspool pool. Both pools have sufficient disk space to hold a 3 GB image...
  9. J

    [SOLVED] Error code 11 when moving storage from zfspool to local: "no space left on device" but there is plenty of space

    These are the versions I am running on the node where this container is located, named proxmox3: # pveversion -v proxmox-ve: 7.3-1 (running kernel: 5.15.85-1-pve) pve-manager: 7.3-6 (running version: 7.3-6/723bb6ec) pve-kernel-helper: 7.3-4 pve-kernel-5.15: 7.3-2 pve-kernel-5.15.85-1-pve...
  10. J

    [SOLVED] Error code 11 when moving storage from zfspool to local: "no space left on device" but there is plenty of space

    Thanks for your reply, but that does not seem to be the problem. The local pool has more than 400 GB of empty space but, indeed, the local pool is just 500 GB big whereas the zfspool pool is 4 TB big. I think that it's attempting to provision more storage than needed on the local pool? Either...
  11. J

    [SOLVED] Error code 11 when moving storage from zfspool to local: "no space left on device" but there is plenty of space

    I have a 5-node cluster with LXC only and I am trying to move 3GB of storage of one of the LXC from the zfspool pool to the local pool on the same node (ext4), but I am getting this error: Formatting '/var/lib/vz/images/106/vm-106-disk-0.raw', fmt=raw size=3221225472 preallocation=off Creating...
  12. J

    Changing the IP address and subnet in all cluster nodes

    Yes, this is the private network for the hosts/nodes of the cluster to talk among themselves.
  13. J

    [SOLVED] NGINX with kTLS on unprivileged LXC on Proxmox 7.3

    Thank you very much for you insightful reply, @cheiss. Yesterday night I managed to get it working, but I had not had the chance to post a reply here. So, what I did in the end was to install Ubuntu 22.04 (Proxmox template) on a new (unprivileged, as always) LXC, as it ships with OpenSSL 3.0.2...
  14. J

    [SOLVED] NGINX with kTLS on unprivileged LXC on Proxmox 7.3

    Good day everyone! I have a 5-node cluster on PVE 7.3-6 with a couple hundred unprivileged LXC, all using the Debian 11 Bullseye template. I was looking at how Netflix can serve 800Gb/s of TLS encrypted video content from a single server, and a large part of it appears to be kTLS. I also saw...
  15. J

    Disabling OTP renders login to WebGUI unusable

    For future reference, I went through the process again and this is a screenshot of what is being shown when trying to log in if TFA is not enabled (because you created a OTP but then unchecked the Enabled mark): To restore the previous status I followed these steps: Moved...
  16. J

    Display latency among nodes

    I didn't know about that one, but yes, that would be a useful tool. Ideally integrated within what Datacenter: Cluster menu option currently offers. Thanks for your feedback!
  17. J

    Display latency among nodes

    Hey everyone! Is there a way to display the latency among nodes, meaning from the node you currently are in (either console or WebGUI) to the rest of the nodes of the cluster? Like running the ping command to each node and displaying the end result together in a quick, useful and fashionably...
  18. J

    Changing the IP address and subnet in all cluster nodes

    Good evening everyone! I have a production cluster with 5 nodes. All of the nodes have one network card with: A dedicated public IP address on eno1. A vmbr4001 bridge for the public subnet of the guests (LXC). A vmbr4002 bridge for the private subnet of the guests (LXC). A vmbr4003 bridge for...
  19. J

    Disabling OTP renders login to WebGUI unusable

    Hey everone! So I had 2FA enabled for my root@pam user and wanted to add a new node to an existing cluster. Last time I had to do that (a couple of minor versions ago, if I recall correctly) I had to remove 2FA, then re-create it after the node was added and set up. So I am now on Proxmox...
  20. J

    Disable OTP via Console

    I was experiencing the same problem on version 7.3 (disabled 2FA of my root user because I wanted to add a new node to the cluster to be enabled afterwards) and was looking for the file. Found it at /etc/pve/priv/tfa.cfg. I move the file to .bak and I could add the node. Then restored it.