[SOLVED] Error "delete user failed: No buffer space available (500)" when trying to delete PVE user

[liddack]

I have a 3-node cluster and yesterday I performed an upgrade in all nodes from PVE 7.4 to PVE 8.0.4. The upgrade went well and all guests are running without issues.
I use WHMCS with the Proxmox VPS for WHMCS module to sell virtualization products in my website.
Today I tried to destroy a LXC-based service using the "Terminate" module action but it couldn't, stating the error message:

Delete user failed: No buffer space available

So I logged in the web GUI to attempt to manually delete the user automatically created by the module when the service was created, but a similar error message appeared:

delete user failed: No buffer space available (500)

In my last attempt I tried to run the command pveum user delete "proxmoxVPS_<user>@pve" on the console and the same error appeared:

❯ pveum user delete "proxmoxVPS_<user>@pve"
delete user failed: No buffer space available

I couldn't find much about this issue. I read in a related issue post that restarting pvestatd before trying again would help, but unfortunately it was not my case.

All three nodes are fully updated:

❯ pveversion 
pve-manager/8.0.4/d258a813cfa6b390 (running kernel: 6.2.16-15-pve)

 

[liddack]

UPDATE: I am also unable to create new users - PVE shows the same error message.
This means I'm currently unable to sell new products in my website since the Proxmox module for WHMCS always creates a new user for each created service.

 

[BobhWasatch]

Is your root filesystem maybe full? What is the output of "df -h"?

 

[liddack]

Is your root filesystem maybe full? What is the output of "df -h"?

Node 1:

root@razor01 /
❯ df -h
Filesystem                             Size  Used Avail Use% Mounted on
udev                                   126G     0  126G   0% /dev
tmpfs                                   26G  2.3M   26G   1% /run
/dev/mapper/pve-root                    94G   16G   74G  18% /
tmpfs                                  126G   66M  126G   1% /dev/shm
tmpfs                                  5.0M     0  5.0M   0% /run/lock
/dev/nvme0n1p2                         511M  312K  511M   1% /boot/efi
/dev/mapper/pve-hs--data                98G   12G   82G  12% /mnt/hs-data
teste-ceph-100g                        768G  128K  768G   1% /teste-ceph-100g
teste2-ceph-100g                       1.7T  128K  1.7T   1% /teste2-ceph-100g
/dev/fuse                              128M  2.0M  127M   2% /etc/pve
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-3
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-0
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-1
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-6
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-4
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-2
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-7
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-5
10.20.30.11,10.20.30.12,10.20.30.13:/  9.0T  712G  8.3T   8% /mnt/pve/images
tmpfs                                   26G     0   26G   0% /run/user/0

Node 2:

root@razor02 /
❯ df -h
Filesystem                             Size  Used Avail Use% Mounted on
udev                                   126G     0  126G   0% /dev
tmpfs                                   26G   31M   26G   1% /run
/dev/mapper/pve-root                    94G   14G   76G  16% /
tmpfs                                  126G   60M  126G   1% /dev/shm
tmpfs                                  5.0M     0  5.0M   0% /run/lock
/dev/nvme0n1p2                         511M  328K  511M   1% /boot/efi
/dev/mapper/pve-hs--data                98G   67G   27G  72% /mnt/hs-data
teste2-ceph-100g                       1.7T  128K  1.7T   1% /teste2-ceph-100g
teste-ceph-100g                        768G  128K  768G   1% /teste-ceph-100g
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-12
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-15
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-13
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-11
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-10
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-9
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-14
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-8
10.20.30.11,10.20.30.12,10.20.30.13:/  9.0T  712G  8.3T   8% /mnt/pve/images
/dev/fuse                              128M  2.0M  127M   2% /etc/pve
tmpfs                                   26G     0   26G   0% /run/user/0

Node 3:

root@razor03 /
❯ df -h
Filesystem                             Size  Used Avail Use% Mounted on
udev                                   126G     0  126G   0% /dev
tmpfs                                   26G   31M   26G   1% /run
/dev/mapper/pve-root                    94G   25G   65G  28% /
tmpfs                                  126G   66M  126G   1% /dev/shm
tmpfs                                  5.0M     0  5.0M   0% /run/lock
/dev/nvme1n1p2                         509M  6.0M  503M   2% /boot/efi
/dev/mapper/pve-hs--data                98G   21G   73G  23% /mnt/hs-data
teste2-ceph-100g                       1.7T  128K  1.7T   1% /teste2-ceph-100g
teste-ceph-100g                        768G  128K  768G   1% /teste-ceph-100g
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-19
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-18
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-17
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-16
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-23
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-20
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-22
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-21
10.20.30.11,10.20.30.12,10.20.30.13:/  9.0T  712G  8.3T   8% /mnt/pve/images
/dev/fuse                              128M  2.0M  127M   2% /etc/pve
tmpfs                                   26G     0   26G   0% /run/user/0

 

[liddack]

[UPDATE] TL;DR: I discovered the /etc/pve/priv/shadow.cfg file got corrupted.

I decided to inspect the Proxmox source code to try to find the root cause.

I did a google search to find out which package pveum belongs to, and I found libpve-access-control.
I opened the directory /usr/share/perl5/PVE inside one of the nodes using VSCode and searched for the error message in the context of deleting an user.



After (guess-)tracing function calls and using prints, I found that the line my $shadow_cfg = cfs_read_file($shadowconfigfile); (/usr/share/perl5/PVE/Auth/PVE.pm:108) is the one that generates the error.

Knowing that the content of $shadowconfigfile is priv/shadow.cfg, I inspected the file using less -M /etc/pve/priv/shadow.cfg and found the following (usernames and hashes are blurred):



Also, the file size is 1 MiB.

root@razor01 /
❯ ls -lh /etc/pve/priv/shadow.cfg
-rw------- 1 root www-data 1.0M Oct 12 13:56 /etc/pve/priv/shadow.cfg

I think it means the file got corrupted somehow during the upgrade.

Currently I don't have a recent backup of the root partition to recover the previous content of this file. Will try to search for older ones to try to recover it.
I already know that the contents of the /etc/pve directory (pmxcfs) is stored in the /var/lib/pve-cluster/config.db file.

 

[liddack]

From a months ago backup, I managed to recreate the shadow.cfg file (although incomplete) and now the error stopped. Now I'm able to delete and create users again.

 

[vector]

 

[vector]

 

[vector]

I encountered a similar problem, ssh can log in, but the WEB UI login failed, prompting no buffer space available, I found that tfa.cfg is also damaged, I don't know why, I deleted tfa.cfg and can log in successfully

 

[Dzung]

I also have this error, anyone have solution to recreate this file

 

[CarstenMartens]

Hallo,

same problem with tfa.cfg here too. Restore from another node helped for us

 

[lkiesow]

Same problem here. The tfa.cfg got corrupted:

# tail -c1000 /etc/pve/priv/tfa.cfg
ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ
 ÂÃÂÃÂÃÂÃÂÃ

 

[lkiesow]

Looking a bit closer at the data, it seems that the corruption begins at a UTF-8 character in the description of a user's TFA config:

...","description":"Zweifaktor fÃ<U+008<U+0083>Â<U+0083>Ã<U+0082>Â<U+0083>Ã<U+0083>Â<U+00

The description probably was something like “Zweifaktor für Nutzer XY”. Maybe the UTF-8 “ü” got parsed as something like latin1 which messed up the JSON?

 

[JohanKarlsson]

I've also found that this can happen with the VM config files themselves in the cipassword field.

 

[luckystrike]

I encountered a similar problem, ssh can log in, but the WEB UI login failed, prompting no buffer space available, I found that tfa.cfg is also damaged, I don't know why, I deleted tfa.cfg and can log in successfully

same problem, delete tfa.cfg， log in syccessfully.

 

[garnoux]

same here, pve 8.1.3