[SOLVED] Error "delete user failed: No buffer space available (500)" when trying to delete PVE user

liddack

Member
Mar 12, 2021
6
1
8
28
I have a 3-node cluster and yesterday I performed an upgrade in all nodes from PVE 7.4 to PVE 8.0.4. The upgrade went well and all guests are running without issues.
I use WHMCS with the Proxmox VPS for WHMCS module to sell virtualization products in my website.
Today I tried to destroy a LXC-based service using the "Terminate" module action but it couldn't, stating the error message:

Code:
Delete user failed: No buffer space available

So I logged in the web GUI to attempt to manually delete the user automatically created by the module when the service was created, but a similar error message appeared:

Code:
delete user failed: No buffer space available (500)

In my last attempt I tried to run the command pveum user delete "proxmoxVPS_<user>@pve" on the console and the same error appeared:

Code:
❯ pveum user delete "proxmoxVPS_<user>@pve"
delete user failed: No buffer space available

I couldn't find much about this issue. I read in a related issue post that restarting pvestatd before trying again would help, but unfortunately it was not my case.

All three nodes are fully updated:
Code:
❯ pveversion 
pve-manager/8.0.4/d258a813cfa6b390 (running kernel: 6.2.16-15-pve)
 
Last edited:
UPDATE: I am also unable to create new users - PVE shows the same error message.
This means I'm currently unable to sell new products in my website since the Proxmox module for WHMCS always creates a new user for each created service.
 
Is your root filesystem maybe full? What is the output of "df -h"?

Node 1:
Code:
root@razor01 /
❯ df -h
Filesystem                             Size  Used Avail Use% Mounted on
udev                                   126G     0  126G   0% /dev
tmpfs                                   26G  2.3M   26G   1% /run
/dev/mapper/pve-root                    94G   16G   74G  18% /
tmpfs                                  126G   66M  126G   1% /dev/shm
tmpfs                                  5.0M     0  5.0M   0% /run/lock
/dev/nvme0n1p2                         511M  312K  511M   1% /boot/efi
/dev/mapper/pve-hs--data                98G   12G   82G  12% /mnt/hs-data
teste-ceph-100g                        768G  128K  768G   1% /teste-ceph-100g
teste2-ceph-100g                       1.7T  128K  1.7T   1% /teste2-ceph-100g
/dev/fuse                              128M  2.0M  127M   2% /etc/pve
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-3
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-0
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-1
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-6
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-4
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-2
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-7
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-5
10.20.30.11,10.20.30.12,10.20.30.13:/  9.0T  712G  8.3T   8% /mnt/pve/images
tmpfs                                   26G     0   26G   0% /run/user/0

Node 2:
Code:
root@razor02 /
❯ df -h
Filesystem                             Size  Used Avail Use% Mounted on
udev                                   126G     0  126G   0% /dev
tmpfs                                   26G   31M   26G   1% /run
/dev/mapper/pve-root                    94G   14G   76G  16% /
tmpfs                                  126G   60M  126G   1% /dev/shm
tmpfs                                  5.0M     0  5.0M   0% /run/lock
/dev/nvme0n1p2                         511M  328K  511M   1% /boot/efi
/dev/mapper/pve-hs--data                98G   67G   27G  72% /mnt/hs-data
teste2-ceph-100g                       1.7T  128K  1.7T   1% /teste2-ceph-100g
teste-ceph-100g                        768G  128K  768G   1% /teste-ceph-100g
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-12
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-15
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-13
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-11
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-10
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-9
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-14
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-8
10.20.30.11,10.20.30.12,10.20.30.13:/  9.0T  712G  8.3T   8% /mnt/pve/images
/dev/fuse                              128M  2.0M  127M   2% /etc/pve
tmpfs                                   26G     0   26G   0% /run/user/0

Node 3:
Code:
root@razor03 /
❯ df -h
Filesystem                             Size  Used Avail Use% Mounted on
udev                                   126G     0  126G   0% /dev
tmpfs                                   26G   31M   26G   1% /run
/dev/mapper/pve-root                    94G   25G   65G  28% /
tmpfs                                  126G   66M  126G   1% /dev/shm
tmpfs                                  5.0M     0  5.0M   0% /run/lock
/dev/nvme1n1p2                         509M  6.0M  503M   2% /boot/efi
/dev/mapper/pve-hs--data                98G   21G   73G  23% /mnt/hs-data
teste2-ceph-100g                       1.7T  128K  1.7T   1% /teste2-ceph-100g
teste-ceph-100g                        768G  128K  768G   1% /teste-ceph-100g
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-19
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-18
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-17
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-16
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-23
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-20
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-22
tmpfs                                  126G   28K  126G   1% /var/lib/ceph/osd/ceph-21
10.20.30.11,10.20.30.12,10.20.30.13:/  9.0T  712G  8.3T   8% /mnt/pve/images
/dev/fuse                              128M  2.0M  127M   2% /etc/pve
tmpfs                                   26G     0   26G   0% /run/user/0
 
[UPDATE] TL;DR: I discovered the /etc/pve/priv/shadow.cfg file got corrupted.

I decided to inspect the Proxmox source code to try to find the root cause.

I did a google search to find out which package pveum belongs to, and I found libpve-access-control.
I opened the directory /usr/share/perl5/PVE inside one of the nodes using VSCode and searched for the error message in the context of deleting an user.

Screenshot_20231015_171916.png

After (guess-)tracing function calls and using prints, I found that the line my $shadow_cfg = cfs_read_file($shadowconfigfile); (/usr/share/perl5/PVE/Auth/PVE.pm:108) is the one that generates the error.

Knowing that the content of $shadowconfigfile is priv/shadow.cfg, I inspected the file using less -M /etc/pve/priv/shadow.cfg and found the following (usernames and hashes are blurred):

Screenshot_20231015_171535.png

Also, the file size is 1 MiB.

Code:
root@razor01 /
❯ ls -lh /etc/pve/priv/shadow.cfg
-rw------- 1 root www-data 1.0M Oct 12 13:56 /etc/pve/priv/shadow.cfg

I think it means the file got corrupted somehow during the upgrade.

Currently I don't have a recent backup of the root partition to recover the previous content of this file. Will try to search for older ones to try to recover it.
I already know that the contents of the /etc/pve directory (pmxcfs) is stored in the /var/lib/pve-cluster/config.db file.
 
  • Like
Reactions: linux
From a months ago backup, I managed to recreate the shadow.cfg file (although incomplete) and now the error stopped. Now I'm able to delete and create users again.
 
I encountered a similar problem, ssh can log in, but the WEB UI login failed, prompting no buffer space available, I found that tfa.cfg is also damaged, I don't know why, I deleted tfa.cfg and can log in successfully
 
  • Like
Reactions: luckystrike
Hallo,

same problem with tfa.cfg here too. Restore from another node helped for us
 
Same problem here. The tfa.cfg got corrupted:

Code:
# tail -c1000 /etc/pve/priv/tfa.cfg
ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ
 ÂÃÂÃÂÃÂÃÂÃ
 
Looking a bit closer at the data, it seems that the corruption begins at a UTF-8 character in the description of a user's TFA config:

Code:
...","description":"Zweifaktor fÃ<U+008<U+0083>Â<U+0083>Ã<U+0082>Â<U+0083>Ã<U+0083>Â<U+00

The description probably was something like “Zweifaktor für Nutzer XY”. Maybe the UTF-8 “ü” got parsed as something like latin1 which messed up the JSON?
 
  • Like
Reactions: linux
I encountered a similar problem, ssh can log in, but the WEB UI login failed, prompting no buffer space available, I found that tfa.cfg is also damaged, I don't know why, I deleted tfa.cfg and can log in successfully
same problem, delete tfa.cfg, log in syccessfully.
1702801925154.png
1702801900589.png
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!