PveProxy : No space left on device - problem with Packer

[zwindler]

Hello,

I'm trying to use Packer to build an image for Proxmox VE following this guide https://image-builder.sigs.k8s.io/capi/providers/proxmox

I'm getting an error when trying to upload the ISO

==> proxmox-iso.ubuntu-2204: Post "https://192.168.1.91:8006/api2/json/nodes/node02/storage/cephfs/upload": write tcp 192.168.1.159:46906->192.168.1.91:8006: use of closed network connection
Build 'proxmox-iso.ubuntu-2204' errored after 1 second 142 milliseconds: Post "https://192.168.1.91:8006/api2/json/nodes/node02/storage/cephfs/upload": write tcp 192.168.1.159:46906->192.168.1.91:8006: use of closed network connection

On the server I get this

root@node02:/var/log/pveproxy# journalctl -u pveproxy -u pvedaemon -f
Jul 26 11:00:42 node02 pveproxy[1249]: worker 1250 finished
Jul 26 11:00:42 node02 pveproxy[1249]: starting 1 worker(s)
Jul 26 11:00:42 node02 pveproxy[1249]: worker 28905 started
Jul 26 11:07:10 node02 pveproxy[28040]: problem with client ::ffff:192.168.1.159; No space left on device
Jul 26 11:09:06 node02 pveproxy[28905]: problem with client ::ffff:192.168.1.159; No space left on device

There is no storage issue on any host (neither client nor server)

Others seem to have had this error before.

Especially https://forum.proxmox.com/threads/pveproxy-no-space-left-on-device-problem-with-storage-api.113451/ which also uses Packer

And there was a bug that was supposedly fixed https://bugzilla.proxmox.com/show_bug.cgi?id=4344

But I'm running 8.2.4 so it shouldn't be an issue on my environment.

I've tried multiple hosts, multiple storages, both token and password authentication (in case it was an authentication error, which I can't rule out but also don't know how to check)

How can I find out more to debug this?

 

[thefu21]

Couldn't you just use the auto installer? [1]

[1] https://pve.proxmox.com/wiki/Automated_Installation

 

[zwindler]

I really don't see how that helps me in my usecase

I'm using packer through the image-builder kubernetes project

 

[zwindler]

Update: For some reason it does seem to work when using root@pam + password, rather than a custom user with custom role + token

So it may be either an issue with packer to "image-builder@pve!capi" authentication (but it doesn't throw a 401 for some reason) OR a permission issue but the role is already pretty wide

pveum user add image-builder@pve --password REDACTED

pveum role add image-builder-role -privs "Datastore.Allocate Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit SDN.Allocate SDN.Audit SDN.Use Sys.AccessNetwork Sys.Audit VM.Allocate VM.Audit VM.Backup VM.Clone VM.Config.CDROM VM.Config.CPU VM.Config.Cloudinit VM.Config.Disk VM.Config.HWType VM.Config.Memory VM.Config.Network VM.Config.Options VM.Console VM.Migrate VM.Monitor VM.PowerMgmt VM.Snapshot VM.Snapshot.Rollback"
pveum aclmod / -user image-builder@pve -role image-builder-role

pveum user token add image-builder@pve capi