PveProxy : No space left on device - problem with Packer

[zwindler]

Hello,

I'm trying to use Packer to build an image for Proxmox VE following this guide https://image-builder.sigs.k8s.io/capi/providers/proxmox

I'm getting an error when trying to upload the ISO

==> proxmox-iso.ubuntu-2204: Post "https://192.168.1.91:8006/api2/json/nodes/node02/storage/cephfs/upload": write tcp 192.168.1.159:46906->192.168.1.91:8006: use of closed network connection
Build 'proxmox-iso.ubuntu-2204' errored after 1 second 142 milliseconds: Post "https://192.168.1.91:8006/api2/json/nodes/node02/storage/cephfs/upload": write tcp 192.168.1.159:46906->192.168.1.91:8006: use of closed network connection

On the server I get this

root@node02:/var/log/pveproxy# journalctl -u pveproxy -u pvedaemon -f
Jul 26 11:00:42 node02 pveproxy[1249]: worker 1250 finished
Jul 26 11:00:42 node02 pveproxy[1249]: starting 1 worker(s)
Jul 26 11:00:42 node02 pveproxy[1249]: worker 28905 started
Jul 26 11:07:10 node02 pveproxy[28040]: problem with client ::ffff:192.168.1.159; No space left on device
Jul 26 11:09:06 node02 pveproxy[28905]: problem with client ::ffff:192.168.1.159; No space left on device

There is no storage issue on any host (neither client nor server)

Others seem to have had this error before.

Especially https://forum.proxmox.com/threads/pveproxy-no-space-left-on-device-problem-with-storage-api.113451/ which also uses Packer

And there was a bug that was supposedly fixed https://bugzilla.proxmox.com/show_bug.cgi?id=4344

But I'm running 8.2.4 so it shouldn't be an issue on my environment.

I've tried multiple hosts, multiple storages, both token and password authentication (in case it was an authentication error, which I can't rule out but also don't know how to check)

How can I find out more to debug this?

 

[thefu21]

Couldn't you just use the auto installer? [1]

[1] https://pve.proxmox.com/wiki/Automated_Installation

 

[zwindler]

I really don't see how that helps me in my usecase

I'm using packer through the image-builder kubernetes project

 

[zwindler]

Update: For some reason it does seem to work when using root@pam + password, rather than a custom user with custom role + token

So it may be either an issue with packer to "image-builder@pve!capi" authentication (but it doesn't throw a 401 for some reason) OR a permission issue but the role is already pretty wide

pveum user add image-builder@pve --password REDACTED

pveum role add image-builder-role -privs "Datastore.Allocate Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit SDN.Allocate SDN.Audit SDN.Use Sys.AccessNetwork Sys.Audit VM.Allocate VM.Audit VM.Backup VM.Clone VM.Config.CDROM VM.Config.CPU VM.Config.Cloudinit VM.Config.Disk VM.Config.HWType VM.Config.Memory VM.Config.Network VM.Config.Options VM.Console VM.Migrate VM.Monitor VM.PowerMgmt VM.Snapshot VM.Snapshot.Rollback"
pveum aclmod / -user image-builder@pve -role image-builder-role

pveum user token add image-builder@pve capi

 

[pksheldon4]

I'm having this same issue. I'm working from a branch new VM so I know there's plenty of space (300GB). I follewed the image-builder instructions

 

[pksheldon4]

Update: For some reason it does seem to work when using root@pam + password, rather than a custom user with custom role + token

So it may be either an issue with packer to "image-builder@pve!capi" authentication (but it doesn't throw a 401 for some reason) OR a permission issue but the role is already pretty wide

pveum user add image-builder@pve --password REDACTED

pveum role add image-builder-role -privs "Datastore.Allocate Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit SDN.Allocate SDN.Audit SDN.Use Sys.AccessNetwork Sys.Audit VM.Allocate VM.Audit VM.Backup VM.Clone VM.Config.CDROM VM.Config.CPU VM.Config.Cloudinit VM.Config.Disk VM.Config.HWType VM.Config.Memory VM.Config.Network VM.Config.Options VM.Console VM.Migrate VM.Monitor VM.PowerMgmt VM.Snapshot VM.Snapshot.Rollback"
pveum aclmod / -user image-builder@pve -role image-builder-role

pveum user token add image-builder@pve capi

Are you sure it's not getting a 401 ? When I look at the logs in /var/log/pveproxy/access.log I get the below error. The first line in the log is a request from the browser, the 2nd is the image-builder upload attempt. I do notice that the browser displays a user but the image-builder request doesn't. However, if I specify a bad user or token, I get an error on the ticket request url.

::ffff:192.168.2.201 - root@pam [18/09/2024:15:12:00 +0000] "GET /api2/json/cluster/resources HTTP/1.1" 200 1105
::ffff:192.168.2.201 - - [18/09/2024:15:22:19 +0000] "POST /api2/json/nodes/pve1/storage/local/upload HTTP/1.1" 401 -

 

[pksheldon4]

For anyone else trying to use image-builder for building a proxmox image, I wasn't able to solve this problem itself, but I was able to get the image to build by switching from an iso_url to an iso_file and manually uploading it to Proxmox before running the `make build-proxmox-ubuntu-2404` command. I made 2 changes to the the packer files Essentially, replacing the 3 iso related properties with the iso_file property

packer/proxmox/ubuntu-2404.json

-  "iso_checksum": "e240e4b801f7bb68c20d1356b60968ad0c33a41d00d828e74ceb3364a0317be9",
-  "iso_checksum_type": "sha256",
-  "iso_url": "https://releases.ubuntu.com/noble/ubuntu-24.04.1-live-server-amd64.iso",
+  "iso_file": "local:iso/ubuntu-24.04.1-live-server-amd64.iso",

and packer/proxmox/packer.json

-      "iso_checksum": "{{user `iso_checksum_type`}}:{{user `iso_checksum`}}",
-      "iso_storage_pool": "{{user `iso_storage_pool`}}",
-      "iso_url": "{{user `iso_url`}}",
+      "iso_file": "{{user `iso_file`}}",