image-builder, 501 when uploading image to storage

[lknite]

I'm putting together a pipeline to build kubernetes images for proxmox.
(image builder url: https://image-builder.sigs.k8s.io/capi/providers/proxmox)

Everything is looking good except when the image-builder script goes to upload an .ISO to my nfs share 'tower', (and i also tried 'local'), and it times out. I granted the role 'Administrator' to the user attempting the upload, I know that's too much permission, for now though I just wanted to eliminate permissions as an issue. There are already .iso files on the tower storage, so it is configured for iso files.

==> proxmox-iso.ubuntu-2204: Retrieving ISO
421==> proxmox-iso.ubuntu-2204: Trying https://releases.ubuntu.com/22.04.5/ubuntu-22.04.5-live-server-amd64.iso
4222024/10/06 03:47:03 packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64 plugin: 2024/10/06 03:47:03 Acquiring lock for: https://releases.ubuntu.com/22.04.5/ubuntu-22.04.5-live-server-amd64.iso?checksum=sha256%3A9bc6028870aef3f74f4e16b900008179e78b130e6b0b9a140635434a46aa98b0 (downloaded_iso_path/c968bbbeb22702b3f10a07276c8ca06720e80c4c.iso.lock)
423==> proxmox-iso.ubuntu-2204: Trying https://releases.ubuntu.com/22.04.5/ubuntu-22.04.5-live-server-amd64.iso?checksum=sha256%3A9bc6028870aef3f74f4e16b900008179e78b130e6b0b9a140635434a46aa98b0
4242024/10/06 03:48:18 packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64 plugin: 2024/10/06 03:48:18 closing
4252024/10/06 03:48:21 packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64 plugin: 2024/10/06 03:48:21 Leaving retrieve loop for ISO
426==> proxmox-iso.ubuntu-2204: https://releases.ubuntu.com/22.04.5/ubuntu-22.04.5-live-server-amd64.iso?checksum=sha256%3A9bc6028870aef3f74f4e16b900008179e78b130e6b0b9a140635434a46aa98b0 => downloaded_iso_path/c968bbbeb22702b3f10a07276c8ca06720e80c4c.iso
427==> proxmox-iso.ubuntu-2204: Post "https://10.0.0.21:8006/nodes/pve-c/storage/tower/upload": write tcp 172.16.89.28:46140->10.0.0.21:8006: write: broken pipe
4282024/10/06 03:48:21 [INFO] (telemetry) ending ubuntu-2204
429==> Wait completed after 1 minute 17 seconds
4302024/10/06 03:48:21 machine readable: error-count []string{"1"}
431==> Some builds didn't complete successfully and had errors:
4322024/10/06 03:48:21 machine readable: proxmox-iso.ubuntu-2204,error []string{"Post \"https://10.0.0.21:8006/nodes/pve-c/storage/tower/upload\": write tcp 172.16.89.28:46140->10.0.0.21:8006: write: broken pipe"}
433==> Builds finished but no artifacts were created.
434Build 'proxmox-iso.ubuntu-2204' errored after 1 minute 17 seconds: Post "https://10.0.0.21:8006/nodes/pve-c/storage/tower/upload": write tcp 172.16.89.28:46140->10.0.0.21:8006: write: broken pipe
435==> Wait completed after 1 minute 17 seconds
4362024/10/06 03:48:21 [INFO] (telemetry) Finalizing.
437==> Some builds didn't complete successfully and had errors:
438--> proxmox-iso.ubuntu-2204: Post "https://10.0.0.21:8006/nodes/pve-c/storage/tower/upload": write tcp 172.16.89.28:46140->10.0.0.21:8006: write: broken pipe
439==> Builds finished but no artifacts were created.
440

Why am I getting a 501? Is there a max file upload size set somewhere that I need to increase maybe?

(I also submitted an Issue at the hashicorp packer github: https://github.com/hashicorp/packer-plugin-proxmox/issues/288)

Additional, I also attempted the same from a linux vm and got the same error, so not something special to the pipeline.

 

[lknite]

Update, and for folks who come by later, the 501 occurs with you specify the proxmox url without "/api2/json".

When I add "/api2/json" the 501 error goes away, and now I'm seeing "use of closed network connection". Anyone have ideas on that one?

proxmox-iso.ubuntu-2204: output will be in this color.

2024/10/09 15:34:12 packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64 plugin: 2024/10/09 15:34:12 using token auth
2024/10/09 15:34:12 packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64 plugin: 2024/10/09 15:34:12 No CD files specified. CD disk will not be made.
==> proxmox-iso.ubuntu-2204: Retrieving ISO
==> proxmox-iso.ubuntu-2204: Trying https://releases.ubuntu.com/22.04.5/ubuntu-22.04.5-live-server-amd64.iso
2024/10/09 15:34:12 packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64 plugin: 2024/10/09 15:34:12 Acquiring lock for: https://releases.ubuntu.com/22.04.5/ubuntu-22.04.5-live-server-amd64.iso?checksum=sha256%!A(MISSING)9bc6028870aef3f74f4e16b900008179e78b130e6b0b9a140635434a46aa98b0 (downloaded_iso_path/c968bbbeb22702b3f10a07276c8ca06720e80c4c.iso.lock)
==> proxmox-iso.ubuntu-2204: Trying https://releases.ubuntu.com/22.04.5/ubuntu-22.04.5-live-server-amd64.iso?checksum=sha256%3A9bc6028870aef3f74f4e16b900008179e78b130e6b0b9a140635434a46aa98b0
==> proxmox-iso.ubuntu-2204: https://releases.ubuntu.com/22.04.5/ubuntu-22.04.5-live-server-amd64.iso?checksum=sha256%3A9bc6028870aef3f74f4e16b900008179e78b130e6b0b9a140635434a46aa98b0 => downloaded_iso_path/c968bbbeb22702b3f10a07276c8ca06720e80c4c.iso
2024/10/09 15:34:43 packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64 plugin: 2024/10/09 15:34:43 Leaving retrieve loop for ISO
==> proxmox-iso.ubuntu-2204: Post "https://10.0.0.21:8006/api2/json/nodes/pve-c/storage/tower/upload": use of closed network connection
2024/10/09 15:34:43 [INFO] (telemetry) ending ubuntu-2204
==> Wait completed after 30 seconds 602 milliseconds
2024/10/09 15:34:43 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:

 

[lknite]

Read this link again and checked this log file: /var/log/pveproxy/access.log .

And, there is a 403 in there:
::ffff:10.0.0.99 - - [09/10/2024:15:41:08 -0600] "POST /api2/json/nodes/pve-a/storage/tower/upload HTTP/1.1" 403 -

But I'm not sure why, the user being used should have full permissions, I've tested with Administrator as well as creating a specific role ImageBuilder.

$ cat go.sh
#!/bin/bash

# configure via env vars
export PACKER_LOG=1
export PROXMOX_BRIDGE="vmbr0"
export PROXMOX_ISO_POOL="tower"
export PROXMOX_NODE="pve-a"
export PROXMOX_STORAGE_POOL="cephfs"
export PROXMOX_TOKEN="...redacted..."
export PROXMOX_URL="https://10.0.0.21:8006/api2/json"
export PROXMOX_USERNAME="capmox@pve!capi"

 

[lknite]

I was granting the needed access to the API Token on the storage, but turns out I hadn't granted the user the needed access. After granting the user the needed permissions then the permission on the storage worked.