syslog filling up drive

[crash987]

i am not to sure what is the cause of this, but i am getting big log files that is filling up the local drive.
I would get the error 500 drive is full error when trying to look at any of the vm's that are running. What i have found is that i am getting something like 45GB for system logs and 44GB for daemon.log all found in the /var/log directory. having a look at the syslog under the web UI, it is flooded with the same log

pve tgtd[1937]: tgtd: graceful_read(66) sg device 11 read failed, errno: 19

i have no idea what is the cause of this is but it would be nice to have free space last more then 30 MIN

Thanks

 

[noel.]

Hi,
Seems to be an iSCSI error. could you attach the output of sudo cat /etc/pve/qemu-server/<VM ID>.conf where <VM ID> is replace with the ID number of the vm?

 

[Gutenberg]

Hi,
I have the same problem. The error messages appear as soon as I start the vm but don't stop when I shut it down:

proxmox tgtd[1255]: tgtd: graceful_read(66) sg device 11 read failed, errno: 19

Here's my configuration file:

root@proxmox:/# cat /etc/pve/qemu-server/109.conf
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 4
efidisk0: local-lvm:vm-109-disk-1,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:01:00
ide2: local:iso/TrueNAS-SCALE-22.12.3.3.iso,media=cdrom,size=1713892K
machine: q35
memory: 8192
meta: creation-qemu=7.2.0,ctime=1696064986
name: pve-trueNAS
net0: virtio=C6:40:4F:F9:CA:2A,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-109-disk-0,size=8G
scsihw: virtio-scsi-pci
smbios1: uuid=c58175e7-f4bd-41bf-9951-ee82f2e0156b
sockets: 1
startup: order=10,up=10
vmgenid: c8a63ce7-72d4-45fc-92c2-0eecd1a3092a

I hope someone can help me.

 

[scyto]

run journalctl with one of these 3 options to reduce the logs, then figure out how they grew so large

     --vacuum-size=BYTES     Reduce disk usage below specified size
     --vacuum-files=INT      Leave only the specified number of journal files
     --vacuum-time=TIME      Remove journal files older than specified time

 

[Gutenberg]

Hi,

thank you for your help!

figure out how they grew so large

That's the tricky part. I'm not sure how. Could the error have something to do with the PCI-Passthrough?

 

[scyto]

That’s the easy part, look at the logs and see what is causing the bloat. On my machine it’s the annoying corosync logging noise bug, limiting the journalctl size more aggressively is key (by default it is supposed to self prune when the drive has less than 10% full).