[SOLVED] Disk 100% in VM after unflag no rebalance.

[totae]

Hello,

I've change value PGs from 128 to 512 .

After that disk inside VM going to 100% and can't write file.

Right now i must flag no rebalance because vm can't used.

proxmox-ve: 7.4-1 (running kernel: 5.15.108-1-pve)
pve-manager: 7.4-15 (running version: 7.4-15/a5d2a31e)
pve-kernel-5.15: 7.4-4
pve-kernel-5.15.108-1-pve: 5.15.108-1
ceph: 17.2.6-pve1
ceph-fuse: 17.2.6-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: not correctly installed
ifupdown2: 3.1.0-1+pmx4
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.2-1
proxmox-backup-file-restore: 2.4.2-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.2
proxmox-widget-toolkit: 3.7.3
pve-cluster: 7.3-3
pve-container: 4.4-6
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+1
pve-firewall: 4.3-4
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1

Network a both ceph use 20 Gbps (Bonding 10 Gbps x 2 port)

I've check error node in cluster found this log (Only node ceph280)
Jul 22 01:23:02 ceph280 ceph-crash[1014]: WARNING:ceph-crashost /var/lib/ceph/crash/2023-07-10T05:20:49.277181Z_4198906b-9b4b-4e10-986d-d0908baf447d tializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Jul 22 01:23:02 ceph280 ceph-crash[1014]: WARNING:ceph-crashost /var/lib/ceph/crash/2023-07-10T05:20:49.277181Z_4198906b-9b4b-4e10-986d-d0908baf447d luster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Jul 22 01:23:02 ceph280 ceph-crash[1014]: WARNING:ceph-crashost /var/lib/ceph/crash/2023-07-10T05:20:49.277181Z_4198906b-9b4b-4e10-986d-d0908baf447d luster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Jul 22 01:23:02 ceph280 ceph-crash[1014]: WARNING:ceph-crashost /var/lib/ceph/crash/2023-07-10T05:20:08.699802Z_34e5bfb4-0ca2-47f8-9904-e832e4093701 tializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Jul 22 01:23:02 ceph280 ceph-crash[1014]: WARNING:ceph-crashost /var/lib/ceph/crash/2023-07-10T05:20:08.699802Z_34e5bfb4-0ca2-47f8-9904-e832e4093701 luster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Jul 22 01:23:02 ceph280 ceph-crash[1014]: WARNING:ceph-crashost /var/lib/ceph/crash/2023-07-10T05:20:08.699802Z_34e5bfb4-0ca2-47f8-9904-e832e4093701 luster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')

Could you please suggest.

 

[itNGO]

Return value back to 128 and increase in smaller steps.
I guess you are using HDDs in CEPH?

 

[totae]

Hello itNGO,

Return value back to 128 and increase in smaller steps.

But PGs is 512 already, I can change back to 128 ?

> I guess you are using HDDs in CEPH?

No, All SSDs

Could you please suggest.

Best Regards,

 

[itNGO]

Hello itNGO,



But PGs is 512 already, I can change back to 128 ?

View attachment 53339

View attachment 53338


No, All SSDs

Could you please suggest.

Best Regards,

What type of SSDs?

You can change back, even it has not finished rebalancing. But it looks like is is already near to finished... maybe just pause your VMs and let it complete might be the faster way?

 

[totae]

You can change back, even it has not finished rebalancing. But it looks like is is already near to finished... maybe just pause your VMs and let it complete might be the faster way?

Thank you, I've unflag and rebalance is done.