[SOLVED] Disk 100% in VM after unflag no rebalance.

totae

New Member
May 27, 2023
10
1
3
Hello,

I've change value PGs from 128 to 512 .

After that disk inside VM going to 100% and can't write file.

Right now i must flag no rebalance because vm can't used.

Code:
proxmox-ve: 7.4-1 (running kernel: 5.15.108-1-pve)
pve-manager: 7.4-15 (running version: 7.4-15/a5d2a31e)
pve-kernel-5.15: 7.4-4
pve-kernel-5.15.108-1-pve: 5.15.108-1
ceph: 17.2.6-pve1
ceph-fuse: 17.2.6-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: not correctly installed
ifupdown2: 3.1.0-1+pmx4
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.2-1
proxmox-backup-file-restore: 2.4.2-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.2
proxmox-widget-toolkit: 3.7.3
pve-cluster: 7.3-3
pve-container: 4.4-6
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+1
pve-firewall: 4.3-4
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1


Network a both ceph use 20 Gbps (Bonding 10 Gbps x 2 port)

I've check error node in cluster found this log (Only node ceph280)
Jul 22 01:23:02 ceph280 ceph-crash[1014]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-07-10T05:20:49.277181Z_4198906b-9b4b-4e10-986d-d0908baf447d tializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Jul 22 01:23:02 ceph280 ceph-crash[1014]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-07-10T05:20:49.277181Z_4198906b-9b4b-4e10-986d-d0908baf447d luster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Jul 22 01:23:02 ceph280 ceph-crash[1014]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-07-10T05:20:49.277181Z_4198906b-9b4b-4e10-986d-d0908baf447d luster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Jul 22 01:23:02 ceph280 ceph-crash[1014]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-07-10T05:20:08.699802Z_34e5bfb4-0ca2-47f8-9904-e832e4093701 tializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Jul 22 01:23:02 ceph280 ceph-crash[1014]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-07-10T05:20:08.699802Z_34e5bfb4-0ca2-47f8-9904-e832e4093701 luster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Jul 22 01:23:02 ceph280 ceph-crash[1014]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-07-10T05:20:08.699802Z_34e5bfb4-0ca2-47f8-9904-e832e4093701 luster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')

Could you please suggest.
 
Return value back to 128 and increase in smaller steps.
I guess you are using HDDs in CEPH?
 
Hello itNGO,



But PGs is 512 already, I can change back to 128 ?

View attachment 53339

View attachment 53338


No, All SSDs

Could you please suggest.

Best Regards,
What type of SSDs?

You can change back, even it has not finished rebalancing. But it looks like is is already near to finished... maybe just pause your VMs and let it complete might be the faster way?
 
You can change back, even it has not finished rebalancing. But it looks like is is already near to finished... maybe just pause your VMs and let it complete might be the faster way?
Thank you, I've unflag and rebalance is done.
 
  • Like
Reactions: itNGO

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!