Problems resizing rbd

A

alexskysilk

Distinguished Member
Oct 16, 2015
1,880
390
153
Chatsworth, CA
www.skysilk.com
Just noticed this on one of my clusters; disk resize is failing with the following error message:

Code: 
Resizing image: 100% complete...done.
mount.nfs: Failed to resolve server rbd: Name or service not known
Failed to update the container's filesystem: command 'unshare -m -- sh -c 'mount --make-rprivate / && mount '\''rbd:rbd3/vm-103-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/rbd3.keyring'\'' /tmp && resize2fs '\''rbd:rbd3/vm-103-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/rbd3.keyring'\'' failed: exit code 32

TASK OK

note the odd "mount.nfs" reference; the disk is rbd.

pveversion:
Code: 
# pveversion -v
proxmox-ve: 5.3-1 (running kernel: 4.15.18-9-pve)
pve-manager: 5.3-7 (running version: 5.3-7/e8ed1e22)
pve-kernel-4.15: 5.2-12
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.17-1-pve: 4.15.17-9
ceph: 12.2.10-pve1
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: not correctly installed
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-43
libpve-guest-common-perl: 2.0-19
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-35
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-2
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
openvswitch-switch: 2.7.0-3
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-33
pve-container: 2.0-33
pve-docs: 5.3-1
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-16
pve-firmware: 2.0-6
pve-ha-manager: 2.0-6
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-44
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1
 
alexskysilk said:
Code: 
Resizing image: 100% complete...done.
mount.nfs: Failed to resolve server rbd: Name or service not known
Failed to update the container's filesystem: command 'unshare -m -- sh -c 'mount --make-rprivate / && mount '\''rbd:rbd3/vm-103-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/rbd3.keyring'\'' /tmp && resize2fs '\''rbd:rbd3/vm-103-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/rbd3.keyring'\'' failed: exit code 32

TASK OK

note the odd "mount.nfs" reference; the disk is rbd.
Click to expand...

Yes, that's strange. But it says "TASK OK" - is the disk now extended?

For considering this issue more details about configuration are necessary ( /etc/pve/lxc/..conf file as well as possibly special settings in /etc/apparmor.d/lxc/...)
 
I guess I missed specifying that the resize process did not complete successfully; This is not a cosmetic bug. the rbd resize actually completed but not the filesystem:

Code: 
rbd du vm-104-disk-0 -p rbd3
warning: fast-diff map is not enabled for vm-104-disk-0. operation may be slow.
NAME          PROVISIONED   USED
vm-104-disk-0       16GiB 724MiB

Code: 
df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/rbd0       7.9G  603M  6.9G   8% /

/etc/pve/lxc/104.conf
Code: 
arch: amd64
cores: 1
hostname: temp
memory: 512
net0: name=eth0,bridge=vmbr0,hwaddr=82:D3:9B:21:3A:24,ip=dhcp,type=veth
ostype: centos
rootfs: rbd3:vm-104-disk-0,size=16G (this value was changed from 8G)
swap: 512

apparmor profiles unmolested from install.

/etc/apparmor.d/lxc/default.
Code: 
# Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc
profile lxc-container-default flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/lxc/container-base>
  # the container may never be allowed to mount devpts.  If it does, it
  # will remount the host's devpts.  We could allow it to do it with
  # the newinstance option (but, right now, we don't).
  deny mount fstype=devpts,
}
 
Last edited:
Alwin said:
No additional mount points configured?
Click to expand...
No, only the rbd is present.

Alwin said:
Can you please check the syslog, there might be some more info on what happend during that time?
Click to expand...

No, no message remotely relating; I attempted a resize again just to verify. FYI, simply mounting the device in an unshare environment and doing a resize2fs works fine- I wonder if its a problem with the command syntax.
 
I experienced the same issue today.

Proxmox executes the command:

`unshare -m -- sh -c 'mount --make-rprivate / && mount '\''rbd:rbd-ssd/vm-105-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/rbd-ssd.keyring'\'' /tmp && resize2fs '\''rbd:rbd-ssd/vm-105-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/rbd-ssd.keyring'\'`

and this command fails with exit code 32 and the message `mount.nfs: Failed to resolve server rbd: Name or service not known`

The disk has been re-sized in the GUI but the filesystem in the container has not been updated.

Edit: I'm running PVE 5.3-6
 
The bug report referenced above has status "PATCH AVAILABLE", but I can't see any details on it. Is this issue solved with a patch?
 
I updated the bug report, patch included in pve-container >= 2.0-36, currently in pve-test repository.
 
Yes I know it is a old post, but It has the same issue.
I have similar issue.
ii pve-container 2.0-40 all Proxmox VE Container management tool

The disk size is increased but the container inside is unchanged.

Code: 
root@vNTDB-host-4:~# rbd info vNTDB-Storage/vm-160-disk-0
rbd image 'vm-160-disk-0':
        size 10GiB in 2560 objects
        order 22 (4MiB objects)
        block_name_prefix: rbd_data.e2abd26b8b4567
        format: 2
        features: layering
        flags:
        create_timestamp: Thu Jul  4 15:13:48 2019
Code: 
root@vNTDB-host-1:~# cat /etc/pve/nodes/vNTDB-host-1/lxc/160.conf
...
rootfs: vNTDB-Storage_ct:vm-160-disk-0,size=10G

Code: 
oam@system-2-225:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/rbd4       4.9G  2.7G  2.0G  59% /
....

Code: 
Oct 15 10:05:18 vNTDB-host-1 pvedaemon[949239]: <root@pam> successful auth for user 'root@pam'
Oct 15 10:05:53 vNTDB-host-1 pvedaemon[975887]: <root@pam> update CT 160: resize --disk rootfs --size 5G
Oct 15 10:05:53 vNTDB-host-1 pvedaemon[975887]: <root@pam> starting task UPID:vNTDB-host-1:000EF2A3:2A1415DA:5DA57DE1:resize:160:root@pam:
Oct 15 10:05:53 vNTDB-host-1 pvedaemon[975887]: <root@pam> end task UPID:vNTDB-host-1:000EF2A3:2A1415DA:5DA57DE1:resize:160:root@pam: OK
 
Run a rbd du vNTDB-Storage/vm-160-disk-0 to get the actual size.
 
Code: 
root@vNTDB-host-1:~# rbd du vNTDB-Storage/vm-160-disk-0
warning: fast-diff map is not enabled for vm-160-disk-0. operation may be slow.
NAME          PROVISIONED    USED
vm-160-disk-0       10GiB 4.96GiB
 
Here is another one. In this case it was working perfectly.

Code: 
Resizing image: 100% complete...done.
resize2fs 1.43.4 (31-Jan-2017)
Filesystem at /dev/rbd/vNTDB-Storage/vm-150-disk-0 is mounted on /tmp; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 2
The filesystem on /dev/rbd/vNTDB-Storage/vm-150-disk-0 is now 2621440 (4k) blocks long.

TASK OK

In the wrong case:
Code: 
Resizing image: 100% complete...done.
resize2fs 1.43.4 (31-Jan-2017)
The filesystem is already 1310720 (4k) blocks long.  Nothing to do!

TASK OK
 
kaltsi said:
The filesystem is already 1310720 (4k) blocks long. Nothing to do!
Click to expand...
What command do you use, as it says here that it already is the wanted size.
 
Can you please post a pveversion -v?
 
# pveversion -v
proxmox-ve: 5.4-2 (running kernel: 4.15.18-18-pve)
pve-manager: 5.4-13 (running version: 5.4-13/aee6f0ec)
pve-kernel-4.15: 5.4-9
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.18-20-pve: 4.15.18-46
pve-kernel-4.15.18-19-pve: 4.15.18-45
pve-kernel-4.15.18-18-pve: 4.15.18-44
ceph: 12.2.12-pve1
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-12
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-55
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-14
libpve-storage-perl: 5.0-44
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-7
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-28
pve-cluster: 5.0-38
pve-container: 2.0-40
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-22
pve-firmware: 2.0-7
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 3.0.1-4
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-54
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2
 
Please try the following on the CLI, pct resize 100 rootfs 15G. Note that the size (eg. 15G) is not an increment, it is the final size. Please post the complete output.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom