No space left while moving disk

B

Blipt

Member
Jan 8, 2021
12
3
8
24
I have LXC container with two disks. When I try to move 2nd disk on other storage I get an error "No space left."

Code: 
Formatting '/thirdDisk/data/images/104/vm-104-disk-0.raw', fmt=raw size=547608330240
mke2fs 1.44.5 (15-Dec-2018)
Discarding device blocks:      4096/133693440                   done                           
Creating filesystem with 133693440 4k blocks and 33423360 inodes
Filesystem UUID: 2db4f90f-6899-47d8-ba7e-a9571a1b87bc
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
    102400000

Allocating group tables:    0/4080         done                           
Writing inode tables:    0/4080         done                           
Creating journal (262144 blocks): done
Multiple mount protection is enabled with update interval 5 seconds.
Writing superblocks and filesystem accounting information:    0/4080         done

rsync: write failed on "/var/lib/lxc/104/.copy-volume-1/rsync/path/to/file/fileerror.log": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3]
TASK ERROR: command '/usr/bin/rsync --stats -X -A --numeric-ids -aH --whole-file --sparse --one-file-system '--bwlimit=0' /var/lib/lxc/104/.copy-volume-2/ /var/lib/lxc/104/.copy-volume-1' failed: exit code 11

I checked free space on target storage and it should be more then enough space on it.

I get very similar error if I try to restore VM from backup.

Should I open bug report?
 
From the looks of it this is on your source node.
Was the container running when you tried to move the disk?
 
Is there enough space on your source storage for a copy of the container's disk?
 
No there is not enough space on source for full disk copy that is reason why I am moving disk. Destination has enough space.
 
Please provide the output of pveversion -v, pct config 104 and cat /etc/pve/storage.cfg
 
Code: 
pveversion -v
proxmox-ve: 6.4-1 (running kernel: 5.4.106-1-pve)
pve-manager: 6.4-5 (running version: 6.4-5/6c7bf5de)
pve-kernel-5.4: 6.4-1
pve-kernel-helper: 6.4-1
pve-kernel-5.0: 6.0-11
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.18-1-pve: 5.0.18-3
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.8
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.4-1
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-2
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-1
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.5-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-3
pve-cluster: 6.4-1
pve-container: 3.3-5
pve-docs: 6.4-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1

Code: 
dir: local
        disable
        path /var/lib/vz
        content iso,backup,vztmpl
        shared 0

zfspool: local-zfs
        pool rpool/data
        content images,rootdir
        sparse 1

dir: local-backup-old
        disable
        path /thirdDisk/vzdumpy
        content iso,backup,vztmpl,rootdir,images,snippets
        prune-backups keep-daily=7,keep-monthly=6,keep-weekly=4
        shared 0

pbs: pp2backup
        datastore pp2backup
        server 10.2.3.2
        content backup
        fingerprint 
        prune-backups keep-all=1
        username pppbs@pbs

pbs: pp3backup
        datastore pp3backup
        server 10.2.3.3
        content backup
        fingerprint 
        prune-backups keep-all=1
        username pppbs@pbs

dir: local-hdd
        path /thirdDisk/data
        content images,rootdir,vztmpl
        prune-backups keep-all=1
        shared 0
 
Last edited:
Which is the source and which the target storage?
 
local-zfs is source and local-hdd is target storage.

I want to move disk-0 to local-hdd

Code: 
df -h
Filesystem                    Size  Used Avail Use% Mounted on
udev                           95G     0   95G   0% /dev
tmpfs                          19G   40M   19G   1% /run
rpool/ROOT/pve-1              322G  3.3G  318G   1% /
tmpfs                          95G   69M   95G   1% /dev/shm
tmpfs                         5.0M     0  5.0M   0% /run/lock
tmpfs                          95G     0   95G   0% /sys/fs/cgroup
/dev/sda1                     2.7T  928G  1.7T  36% /thirdDisk
rpool                         318G  128K  318G   1% /rpool
...
rpool/data/subvol-104-disk-2   32G   22G   11G  68% /rpool/data/subvol-104-disk-2
...
rpool/data/subvol-104-disk-0  530G  437G   94G  83% /rpool/data/subvol-104-disk-0
...

Code: 
df -i
Filesystem                      Inodes   IUsed     IFree IUse% Mounted on
udev                          24748067     537  24747530    1% /dev
tmpfs                         24755482    1272  24754210    1% /run
rpool/ROOT/pve-1             667155636   95524 667060112    1% /
tmpfs                         24755482     136  24755346    1% /dev/shm
tmpfs                         24755482      18  24755464    1% /run/lock
tmpfs                         24755482      18  24755464    1% /sys/fs/cgroup
/dev/sda1                    183148544  234722 182913822    1% /thirdDisk
rpool                        667060121       9 667060112    1% /rpool
...
rpool/data                   667060130      18 667060112    1% /rpool/data
rpool/ROOT                   667060119       7 667060112    1% /rpool/ROOT
rpool/data/subvol-104-disk-2  22374935  751983  21622952    4% /rpool/data/subvol-104-disk-2
...
rpool/data/subvol-104-disk-0 202220796 5357324 196863472    3% /rpool/data/subvol-104-disk-0
...
 
Can you run ls -l /path/to/file for the file that could not be written on the source storage?
In addition, please run du -sh at the top level of the rootfs.

Also post the CT config (pct cofig 104).
 
It is not always same file after many tries and file was deleted after unsuccessful move.

Code: 
ls -l /var/lib/lxc/104/.copy-volume-1/rsync/....log
ls: cannot access '/var/lib/lxc/104/.copy-volume-1/rsync/...log': No such file or directory

Code: 
arch: amd64
cores: 8
hostname:
memory: 7000
mp1: local-zfs:subvol-104-disk-0,mp=/backup,size=530G
nameserver:
net0: name=eth0,bridge=vmbr0,gw=,hwaddr=,ip=/24,ip6=auto,type=veth
net1: name=eth1,bridge=vmbr2,hwaddr=,ip=/24,ip6=auto,tag=,type=veth
onboot: 1
ostype: centos
rootfs: local-zfs:subvol-104-disk-2,size=32G
searchdomain:
swap: 512
 
The issue could be because of compression.
Please provide the output of:
Code: 
zfs get all rpool/data/subvol-104-disk-0
zfs get all rpool/data/subvol-104-disk-2
 
Code: 
zfs get all rpool/data/subvol-104-disk-0
NAME                          PROPERTY              VALUE                          SOURCE
rpool/data/subvol-104-disk-0  type                  filesystem                     -
rpool/data/subvol-104-disk-0  creation              Thu Jan 21 16:11 2021          -
rpool/data/subvol-104-disk-0  used                  429G                           -
rpool/data/subvol-104-disk-0  available             101G                           -
rpool/data/subvol-104-disk-0  referenced            429G                           -
rpool/data/subvol-104-disk-0  compressratio         1.61x                          -
rpool/data/subvol-104-disk-0  mounted               yes                            -
rpool/data/subvol-104-disk-0  quota                 none                           default
rpool/data/subvol-104-disk-0  reservation           none                           default
rpool/data/subvol-104-disk-0  recordsize            128K                           default
rpool/data/subvol-104-disk-0  mountpoint            /rpool/data/subvol-104-disk-0  default
rpool/data/subvol-104-disk-0  sharenfs              off                            default
rpool/data/subvol-104-disk-0  checksum              on                             default
rpool/data/subvol-104-disk-0  compression           on                             inherited from rpool
rpool/data/subvol-104-disk-0  atime                 off                            inherited from rpool
rpool/data/subvol-104-disk-0  devices               on                             default
rpool/data/subvol-104-disk-0  exec                  on                             default
rpool/data/subvol-104-disk-0  setuid                on                             default
rpool/data/subvol-104-disk-0  readonly              off                            default
rpool/data/subvol-104-disk-0  zoned                 off                            default
rpool/data/subvol-104-disk-0  snapdir               hidden                         default
rpool/data/subvol-104-disk-0  aclmode               discard                        default
rpool/data/subvol-104-disk-0  aclinherit            restricted                     default
rpool/data/subvol-104-disk-0  createtxg             8957745                        -
rpool/data/subvol-104-disk-0  canmount              on                             default
rpool/data/subvol-104-disk-0  xattr                 sa                             received
rpool/data/subvol-104-disk-0  copies                1                              default
rpool/data/subvol-104-disk-0  version               5                              -
rpool/data/subvol-104-disk-0  utf8only              off                            -
rpool/data/subvol-104-disk-0  normalization         none                           -
rpool/data/subvol-104-disk-0  casesensitivity       sensitive                      -
rpool/data/subvol-104-disk-0  vscan                 off                            default
rpool/data/subvol-104-disk-0  nbmand                off                            default
rpool/data/subvol-104-disk-0  sharesmb              off                            default
rpool/data/subvol-104-disk-0  refquota              530G                           local
rpool/data/subvol-104-disk-0  refreservation        none                           default
rpool/data/subvol-104-disk-0  guid                  5048813586118213732            -
rpool/data/subvol-104-disk-0  primarycache          all                            default
rpool/data/subvol-104-disk-0  secondarycache        all                            default
rpool/data/subvol-104-disk-0  usedbysnapshots       0B                             -
rpool/data/subvol-104-disk-0  usedbydataset         429G                           -
rpool/data/subvol-104-disk-0  usedbychildren        0B                             -
rpool/data/subvol-104-disk-0  usedbyrefreservation  0B                             -
rpool/data/subvol-104-disk-0  logbias               latency                        default
rpool/data/subvol-104-disk-0  objsetid              49695                          -
rpool/data/subvol-104-disk-0  dedup                 off                            default
rpool/data/subvol-104-disk-0  mlslabel              none                           default
rpool/data/subvol-104-disk-0  sync                  standard                       inherited from rpool
rpool/data/subvol-104-disk-0  dnodesize             legacy                         default
rpool/data/subvol-104-disk-0  refcompressratio      1.61x                          -
rpool/data/subvol-104-disk-0  written               0                              -
rpool/data/subvol-104-disk-0  logicalused           669G                           -
rpool/data/subvol-104-disk-0  logicalreferenced     669G                           -
rpool/data/subvol-104-disk-0  volmode               default                        default
rpool/data/subvol-104-disk-0  filesystem_limit      none                           default
rpool/data/subvol-104-disk-0  snapshot_limit        none                           default
rpool/data/subvol-104-disk-0  filesystem_count      none                           default
rpool/data/subvol-104-disk-0  snapshot_count        none                           default
rpool/data/subvol-104-disk-0  snapdev               hidden                         default
rpool/data/subvol-104-disk-0  acltype               posix                          received
rpool/data/subvol-104-disk-0  context               none                           default
rpool/data/subvol-104-disk-0  fscontext             none                           default
rpool/data/subvol-104-disk-0  defcontext            none                           default
rpool/data/subvol-104-disk-0  rootcontext           none                           default
rpool/data/subvol-104-disk-0  relatime              off                            default
rpool/data/subvol-104-disk-0  redundant_metadata    all                            default
rpool/data/subvol-104-disk-0  overlay               on                             default
rpool/data/subvol-104-disk-0  encryption            off                            default
rpool/data/subvol-104-disk-0  keylocation           none                           default
rpool/data/subvol-104-disk-0  keyformat             none                           default
rpool/data/subvol-104-disk-0  pbkdf2iters           0                              default
rpool/data/subvol-104-disk-0  special_small_blocks  0                              default

Code: 
zfs get all rpool/data/subvol-104-disk-2
NAME                          PROPERTY              VALUE                          SOURCE
rpool/data/subvol-104-disk-2  type                  filesystem                     -
rpool/data/subvol-104-disk-2  creation              Tue Aug 13 15:05 2019          -
rpool/data/subvol-104-disk-2  used                  21.7G                          -
rpool/data/subvol-104-disk-2  available             10.3G                          -
rpool/data/subvol-104-disk-2  referenced            21.7G                          -
rpool/data/subvol-104-disk-2  compressratio         2.02x                          -
rpool/data/subvol-104-disk-2  mounted               yes                            -
rpool/data/subvol-104-disk-2  quota                 none                           default
rpool/data/subvol-104-disk-2  reservation           none                           default
rpool/data/subvol-104-disk-2  recordsize            128K                           default
rpool/data/subvol-104-disk-2  mountpoint            /rpool/data/subvol-104-disk-2  default
rpool/data/subvol-104-disk-2  sharenfs              off                            default
rpool/data/subvol-104-disk-2  checksum              on                             default
rpool/data/subvol-104-disk-2  compression           on                             inherited from rpool
rpool/data/subvol-104-disk-2  atime                 off                            inherited from rpool
rpool/data/subvol-104-disk-2  devices               on                             default
rpool/data/subvol-104-disk-2  exec                  on                             default
rpool/data/subvol-104-disk-2  setuid                on                             default
rpool/data/subvol-104-disk-2  readonly              off                            default
rpool/data/subvol-104-disk-2  zoned                 off                            default
rpool/data/subvol-104-disk-2  snapdir               hidden                         default
rpool/data/subvol-104-disk-2  aclmode               discard                        default
rpool/data/subvol-104-disk-2  aclinherit            restricted                     default
rpool/data/subvol-104-disk-2  createtxg             3031                           -
rpool/data/subvol-104-disk-2  canmount              on                             default
rpool/data/subvol-104-disk-2  xattr                 sa                             received
rpool/data/subvol-104-disk-2  copies                1                              default
rpool/data/subvol-104-disk-2  version               5                              -
rpool/data/subvol-104-disk-2  utf8only              off                            -
rpool/data/subvol-104-disk-2  normalization         none                           -
rpool/data/subvol-104-disk-2  casesensitivity       sensitive                      -
rpool/data/subvol-104-disk-2  vscan                 off                            default
rpool/data/subvol-104-disk-2  nbmand                off                            default
rpool/data/subvol-104-disk-2  sharesmb              off                            default
rpool/data/subvol-104-disk-2  refquota              32G                            received
rpool/data/subvol-104-disk-2  refreservation        none                           default
rpool/data/subvol-104-disk-2  guid                  6801227295565156267            -
rpool/data/subvol-104-disk-2  primarycache          all                            default
rpool/data/subvol-104-disk-2  secondarycache        all                            default
rpool/data/subvol-104-disk-2  usedbysnapshots       4.93M                          -
rpool/data/subvol-104-disk-2  usedbydataset         21.7G                          -
rpool/data/subvol-104-disk-2  usedbychildren        0B                             -
rpool/data/subvol-104-disk-2  usedbyrefreservation  0B                             -
rpool/data/subvol-104-disk-2  logbias               latency                        default
rpool/data/subvol-104-disk-2  objsetid              899                            -
rpool/data/subvol-104-disk-2  dedup                 off                            default
rpool/data/subvol-104-disk-2  mlslabel              none                           default
rpool/data/subvol-104-disk-2  sync                  standard                       inherited from rpool
rpool/data/subvol-104-disk-2  dnodesize             legacy                         default
rpool/data/subvol-104-disk-2  refcompressratio      2.02x                          -
rpool/data/subvol-104-disk-2  written               0                              -
rpool/data/subvol-104-disk-2  logicalused           40.9G                          -
rpool/data/subvol-104-disk-2  logicalreferenced     40.9G                          -
rpool/data/subvol-104-disk-2  volmode               default                        default
rpool/data/subvol-104-disk-2  filesystem_limit      none                           default
rpool/data/subvol-104-disk-2  snapshot_limit        none                           default
rpool/data/subvol-104-disk-2  filesystem_count      none                           default
rpool/data/subvol-104-disk-2  snapshot_count        none                           default
rpool/data/subvol-104-disk-2  snapdev               hidden                         default
rpool/data/subvol-104-disk-2  acltype               posix                          received
rpool/data/subvol-104-disk-2  context               none                           default
rpool/data/subvol-104-disk-2  fscontext             none                           default
rpool/data/subvol-104-disk-2  defcontext            none                           default
rpool/data/subvol-104-disk-2  rootcontext           none                           default
rpool/data/subvol-104-disk-2  relatime              off                            default
rpool/data/subvol-104-disk-2  redundant_metadata    all                            default
rpool/data/subvol-104-disk-2  overlay               on                             default
rpool/data/subvol-104-disk-2  encryption            off                            default
rpool/data/subvol-104-disk-2  keylocation           none                           default
rpool/data/subvol-104-disk-2  keyformat             none                           default
rpool/data/subvol-104-disk-2  pbkdf2iters           0                              default
rpool/data/subvol-104-disk-2  special_small_blocks  0                              default
 
The big subvol has 1.61x compression ratio, and the smaller one 2.02x. Both won't fit in an ext4 filesystem with the exact same size.
What you can do is resize both subvols so that all the data would fit uncompressed and then move them to the other storage.
 
  • Like
Reactions: ramrot
@mira thank you for your suggestion.
Please let me know if I understand it correctly.

You suggest to resize both volumes (x 1.7 and x 2.4) and then move the volumes from ZFS to ext4?

You say, newly created raw disk on ext4, which should have the exact size as the previous zvol (530 GB), is not big enough for uncompressed data from ZVOL, correct?

This would imply, that inside LXC, users also see the compressed data usage instead of actual data usage, correct?

I guess PM should at least implement a warning when moving LXCs from compressed FS to noncompressed FS in PM GUI.
However, even better it would be, to have GUI calculate and propose new size and resize it before moving data.
I guess I could open a bug report of feature request.
 
I did resize disk for 200 GB and I will see tomorrow if disk was successfully moved.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back