Duplicated disk set at VM

[369]

Hello

I am pretty new to Proxmox and currently testing if we can use Proxmox as alternative to VMWare.

Now I discovered something strange at one test VM. The VM has three disks:

• 32GB - Ubuntu Server
• 1TB ZFS for department data
• 2TB ZFS for another department

All disks have discard=on (thin provisioning). The VM has been created at PVE host 1. Basic setup (OS, updates, filesystems) has been done there.
Then the VM has been moved to PVE host 2.
On host 2 I filled the data disks with rsync from our production systems (500GB & 1.3TB)
Then I tried to move the VM back to host 1. This failed because host 1 run out of free space (4TB ZFS local storage)
Now I recognized that there are 9 disks for this VM. vm-104-disk-0 to vm-104-disk-8. Each disk three times. The last three are connected to the VM.

What happend there? And can it be, that the discard setting will be lost on a VM migration? Sparse flag is set on each ZFS storage.

Thank you

 

[ozdjh]

Are you running shared storage (ceph or nfs or something) ? Or are you using local storage and expecting the VM migration to physically copy all that data between the cluster nodes?

 

[369]

At the moment only local storage. No ceph or nfs. All I tried was to move the VM from one host to another one by copy the data to the local storage on the other host.

The final setup should work with ZFS over iSCSI. But for the moment I have just 2 PVE hosts with 240 GB ZFS mirror for the system and 4TB ZFS mirror as local storage for VMs

 

[dcsapak]

please post your vm config (qm config ID) and storage config (/etc/pve/storage.cfg)

 

[369]

/etc/pve/storage.cfg:

dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

zfspool: LO-ZFS-6TB
        pool LO-ZFS-6TB
        content rootdir,images
        mountpoint /LO-ZFS-6TB
        nodes pve00
        sparse 1

zfspool: LO-ZFS-4TB
        pool LO-ZFS-4TB
        content rootdir,images
        mountpoint /LO-ZFS-4TB
        nodes pve10
        sparse 1

zfspool: PVE01-Z0-4TB
        pool PVE01-Z0-4TB
        content images,rootdir
        mountpoint /PVE01-Z0-4TB
        nodes pve01
        sparse 1

pve10 is the cluster node on which the VM is at the moment. pve01 was the designated destination for the live migration.

qm config:

agent: 1
balloon: 2048
bootdisk: scsi0
cores: 4
ide2: none,media=cdrom
memory: 4096
name: DEXXXXX01
net0: e1000=06:6C:3A:3E:7A:4E,bridge=vmbr100
numa: 0
ostype: l26
scsi0: LO-ZFS-4TB:vm-104-disk-6,discard=on,format=raw,size=32G
scsi1: LO-ZFS-4TB:vm-104-disk-7,discard=on,format=raw,size=1T
scsi2: LO-ZFS-4TB:vm-104-disk-8,discard=on,format=raw,size=2T
smbios1: uuid=e043c78c-61ad-488f-bd6f-2c17b2fdcec4
sockets: 1
vmgenid: a72a0bca-1f53-4dd0-b182-16598e71db3b

In the filestorage I have also the disks: vm-104-disk-0, vm-104-disk-1, vm-104-disk-2, vm-104-disk-3, vm-104-disk-4 & vm-104-disk-5. When I created the VM it had only the disks -0 to -2 connected (and created).

 

[gpsantos]

Hello Proxmox community.

I have found a similar issue while migrating one VM with just one disk to another node. Only for this machine, the system seems to find other disks, as you see in the following log:

task started by HA resource agent
2021-04-12 14:28:04 starting migration of VM 104 to node 'pve2' (172.27.132.22)
2021-04-12 14:28:04 found local disk 'local-zfs:vm-104-disk-0' (via storage)
2021-04-12 14:28:04 found local, replicated disk 'local-zfs:vm-104-disk-1' (in current VM config)
2021-04-12 14:28:04 found local disk 'local-zfs:vm-104-disk-2' (via storage)
2021-04-12 14:28:04 found local disk 'local-zfs:vm-104-disk-3' (via storage)
2021-04-12 14:28:04 scsi0: start tracking writes using block-dirty-bitmap 'repl_scsi0'
2021-04-12 14:28:04 replicating disk images
2021-04-12 14:28:04 start replication job
...

Only the first one is declared in the machine configuration:

root@pve2:~# qm config 104
agent: 1
bootdisk: scsi0
cores: 6
ide2: none,media=cdrom
memory: 15251
name: airis-pais-prod-worker-02
net0: virtio=36:50:0F:EF:C3:A3,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: local-zfs:vm-104-disk-1,format=raw,size=100G
scsihw: virtio-scsi-pci
smbios1: uuid=64bd283c-548f-4d45-99e2-7a86d70c4bff
sockets: 1
vmgenid: 527665e6-0fad-4ae7-90c7-292604ecbd05

This is my /etc/pve/storage.cfg:

dir: local
    path /var/lib/vz
    content iso,vztmpl,backup

zfspool: local-zfs
        pool rpool/data
        sparse
        nodes pve1,pve2
        content images,rootdir

lvmthin: local-lvm

 

[gpsantos]

This is what happens when I try to remove the unused disks:



How can I force this operation? The only one important here is the one with 100GB of capacity. I guess this problem comes from a renaming of the disk at some point...

 

[ETS_FTW]

Would like to bump this.

I am encountering the same issue after a replication. I have 2 disks in my VM (originally 0 and1, now 2 and 3) and an exact duplicate of each, so now totaling 4. I cannot find any reference to 0 and 1 anywhere (<vmid>.conf, storage.cfg) except for the storage page of the pool on the node where the VM resides.

What aren't I seeing?

 

[iffster]

Hello

I am pretty new to Proxmox and currently testing if we can use Proxmox as alternative to VMWare.

Now I discovered something strange at one test VM. The VM has three disks:

• 32GB - Ubuntu Server
• 1TB ZFS for department data
• 2TB ZFS for another department

All disks have discard=on (thin provisioning). The VM has been created at PVE host 1. Basic setup (OS, updates, filesystems) has been done there.
Then the VM has been moved to PVE host 2.
On host 2 I filled the data disks with rsync from our production systems (500GB & 1.3TB)
Then I tried to move the VM back to host 1. This failed because host 1 run out of free space (4TB ZFS local storage)
Now I recognized that there are 9 disks for this VM. vm-104-disk-0 to vm-104-disk-8. Each disk three times. The last three are connected to the VM.

What happend there? And can it be, that the discard setting will be lost on a VM migration? Sparse flag is set on each ZFS storage.

Thank you

Have you managed to solve it? Same issue (proxmox 7.4)

 

[greenie]

bump again for this, big is still present. ill add my own to prove that this is still present.

error after migrating 3 x of one disk on one vm and 2 x on a different vm. ill sent configs + data for vm 200:



atempting to delete this gives me the following:


/etc/pve/storage.cfg:
```
nodes pve1
sparse 1

zfspool: Local_NVME
pool Local_NVME
content rootdir,images
mountpoint /Local_NVME
nodes pve3
sparse 0

cifs: Proxmox-VMs-Server1
path /mnt/pve/Proxmox-VMs-Server1
server 192.168.0.35
share Proxmox-VMs-Server1
content vztmpl,snippets,iso,backup,images,rootdir
prune-backups keep-last=3
username admin

cifs: Proxmox-VMs-Server2
path /mnt/pve/Proxmox-VMs-Server2
server 192.168.0.35
share Proxmox-VMs-Server2
content vztmpl,iso,snippets,images,backup,rootdir
prune-backups keep-all=1
username admin

zfspool: Local-NVME
pool Local-NVME
content images,rootdir
mountpoint /Local-NVME
nodes pve1,pve4,pve2
```

vm config
```
#192.168.0.5
agent: 1
balloon: 8192
boot: order=scsi0;net0
cores: 10
memory: 32768
meta: creation-qemu=6.1.0,ctime=1648391148
name: Game-Server-V1
net0: virtio=A6:E7:9A:F6:B9:FE,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: Local-NVME:vm-200-disk-2,cache=writeback,discard=on,size=300G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=07ee633b-4540-4d6a-9577-8346efc6e5f3
sockets: 1
startup: order=9,up=5,down=10
tags: gamehosting
vmgenid: 379324ef-6379-4a92-996e-9db6c155c049
```

 

[alexgauth]

This post is old but me and my boss found something. We had some vm disk in 2 different pools but they were only showing at one place in the ui in the vm page. In the ui in the pools were 2 vm disk using same spaces so it was filling the disk spaces. We searched online and found nothing.

Finally, we just tought, vm are there, disk are there but not linked, so we tought qm rescan and it resolved it. All the unused disk showed up in the command return and they were now appearing in the ui.

In all qm rescan