Migration Warnings

G

Grunt

Member
Sep 6, 2022
29
5
8
I have a two node (with a qnode) cluster using ZFS over iSCSI to TrueNAS for storage. There is a dedicated 10Gb network used to connect the nodes to storage. When performing a migration from one node to the other I get the below output. The migrations succeed, but what are these warnings?

Code: 
23-04-21 22:52:32 use dedicated network address for sending migration traffic (10.1.0.2)
2023-04-21 22:52:32 starting migration of VM 110 to node 'proxve2' (10.1.0.2)
  WARNING: Not using device /dev/sds3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdd3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdt3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdaj3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdf3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdv3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdh3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdk3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdl3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdn3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdp3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: Not using device /dev/sds3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdd3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdt3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdaj3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdf3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdv3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdh3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdk3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdl3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdn3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdp3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
2023-04-21 22:52:32 starting VM 110 on remote node 'proxve2'
 
Hello,

Can you please provide us with more information, like the VM config `qm config 110` and the output of `pveversion -v` command, and also the list storage `cat /etc/pve/storage.cfg`?

Did you see anything in the syslog during the migration operation time?
 
Code: 
root@proxve2:~# qm config 110
agent: 1
balloon: 0
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 4
cpu: host
efidisk0: TrueNAS:vm-110-disk-0,efitype=4m,pre-enrolled-keys=1,size=528K
ide2: none,media=cdrom
machine: q35
memory: 3072
meta: creation-qemu=7.2.0,ctime=1681497018
name: PrivacyIDEA
net0: virtio=E6:51:B9:F8:33:91,bridge=vmbr1
numa: 1
ostype: l26
scsi0: TrueNAS:vm-110-disk-1,backup=0,iothread=1,size=25G
scsihw: virtio-scsi-single
smbios1: uuid=d5e472a6-f735-448c-8dd0-687cc73786dd
sockets: 1
vmgenid: cb241458-ba09-470c-a512-56bc4c8aeb03


Code: 
root@proxve2:~# pveversion -v
proxmox-ve: 7.4-1 (running kernel: 5.15.104-1-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.4-1
pve-kernel-5.15.104-1-pve: 5.15.104-2
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-4
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.1-1
proxmox-backup-file-restore: 2.4.1-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.6.5
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-2
pve-firewall: 4.3-1
pve-firmware: 3.6-4
pve-ha-manager: 3.6.0
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1


Code: 
root@proxve2:~# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content iso,backup,vztmpl


lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images


zfs: TrueNAS
        blocksize 4k
        iscsiprovider freenas
        pool VMPool
        portal 10.1.0.253
        target iqn.2005-10.org.freenas.ctl:vmpool
        content images
        freenas_apiv4_host 10.1.0.253
        freenas_password
        freenas_use_ssl 0
        freenas_user root
        nowritecache 0
        sparse 0
 
I just performed a migration of the VM while tailing syslog, below is what occurred.

Code: 
Apr 24 16:07:48 proxve2 pvedaemon[603608]: <root@pam> starting task UPID:proxve2:00093860:01855AE9:6446EFA4:qmigrate:110:root@pam:
Apr 24 16:07:51 proxve2 pmxcfs[2304]: [status] notice: received log
Apr 24 16:08:13 proxve2 pmxcfs[2304]: [status] notice: received log
Apr 24 16:08:17 proxve2 QEMU[50083]: kvm: iSCSI CheckCondition: SENSE KEY:UNIT_ATTENTION(6) ASCQ:(null)(0x3f0e)
Apr 24 16:08:19 proxve2 kernel: [255194.570969] vmbr1: port 11(tap110i0) entered disabled state
Apr 24 16:08:19 proxve2 qmeventd[723]: read: Connection reset by peer
Apr 24 16:08:19 proxve2 systemd[1]: 110.scope: Succeeded.
Apr 24 16:08:19 proxve2 systemd[1]: 110.scope: Consumed 27min 52.157s CPU time.
Apr 24 16:08:20 proxve2 pvedaemon[603608]: <root@pam> end task UPID:proxve2:00093860:01855AE9:6446EFA4:qmigrate:110:root@pam: OK
 
Thank you for the above information!

The issue looks like from the LVM storage? since I can see that everything is good in the above outputs, I would ask what about if you run `pvs` command in the Proxmox VE shell? - And do you see this warning on both nodes?
 
you might need to filter your iscsi volumes so that they are not mistaken for host PVs (I assume you are using LVM in (some) VMs? possibly using clones, so the same LVM setup is "visible" multiple times on the PVE host?) - that's done via lvm.conf. add appropriate filters that match volumes that should not be used for LVM directly on the host. by default, the pve volume group and all ZVOLs are added there, but of course we cannot know about all possible custom configurations and filter them out pro-actively ;))
 
  • Like
Reactions: Moayad
Moayad said:
Thank you for the above information!

The issue looks like from the LVM storage? since I can see that everything is good in the above outputs, I would ask what about if you run `pvs` command in the Proxmox VE shell? - And do you see this warning on both nodes?
Click to expand...
Hmmm...not sure why it's using LVM for storage when I'm using ZFS over iSCSI.

Code: 
root@proxve1:~# pvs
  WARNING: Not using device /dev/sds3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdd3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdt3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdaj3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdf3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdv3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdh3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdk3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdl3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdn3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdp3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  PV             VG        Fmt  Attr PSize   PFree 
  /dev/nvme0n1p3 pve       lvm2 a--  232.38g <16.00g
  /dev/sdb3      ubuntu-vg lvm2 a--  <21.95g  10.97g

Code: 
root@proxve2:~# pvs
  WARNING: Not using device /dev/sds3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdd3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdt3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdaj3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdf3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdv3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdh3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdk3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdl3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdn3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: Not using device /dev/sdp3 for PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  WARNING: PV 9TA5rB-8rdj-YYuv-hP2a-WVTo-YVEt-Iyk3zR prefers device /dev/sdb3 because device was seen first.
  PV             VG        Fmt  Attr PSize   PFree 
  /dev/nvme0n1p3 pve       lvm2 a--  232.38g <16.00g
  /dev/sdb3      ubuntu-vg lvm2 a--  <21.95g  10.97g
 
Well, I removed the default local LVM from the storage config (in the GUI), and those warning messages have stopped.
 
the underlying issue is likely still there, it's just not visible in the migration log because no part of the migration will query LVM now. if you have block devices visible on the host that contain LVM data, they might be used by the host. this is dangerous if the LVM data actually belongs to guests. you should setup LVM filtering for any block devices that are visible on the host that belong to guests, or you risk corrupting them at some point by accident.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom