Failed to migrate disk (Device /dev/dm-xx not initialized in udev database even after waiting 10000

dumdum

New Member
May 10, 2019
5
0
1
37
pveversion:
pve-manager/6.0-4/2a719255 (running kernel: 5.0.15-1-pve)

Migrate disk fails to every time with the below message.
I am suspecting a kernel related issue with LVM thinpool on Proxmox 6.0, see further below for kern.log.
I see the same error for timeout from udev also when creating a new VM in LVM thinpool storage.


Task Log
Code:
...
Transferred: 21474836480 bytes remaining: 0 bytes total: 21474836480 bytes progression: 100.00 %
  WARNING: Device /dev/dm-17 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-17 not initialized in udev database even after waiting 10000000 microseconds.
  Logical volume "vm-100-disk-0" successfully removed
  WARNING: Device /dev/dm-17 not initialized in udev database even after waiting 10000000 microseconds.
TASK ERROR: storage migration failed: command '/sbin/lvs --separator : --noheadings --units b --unbuffered --nosuffix --options lv_size /dev/vg01/vm-100-disk-0' failed: got timeout
kern.log
Code:
kernel: [ 1453.178236] INFO: task systemd-udevd:4712 blocked for more than 120 seconds.
kernel: [ 1453.178266]       Tainted: P           O      5.0.15-1-pve #1
kernel: [ 1453.178284] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: [ 1453.178307] systemd-udevd   D    0  4712    934 0x00000100
kernel: [ 1453.178310] Call Trace:
kernel: [ 1453.178322]  __schedule+0x2d4/0x870
kernel: [ 1453.178325]  schedule+0x2c/0x70
kernel: [ 1453.178327]  schedule_preempt_disabled+0xe/0x10
kernel: [ 1453.178329]  __mutex_lock.isra.10+0x2e4/0x4c0
kernel: [ 1453.178338]  ? exact_lock+0x11/0x20
kernel: [ 1453.178339]  ? disk_map_sector_rcu+0x70/0x70
kernel: [ 1453.178341]  __mutex_lock_slowpath+0x13/0x20
kernel: [ 1453.178342]  mutex_lock+0x2c/0x30
kernel: [ 1453.178347]  __blkdev_get+0x7b/0x550
kernel: [ 1453.178348]  ? bd_acquire+0xd0/0xd0
kernel: [ 1453.178350]  blkdev_get+0x10c/0x330
kernel: [ 1453.178351]  ? bd_acquire+0xd0/0xd0
kernel: [ 1453.178352]  blkdev_open+0x92/0x100
kernel: [ 1453.178356]  do_dentry_open+0x143/0x3a0
kernel: [ 1453.178359]  vfs_open+0x2d/0x30
kernel: [ 1453.178361]  path_openat+0x2d4/0x16d0
kernel: [ 1453.178366]  ? page_add_file_rmap+0x5f/0x220
kernel: [ 1453.178370]  ? alloc_set_pte+0x104/0x5b0
kernel: [ 1453.178373]  do_filp_open+0x93/0x100
kernel: [ 1453.178381]  ? strncpy_from_user+0x56/0x1b0
kernel: [ 1453.178397]  ? __alloc_fd+0x46/0x150
kernel: [ 1453.178399]  do_sys_open+0x177/0x280
kernel: [ 1453.178400]  __x64_sys_openat+0x20/0x30
kernel: [ 1453.178407]  do_syscall_64+0x5a/0x110
kernel: [ 1453.178410]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
kernel: [ 1453.178413] RIP: 0033:0x7fe8ae3cc1ae
kernel: [ 1453.178418] Code: Bad RIP value.
kernel: [ 1453.178419] RSP: 002b:00007ffe83be4780 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
kernel: [ 1453.178420] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fe8ae3cc1ae
kernel: [ 1453.178421] RDX: 0000000000080000 RSI: 00005631364785c0 RDI: 00000000ffffff9c
kernel: [ 1453.178421] RBP: 00007fe8adbebc60 R08: 0000563135465270 R09: 000000000000000f
kernel: [ 1453.178422] R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
kernel: [ 1453.178423] R13: 0000000000000000 R14: 0000000000000000 R15: 0000563136454dc0
 

inetshell

New Member
Aug 16, 2019
3
0
1
31
Please try using this command and see if it works fine after that:
udevadm trigger
 

kernel2008

New Member
Oct 21, 2019
1
0
1
41
pveversion:
pve-manager/6.0-4/2a719255 (running kernel: 5.0.15-1-pve)

Migrate disk fails to every time with the below message.
I am suspecting a kernel related issue with LVM thinpool on Proxmox 6.0, see further below for kern.log.
I see the same error for timeout from udev also when creating a new VM in LVM thinpool storage.


Task Log
Code:
...
Transferred: 21474836480 bytes remaining: 0 bytes total: 21474836480 bytes progression: 100.00 %
  WARNING: Device /dev/dm-17 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-17 not initialized in udev database even after waiting 10000000 microseconds.
  Logical volume "vm-100-disk-0" successfully removed
  WARNING: Device /dev/dm-17 not initialized in udev database even after waiting 10000000 microseconds.
TASK ERROR: storage migration failed: command '/sbin/lvs --separator : --noheadings --units b --unbuffered --nosuffix --options lv_size /dev/vg01/vm-100-disk-0' failed: got timeout
kern.log
Code:
kernel: [ 1453.178236] INFO: task systemd-udevd:4712 blocked for more than 120 seconds.
kernel: [ 1453.178266]       Tainted: P           O      5.0.15-1-pve #1
kernel: [ 1453.178284] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: [ 1453.178307] systemd-udevd   D    0  4712    934 0x00000100
kernel: [ 1453.178310] Call Trace:
kernel: [ 1453.178322]  __schedule+0x2d4/0x870
kernel: [ 1453.178325]  schedule+0x2c/0x70
kernel: [ 1453.178327]  schedule_preempt_disabled+0xe/0x10
kernel: [ 1453.178329]  __mutex_lock.isra.10+0x2e4/0x4c0
kernel: [ 1453.178338]  ? exact_lock+0x11/0x20
kernel: [ 1453.178339]  ? disk_map_sector_rcu+0x70/0x70
kernel: [ 1453.178341]  __mutex_lock_slowpath+0x13/0x20
kernel: [ 1453.178342]  mutex_lock+0x2c/0x30
kernel: [ 1453.178347]  __blkdev_get+0x7b/0x550
kernel: [ 1453.178348]  ? bd_acquire+0xd0/0xd0
kernel: [ 1453.178350]  blkdev_get+0x10c/0x330
kernel: [ 1453.178351]  ? bd_acquire+0xd0/0xd0
kernel: [ 1453.178352]  blkdev_open+0x92/0x100
kernel: [ 1453.178356]  do_dentry_open+0x143/0x3a0
kernel: [ 1453.178359]  vfs_open+0x2d/0x30
kernel: [ 1453.178361]  path_openat+0x2d4/0x16d0
kernel: [ 1453.178366]  ? page_add_file_rmap+0x5f/0x220
kernel: [ 1453.178370]  ? alloc_set_pte+0x104/0x5b0
kernel: [ 1453.178373]  do_filp_open+0x93/0x100
kernel: [ 1453.178381]  ? strncpy_from_user+0x56/0x1b0
kernel: [ 1453.178397]  ? __alloc_fd+0x46/0x150
kernel: [ 1453.178399]  do_sys_open+0x177/0x280
kernel: [ 1453.178400]  __x64_sys_openat+0x20/0x30
kernel: [ 1453.178407]  do_syscall_64+0x5a/0x110
kernel: [ 1453.178410]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
kernel: [ 1453.178413] RIP: 0033:0x7fe8ae3cc1ae
kernel: [ 1453.178418] Code: Bad RIP value.
kernel: [ 1453.178419] RSP: 002b:00007ffe83be4780 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
kernel: [ 1453.178420] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fe8ae3cc1ae
kernel: [ 1453.178421] RDX: 0000000000080000 RSI: 00005631364785c0 RDI: 00000000ffffff9c
kernel: [ 1453.178421] RBP: 00007fe8adbebc60 R08: 0000563135465270 R09: 000000000000000f
kernel: [ 1453.178422] R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
kernel: [ 1453.178423] R13: 0000000000000000 R14: 0000000000000000 R15: 0000563136454dc0
I have the same problem but the command "udevadm trigger " doesn't work.
I have two LVM-Thin: "local-lvm" and "my-add-lvm".The problem disaper when cloning vm from local-lvm to local-lvm and creating new vm on my-add-lvm.Just I clone vm from my-add-lvm to my-add-lvm, and clone vm from local-lvm to my-add-lvm,the task must failure.The error message in kernel.log is same as you.
Code:
... 
WARNING: Device /dev/dm-23 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/dm-23 not initialized in udev database even after waiting 10000000 microseconds.
  Logical volume "vm-123-disk-0" successfully removed
  WARNING: Device /dev/dm-23 not initialized in udev database even after waiting 10000000 microseconds.
TASK ERROR: clone failed: command '/sbin/lvs --separator : --noheadings --units b --unbuffered --nosuffix --options lv_size /dev/vgdata/vm-123-disk-0' failed: got timeout
kern.log
Code:
... 
kernel: [2330700.135191] INFO: task qemu-img:13646 blocked for more than 120 seconds.
kernel: [2330700.135203]       Tainted: P        W  O      5.0.15-1-pve #1
kernel: [2330700.135207] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: [2330700.135213] qemu-img        D    0 13646  13619 0x00000000
kernel: [2330700.135218] Call Trace:
kernel: [2330700.135232]  __schedule+0x2d4/0x870
kernel: [2330700.135237]  schedule+0x2c/0x70
kernel: [2330700.135244]  io_schedule+0x16/0x40
kernel: [2330700.135251]  wait_on_page_bit+0x141/0x210
kernel: [2330700.135256]  ? file_check_and_advance_wb_err+0xe0/0xe0
kernel: [2330700.135262]  write_cache_pages+0x381/0x4d0
kernel: [2330700.135266]  ? __wb_calc_thresh+0x130/0x130
kernel: [2330700.135272]  generic_writepages+0x56/0x90
kernel: [2330700.135278]  blkdev_writepages+0xe/0x10
kernel: [2330700.135281]  do_writepages+0x41/0xd0
kernel: [2330700.135287]  ? __wake_up_common_lock+0x8e/0xc0
kernel: [2330700.135292]  __filemap_fdatawrite_range+0xc5/0x100
kernel: [2330700.135297]  filemap_write_and_wait+0x31/0x90
kernel: [2330700.135301]  __blkdev_put+0x72/0x1e0
kernel: [2330700.135304]  ? fsnotify+0x28b/0x3c0
kernel: [2330700.135307]  ? fsnotify+0x2ef/0x3c0
kernel: [2330700.135311]  blkdev_put+0x4c/0xd0
kernel: [2330700.135314]  blkdev_close+0x34/0x70
kernel: [2330700.135320]  __fput+0xbc/0x230
kernel: [2330700.135325]  ____fput+0xe/0x10
kernel: [2330700.135331]  task_work_run+0x9d/0xc0
kernel: [2330700.135338]  exit_to_usermode_loop+0xf2/0x100
kernel: [2330700.135342]  do_syscall_64+0xf0/0x110
kernel: [2330700.135348]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
kernel: [2330700.135351] RIP: 0033:0x7f46b99ee5d7
kernel: [2330700.135359] Code: Bad RIP value.
kernel: [2330700.135361] RSP: 002b:00007ffdc0680100 EFLAGS: 00000293 ORIG_RAX: 0000000000000003
kernel: [2330700.135364] RAX: 0000000000000000 RBX: 000000000000000a RCX: 00007f46b99ee5d7
kernel: [2330700.135366] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000000a
kernel: [2330700.135367] RBP: 0000000000000000 R08: 0000000000000001 R09: 00007f46ad6d2960
kernel: [2330700.135368] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000000
kernel: [2330700.135370] R13: 0000000000000000 R14: ffffffffffffffff R15: 0000000000000000
 
Last edited:

MPG

New Member
Nov 7, 2019
1
0
1
52
I have the same problem but the command "udevadm trigger " doesn't work.
I have two LVM-Thin: "local-lvm" and "my-add-lvm".The problem disaper when cloning vm from local-lvm to local-lvm and creating new vm on my-add-lvm.Just I clone vm from my-add-lvm to my-add-lvm, and clone vm from local-lvm to my-add-lvm,the task must failure.The error message in kernel.log is same as you.
Same problem here.

Any news ?
 

Christos Dalamagkas

New Member
Aug 6, 2019
1
0
1
25
I was having the same problem, not able to clone/migrate storage to my second lvm-thin stogare (2.18 TiB RAID10 Logical Volume composed of 4 SAS HDDs). The primary lvm-thin (RAID1 logical volume composed of two SSDs) was working fin. The problem does not appear if I configure the second storage as a directory, but is desirable to make lvm-thin work on the second storage.

Although, I had to order a second server, exactly with the same specifications (HPE ProLiant ML350), and lvm-thin on the second storage worked like a charm!

After comparing the hw/sw specifications of the two servers one-by-one, the first server has different RAID controller (HPE Smart Array E208e-p SR Gen10) instead of the HPE Smart Array P408i-a SR Gen10 that is on the second server. That was the only difference between the two servers.

So, what is your RAID controller? Is it possible that the controller is incompatible with lvm-thin on SAS HDDs?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!