Annoying disk noise tracked down, but what to do?

[Neil Duffy]

Hi,

I've been using Proxmox for a few weeks now and since changing from an Ubuntu server running KVM and Proxmox, I've noticed an annoying, periodic noise coming from my additional drives (i.e. not where Proxmox is installed).

I have Proxmox installed to my 250gb SSD, but the system also has 2 off 4TB drives (one internal, one USB).

Anyways, by echo 1 > /proc/sys/vm/block_dump and tailing /var/log/syslog I've found that the journaling daemon, JBD is constantly writing to dm-1, which is infact the /pve-root LVM

Jan 30 04:52:01 pve kernel: [ 2703.377913] jbd2/dm-1-8(308): WRITE block 59025936 on dm-1 (8 sectors)
Jan 30 04:52:01 pve kernel: [ 2703.377926] jbd2/dm-1-8(308): WRITE block 59025944 on dm-1 (8 sectors)
Jan 30 04:52:01 pve kernel: [ 2703.377930] jbd2/dm-1-8(308): WRITE block 59025952 on dm-1 (8 sectors)
Jan 30 04:52:01 pve kernel: [ 2703.377932] jbd2/dm-1-8(308): WRITE block 59025960 on dm-1 (8 sectors)
Jan 30 04:52:01 pve kernel: [ 2703.377934] jbd2/dm-1-8(308): WRITE block 59025968 on dm-1 (8 sectors)
Jan 30 04:52:01 pve kernel: [ 2703.377936] jbd2/dm-1-8(308): WRITE block 59025976 on dm-1 (8 sectors)
Jan 30 04:52:01 pve kernel: [ 2703.377945] jbd2/dm-1-8(308): WRITE block 59025984 on dm-1 (8 sectors)
Jan 30 04:52:01 pve kernel: [ 2703.377947] jbd2/dm-1-8(308): WRITE block 59025992 on dm-1 (8 sectors)
Jan 30 04:52:01 pve kernel: [ 2703.377949] jbd2/dm-1-8(308): WRITE block 59026000 on dm-1 (8 sectors)
Jan 30 04:52:01 pve kernel: [ 2703.377951] jbd2/dm-1-8(308): WRITE block 59026008 on dm-1 (8 sectors)
Jan 30 04:52:01 pve kernel: [ 2703.377953] jbd2/dm-1-8(308): WRITE block 59026016 on dm-1 (8 sectors)
Jan 30 04:52:01 pve kernel: [ 2703.377955] jbd2/dm-1-8(308): WRITE block 59026024 on dm-1 (8 sectors)
Jan 30 04:52:01 pve kernel: [ 2703.377957] jbd2/dm-1-8(308): WRITE block 59026032 on dm-1 (8 sectors)
Jan 30 04:52:01 pve kernel: [ 2703.378569] jbd2/dm-1-8(308): WRITE block 59026040 on dm-1 (8 sectors)

lrwxrwxrwx 1 root root 7 Jan 30 04:07 pve-data -> ../dm-5
lrwxrwxrwx 1 root root 7 Jan 30 04:07 pve-data_tdata -> ../dm-3
lrwxrwxrwx 1 root root 7 Jan 30 04:07 pve-data_tmeta -> ../dm-2
lrwxrwxrwx 1 root root 7 Jan 30 04:07 pve-data-tpool -> ../dm-4
lrwxrwxrwx 1 root root 7 Jan 30 04:07 pve-root -> ../dm-1
lrwxrwxrwx 1 root root 7 Jan 30 04:07 pve-swap -> ../dm-0
lrwxrwxrwx 1 root root 7 Jan 30 04:07 pve-vm--100--disk--1 -> ../dm-6

I understand the need for the journal and how it works, but what I cannot understand is why its causing IO operations on /dev/sda (external 4TB) and /dev/sdb (internal 4TB) at the same time?

The drives are not even mounted and don't show explicitly in the above logs, but the disk noise (and flashing led on the usb) drive are a clear indication of what's going on.

And it all happens perfectly in time with the JBD DM-1 process doing its thing.

Is this a kernel bug?

The only way I can stop it is to pass through all affected hardware to my windows 7 guest.

Any thoughts?

Thanks

Neil

 

[Brian Read]

It is possible I have the same error - only in my case it seems to try to write to the CD ROM, I get a "buzz" every few minutes from the server.

My server is ZFS based (three drives), plus an SSD as a cache.

On latest 5.1. although i feel sure it used to do it on V4 as well..

 

[Neil Duffy]

Hi Brian,

You might well have, although I have a CDROM and don't see any such activity like I'm seeing with the HD's.

From what I've read, this is a problem with the general linux kernel, and at one time was fixed, but the bug has re-emerged...

https://askubuntu.com/questions/735376/14-04-jbd2-and-kworker-constantly-writing-to-disk

What I cannot understand, is that even though this journalling activity correctly relates to the LVM install, shown as dm-1 in my original post, but it bother's the unmounted disks I have connected too.

As I said, I have to physically remove them from the debian / linux install by passing through the sata and usb3 interfaces the drives connect to.

Definitely something a miss and I'm sure its kernel related.

Neil