Hi,
we're currently experiencing heavy trouble with kvm-guest running cloudlinux handling lots of I/O.
Concrete problem:
The server is running with a normal load and has enough memory free, but suddenly there is a page allocation failure of order 4 which is not too big of a chunk of memory. Few moments later the kernel panics at a Null pointer exception while trying to free memory.
As far as I understand, this happens when __list_del_entry is called, as this is where the Instruction Pointer points to. (dmesg-output below)
We see this behavior on all servers running a modified CentOS-Kernel, but on one Guest, which I'm going to use as reference here I noticed one thing:
The server always crashes at a systemd-logind process of a certain user. The user has noshell in /etc/passwd but has a cronjob enabled to call owncloud's cron.php.
System Configuration:
host:
package versions:
proxmox-ve: 4.3-72 (running kernel: 4.4.24-1-pve)
pve-manager: 4.3-12 (running version: 4.3-12/6894c9d9)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.4.24-1-pve: 4.4.24-72
pve-kernel-4.2.2-1-pve: 4.2.2-16
pve-kernel-4.2.3-2-pve: 4.2.3-22
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-47
qemu-server: 4.0-96
pve-firmware: 1.1-10
libpve-common-perl: 4.0-83
libpve-access-control: 4.0-19
libpve-storage-perl: 4.0-68
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-docs: 4.3-17
pve-qemu-kvm: 2.7.0-8
pve-container: 1.0-85
pve-firewall: 2.0-31
pve-ha-manager: 1.0-38
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 2.0.6-1
lxcfs: 2.0.5-pve1
criu: 1.6.0-1
novnc-pve: 0.5-8
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.8-pve13~bpo80
ceph: 0.94.9-1~bpo80+1
misc:
guest-system:
qemu-config:
boot: cdn
bootdisk: scsi0
cores: 8
cpu: host
ide2: none,media=cdrom
memory: 20000
name: some-server
net0: e1000=36:63:33:35:63:34,bridge=vmbr0
net1: e1000=36:38:66:39:65:38,bridge=vmbr1
numa: 1
ostype: l24
scsi0: pool:disk,discard=on,size=100G
scsi1: pool:disk1,discard=on,size=2000G
scsi2: other-pool:disk2,discard=on,size=2000G
scsi3: other-pool:disk3,discard=on,size=1500G
scsi4: ssd-pool:disk4,discard=on,size=5G
scsihw: virtio-scsi-pci
smbios1: uuid=c71b9c08-fdd3-4d2d-967f-26550d996eba
sockets: 2
kernel of the guest:
3.10.0-427.18.2.lve1.4.27.el7.x86_64
guest-dmesg at crash:
we're currently experiencing heavy trouble with kvm-guest running cloudlinux handling lots of I/O.
Concrete problem:
The server is running with a normal load and has enough memory free, but suddenly there is a page allocation failure of order 4 which is not too big of a chunk of memory. Few moments later the kernel panics at a Null pointer exception while trying to free memory.
As far as I understand, this happens when __list_del_entry is called, as this is where the Instruction Pointer points to. (dmesg-output below)
We see this behavior on all servers running a modified CentOS-Kernel, but on one Guest, which I'm going to use as reference here I noticed one thing:
The server always crashes at a systemd-logind process of a certain user. The user has noshell in /etc/passwd but has a cronjob enabled to call owncloud's cron.php.
System Configuration:
host:
package versions:
proxmox-ve: 4.3-72 (running kernel: 4.4.24-1-pve)
pve-manager: 4.3-12 (running version: 4.3-12/6894c9d9)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.4.24-1-pve: 4.4.24-72
pve-kernel-4.2.2-1-pve: 4.2.2-16
pve-kernel-4.2.3-2-pve: 4.2.3-22
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-47
qemu-server: 4.0-96
pve-firmware: 1.1-10
libpve-common-perl: 4.0-83
libpve-access-control: 4.0-19
libpve-storage-perl: 4.0-68
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-docs: 4.3-17
pve-qemu-kvm: 2.7.0-8
pve-container: 1.0-85
pve-firewall: 2.0-31
pve-ha-manager: 1.0-38
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 2.0.6-1
lxcfs: 2.0.5-pve1
criu: 1.6.0-1
novnc-pve: 0.5-8
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.8-pve13~bpo80
ceph: 0.94.9-1~bpo80+1
misc:
Code:
# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
cat /sys/kernel/mm/transparent_hugepage/defrag
[always] madvise never
# cat /proc/meminfo
MemTotal: 32942344 kB
MemFree: 997232 kB
MemAvailable: 3263744 kB
Buffers: 72468 kB
Cached: 1932180 kB
SwapCached: 477304 kB
Active: 24489424 kB
Inactive: 5115564 kB
Active(anon): 23669200 kB
Inactive(anon): 4063368 kB
Active(file): 820224 kB
Inactive(file): 1052196 kB
Unevictable: 3520 kB
Mlocked: 3520 kB
SwapTotal: 8388604 kB
SwapFree: 4721236 kB
Dirty: 1276 kB
Writeback: 0 kB
AnonPages: 27275920 kB
Mapped: 108412 kB
Shmem: 129796 kB
Slab: 1216424 kB
SReclaimable: 519520 kB
SUnreclaim: 696904 kB
KernelStack: 25776 kB
PageTables: 89112 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 24859776 kB
Committed_AS: 42287420 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 254592 kB
DirectMap2M: 29104128 kB
DirectMap1G: 4194304 kBguest-system:
qemu-config:
boot: cdn
bootdisk: scsi0
cores: 8
cpu: host
ide2: none,media=cdrom
memory: 20000
name: some-server
net0: e1000=36:63:33:35:63:34,bridge=vmbr0
net1: e1000=36:38:66:39:65:38,bridge=vmbr1
numa: 1
ostype: l24
scsi0: pool:disk,discard=on,size=100G
scsi1: pool:disk1,discard=on,size=2000G
scsi2: other-pool:disk2,discard=on,size=2000G
scsi3: other-pool:disk3,discard=on,size=1500G
scsi4: ssd-pool:disk4,discard=on,size=5G
scsihw: virtio-scsi-pci
smbios1: uuid=c71b9c08-fdd3-4d2d-967f-26550d996eba
sockets: 2
kernel of the guest:
3.10.0-427.18.2.lve1.4.27.el7.x86_64
guest-dmesg at crash:
Code:
[55604.405717] systemd-logind[720]: New user some-user logged in.
[55604.409706] systemd-logind: page allocation failure: order:4, mode:0x40d0
[55604.409723] CPU: 9 PID: 720 Comm: systemd-logind ve: 0 Tainted: G W OE ------------ 3.10.0-427.18.2.lve1.4.27.el7.x86_64 #1 15.2
[55604.409726] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.3-0-ge2fc41e-prebuilt.qemu-project.org 04/01/2014
[55604.409733] 0000000000000004 00000000048698de ffff88051215bac0 ffffffff816372d7
[55604.409742] ffff88051215bb50 ffffffff81180a80 0000000000000000 00000000ffffffff
[55604.409746] ffff880521fd8000 ffffffff811834f6 ffff88051215bb20 00000000048698de
[55604.409750] Call Trace:
[55604.409810] [<ffffffff816372d7>] dump_stack+0x19/0x1b
[55604.409854] [<ffffffff81180a80>] warn_alloc_failed+0x110/0x180
[55604.409880] [<ffffffff811834f6>] ? drain_local_pages+0x16/0x20
[55604.409885] [<ffffffff81185274>] __alloc_pages_nodemask+0xa54/0xc10
[55604.409893] [<ffffffff811cb349>] alloc_pages_current+0xa9/0x170
[55604.410597] [<ffffffff8119f918>] kmalloc_order+0x18/0x50
[55604.410897] [<ffffffff8119f976>] kmalloc_order_trace+0x26/0xa0
[55604.411232] [<ffffffff811d8cb9>] __kmalloc+0x259/0x270
[55604.411249] [<ffffffff811a4e72>] __list_lru_init+0xf2/0x1a0
[55604.411261] [<ffffffff811fcb64>] sget+0x184/0x3e0
[55604.411264] [<ffffffff811fc070>] ? get_anon_bdev+0x120/0x120
[55604.411269] [<ffffffff81194a80>] ? shmem_rename+0x20/0x20
[55604.411272] [<ffffffff811fd0f0>] mount_nodev+0x30/0xb0
[55604.411276] [<ffffffff81194708>] shmem_mount+0x18/0x20
[55604.411279] [<ffffffff811fdb65>] mount_fs+0x15/0xc0
[55604.411290] [<ffffffff8121a1ff>] vfs_kern_mount+0x5f/0xf0
[55604.411294] [<ffffffff8121ca39>] do_mount+0x219/0xa20
[55604.411297] [<ffffffff8117f9fe>] ? __get_free_pages+0xe/0x50
[55604.411301] [<ffffffff8121d2d6>] SyS_mount+0x96/0xf0
[55604.411311] [<ffffffff81647bc9>] system_call_fastpath+0x16/0x1b
[55604.411314] Mem-Info:
[55604.411316] Node 0 DMA per-cpu:
[55604.411319] CPU 0: hi: 0, btch: 1 usd: 0
[55604.411321] CPU 1: hi: 0, btch: 1 usd: 0
[55604.411323] CPU 2: hi: 0, btch: 1 usd: 0
[55604.411327] CPU 3: hi: 0, btch: 1 usd: 0
[55604.411329] CPU 4: hi: 0, btch: 1 usd: 0
[55604.411331] CPU 5: hi: 0, btch: 1 usd: 0
[55604.411333] CPU 6: hi: 0, btch: 1 usd: 0
[55604.411334] CPU 7: hi: 0, btch: 1 usd: 0
[55604.411336] CPU 8: hi: 0, btch: 1 usd: 0
[55604.411338] CPU 9: hi: 0, btch: 1 usd: 0
[55604.411340] CPU 10: hi: 0, btch: 1 usd: 0
[55604.411342] CPU 11: hi: 0, btch: 1 usd: 0
[55604.411344] CPU 12: hi: 0, btch: 1 usd: 0
[55604.411346] CPU 13: hi: 0, btch: 1 usd: 0
[55604.411347] CPU 14: hi: 0, btch: 1 usd: 0
[55604.411349] CPU 15: hi: 0, btch: 1 usd: 0
[55604.411350] Node 0 DMA32 per-cpu:
[55604.411368] CPU 0: hi: 186, btch: 31 usd: 11
[55604.411372] CPU 1: hi: 186, btch: 31 usd: 32
[55604.411374] CPU 2: hi: 186, btch: 31 usd: 26
[55604.411376] CPU 3: hi: 186, btch: 31 usd: 0
[55604.411377] CPU 4: hi: 186, btch: 31 usd: 52
[55604.411379] CPU 5: hi: 186, btch: 31 usd: 41
[55604.411381] CPU 6: hi: 186, btch: 31 usd: 0
[55604.411383] CPU 7: hi: 186, btch: 31 usd: 27
[55604.411385] CPU 8: hi: 186, btch: 31 usd: 0
[55604.411387] CPU 9: hi: 186, btch: 31 usd: 0
[55604.411388] CPU 10: hi: 186, btch: 31 usd: 47
[55604.411390] CPU 11: hi: 186, btch: 31 usd: 0
[55604.411392] CPU 12: hi: 186, btch: 31 usd: 0
[55604.411394] CPU 13: hi: 186, btch: 31 usd: 0
[55604.411397] CPU 14: hi: 186, btch: 31 usd: 0
[55604.411399] CPU 15: hi: 186, btch: 31 usd: 0
[55604.411400] Node 0 Normal per-cpu:
[55604.411402] CPU 0: hi: 186, btch: 31 usd: 0
[55604.411411] CPU 1: hi: 186, btch: 31 usd: 0
[55604.411413] CPU 2: hi: 186, btch: 31 usd: 0
[55604.411415] CPU 3: hi: 186, btch: 31 usd: 0
[55604.411416] CPU 4: hi: 186, btch: 31 usd: 0
[55604.411418] CPU 5: hi: 186, btch: 31 usd: 0
[55604.411420] CPU 6: hi: 186, btch: 31 usd: 0
[55604.411422] CPU 7: hi: 186, btch: 31 usd: 0
[55604.411424] CPU 8: hi: 186, btch: 31 usd: 0
[55604.411425] CPU 9: hi: 186, btch: 31 usd: 0
[55604.411427] CPU 10: hi: 186, btch: 31 usd: 0
[55604.411429] CPU 11: hi: 186, btch: 31 usd: 0
[55604.411432] CPU 12: hi: 186, btch: 31 usd: 0
[55604.411436] CPU 13: hi: 186, btch: 31 usd: 0
[55604.411438] CPU 14: hi: 186, btch: 31 usd: 0
[55604.411439] CPU 15: hi: 186, btch: 31 usd: 0
[55604.411441] Node 1 Normal per-cpu:
[55604.411443] CPU 0: hi: 186, btch: 31 usd: 0
[55604.411445] CPU 1: hi: 186, btch: 31 usd: 0
[55604.411447] CPU 2: hi: 186, btch: 31 usd: 0
[55604.411449] CPU 3: hi: 186, btch: 31 usd: 0
[55604.411450] CPU 4: hi: 186, btch: 31 usd: 1
[55604.411452] CPU 5: hi: 186, btch: 31 usd: 0
[55604.411456] CPU 6: hi: 186, btch: 31 usd: 0
[55604.411458] CPU 7: hi: 186, btch: 31 usd: 0
[55604.411460] CPU 8: hi: 186, btch: 31 usd: 0
[55604.411462] CPU 9: hi: 186, btch: 31 usd: 0
[55604.411464] CPU 10: hi: 186, btch: 31 usd: 29
[55604.411465] CPU 11: hi: 186, btch: 31 usd: 30
[55604.411467] CPU 12: hi: 186, btch: 31 usd: 0
[55604.411469] CPU 13: hi: 186, btch: 31 usd: 19
[55604.411471] CPU 14: hi: 186, btch: 31 usd: 21
[55604.411474] CPU 15: hi: 186, btch: 31 usd: 0
[55604.411480] active_anon:147449 inactive_anon:287071 isolated_anon:64
active_file:1297168 inactive_file:1594428 isolated_file:0
unevictable:1 dirty:663 writeback:77 wbtmp:0 unstable:0
free:561839 slab_reclaimable:775298 slab_unreclaimable:201570
mapped:15899 shmem:64507 pagetables:22828 bounce:0
free_cma:0
[55604.411484] Node 0 DMA free:15892kB min:72kB low:88kB high:108kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[55604.411498] lowmem_reserve[]: 0 2810 9578 9578
[55604.411503] Node 0 DMA32 free:287876kB min:13028kB low:16284kB high:19540kB active_anon:94652kB inactive_anon:223420kB active_file:831636kB inactive_file:835992kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:3129200kB managed:2879632kB mlocked:0kB dirty:312kB writeback:304kB mapped:9876kB shmem:39280kB slab_reclaimable:422704kB slab_unreclaimable:126968kB kernel_stack:1232kB pagetables:21380kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[55604.411517] lowmem_reserve[]: 0 0 6768 6768
[55604.411520] Node 0 Normal free:448360kB min:31376kB low:39220kB high:47064kB active_anon:271196kB inactive_anon:541004kB active_file:1979388kB inactive_file:2086756kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:7094272kB managed:6930708kB mlocked:0kB dirty:968kB writeback:0kB mapped:26292kB shmem:98924kB slab_reclaimable:1108348kB slab_unreclaimable:298536kB kernel_stack:3744kB pagetables:41852kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[55604.411526] lowmem_reserve[]: 0 0 0 0
[55604.411530] Node 1 Normal free:1495228kB min:45628kB low:57032kB high:68440kB active_anon:223948kB inactive_anon:383860kB active_file:2377648kB inactive_file:3454964kB unevictable:4kB isolated(anon):128kB isolated(file):0kB present:10240000kB managed:10077924kB mlocked:4kB dirty:1372kB writeback:4kB mapped:27428kB shmem:119824kB slab_reclaimable:1570140kB slab_unreclaimable:380760kB kernel_stack:2640kB pagetables:28080kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[55604.411538] lowmem_reserve[]: 0 0 0 0
[55604.411541] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB
[55604.411581] Node 0 DMA32: 30057*4kB (UEM) 16700*8kB (UEM) 2035*16kB (UEM) 6*32kB (MR) 0*64kB 1*128kB (R) 1*256kB (R) 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 287476kB
[55604.411596] Node 0 Normal: 53133*4kB (UEM) 23633*8kB (UEM) 2858*16kB (UEM) 7*32kB (MR) 0*64kB 1*128kB (R) 0*256kB 0*512kB 0*1024kB 1*2048kB (R) 0*4096kB = 449724kB
[55604.411609] Node 1 Normal: 198856*4kB (UEM) 66254*8kB (UEM) 10396*16kB (UEM) 92*32kB (UMR) 5*64kB (R) 3*128kB (R) 2*256kB (R) 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 1496464kB
[55604.411640] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[55604.411655] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[55604.411657] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[55604.411659] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[55604.411660] 2956431 total pagecache pages
[55604.411663] 328 pages in swap cache
[55604.411665] Swap cache stats: add 749, delete 421, find 47816/47840
[55604.411668] Free swap = 617040kB
[55604.411671] Total swap = 619136kB
[55604.411678] 5119866 pages RAM
[55604.411680] 0 pages HighMem/MovableOnly
[55604.411681] 143823 pages reserved
[55604.412870] BUG: unable to handle kernel NULL pointer dereference at (null)
[55604.412982] IP: [<ffffffff8130b419>] __list_del_entry+0x29/0xd0
[55604.413012] PGD 0
[55604.413012] Oops: 0000 [#1] SMP
[55604.423458] systemd[1]: Got message type=signal sender=org.freedesktop.DBus destination=n/a object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=NameOwnerChanged cookie=9542 reply_cookie=0 error=n/a
[55604.413012] Modules linked in: kmodlve(OE) nf_conntrack_ftp ip6t_rpfilter ipt_REJECT ip6t_REJECT xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vfat fat isofs xfs libcrc32c loop kvm_amd kvm pcspkr i2c_piix4 sg ppdev parport_pc shpchp virtio_balloon parport ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif sr_mod crct10dif_generic cdrom crct10dif_common ata_generic pata_acpi virtio_scsi cirrus drm_kms_helper ttm ata_piix drm virtio_pci virtio_ring libata e1000 serio_raw i2c_core virtio floppy
[55604.413012] CPU: 9 PID: 720 Comm: systemd-logind ve: 0 Tainted: G W OE ------------ 3.10.0-427.18.2.lve1.4.27.el7.x86_64 #1 15.2
[55604.413012] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.3-0-ge2fc41e-prebuilt.qemu-project.org 04/01/2014
[55604.413012] task: ffff880511670cb0 ti: ffff880512158000 task.ti: ffff880512158000
[55604.462032] RIP: 0010:[<ffffffff8130b419>] [<ffffffff8130b419>] __list_del_entry+0x29/0xd0
[55604.462032] RSP: 0018:ffff88051215bd50 EFLAGS: 00010207
[55604.462032] RAX: 0000000000000000 RBX: ffff8802c2ac93c8 RCX: dead000000200200
[55604.462032] RDX: 0000000000000000 RSI: 000000007280727e RDI: ffff8802c2ac93c8
[55604.462032] RBP: ffff88051215bd50 R08: 0000000000021fa0 R09: ffff880521c61fa0
[55604.462032] R10: ffffea00040beb80 R11: ffffffff811a4eca R12: ffff8802c2ac93c0
[55604.462032] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff819c8c40
[55604.462032] FS: 00007f37cb102900(0000) GS:ffff880521c40000(0000) knlGS:0000000000000000
[55604.462032] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[55604.462032] CR2: 0000000000000000 CR3: 0000000510a61000 CR4: 00000000000007e0
[55604.462032] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[55604.462032] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[55604.462032] Stack:
[55604.462032] ffff88051215bd68 ffffffff8130b4cd ffff8802c2ac9000 ffff88051215bd88
[55604.462032] ffffffff811a4d20 ffff8802c2ac9000 ffff8802c2ac93c0 ffff88051215bda0
[55604.462032] ffffffff811fbc79 ffff8802c2ac9000 ffff88051215bdf0 ffffffff811fcce8
[55604.462032] Call Trace:
[55604.462032] [<ffffffff8130b4cd>] list_del+0xd/0x30
[55604.462032] [<ffffffff811a4d20>] list_lru_destroy+0x30/0x90
[55604.462032] [<ffffffff811fbc79>] destroy_super+0x19/0x90
[55604.462032] [<ffffffff811fcce8>] sget+0x308/0x3e0
[55604.462032] [<ffffffff811fc070>] ? get_anon_bdev+0x120/0x120
[55604.462032] [<ffffffff81194a80>] ? shmem_rename+0x20/0x20
[55604.517110] systemd[1]: Got message type=signal sender=org.freedesktop.DBus destination=n/a object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=NameOwnerChanged cookie=9543 reply_cookie=0 error=n/a
[55604.462032] [<ffffffff811fd0f0>] mount_nodev+0x30/0xb0
[55604.462032] [<ffffffff81194708>] shmem_mount+0x18/0x20
[55604.462032] [<ffffffff811fdb65>] mount_fs+0x15/0xc0
[55604.462032] [<ffffffff8121a1ff>] vfs_kern_mount+0x5f/0xf0
[55604.462032] [<ffffffff8121ca39>] do_mount+0x219/0xa20
[55604.462032] [<ffffffff8117f9fe>] ? __get_free_pages+0xe/0x50
[55604.462032] [<ffffffff8121d2d6>] SyS_mount+0x96/0xf0
[55604.462032] [<ffffffff81647bc9>] system_call_fastpath+0x16/0x1b
[55604.462032] Code: 00 00 55 48 8b 17 48 b9 00 01 10 00 00 00 ad de 48 8b 47 08 48 89 e5 48 39 ca 74 29 48 b9 00 02 20 00 00 00 ad de 48 39 c8 74 7a <4c> 8b 00 4c 39 c7 75 53 4c 8b 42 08 4c 39 c7 75 2b 48 89 42 08
[55604.462032] RIP [<ffffffff8130b419>] __list_del_entry+0x29/0xd0
[55604.462032] RSP <ffff88051215bd50>
[55604.462032] CR2: 0000000000000000