OpenVZ creation fails // CRASH

[JohnDoe]

Debian Squeeze // EXT4 // OpenVZ creation fails // CRASH

Hello!

I generated a debian lenny image / a proxmox-template with DAB.
It works perfectly on our production server. Now I wanted to use
it on a test-server, but container creation seems not to work.

The only difference seems to be that the production system is
build on a preinstalled debian lenny (5.0.x) an the test system,
that crashes, runs on a debian squeeze beta installation. kvm
seems to run perfectly, but VZ-creation (at all, also on proxmox
certified templates) fails.

$ sudo pveversion -v
pve-manager: 1.7-10 (pve-manager/1.7/5323)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.7-30
pve-kernel-2.6.32-4-pve: 2.6.32-30
qemu-server: 1.1-28
pve-firmware: 1.0-10
libpve-storage-perl: 1.0-16
vncterm: 0.9-2
vzctl: 3.0.24-11
vzdump: 1.2.6-1
vzprocps: 2.0.11-1dso2
vzquota: 3.0.12-3
pve-qemu-kvm: 0.13.0-3
ksm-control-daemon: 1.0-4

I added the crash report:

 

[JohnDoe]

BUG: unable to handle kernel NULL pointer dereference at (null)       
IP: [<(null)>] (null)                                                                 
PGD 22a47d067 PUD 22a518067 PMD 0                                                     
Oops: 0010 [#1] SMP                                                                   
last sysfs file: /sys/kernel/uevent_seqnum                                            
CPU 0                                                                                 
Modules linked in: vzethdev vznetdev simfs vzrst vzcpt vzdquota vzmon vzdev xt_tcpudp]
Pid: 2237, comm: tar Not tainted 2.6.32-4-pve #1 dzhanibekov PowerEdge T110           
RIP: 0010:[<0000000000000000>]  [<(null)>] (null)                                     
RSP: 0018:ffff88022a405a20  EFLAGS: 00010246                                          
RAX: ffffffffa02b9700 RBX: ffff88023011fcc0 RCX: 000000000000000c                     
RDX: 0000000000000000 RSI: 0000000000002000 RDI: ffff88023011fcc0                     
RBP: ffff880239e4d770 R08: 0000000000000000 R09: ffff88023011fc00                     
R10: ffff88023011f8c0 R11: 0000000000000000 R12: ffff88023011fc00                     
R13: 0000000000000002 R14: 0000000000000000 R15: ffff88023011ffc8                     
FS:  00007fe5972c5700(0000) GS:ffff88000aa00000(0000) knlGS:0000000000000000          
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033                                     
CR2: 0000000000000000 CR3: 000000022a474000 CR4: 00000000000026f0                     
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000                     
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400                     
Process tar (pid: 2237, veid=0, threadinfo ffff88022a404000, task ffff88023341c000)   
Stack:                                                                                
 ffffffffa014e4c4 ffff880239e4d770 0000000000000000 0000004000000000                  
<0> fffffffffffffff3 fffffffffffffff4 0000000000000000 ffffffffffff0000               
<0> 0000000000000001 ffff88023b6dbc00 0000000000000000 0000000009b69148               
Call Trace:                                                                           
 [<ffffffffa014e4c4>] ? ext4_da_get_block_prep+0x20c/0x318 [ext4]                     
 [<ffffffff81112230>] ? __block_prepare_write+0x14c/0x2c0                             
 [<ffffffffa014e2b8>] ? ext4_da_get_block_prep+0x0/0x318 [ext4]                       
 [<ffffffff810b7015>] ? add_to_page_cache_locked+0x76/0xc1                            
 [<ffffffff811124ff>] ? block_write_begin+0x7a/0xc7                                   
 [<ffffffffa01511db>] ? ext4_da_write_begin+0x1b9/0x272 [ext4]                        
 [<ffffffffa014e2b8>] ? ext4_da_get_block_prep+0x0/0x318 [ext4]                       
 [<ffffffffa0175af6>] ? ext4_xattr_get+0x1fa/0x27c [ext4]                             
 [<ffffffff810b798a>] ? generic_file_buffered_write+0x118/0x278                       
 [<ffffffff810b7e9b>] ? __generic_file_aio_write+0x25f/0x293                          
 [<ffffffffa01488ae>] ? ext4_file_open+0xc4/0xe8 [ext4]                               
 [<ffffffffa02b3846>] ? vzquota_qlnk_destroy+0x16/0xbe [vzdquota]                     
 [<ffffffff810b7f28>] ? generic_file_aio_write+0x59/0x9f                              
 [<ffffffffa01487ea>] ? ext4_file_open+0x0/0xe8 [ext4]                                
 [<ffffffff810f1282>] ? do_sync_write+0xce/0x113                                      
 [<ffffffff81066932>] ? autoremove_wake_function+0x0/0x2e                             
 [<ffffffff810e7853>] ? virt_to_head_page+0x9/0x2a                                    
 [<ffffffff810fc23b>] ? sys_mkdirat+0xc2/0xd1                                         
 [<ffffffff810f1c82>] ? vfs_write+0xa9/0x102                                          
 [<ffffffff810f1dee>] ? sys_write+0x49/0xc1                                           
 [<ffffffff81010c12>] ? system_call_fastpath+0x16/0x1b                                
Code:  Bad RIP value.                                                                 
RIP  [<(null)>] (null)                                                                
 RSP <ffff88022a405a20>                                                               
CR2: 0000000000000000                                                                 
---[ end trace 9bbce44ed8e26e65 ]---                                                  
BUG: unable to handle kernel NULL pointer dereference at (null)                       
IP: [<(null)>] (null)                                                                 
PGD 22e7cb067 PUD 22e6dc067 PMD 0                                                     
Oops: 0010 [#2] SMP                                                                   
last sysfs file: /sys/kernel/uevent_seqnum                                            
CPU 0                                                                                 
Modules linked in: vzethdev vznetdev simfs vzrst vzcpt vzdquota vzmon vzdev xt_tcpudp]
Pid: 2223, comm: vzctl Tainted: G      D    2.6.32-4-pve #1 dzhanibekov PowerEdge T110
RIP: 0010:[<0000000000000000>]  [<(null)>] (null)                                     
RSP: 0018:ffff88022a463d30  EFLAGS: 00010246                                          
RAX: ffffffffa02b9700 RBX: 0000000000000002 RCX: 000000000000000c                     
RDX: ffff88023b6de000 RSI: 0000000000002000 RDI: ffff88023006a0c0                     
RBP: ffff88023006a000 R08: 0000000000000003 R09: ffff8802300564c0                     
R10: 0000000000000000 R11: 0000000000000000 R12: ffff88023006a0c0                     
R13: ffffea0009bceb30 R14: ffff88023b6dbc00 R15: 0000000000000000                     
FS:  00007fcbe6549700(0000) GS:ffff88000aa00000(0000) knlGS:0000000000000000          
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033                                     
CR2: 0000000000000000 CR3: 000000022e5c8000 CR4: 00000000000026f0                     
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000                     
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400                     
Process vzctl (pid: 2223, veid=0, threadinfo ffff88022a462000, task ffff88023341d000) 
Stack:                                                                                
 ffffffffa014d05a 0000000000000000 ffffea0009bceb30 ffff88023006a1e0                  
<0> 0000000000000000 0000000000000000 ffff88022a463dd8 0000000000000000               
<0> ffffffff810bfd37 0000000000000000 ffffea0009bceb30 0000000000000000               
Call Trace:                                                                           
 [<ffffffffa014d05a>] ? ext4_da_invalidatepage+0x134/0x14b [ext4]                     
 [<ffffffff810bfd37>] ? truncate_inode_page+0x45/0x84                                 
 [<ffffffff810bfe20>] ? truncate_inode_pages_range+0xaa/0x2b0                         
 [<ffffffffa0122caf>] ? jbd2_journal_stop+0x20b/0x21e [jbd2]                          
 [<ffffffffa02b4113>] ? vzquota_inode_data+0x3b/0xa3 [vzdquota]                       
 [<ffffffffa0151294>] ? ext4_delete_inode+0x0/0x21c [ext4]                            
 [<ffffffffa02b4375>] ? vzquota_inode_init_call+0x3a/0x6d [vzdquota]                  
 [<ffffffffa01512d7>] ? ext4_delete_inode+0x43/0x21c [ext4]                           
 [<ffffffffa0151294>] ? ext4_delete_inode+0x0/0x21c [ext4]                            
 [<ffffffffa0151294>] ? ext4_delete_inode+0x0/0x21c [ext4]                            
 [<ffffffff81103a2c>] ? generic_delete_inode+0xd4/0x160                               
 [<ffffffff810fc008>] ? do_unlinkat+0xe2/0x134                                        
 [<ffffffff81317574>] ? do_page_fault+0x2e0/0x2fc                                     
 [<ffffffff813153f5>] ? page_fault+0x25/0x30                                          
 [<ffffffff81010c12>] ? system_call_fastpath+0x16/0x1b                                
Code:  Bad RIP value.                                                                 
RIP  [<(null)>] (null)                                                                
 RSP <ffff88022a463d30>                                                               
CR2: 0000000000000000                                                                 
---[ end trace 9bbce44ed8e26e66 ]---

 

[tom]

Debian Squeeze is not working/tested with the Proxmox VE 1.x packages. Proxmox VE 2.x is based on Squeeze (no release date set).

 

[JohnDoe]

Same error now on debian lenny

Okay. Proxmox is working under Debian Squeeze, but OpenVZ crashes on
ext4-handling with a null-pointer exception.

The only alternative would be XFS for me, because of performance aspects.
But as discribed here: http://wiki.openvz.org/Disk_quota , disk quotas only
work on ext filesystems. in detail only working on ext2 and ext3, crashing
on ext4.

The problem is related to a 2.6.32 kernel bug.
For ext4 there is currently a workaround:
http://serverfault.com/questions/173097/openvz-quota-error-on-gentoo
http://www.mail-archive.com/users@openvz.org/msg03482.html
http://lists.debian.org/debian-kernel/2010/11/msg00058.html
http://forum.proxmox.com/threads/4931-2.6.32-and-ext4-bug
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=587905
http://bugzilla.openvz.org/show_bug.cgi?id=1509
http://bugzilla.openvz.org/show_bug.cgi?id=1510

Option 1:
Disable Disk Quota in /etc/vz/vz.conf:
DISK_QUOTA=yes ---> DISK_QUOTA=no

Option 2:
mount ext4 with nodelalloc flag to disable delayed allocation
in /etc/fstab:
change, e.g.:
/dev/mapper/vg01-data /var/lib/vz/ ext4 noatime,relatime 0 2
to:
/dev/mapper/vg01-data /var/lib/vz/ ext4 noatime,relatime,nodelalloc 0 2

both options are known to work.

there would be a third option:
switch to a newer kernel
but, currently proxmox pve-kernel 2.6.35 not supports openvz

 

[tom]

why is ext3 not suited? you are right, its the only supported and reliable filesystem for OpenVZ (in our current system)

OpenVZ is not available for Kernels higher 2.6.32. Proxmox VE 2.x series should work with ext4, but its not available yet.

 

[JohnDoe]

why is ext3 not suited?

- performance and stability of xfs or ext4 is definitely better on many benchmarks, confirmed by own benchmarks
- ever had errors on an ext3 filesystem? journal checking and replaying can take days on big hdds, ext4 finished in minutes

OpenVZ is not available for Kernels higher 2.6.32.

i know.

 

[tom]

- performance and stability of xfs or ext4 is definitely better on many benchmarks, confirmed by own benchmarks
- ever had errors on an ext3 filesystem? journal checking and replaying can take days on big hdds, ext4 finished in minutes

yes, but depends on the use case. for OpenVZ ext3 is the most stable.

i know. developer branch needed.

based on the history of all OpenVZ development branches (after 2.6.18) I do not see any good chance for this. My personal opinion is that OpenVZ will be stable for 2.6.32 (RHEL6 branch) but there will be no usable OpenVZ in future kernels soon - hopefully lxc can fill this gap sometime.

 

[JohnDoe]

yes, but depends on the use case. for OpenVZ ext3 is the most stable.

but it's critical when the server filesystem is damaged und have to be rebuilded or checked and a production server then is offline for hours or days.
i prefer xfs and ext4 (since debian squeeze, where ext4 is stable module, ext4dev not needed anymore)

do you know about further issues/problems on ext4 or xfs usage (on openvz)?
for me xfs seems to be very stable. ok, disk quota seems not be applicable / ignored, but in our case, that's managable.
for kvm we're using lvm storage(s)

 

[tom]

AFAIK xfs works but does not have support for disk quotas inside containers - but I do not have personal experience, we run all on ext3.

 

[JohnDoe]

yes, but depends on the use case. for OpenVZ ext3 is the most stable.

AFAIK xfs works but does not have support for disk quotas inside containers - but I do not have personal experience, we run all on ext3.

uh! oh! ambivalent!
a little inconsistent and conflicting statement?!

"ext3 is the best and stable filesystem on openvz. we didn't use or test any other."

 

[tom]

uh! oh! ambivalent!
a little inconsistent and conflicting statement?!

No, I really know what I am talking about and I know our Kernel and our software. Currently OpenVZ stable is only available for our 2.6.18 branch, therefore its ext3. We cannot change facts just because you don´t like them. And yes, in future we will support OpenVZ and ext4. So what do you want NOW?

 

[JohnDoe]

Then you would know, that ext4 is also working on Proxmox/OpenVZ 2.6.32-kernel, since 2.6.28 it is in the linux kernel.
@see : http://wiki.openvz.org/Disk_quota
ext4 is known by the community to work (surely, only on a supporting kernel), isn't it?

 

[tom]

No, ext4 is not supported in our 2.6.32 branch - this is wrong. Proxmox VE 1.x works with ext3. Its the same like softraid. Could work but its not supported here.