New Kernel for Proxmox VE 3.0 (pvetest)

[informant]

Hi, thanks for yesterday update. the firtst manually backups are working fine but auto-backup in the night make following error again with hight load:

...
Aug 14 02:57:00 pegasus rrdcached[2462]: rotating journals
Aug 14 02:57:00 pegasus rrdcached[2462]: started new journal /var/lib/rrdcached/journal/rrd.journal.1376441820.048185
Aug 14 02:57:00 pegasus rrdcached[2462]: removing old journal /var/lib/rrdcached/journal/rrd.journal.1376434620.048197
Aug 14 02:57:00 pegasus pmxcfs[2475]: [dcdb] notice: data verification successful
Aug 14 03:17:01 pegasus /USR/SBIN/CRON[175497]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Aug 14 03:57:00 pegasus rrdcached[2462]: flushing old values
Aug 14 03:57:00 pegasus rrdcached[2462]: rotating journals
Aug 14 03:57:00 pegasus rrdcached[2462]: started new journal /var/lib/rrdcached/journal/rrd.journal.1376445420.048208
Aug 14 03:57:00 pegasus rrdcached[2462]: removing old journal /var/lib/rrdcached/journal/rrd.journal.1376438220.048171
Aug 14 03:57:00 pegasus pmxcfs[2475]: [dcdb] notice: data verification successful
Aug 14 04:17:01 pegasus /USR/SBIN/CRON[200881]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Aug 14 04:30:05 pegasus vzdump[136439]: ERROR: can't aquire lock '/var/run/vzdump.lock' - got timeout
Aug  14 04:30:06 pegasus vzdump[136344]: <root@pam> end task  UPID:pegasus:000214F7:00190635:520AC17D:vzdump::root@pam: unexpected  status
Aug 14 04:57:00 pegasus rrdcached[2462]: flushing old values
Aug 14 04:57:00 pegasus rrdcached[2462]: rotating journals
Aug 14 04:57:00 pegasus rrdcached[2462]: started new journal /var/lib/rrdcached/journal/rrd.journal.1376449020.048160
Aug 14 04:57:00 pegasus rrdcached[2462]: removing old journal /var/lib/rrdcached/journal/rrd.journal.1376441820.048185
Aug 14 04:57:00 pegasus pmxcfs[2475]: [dcdb] notice: data verification successful
Aug 14 04:58:32 pegasus kernel: device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
Aug 14 04:58:32 pegasus kernel: EXT3-fs error (device dm-3): ext3_find_entry: reading directory #232710368 offset 0
Aug 14 04:58:32 pegasus kernel: Buffer I/O error on device dm-3, logical block 0
Aug 14 04:58:32 pegasus kernel: lost page write due to I/O error on dm-3
Aug 14 04:58:32 pegasus kernel: EXT3-fs (dm-3): I/O error while writing superblock
Aug 14 04:58:32 pegasus kernel: EXT3-fs error (device dm-3): ext3_find_entry: reading directory #232710368 offset 0
Aug 14 04:58:32 pegasus kernel: Buffer I/O error on device dm-3, logical block 0
Aug 14 04:58:32 pegasus kernel: lost page write due to I/O error on dm-3
Aug 14 04:58:32 pegasus kernel: EXT3-fs (dm-3): I/O error while writing superblock
Aug 14 04:58:32 pegasus kernel: EXT3-fs error (device dm-3): ext3_find_entry: reading directory #232710368 offset 0
Aug 14 04:58:32 pegasus kernel: Buffer I/O error on device dm-3, logical block 0
Aug 14 04:58:32 pegasus kernel: lost page write due to I/O error on dm-3
Aug 14 04:58:32 pegasus kernel: EXT3-fs (dm-3): I/O error while writing superblock
Aug 14 04:58:32 pegasus kernel: EXT3-fs error (device dm-3): ext3_find_entry: reading directory #232710368 offset 0
Aug 14 04:58:32 pegasus kernel: Buffer I/O error on device dm-3, logical block 0
Aug 14 04:58:32 pegasus kernel: lost page write due to I/O error on dm-3
Aug 14 04:58:32 pegasus kernel: EXT3-fs (dm-3): I/O error while writing superblock
Aug 14 04:58:32 pegasus kernel: EXT3-fs error (device dm-3): ext3_find_entry: reading directory #232710368 offset 0
Aug 14 04:58:32 pegasus kernel: Buffer I/O error on device dm-3, logical block 0
Aug 14 04:58:32 pegasus kernel: lost page write due to I/O error on dm-3
Aug 14 04:58:32 pegasus kernel: EXT3-fs (dm-3): I/O error while writing superblock
Aug 14 04:58:32 pegasus kernel: EXT3-fs error (device dm-3): ext3_find_entry: reading directory #232710368 offset 0
Aug 14 04:58:32 pegasus kernel: Buffer I/O error on device dm-3, logical block 0
Aug 14 04:58:32 pegasus kernel: lost page write due to I/O error on dm-3
...

does we proxmox team have any ideas to fix? very thanks.

regards

 

[dietmar]

Aug 14 04:58:32 pegasus kernel: device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
[/QUOTE]

Seems you simply run out of snapshot space ==> increase snapshot size.

 

[informant]

Hi dietmar,

/etc/vzdump.conf
size: 8192

but at the backup(time) files are not more changing as 100mb, 8 gb would thus range yes. or do you mean a other configuration?

regards

 

[dietmar]

The log says snapshot runs full - so please check with lvs/lvdisplay during backup.

Note: All container run on same LV by default, so any change from any container increase snapshot usage.

 

[informant]

OK, thanks for your information. we would check it again. can you implement a check, if it full, stop the backup, well if one backup make a error, all other ct´s and vm´s are not reachable and server hang with high load.

best regards

 

[dietmar]

OK, thanks for your information. we would check it again. can you implement a check

I would accept a patch if somebody implements it.

 

[lynchie]

did you a full-upgrade or just install the specific .deb files?

for me a full-upgrade offers to install also new packages:
glusterfs-client{a} glusterfs-common{a} libibverbs1{a} liblvm2app2.2{a} librdmacm1{a} socat{a}

where I'm not sure if I want to go down this road

regards
hk

I just updated the specific files, i.e. apt-get install lvm2 clvm dmsetup libdevmapper1.02.1 libdevmapper-event1.02.1

 

[hk@]

hi
not to break things we did a full-upgrade on four systems and backups seemed to work fine using lvm-snapshots.

we disabled the pvetest-repo afterwards and hope we do not break things this way. Any hint if we could break things would be appreciated.

and yes, a check during backup for any snapshot running full would be much appreciated - at least a warning would be nice.

regards
hk

 

[dietmar]

we disabled the pvetest-repo afterwards and hope we do not break things this way. Any hint if we could break things would be appreciated.

IMHO the code in pvetest is quite stable now, and we plan to release 3.1 quite soon (with code from pvetest).