Filesystem errors during backup - is this normal??

O

oeginc

Member
Mar 21, 2009
133
0
16
I just built 2 brand new machines, fresh everything:

Gigabyte Motherboards
AMD Phenom(tm) 9850 Quad-Core Processors
Highpoint 2640x4 RAID controller w/4 seagate drives in RAID-10 configuration
1x500GB Hitachi drive for booting
1x2TB Hitachi drive for backups

I get tons of these:
Code: 
EXT4-fs error (device dm-5): __ext4_get_inode_loc: unable to read inode block - inode=13811696, block=55214592
and tons of these:
Code: 
EXT4-fs error (device dm-5): ext4_find_entry: reading directory #12730436 offset 0
and a few of these:
Code: 
EXT4-fs: mballoc: 1 blocks 1 reqs (0 success)
EXT4-fs: mballoc: 1 extents scanned, 0 goal hits, 1 2^N hits, 0 breaks, 0 lost
EXT4-fs: mballoc: 11 generated and it took 457676
EXT4-fs: mballoc: 512 preallocated, 0 discarded
ext4_abort called.
EXT4-fs error (device dm-5): ext4_put_super: Couldn't clean up the journal
EXT4-fs: barriers enabled
kjournald2 starting: pid 23341, dev dm-5:8, commit interval 5 seconds
EXT4 FS on dm-5, internal journal on dm-5:8
EXT4-fs: mballoc enabled
ext4_orphan_cleanup: deleting unreferenced inode 12592186
ext4_orphan_cleanup: deleting unreferenced inode 12553793
ext4_orphan_cleanup: deleting unreferenced inode 12553792
ext4_orphan_cleanup: deleting unreferenced inode 12553791
ext4_orphan_cleanup: deleting unreferenced inode 12550908
ext4_orphan_cleanup: deleting unreferenced inode 12550907
ext4_orphan_cleanup: deleting unreferenced inode 12435833
EXT4-fs: dm-5: 7 orphan inodes deleted
EXT4-fs: recovery complete.
EXT4-fs: mounted filesystem dm-5 with ordered data mode
EXT4-fs: mballoc: 1 blocks 1 reqs (0 success)
EXT4-fs: mballoc: 1 extents scanned, 0 goal hits, 1 2^N hits, 0 breaks, 0 lost
EXT4-fs: mballoc: 11 generated and it took 441812
EXT4-fs: mballoc: 512 preallocated, 0 discarded
EXT4-fs: barriers enabled
Quite often (almost everytime the backup runs)... Now is this normal and happening just because it's cleaning up the snapshot, or is this something to be concerned about? It's happening on both servers...

Additional Information... Just checked this mornings logs and noticed this... It happens on this particular VM fairly often..

Code: 
Apr 09 03:19:45 INFO: tar: ./etc/postfix/main.cf.bak: Warning: Read error at byte 0, while reading 1627 bytes: Input/output error
Apr 09 03:19:45 INFO: tar: ./etc/postfix/rbl_client_exceptions.db: Warning: Read error at byte 0, while reading 3584 bytes: Input/output e
rror
Apr 09 03:19:45 INFO: tar: ./etc/postfix/master.cf: Warning: Read error at byte 0, while reading 1024 bytes: Input/output error
Apr 09 03:19:45 INFO: tar: ./etc/postfix/postfix-files: Warning: Read error at byte 0, while reading 5120 bytes: Input/output error
Apr 09 03:19:45 INFO: tar: ./etc/postfix/mysql_virtual_alias_maps.cf: Warning: Read error at byte 0, while reading 177 bytes: Input/output
 error
Apr 09 03:19:45 INFO: tar: ./etc/postfix/main.cf: Warning: Read error at byte 0, while reading 2085 bytes: Input/output error
Apr 09 03:19:45 INFO: tar: ./etc/postfix/reject_over_quota.db: Warning: Read error at byte 0, while reading 1536 bytes: Input/output error
Apr 09 03:19:45 INFO: tar: ./etc/postfix/sasl/: Warning: Cannot savedir: Input/output error
Apr 09 03:19:45 INFO: tar: ./etc/postfix/sasl: Warning: Cannot close: Bad file descriptor
Apr 09 03:19:45 INFO: tar: ./etc/postfix/rbl_recipient_exceptions.db: Warning: Read error at byte 0, while reading 9216 bytes: Input/outpu
t error
Apr 09 03:19:45 INFO: tar: ./etc/postfix/virtual.db: Warning: Read error at byte 0, while reading 6144 bytes: Input/output error
Apr 09 03:19:45 INFO: tar: ./etc/postfix/post-install: Warning: Read error at byte 0, while reading 3584 bytes: Input/output error
Apr 09 03:19:45 INFO: tar: ./etc/sgml/: Warning: Cannot savedir: Input/output error
Apr 09 03:19:45 INFO: tar: ./etc/sgml: Warning: Cannot close: Bad file descriptor
Apr 09 03:19:45 INFO: tar: ./etc/sgml/catalog.old: Warning: Read error at byte 0, while reading 335 bytes: Input/output error
Apr 09 03:19:45 INFO: tar: ./etc/sgml/catalog: Warning: Read error at byte 0, while reading 366 bytes: Input/output error
Apr 09 03:19:45 INFO: tar: ./etc/sgml/xml-core.cat: Warning: Read error at byte 0, while reading 391 bytes: Input/output error
Apr 09 03:19:45 INFO: tar: ./etc/ca-certificates/: Warning: Cannot savedir: Input/output error
Apr 09 03:19:45 INFO: tar: ./etc/ca-certificates: Warning: Cannot close: Bad file descriptor
Apr 09 03:19:45 INFO: tar: ./etc/ca-certificates/update.d/: Warning: Cannot savedir: Input/output error
Apr 09 03:19:45 INFO: tar: ./etc/ca-certificates/update.d: Warning: Cannot close: Bad file descriptor
Apr 09 03:19:45 INFO: tar: ./etc/amavis/: Warning: Cannot savedir: Input/output error
Apr 09 03:19:45 INFO: tar: ./etc/amavis: Warning: Cannot close: Bad file descriptor
Apr 09 03:19:45 INFO: tar: ./etc/amavis/README.l10n: Warning: Read error at byte 0, while reading 978 bytes: Input/output error
Apr 09 03:19:45 INFO: tar: ./etc/amavis/en_US/: Warning: Cannot savedir: Input/output error
Apr 09 03:19:45 INFO: tar: ./etc/amavis/en_US: Warning: Cannot close: Bad file descriptor
Apr 09 03:19:45 INFO: tar: ./etc/amavis/en_US/template-spam-admin.txt: Warning: Read error at byte 0, while reading 1123 bytes: Input/outp
ut error
Apr 09 03:19:45 INFO: tar: ./etc/amavis/en_US/template-virus-admin.txt: Warning: Read error at byte 0, while reading 1731 bytes: Input/out
put error
Apr 09 03:19:45 INFO: tar: ./etc/amavis/en_US/template-spam-sender.txt: Warning: Read error at byte 0, while reading 1524 bytes: Input/out
put error
Apr 09 03:19:45 INFO: tar: ./etc/amavis/en_US/template-dsn.txt: Warning: Read error at byte 0, while reading 4713 bytes: Input/output erro
r
Apr 09 03:19:45 INFO: tar: ./etc/amavis/en_US/template-virus-sender.txt: Warning: Read error at byte 0, while reading 3229 bytes: Input/ou
tput error
Apr 09 03:19:45 INFO: tar: ./etc/amavis/en_US/template-virus-recipient.txt: Warning: Read error at byte 0, while reading 1543 bytes: Input
/output error
Apr 09 03:19:45 INFO: tar: ./etc/amavis/en_US/charset: Warning: Read error at byte 0, while reading 170 bytes: Input/output error
Apr 09 03:19:45 INFO: tar: ./etc/amavis/conf.d/: Warning: Cannot savedir: Input/output error
There are THOUSANDS of similar lines (the log file itself is 372MB!)... All of the other VM's backup fine with ZERO errors... This is an OpenVZ container, not a QEMU (otherwise I'd think the disk image might be bad)... Ideas? It's a fairly high-volume mailserver...

I have 16GB set aside for the snapshot, and this starts happening ~30 minutes into the backup...
 
Last edited:
what is the output of 'pveversion -v'? and why don´t you use the recommended ext3? any other useful info about your custom setup?
 
Code: 
# pveversion -v

pve-manager: 1.5-8 (pve-manager/1.5/4674)
running kernel: 2.6.18-2-pve
proxmox-ve-2.6.18: 1.5-5
pve-kernel-2.6.18-2-pve: 2.6.18-5
qemu-server: 1.1-11
pve-firmware: 1.0-3
libpve-storage-perl: 1.0-10
vncterm: 0.9-2
vzctl: 3.0.23-1pve8
vzdump: 1.2-5
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm-2.6.18: 0.9.1-5
I am using EXT4 on this server because in my tests it appeared to show a better read/write speed than EXT3. The 2nd server is using EXT3 and having the same exact problem when this particular container is running on it.

Other than that, only thing I might be doing differently is mounting my /var/lib/vz and /backup without atime support to bump up the speed a little bit..

A longer description of the build process went like this:

1. Installed ProxMox v1.5-3 on the 500GB Hitachi drive.
2. Unmounted /dev/pve/data and changed it's mount point from /var/lib/vz to /home
3. Created single partition spanning the entire drive for the RAID-10 drive setup (4 x 500GB Seagates) and the backup drive (1 x 2TB Hitachi)
4. Ran pvcreate on both of the above partitions
5. Ran vgcreate pve2 on the RAID-10 partition
6. Ran vgcreate pve3 on the backup partition
7. Ran lvcreate on pve2 leaving 16GB free space calling it machines
8. Ran lvcreate on pve3 using 100% space calling it backup
9. Formatted both partitions (mkfs.ext4 /dev/pve2/machines & mkfs.ext4 /dev/pve3/backup)
10. Setup mount point for /dev/pve2/machines on /var/lib/vz
11. Setup mount point for /dev/pve2/backup on /backup
 
Last edited:
Nobody has any idea on this? It's happening on two machines, it can't just be a coincidence...
 
For what it's worth, I temporarily switched the backup type to SUSPEND and all of these errors went away and everything is backing up fine now... With the exception that each machine is down for ~1-2 hours per night now...

Any ideas on how I can get snapshots to work properly? I've left 16GB of free space in LVM, I've heard that 4GB should be more than enough? The particular server in question is a high volume mail server, but I'm pretty sure we don't send/receive 16GB's within the backup period, but we do alot of adding/removing/renaming of files 24/7 - do you think I'm running out of snapshot space, or would that be a difference error??
 
oeginc said:
Any ideas on how I can get snapshots to work properly? I've left 16GB of free space in LVM, I've heard that 4GB should be more than enough? The particular server in question is a high volume mail server, but I'm pretty sure we don't send/receive 16GB's within the backup period, but we do alot of adding/removing/renaming of files 24/7 - do you think I'm running out of snapshot space, or would that be a difference error??
Click to expand...

We all use ext3.
 
This post is very old,
however today i have the same problem with version 2.1 and 2.2 and errors like this:
exT4-fs error (device dm-4): ext4_find_entry: reading directory offset 0
The backup is not finished.
After some search i have disable the USB3 feature in the BIOS and all problems is gone.
I not have any USB device, all hard disks is SATA, in USD is connected only the keyboard.
May and other people find this problem so i post this. When i have time i testit and in other motherboard, if happen always may it is better to have this info in installation guide.
 
Hi there!

I have the same problem, but with QEMU and the thing is that this problem appears only to VM, that are bigger than 25GB during backup process.
VM's are Ubuntu 12.04.2 LTS 64bit.
Will try to disable this USB3 feature and write down if problem is gone!
pveversion -v
pve-manager: 2.3-13 (pve-manager/2.3/7946f1f1)
running kernel: 2.6.32-19-pve
proxmox-ve-2.6.32: 2.3-95
pve-kernel-2.6.32-16-pve: 2.6.32-82
pve-kernel-2.6.32-19-pve: 2.6.32-95
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-4
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-36
qemu-server: 2.3-20
pve-firmware: 1.0-21
libpve-common-perl: 1.0-49
libpve-access-control: 1.0-26
libpve-storage-perl: 2.3-7
vncterm: 1.0-4
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.4-10
ksm-control-daemon: 1.1-1
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back