Backuping bottleneck

[yarii]

It's all about time.

The time server spent on doing backups.
The time we spent on recovering files.

So my three question:
1. Is this possible to do compressed backup without tar? Is tar really needed this days?
2. I think that tar is my bottleneck when doing backups (it uses only one core of my CPU) at nearly 100%.
3. Is it possible to do some statistics by ZFS dataset or per LXC container/KVM machine?

 

[fireon]

Activate pigz in the config "/etc/vzdump.conf". And all cores will be used to compress. Or use lzo (default) it is much more better. Is the backuptarget ZFS?

 

[yarii]

I use lzo. Even using lzo all files are tared and then packed by lzo.

The backup target is not ZFS. It's a NAS with ext4 mounted in directory using samba.

 

[fireon]

I think that tar is my bottleneck when doing backups (it uses only one core of my CPU) at nearly 100%.

Never had a problem with. Doing Terrabytes backups every day. So what hardware do you have on Proxmox and on your Backupmachine? CPU Memory, Disks, Raid.
Please do also an "pveperf" on you proxmox vm-target.

 

[yarii]

Ok this was my mistake. Tar is not bottleneck. I checked iotop values not top values.

So my two question are:
1. Is this possible to do compressed backup without tar? Is tar really needed this days?
2. Is it possible to do some statistics by ZFS dataset or per LXC container/KVM machine?

 

[czechsys]

Never had a problem with. Doing Terrabytes backups every day. So what hardware do you have on Proxmox and on your Backupmachine? CPU Memory, Disks, Raid.
Please do also an "pveperf" on you proxmox vm-target.

Well, whats your backup speed? Because if i backup from any source (nfs,local lvm) to local ext4 (sas Raid10 x4), calculated speed is ~150MBps in the backup log...

 

[yarii]

Hey guys please don't do offtopic. The bottleneck was not in tar.

Please go back to my two question are:
1. Is this possible to do compressed backup without tar? Is tar really needed this days?
(If You got 1MB file to recover and 1TB backup file, this is really frustrating to unpack and then dig into tar file)

2. Is it possible to do some statistics by ZFS dataset or per LXC container/KVM machine?
(I need to know what lxc/vm do the most io operations)

 

[fireon]

1. Is this possible to do compressed backup without tar? Is tar really needed this days?
(If You got 1MB file to recover and 1TB backup file, this is really frustrating to unpack and then dig into tar file)

I think this is so implemented and not useable without tar. But you can do an Feature Request on https://bugzilla.proxmox.com

What you are looking is to easy to an Filebackup. So we use here Backuppc. If you are looking for an ready Ubuntupackage you can use us repository.

Well, whats your backup speed? Because if i backup from any source (nfs,local lvm) to local ext4 (sas Raid10 x4), calculated speed is ~150MBps in the backup log...

Maximum vor Gigabit, always. But only if your machine have enough power.