file restore test

[godfather007]

Hi,

we just moved our biggest Windows fileserver from VMware to proxmox ..
Before moving 20+TB we discovered not to use REFS as this would give problems to restore files from it.

So we build an NTFS volume for it with clustersize 16KB to exceed the standard 16TB volume limit on 4KB size.

Now we are getting error 400 during a restore procedure. See attachment.
Sometimes we get connection timeout 596

Already trying to run a chkdsk atm. No problems found

What else could i do to get around this?


Using PVE 8.0.3 and PBS 3.0.1

 

[tom]

Using PVE 8.0.3 and PBS 3.0.1

pls upgrade to latest (of both) and try again.

 

[godfather007]

I will look into this and plan some upgrades and provide results. Thx.

 

[godfather007]

I will look into this and plan some upgrades and provide results. Thx.

Had to replace the backuphost (hardware failure). Initial backup has been made and problem still exists with PBS 3.1.2. Still going to research this after upgrading PVE to 8.1.3

 

[godfather007]

After replacing our backup host (it got a corrupt controller) i finally managed to do the upgrade to 8.1.3 (PVE)

Unfortunately the message remains. (see attachment)

What could it be? Maybe the clustersize of 16KB?

 

[fabian]

could you post the log for such an attempted restore? see https://pve.proxmox.com/pve-docs/chapter-vzdump.html#_single_file_restore

 

[godfather007]

Thx, i can see the directory:

/var/log/proxmox-backup

It only contains 2 sub-dirs called: API and TASKS

 

[fabian]

you need to check on the PVE host where you attempted the file restore..

 

[godfather007]

you need to check on the PVE host where you attempted the file restore..

Are you certain? I can only find PVE and system stuff in /var/log on my pve host...

 

[godfather007]

Found it, sorry ... was looking at another host than my browser was leaning focusing at.
It is the qemu.log as attached:


There is some complaining about a corrupt $MFT, but chkdsk cannot find anything.

 

[fabian]

thanks! it's possible that such large cluster sizes are not supported by the Linux NTFS driver. it might be possible to improve support by switching to the newer ntfs3 driver.

edit: quick test, 16k cluster size is not the issue, maybe the disk size is, will report back..

edit: also works with a 20TB disk with 32k cluster size - could you maybe try updating your PVE system to the current version? if it still doesn't work then, more details regarding the disk initialization and contents would maybe help to reproduce..

 

[godfather007]

Hmmz.. I have to upgrade again then, strange...

Did not see it coming, of course the machiine is in use... too many applications have a permanent mount to it :-(

By any chance, could windows dedup be the cause?

 

[fabian]

By any chance, could windows dedup be the cause?

yes, that is very likely (to not work, whether it causes your particular issue is hard to tell).

 

[godfather007]

It took almost 4 days to dedup the data on the clone VM.

https://learn.microsoft.com/en-us/answers/questions/1352079/turn-off-dedup-and-delete-chunkstore

Made a new backup from it... no success... even after a reboot.

So now making a fresh backup... back in a jiffy (20 hours or so on those spinners)

Thx four your time! When this is all behind i'll write something about it... for a wiki / requirements page...

 

[fabian]

I definitely want to evaluate whether switching to ntfs3 (the newer in-kernel driver) makes sense, for that it would help to narrow down which parameters are currently not working so that we can test them one by one.

e.g.,
- large filesystem
- deduplication
- compression
- redundant/split volumes
- ... ?

I am not an NTFS expert (yet ), so I am probably missing some details!

there will always be some setups that cannot work out of the box with the generic file-restore VM, e.g. anything using encryption or similar things is out of scope by design.

 

[godfather007]

Unfortuantely... after removing the windows roles, duplicating the data, making a new backup .... the file restore returns the error.
If i can help, i would be glad to do some extra tests.

regards, Martijn

 

[fabian]

could you provide the exact parameters used to create and fill the volume?

- size
- steps to create the volume (e.g., which tool is used to format, which options are selected, ..)
- does it fail to be browseable when backed up while empty?
- if not, what kind of data is on it when it becomes "unbrowseable"?

thanks!

 

[godfather007]

Good morning,

i'll create a new machine (smaller one) with the same steps (they were manual but i think i can manage).
This will speed up the backup attempt.

 

[fabian]

thanks!

https://bugzilla.proxmox.com/show_bug.cgi?id=5187 might also be of interest to you - although no concrete work on that has started yet

 

[godfather007]

Damn... this one works:

Created the VM from the same original template:

-add virtio device (TST 200GB), full (no discard) (later changed to discard)
-in windows: disk manager (old school NT4) selected gpt device
-format: NTFS from default to 16 KB allocation unit (quick format)
-made a test-dir with a testfile
-did a backup and it worked.


i will change the size to 2000GB 10000GB 20000GB 30000GB and run the backup again in between.