NFS Datastore creation fails at 99% (time safety check)

sergioposadam · Friday at 17:37

Hello everyone,

I'm having an issue while trying to create a datastore in PBS using NFS. The process fails at the final step after 99% of the chunks are created. However, when I create a datastore on a local drive, such as /tmp/DStest, it works perfectly fine.

I already upgraded PBS to the latest version

Error Log:

Code:

...
Chunkstore create: 95%
Chunkstore create: 96%
Chunkstore create: 97%
Chunkstore create: 98%
Chunkstore create: 99%
Chunk metadata was not correctly updated during access time safety check:
Timestamps before update: accessed Some(SystemTime { tv_sec: 1746199830, tv_nsec: 792995000 }), modified Some(SystemTime { tv_sec: 1746199830, tv_nsec: 792995000 }), created None
Timestamps after update: accessed Some(SystemTime { tv_sec: 1746199830, tv_nsec: 792995000 }), modified Some(SystemTime { tv_sec: 1746199830, tv_nsec: 792995000 }), created None
TASK ERROR: access time safety check failed: access time safety check using newly inserted chunk failed, aborting GC!
Error: task failed (status access time safety check failed: access time safety check using newly inserted chunk failed, aborting GC!)

Has anyone else experienced this issue or have any suggestions on how to resolve it? Any help would be greatly appreciated!

Thank you in advance!

cl1705 · Friday at 20:55

I am having the same exact issue.

I have tested this on 2 different NFS shares we will call share A and share B.

Share A is in the same data center in the same relative network as the PBS server
Share B is on a different subnet in a different location that has to go through a firewall

I can successfully mount both NFSs and I was able to create the datastore on share A no issues
Share B however does mount and I can copy data to it, I can finish the chunking, but then fails with that same error.

The only difference I can see is that the latency to share B is two times that of latency to A.

We have tried to adjust the NFS mount parameters with things like lazytime, but to no avail.

waltar · Saturday at 10:55

sergioposadam said:
access time safety check failed

Did you mount your XYZ filesystem inside your NAS with "noatime" and/or "nodiratime" (which then is exposed to your PBS with nfs) ?

Edit: Or did you mount your nfs share on PBS with "noatime" and/or "nodiratime" ?

Chris · Saturday at 13:23

Hi, this check was introduced to exactly for this reason: to early on detect issues with filesystems not honoring access timestamp updates, which are essential for the garbage collection to work correctly (see https://pbs.proxmox.com/docs/maintenance.html#gc-background). Please do share the mount parameters you are using for mounting the NAS as well as details about the filesystem and appliance used for the NAS.

LSX3 · Monday at 10:30

Hello everyone,
I'm experiencing the same issue with an SMB share.
The share is mounted via fstab, and I'd like to share my mount parameters.
My PBS and NAS are in the same subnet.

Bash:

//10.51.46.21/Proxmox/PBS /mnt/Backup-NAS cifs credentials=/etc/smbmount/smbcredentials_Backup-NAS,vers=3.0,uid=34,gid=34,noserverino,iocharset=utf8,file_mode=0777,dir_mode=0777,noperm,nofail,_netdev 0 0

Chris · Monday at 10:58

LSX3 said:
Hello everyone,
I'm experiencing the same issue with an SMB share.
The share is mounted via fstab, and I'd like to share my mount parameters.
My PBS and NAS are in the same subnet.

Bash:

//10.51.46.21/Proxmox/PBS /mnt/Backup-NAS cifs credentials=/etc/smbmount/smbcredentials_Backup-NAS,vers=3.0,uid=34,gid=34,noserverino,iocharset=utf8,file_mode=0777,dir_mode=0777,noperm,nofail,_netdev 0 0

Please also share the output of the task log and tell us more about your SMB share... What appliance is used? What OS and what filesystem?

LSX3 · Monday at 12:07

Hi Chris,

Code:

2025-05-05T10:42:05+02:00: Chunkstore create: 1%
2025-05-05T10:42:07+02:00: Chunkstore create: 2%
2025-05-05T10:42:09+02:00: Chunkstore create: 3%
2025-05-05T10:42:11+02:00: Chunkstore create: 4%
2025-05-05T10:42:13+02:00: Chunkstore create: 5%
2025-05-05T10:42:15+02:00: Chunkstore create: 6%
2025-05-05T10:42:17+02:00: Chunkstore create: 7%
2025-05-05T10:42:19+02:00: Chunkstore create: 8%
2025-05-05T10:42:20+02:00: Chunkstore create: 9%
2025-05-05T10:42:22+02:00: Chunkstore create: 10%
2025-05-05T10:42:24+02:00: Chunkstore create: 11%
2025-05-05T10:42:26+02:00: Chunkstore create: 12%
2025-05-05T10:42:28+02:00: Chunkstore create: 13%
2025-05-05T10:42:30+02:00: Chunkstore create: 14%
2025-05-05T10:42:32+02:00: Chunkstore create: 15%
2025-05-05T10:42:34+02:00: Chunkstore create: 16%
2025-05-05T10:42:36+02:00: Chunkstore create: 17%
2025-05-05T10:42:38+02:00: Chunkstore create: 18%
2025-05-05T10:42:40+02:00: Chunkstore create: 19%
2025-05-05T10:42:41+02:00: Chunkstore create: 20%
2025-05-05T10:42:43+02:00: Chunkstore create: 21%
2025-05-05T10:42:45+02:00: Chunkstore create: 22%
2025-05-05T10:42:47+02:00: Chunkstore create: 23%
2025-05-05T10:42:49+02:00: Chunkstore create: 24%
2025-05-05T10:42:51+02:00: Chunkstore create: 25%
2025-05-05T10:42:53+02:00: Chunkstore create: 26%
2025-05-05T10:42:55+02:00: Chunkstore create: 27%
2025-05-05T10:42:57+02:00: Chunkstore create: 28%
2025-05-05T10:42:58+02:00: Chunkstore create: 29%
2025-05-05T10:43:00+02:00: Chunkstore create: 30%
2025-05-05T10:43:02+02:00: Chunkstore create: 31%
2025-05-05T10:43:04+02:00: Chunkstore create: 32%
2025-05-05T10:43:06+02:00: Chunkstore create: 33%
2025-05-05T10:43:08+02:00: Chunkstore create: 34%
2025-05-05T10:43:10+02:00: Chunkstore create: 35%
2025-05-05T10:43:11+02:00: Chunkstore create: 36%
2025-05-05T10:43:13+02:00: Chunkstore create: 37%
2025-05-05T10:43:15+02:00: Chunkstore create: 38%
2025-05-05T10:43:17+02:00: Chunkstore create: 39%
2025-05-05T10:43:19+02:00: Chunkstore create: 40%
2025-05-05T10:43:21+02:00: Chunkstore create: 41%
2025-05-05T10:43:23+02:00: Chunkstore create: 42%
2025-05-05T10:43:25+02:00: Chunkstore create: 43%
2025-05-05T10:43:26+02:00: Chunkstore create: 44%
2025-05-05T10:43:28+02:00: Chunkstore create: 45%
2025-05-05T10:43:30+02:00: Chunkstore create: 46%
2025-05-05T10:43:32+02:00: Chunkstore create: 47%
2025-05-05T10:43:34+02:00: Chunkstore create: 48%
2025-05-05T10:43:36+02:00: Chunkstore create: 49%
2025-05-05T10:43:38+02:00: Chunkstore create: 50%
2025-05-05T10:43:40+02:00: Chunkstore create: 51%
2025-05-05T10:43:42+02:00: Chunkstore create: 52%
2025-05-05T10:43:43+02:00: Chunkstore create: 53%
2025-05-05T10:43:45+02:00: Chunkstore create: 54%
2025-05-05T10:43:47+02:00: Chunkstore create: 55%
2025-05-05T10:43:49+02:00: Chunkstore create: 56%
2025-05-05T10:43:51+02:00: Chunkstore create: 57%
2025-05-05T10:43:53+02:00: Chunkstore create: 58%
2025-05-05T10:43:55+02:00: Chunkstore create: 59%
2025-05-05T10:43:56+02:00: Chunkstore create: 60%
2025-05-05T10:43:58+02:00: Chunkstore create: 61%
2025-05-05T10:44:00+02:00: Chunkstore create: 62%
2025-05-05T10:44:02+02:00: Chunkstore create: 63%
2025-05-05T10:44:04+02:00: Chunkstore create: 64%
2025-05-05T10:44:06+02:00: Chunkstore create: 65%
2025-05-05T10:44:08+02:00: Chunkstore create: 66%
2025-05-05T10:44:09+02:00: Chunkstore create: 67%
2025-05-05T10:44:11+02:00: Chunkstore create: 68%
2025-05-05T10:44:13+02:00: Chunkstore create: 69%
2025-05-05T10:44:15+02:00: Chunkstore create: 70%
2025-05-05T10:44:17+02:00: Chunkstore create: 71%
2025-05-05T10:44:19+02:00: Chunkstore create: 72%
2025-05-05T10:44:21+02:00: Chunkstore create: 73%
2025-05-05T10:44:23+02:00: Chunkstore create: 74%
2025-05-05T10:44:25+02:00: Chunkstore create: 75%
2025-05-05T10:44:27+02:00: Chunkstore create: 76%
2025-05-05T10:44:29+02:00: Chunkstore create: 77%
2025-05-05T10:44:31+02:00: Chunkstore create: 78%
2025-05-05T10:44:33+02:00: Chunkstore create: 79%
2025-05-05T10:44:35+02:00: Chunkstore create: 80%
2025-05-05T10:44:37+02:00: Chunkstore create: 81%
2025-05-05T10:44:39+02:00: Chunkstore create: 82%
2025-05-05T10:44:41+02:00: Chunkstore create: 83%
2025-05-05T10:44:43+02:00: Chunkstore create: 84%
2025-05-05T10:44:45+02:00: Chunkstore create: 85%
2025-05-05T10:44:46+02:00: Chunkstore create: 86%
2025-05-05T10:44:48+02:00: Chunkstore create: 87%
2025-05-05T10:44:50+02:00: Chunkstore create: 88%
2025-05-05T10:44:52+02:00: Chunkstore create: 89%
2025-05-05T10:44:54+02:00: Chunkstore create: 90%
2025-05-05T10:44:56+02:00: Chunkstore create: 91%
2025-05-05T10:44:58+02:00: Chunkstore create: 92%
2025-05-05T10:45:00+02:00: Chunkstore create: 93%
2025-05-05T10:45:02+02:00: Chunkstore create: 94%
2025-05-05T10:45:03+02:00: Chunkstore create: 95%
2025-05-05T10:45:05+02:00: Chunkstore create: 96%
2025-05-05T10:45:07+02:00: Chunkstore create: 97%
2025-05-05T10:45:09+02:00: Chunkstore create: 98%
2025-05-05T10:45:11+02:00: Chunkstore create: 99%
2025-05-05T10:45:14+02:00: Chunk metadata was not correctly updated during access time safety check:
2025-05-05T10:45:14+02:00: Timestamps before update: accessed Some(SystemTime { tv_sec: 1746434749, tv_nsec: 277418600 }), modified Some(SystemTime { tv_sec: 1746434749, tv_nsec: 277418600 }), created Some(SystemTime { tv_sec: 1746434749, tv_nsec: 272286500 })
2025-05-05T10:45:14+02:00: Timestamps after update: accessed Some(SystemTime { tv_sec: 1746434714, tv_nsec: 346284800 }), modified Some(SystemTime { tv_sec: 1746434749, tv_nsec: 277418600 }), created Some(SystemTime { tv_sec: 1746434749, tv_nsec: 272286500 })
2025-05-05T10:45:14+02:00: TASK ERROR: access time safety check failed: access time safety check using newly inserted chunk failed, aborting GC!

I’m using a Synology NAS that provides an SMB share.
My PBS (installed via the ISO, version 3.4) accesses it via CIFS.

Chris · Monday at 13:10

LSX3 said:

Hi Chris,

Code:

2025-05-05T10:42:05+02:00: Chunkstore create: 1%
2025-05-05T10:42:07+02:00: Chunkstore create: 2%
2025-05-05T10:42:09+02:00: Chunkstore create: 3%
2025-05-05T10:42:11+02:00: Chunkstore create: 4%
2025-05-05T10:42:13+02:00: Chunkstore create: 5%
2025-05-05T10:42:15+02:00: Chunkstore create: 6%
2025-05-05T10:42:17+02:00: Chunkstore create: 7%
2025-05-05T10:42:19+02:00: Chunkstore create: 8%
2025-05-05T10:42:20+02:00: Chunkstore create: 9%
2025-05-05T10:42:22+02:00: Chunkstore create: 10%
2025-05-05T10:42:24+02:00: Chunkstore create: 11%
2025-05-05T10:42:26+02:00: Chunkstore create: 12%
2025-05-05T10:42:28+02:00: Chunkstore create: 13%
2025-05-05T10:42:30+02:00: Chunkstore create: 14%
2025-05-05T10:42:32+02:00: Chunkstore create: 15%
2025-05-05T10:42:34+02:00: Chunkstore create: 16%
2025-05-05T10:42:36+02:00: Chunkstore create: 17%
2025-05-05T10:42:38+02:00: Chunkstore create: 18%
2025-05-05T10:42:40+02:00: Chunkstore create: 19%
2025-05-05T10:42:41+02:00: Chunkstore create: 20%
2025-05-05T10:42:43+02:00: Chunkstore create: 21%
2025-05-05T10:42:45+02:00: Chunkstore create: 22%
2025-05-05T10:42:47+02:00: Chunkstore create: 23%
2025-05-05T10:42:49+02:00: Chunkstore create: 24%
2025-05-05T10:42:51+02:00: Chunkstore create: 25%
2025-05-05T10:42:53+02:00: Chunkstore create: 26%
2025-05-05T10:42:55+02:00: Chunkstore create: 27%
2025-05-05T10:42:57+02:00: Chunkstore create: 28%
2025-05-05T10:42:58+02:00: Chunkstore create: 29%
2025-05-05T10:43:00+02:00: Chunkstore create: 30%
2025-05-05T10:43:02+02:00: Chunkstore create: 31%
2025-05-05T10:43:04+02:00: Chunkstore create: 32%
2025-05-05T10:43:06+02:00: Chunkstore create: 33%
2025-05-05T10:43:08+02:00: Chunkstore create: 34%
2025-05-05T10:43:10+02:00: Chunkstore create: 35%
2025-05-05T10:43:11+02:00: Chunkstore create: 36%
2025-05-05T10:43:13+02:00: Chunkstore create: 37%
2025-05-05T10:43:15+02:00: Chunkstore create: 38%
2025-05-05T10:43:17+02:00: Chunkstore create: 39%
2025-05-05T10:43:19+02:00: Chunkstore create: 40%
2025-05-05T10:43:21+02:00: Chunkstore create: 41%
2025-05-05T10:43:23+02:00: Chunkstore create: 42%
2025-05-05T10:43:25+02:00: Chunkstore create: 43%
2025-05-05T10:43:26+02:00: Chunkstore create: 44%
2025-05-05T10:43:28+02:00: Chunkstore create: 45%
2025-05-05T10:43:30+02:00: Chunkstore create: 46%
2025-05-05T10:43:32+02:00: Chunkstore create: 47%
2025-05-05T10:43:34+02:00: Chunkstore create: 48%
2025-05-05T10:43:36+02:00: Chunkstore create: 49%
2025-05-05T10:43:38+02:00: Chunkstore create: 50%
2025-05-05T10:43:40+02:00: Chunkstore create: 51%
2025-05-05T10:43:42+02:00: Chunkstore create: 52%
2025-05-05T10:43:43+02:00: Chunkstore create: 53%
2025-05-05T10:43:45+02:00: Chunkstore create: 54%
2025-05-05T10:43:47+02:00: Chunkstore create: 55%
2025-05-05T10:43:49+02:00: Chunkstore create: 56%
2025-05-05T10:43:51+02:00: Chunkstore create: 57%
2025-05-05T10:43:53+02:00: Chunkstore create: 58%
2025-05-05T10:43:55+02:00: Chunkstore create: 59%
2025-05-05T10:43:56+02:00: Chunkstore create: 60%
2025-05-05T10:43:58+02:00: Chunkstore create: 61%
2025-05-05T10:44:00+02:00: Chunkstore create: 62%
2025-05-05T10:44:02+02:00: Chunkstore create: 63%
2025-05-05T10:44:04+02:00: Chunkstore create: 64%
2025-05-05T10:44:06+02:00: Chunkstore create: 65%
2025-05-05T10:44:08+02:00: Chunkstore create: 66%
2025-05-05T10:44:09+02:00: Chunkstore create: 67%
2025-05-05T10:44:11+02:00: Chunkstore create: 68%
2025-05-05T10:44:13+02:00: Chunkstore create: 69%
2025-05-05T10:44:15+02:00: Chunkstore create: 70%
2025-05-05T10:44:17+02:00: Chunkstore create: 71%
2025-05-05T10:44:19+02:00: Chunkstore create: 72%
2025-05-05T10:44:21+02:00: Chunkstore create: 73%
2025-05-05T10:44:23+02:00: Chunkstore create: 74%
2025-05-05T10:44:25+02:00: Chunkstore create: 75%
2025-05-05T10:44:27+02:00: Chunkstore create: 76%
2025-05-05T10:44:29+02:00: Chunkstore create: 77%
2025-05-05T10:44:31+02:00: Chunkstore create: 78%
2025-05-05T10:44:33+02:00: Chunkstore create: 79%
2025-05-05T10:44:35+02:00: Chunkstore create: 80%
2025-05-05T10:44:37+02:00: Chunkstore create: 81%
2025-05-05T10:44:39+02:00: Chunkstore create: 82%
2025-05-05T10:44:41+02:00: Chunkstore create: 83%
2025-05-05T10:44:43+02:00: Chunkstore create: 84%
2025-05-05T10:44:45+02:00: Chunkstore create: 85%
2025-05-05T10:44:46+02:00: Chunkstore create: 86%
2025-05-05T10:44:48+02:00: Chunkstore create: 87%
2025-05-05T10:44:50+02:00: Chunkstore create: 88%
2025-05-05T10:44:52+02:00: Chunkstore create: 89%
2025-05-05T10:44:54+02:00: Chunkstore create: 90%
2025-05-05T10:44:56+02:00: Chunkstore create: 91%
2025-05-05T10:44:58+02:00: Chunkstore create: 92%
2025-05-05T10:45:00+02:00: Chunkstore create: 93%
2025-05-05T10:45:02+02:00: Chunkstore create: 94%
2025-05-05T10:45:03+02:00: Chunkstore create: 95%
2025-05-05T10:45:05+02:00: Chunkstore create: 96%
2025-05-05T10:45:07+02:00: Chunkstore create: 97%
2025-05-05T10:45:09+02:00: Chunkstore create: 98%
2025-05-05T10:45:11+02:00: Chunkstore create: 99%
2025-05-05T10:45:14+02:00: Chunk metadata was not correctly updated during access time safety check:
2025-05-05T10:45:14+02:00: Timestamps before update: accessed Some(SystemTime { tv_sec: 1746434749, tv_nsec: 277418600 }), modified Some(SystemTime { tv_sec: 1746434749, tv_nsec: 277418600 }), created Some(SystemTime { tv_sec: 1746434749, tv_nsec: 272286500 })
2025-05-05T10:45:14+02:00: Timestamps after update: accessed Some(SystemTime { tv_sec: 1746434714, tv_nsec: 346284800 }), modified Some(SystemTime { tv_sec: 1746434749, tv_nsec: 277418600 }), created Some(SystemTime { tv_sec: 1746434749, tv_nsec: 272286500 })
2025-05-05T10:45:14+02:00: TASK ERROR: access time safety check failed: access time safety check using newly inserted chunk failed, aborting GC!

I’m using a Synology NAS that provides an SMB share.
My PBS (installed via the ISO, version 3.4) accesses it via CIFS.

Hmm, so according to the task log you access timestamp jumped into the past and was not incremented as the PBS expects it to. Do you have NTP enabled on the Synology NAS? Are there other concurrent processes accessing the same share? Also, you could try to see if the issue persists when explicitly setting cache=strict as mount parameter.

Also, is this reproducible when trying to create the datastore again?

LSX3 · Monday at 14:02

Sure! Here's your corrected and polished English version:

Hi Chris,
yes, both devices use the same NTP server.
The time is exactly the same on both devices, down to the second, including the time zone.
No, the share is dedicated exclusively to PBS, with a separate user created just for that purpose. No other users or processes access it.
In PVE, the share can be mounted without any issues and backups work as expected — I tested that (of course, I deleted everything afterward and returned the system to its original state).
The issue occurs every time I try to add the share, so it's fully reproducible.
With the cache=strict parameter, it works perfectly.

Thanks a lot for the quick solution!
Maybe this could also be the solution for the NFS mount issue?

cl1705 · Monday at 15:55

Hi Chris,

cache=strict is not an option with NFS
Also according to https://manpages.debian.org/bullseye/nfs-common/nfs.5.en.html

In particular, the atime/noatime, diratime/nodiratime, relatime/norelatime, and strictatime/nostrictatime mount options have no effect on NFS mounts.

Please advise

ftsab ---- 172.17.7.216:/zpl01/dls-pbs /mnt/dls-pbs nfs defaults 0 0

exports ----- /zpl01/dls-pbs/ 172.17.40.181/32(rw,sync,no_subtree_check,sec=sys,secure,no_root_squash,no_all_squash)

as well as the result

2025-05-05T09:48:01-04:00: Chunkstore create: 1%
2025-05-05T09:48:02-04:00: Chunkstore create: 2%
2025-05-05T09:48:03-04:00: Chunkstore create: 3%
2025-05-05T09:48:03-04:00: Chunkstore create: 4%
2025-05-05T09:48:04-04:00: Chunkstore create: 5%
2025-05-05T09:48:05-04:00: Chunkstore create: 6%
2025-05-05T09:48:05-04:00: Chunkstore create: 7%
2025-05-05T09:48:06-04:00: Chunkstore create: 8%
2025-05-05T09:48:07-04:00: Chunkstore create: 9%
2025-05-05T09:48:07-04:00: Chunkstore create: 10%
2025-05-05T09:48:08-04:00: Chunkstore create: 11%
2025-05-05T09:48:09-04:00: Chunkstore create: 12%
2025-05-05T09:48:09-04:00: Chunkstore create: 13%
2025-05-05T09:48:10-04:00: Chunkstore create: 14%
2025-05-05T09:48:11-04:00: Chunkstore create: 15%
2025-05-05T09:48:12-04:00: Chunkstore create: 16%
2025-05-05T09:48:12-04:00: Chunkstore create: 17%
2025-05-05T09:48:13-04:00: Chunkstore create: 18%
2025-05-05T09:48:14-04:00: Chunkstore create: 19%
2025-05-05T09:48:14-04:00: Chunkstore create: 20%
2025-05-05T09:48:15-04:00: Chunkstore create: 21%
2025-05-05T09:48:16-04:00: Chunkstore create: 22%
2025-05-05T09:48:16-04:00: Chunkstore create: 23%
2025-05-05T09:48:17-04:00: Chunkstore create: 24%
2025-05-05T09:48:18-04:00: Chunkstore create: 25%
2025-05-05T09:48:18-04:00: Chunkstore create: 26%
2025-05-05T09:48:19-04:00: Chunkstore create: 27%
2025-05-05T09:48:20-04:00: Chunkstore create: 28%
2025-05-05T09:48:20-04:00: Chunkstore create: 29%
2025-05-05T09:48:21-04:00: Chunkstore create: 30%
2025-05-05T09:48:22-04:00: Chunkstore create: 31%
2025-05-05T09:48:22-04:00: Chunkstore create: 32%
2025-05-05T09:48:23-04:00: Chunkstore create: 33%
2025-05-05T09:48:24-04:00: Chunkstore create: 34%
2025-05-05T09:48:25-04:00: Chunkstore create: 35%
2025-05-05T09:48:25-04:00: Chunkstore create: 36%
2025-05-05T09:48:26-04:00: Chunkstore create: 37%
2025-05-05T09:48:27-04:00: Chunkstore create: 38%
2025-05-05T09:48:27-04:00: Chunkstore create: 39%
2025-05-05T09:48:28-04:00: Chunkstore create: 40%
2025-05-05T09:48:29-04:00: Chunkstore create: 41%
2025-05-05T09:48:29-04:00: Chunkstore create: 42%
2025-05-05T09:48:30-04:00: Chunkstore create: 43%
2025-05-05T09:48:31-04:00: Chunkstore create: 44%
2025-05-05T09:48:32-04:00: Chunkstore create: 45%
2025-05-05T09:48:32-04:00: Chunkstore create: 46%
2025-05-05T09:48:33-04:00: Chunkstore create: 47%
2025-05-05T09:48:34-04:00: Chunkstore create: 48%
2025-05-05T09:48:34-04:00: Chunkstore create: 49%
2025-05-05T09:48:35-04:00: Chunkstore create: 50%
2025-05-05T09:48:36-04:00: Chunkstore create: 51%
2025-05-05T09:48:36-04:00: Chunkstore create: 52%
2025-05-05T09:48:37-04:00: Chunkstore create: 53%
2025-05-05T09:48:38-04:00: Chunkstore create: 54%
2025-05-05T09:48:38-04:00: Chunkstore create: 55%
2025-05-05T09:48:39-04:00: Chunkstore create: 56%
2025-05-05T09:48:40-04:00: Chunkstore create: 57%
2025-05-05T09:48:40-04:00: Chunkstore create: 58%
2025-05-05T09:48:41-04:00: Chunkstore create: 59%
2025-05-05T09:48:42-04:00: Chunkstore create: 60%
2025-05-05T09:48:43-04:00: Chunkstore create: 61%
2025-05-05T09:48:43-04:00: Chunkstore create: 62%
2025-05-05T09:48:44-04:00: Chunkstore create: 63%
2025-05-05T09:48:45-04:00: Chunkstore create: 64%
2025-05-05T09:48:45-04:00: Chunkstore create: 65%
2025-05-05T09:48:46-04:00: Chunkstore create: 66%
2025-05-05T09:48:47-04:00: Chunkstore create: 67%
2025-05-05T09:48:47-04:00: Chunkstore create: 68%
2025-05-05T09:48:48-04:00: Chunkstore create: 69%
2025-05-05T09:48:49-04:00: Chunkstore create: 70%
2025-05-05T09:48:49-04:00: Chunkstore create: 71%
2025-05-05T09:48:50-04:00: Chunkstore create: 72%
2025-05-05T09:48:51-04:00: Chunkstore create: 73%
2025-05-05T09:48:51-04:00: Chunkstore create: 74%
2025-05-05T09:48:52-04:00: Chunkstore create: 75%
2025-05-05T09:48:53-04:00: Chunkstore create: 76%
2025-05-05T09:48:54-04:00: Chunkstore create: 77%
2025-05-05T09:48:54-04:00: Chunkstore create: 78%
2025-05-05T09:48:55-04:00: Chunkstore create: 79%
2025-05-05T09:48:56-04:00: Chunkstore create: 80%
2025-05-05T09:48:56-04:00: Chunkstore create: 81%
2025-05-05T09:48:57-04:00: Chunkstore create: 82%
2025-05-05T09:48:58-04:00: Chunkstore create: 83%
2025-05-05T09:48:58-04:00: Chunkstore create: 84%
2025-05-05T09:48:59-04:00: Chunkstore create: 85%
2025-05-05T09:49:00-04:00: Chunkstore create: 86%
2025-05-05T09:49:00-04:00: Chunkstore create: 87%
2025-05-05T09:49:01-04:00: Chunkstore create: 88%
2025-05-05T09:49:02-04:00: Chunkstore create: 89%
2025-05-05T09:49:03-04:00: Chunkstore create: 90%
2025-05-05T09:49:03-04:00: Chunkstore create: 91%
2025-05-05T09:49:04-04:00: Chunkstore create: 92%
2025-05-05T09:49:05-04:00: Chunkstore create: 93%
2025-05-05T09:49:05-04:00: Chunkstore create: 94%
2025-05-05T09:49:06-04:00: Chunkstore create: 95%
2025-05-05T09:49:07-04:00: Chunkstore create: 96%
2025-05-05T09:49:07-04:00: Chunkstore create: 97%
2025-05-05T09:49:08-04:00: Chunkstore create: 98%
2025-05-05T09:49:09-04:00: Chunkstore create: 99%
2025-05-05T09:49:10-04:00: Chunk metadata was not correctly updated during access time safety check:
2025-05-05T09:49:10-04:00: Timestamps before update: accessed Some(SystemTime { tv_sec: 0, tv_nsec: 0 }), modified Some(SystemTime { tv_sec: 1746452949, tv_nsec: 964037408 }), created None
2025-05-05T09:49:10-04:00: Timestamps after update: accessed Some(SystemTime { tv_sec: 0, tv_nsec: 0 }), modified Some(SystemTime { tv_sec: 1746452949, tv_nsec: 964037408 }), created None
2025-05-05T09:49:10-04:00: TASK ERROR: access time safety check failed: access time safety check using newly inserted chunk failed, aborting GC!

But yet when I do this same exact export from a server on the same subnet - 172.17.40.131 it works fine, and the only difference I can see is the latency is doubled, so could that be the issue?

Chris · Monday at 16:20

cl1705 said:
cache=strict is not an option with NFS

Yes, that was proposed for CIFS shares only.

cl1705 said:
2025-05-05T09:49:10-04:00: Timestamps before update: accessed Some(SystemTime { tv_sec: 0, tv_nsec: 0 }), modified Some(SystemTime { tv_sec: 1746452949, tv_nsec: 964037408 }), created None
2025-05-05T09:49:10-04:00: Timestamps after update: accessed Some(SystemTime { tv_sec: 0, tv_nsec: 0 }), modified Some(SystemTime { tv_sec: 1746452949, tv_nsec: 964037408 }), created None

Well, this seems completely wrong...!

Your NFS share seems to only return the unix epoch as accessed timestamp and no created timestamp at all! As is, this does not satisfy the requirements to act as reliable PBS datastore backing storage.

What appliance, OS and filesystem is used on the NFS server side?

waltar · Monday at 16:42

Wondering that extra defining cache=strict has any effect as it's default since a very long time and if not other one set ...
man mount.cifs:
cache=arg
Cache mode. See the section below on CACHE COHERENCY for details. Allowed values are:
• none - do not cache file data at all
• strict - follow the CIFS/SMB2 protocol strictly
• loose - allow loose caching semantics
The default in kernels prior to 3.7 was loose. As of kernel 3.7 the default is strict.

cl1705 · Monday at 17:45

filesystem is ZFS

lsb_release -a
Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: noble

dpkg -l | grep nfs
ii libnfsidmap1:amd64 1:2.6.4-3ubuntu5 amd64 NFS idmapping library
ii nfs-common 1:2.6.4-3ubuntu5 amd64 NFS support files common to client and server
ii nfs-kernel-server 1:2.6.4-3ubuntu5 amd64 support for NFS kernel server

Chris · Monday at 17:58

cl1705 said:
lsb_release -a
Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: noble

dpkg -l | grep nfs
ii libnfsidmap1:amd64 1:2.6.4-3ubuntu5 amd64 NFS idmapping library
ii nfs-common 1:2.6.4-3ubuntu5 amd64 NFS support files common to client and server
ii nfs-kernel-server 1:2.6.4-3ubuntu5 amd64 support for NFS kernel server

What filesystem? Maybe related to the user report in this thread https://forum.proxmox.com/threads/garbage-collection-deletes-all-chunks.151582/?

sergioposadam · Monday at 18:52

I am using

Chris said:
Hi, this check was introduced to exactly for this reason: to early on detect issues with filesystems not honoring access timestamp updates, which are essential for the garbage collection to work correctly (see https://pbs.proxmox.com/docs/maintenance.html#gc-background). Please do share the mount parameters you are using for mounting the NAS as well as details about the filesystem and appliance used for the NAS.

We are currently using a Dell PowerScale H700 ISILON running version 9.17.14. Also PBS is a VM running lastet version
proxmox-backup-server 3.4.0-1 running version: 3.4.0

waltar · Monday at 20:53

Isilon ... looks interesting ... is "enable access time tracking" active (which should) or disabled ?

sergioposadam · Monday at 22:32

waltar said:
Isilon ... looks interesting ... is "enable access time tracking" active (which should) or disabled ? View attachment 85723

Yes, that was the issue, Thank you so much!

Chris · 2025-05-06T08:30:27+0200

waltar said:
Isilon ... looks interesting ... is "enable access time tracking" active (which should) or disabled ? View attachment 85723

Thanks for sharing the required setting for OneFS configuration! Access time tracking must be enabled on these storage boxes for GC to work correctly.

sergioposadam said:
Yes, that was the issue, Thank you so much!

Please be aware: In case you would like to reduce the garbage collection cutoff time [0], you should most definitely adapt the precision in the OneFS settings [1].

[0] https://pbs.proxmox.com/docs/storage.html#tuning
[1] https://infohub.delltechnologies.co...r-eda-design-environments/access-time-tuning/

Chris · 2025-05-06T08:35:19+0200

cl1705 said:
filesystem is ZFS

lsb_release -a
Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: noble

dpkg -l | grep nfs
ii libnfsidmap1:amd64 1:2.6.4-3ubuntu5 amd64 NFS idmapping library
ii nfs-common 1:2.6.4-3ubuntu5 amd64 NFS support files common to client and server
ii nfs-kernel-server 1:2.6.4-3ubuntu5 amd64 support for NFS kernel server

Check your ZFS version and make sure your system is up to date. You might be running into this issue https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/2077487

Chris · 2025-05-06T08:41:55+0200

waltar said:
Wondering that extra defining cache=strict has any effect as it's default since a very long time and if not other one set ...
man mount.cifs:
cache=arg
Cache mode. See the section below on CACHE COHERENCY for details. Allowed values are:
• none - do not cache file data at all
• strict - follow the CIFS/SMB2 protocol strictly
• loose - allow loose caching semantics
The default in kernels prior to 3.7 was loose. As of kernel 3.7 the default is strict.

True, makes me wonder why it had an effect. @LSX3 can you verify that this was indeed solved by adding the mount option by e.g. creating a new datastore after mounting without the cache mount parameters? It could be that maybe just the remount after adding the mount parameter had an effect?

NFS Datastore creation fails at 99% (time safety check)

New Member

New Member

Renowned Member

Proxmox Staff Member

New Member

Proxmox Staff Member

New Member

Proxmox Staff Member

New Member

New Member

Proxmox Staff Member

Renowned Member

New Member

Proxmox Staff Member

New Member

Renowned Member

New Member

Proxmox Staff Member

Proxmox Staff Member

Proxmox Staff Member

We value your privacy