NFS and ZeroFS

J

jfmatth

New Member
Apr 25, 2026
7
0
1
Hello,

Love Proxmox, been using for years but first post :)

I am using ZeroFS as my NFS server and am running into odd issues trying to mount NFS as a storage entry.

from the command line on a proxmox node, I can mount the NFS server point

Code: 
root@prox-nfs:/etc/pve# mount -o port=2049,mountport=2049,tcp,nolock 192.168.100.50:/proxmox /mnt/zerofs


But when added to /etc/pve/storage.cfg it doesn't work, just shows offline

Code: 
nfs: zerofs
        export /proxmox
        path /mnt/pve/zerofs
        server 192.168.100.50
        content backup,iso
        options proto=tcp,port=2049,mountport=2049,nolock,vers=3

Any help is appreciated
 
Last edited:
Comment your "options" line with # in front and it should mount, "mountport" prevent the mount.
Btw. you need "/proxmox" no_root_squash exported otherwise the content line couldn't work.
 
@waltar Thank you for the prompt reply.

Unfortunately, those changes didn't work.

In asking AI what it could be, the CLI options tell Linux to skip any standard NFS checks and just mount the export. Proxmox seems stricter in what it will do / allow in mounting NFS.

In trying your idea, I get the typical "ZeroFS is offline" in journalctl messages.

ZeroFS is a golang implemented NFS server and doesn't have the typical RPCBind ports, etc. implemented.

I was hoping since the CLI mounted it, it would work with some options for Proxmox.

Any other ideas are appreciated.
 
Your manual mount use a kind of nfs version (4.0/4.1/4.2) while in storage.cfg you define version 3 which depends on rpcbind which your zerofs doesn't have as you wrote.
Btw. why use a uncommon nfs server if any linux has a functional kernel nfs server and even an optional ganesha nfs server do it's work.
 
  • Like
Reactions: bl1mp, Kingneutron and Johannes S
Thank you Waltar,

I'm not sure the two attempts that that different, but ZeroFS only supports a subset of NFS3

I'm trying to find a solution to 'unlimited' home storage, i.e. Cache the recent stuff on home storage < 2TB and put the rest in S3. There a several projects that do this: SeaweedFS, JuiceFS and ZeroFS. Zero is the one I got working, the others didn't fit my needs.

If you have other suggestions, I'm all ears.

Thanks again for your help.

J
 
czechsys said:
Option: You can mount nfs via /etc/fstab and use storage -> directory variant on the PVE side.
Click to expand...
Thank you @czechsys I tried that and I ended up resharing the "NFS" mount on the ZeroFS box as "true" NFS, so all my complexity is in one place.

Sharing a directory means I have to mount that NFS share everywhere :O
 
s3 is NOT a filesystem but an object storage intended to be used via an API ( or tools leveraging this api). It's semantics work quite differently to traditional filesystems and any of the attempts (s3fs, zerofs etc) to use it as filesystem are basically hacky approaches who try to do something S3 was never intended for. What you could do is to reimplement what amazon did with s3 files: Have a NFS server with enough storage to contain all the data and implement a sync mechanism who regulary sync data between the NFS server and the s3 buckets. rclone might be one way to implement this.
How much data do you have that you need "unlimited storage"? For most home users some large-capacity HDDs together in a ZFS mirror/striped mirror/RAIDZ-Vpool should be more than enough. Of couse you would still need backup. Since there are enough backup tools to backup bulk data to s3 storage (for backup cloud storage is a good option) both things together should cover most homeusers needs.

For details on on s3 and s3 files see following piece by Corey Quinn:
https://www.lastweekinaws.com/blog/s3-is-not-a-filesystem-but-now-theres-one-in-front-of-it/

And a less tongue-in-cheek news article by German IT news site Heise: https://www.heise.de/en/news/AWS-makes-S3-a-file-system-11249368.html

Edit: Added link to Heise article
 
Last edited:
  • Like
Reactions: UdoB
Johannes S said:
s3 is NOT a filesystem but an object storage intended to be used via an API ( or tools leveraging this api). It's semantics work quite differently to traditional filesystems and any of the attempts (s3fs, zerofs etc) to use it as filesystem are basically hacky approaches who try to do something S3 was never intended for. What you could do is to reimplement what amazon did with s3 files: Have a NFS server with enough storage to contain all the data and implement a sync mechanism who regulary sync data between the NFS server and the s3 buckets. rclone might be one way to implement this.
How much data do you have that you need "unlimited storage"? For most home users some large-capacity HDDs together in a ZFS mirror/striped mirror/RAIDZ-Vpool should be more than enough. Of couse you would still need backup. Since there are enough backup tools to backup bulk data to s3 storage (for backup cloud storage is a good option) both things together should cover most homeusers needs.

For details on on s3 and s3 files see following piece by Corey Quinn:
https://www.lastweekinaws.com/blog/s3-is-not-a-filesystem-but-now-theres-one-in-front-of-it/

And a less tongue-in-cheek news article by German IT news site Heise: https://www.heise.de/en/news/AWS-makes-S3-a-file-system-11249368.html

Edit: Added link to Heise article
Click to expand...
@Johannes S I do appreciate the thorough response.

Yes, S3 isn't a filesystem and that's why I need a front-end that exposes it as NFS.

Much of this question now is moot since I found a better solution - JuiceFS.

There are several similar 'servers' out there that do a variety of things like what I want, basically front S3 with another filesystem that Proxmox and K8s can use natively.

In the case of JuiceFS, it's the simplest and fastest system I've found. You setup the server, share the JuiceFS filesystem via NFS and viola, unlimited and cached storage.

The question of "why" comes up a lot, basically I want to have a small amount of storage locally as cache (~2TB) and the rest in the cloud with S3. Ultimately I hope to replace Google photos with Immich and other services, but before I get there I wanted to have something like this.

Thank you all for your responses.
 
jfmatth said:
@Johannes S I do appreciate the thorough response.

Yes, S3 isn't a filesystem and that's why I need a front-end that exposes it as NFS.
Click to expand...

You didn't get my point: S3 is not a filesystem and any attempt to try to treat it as one is calling for trouble.

jfmatth said:
Much of this question now is moot since I found a better solution - JuiceFS.

There are several similar 'servers' out there that do a variety of things like what I want, basically front S3 with another filesystem that Proxmox and K8s can use natively.
Click to expand...

If you want to use S3 "natively" from K8S then use S3 for it, after all that's what it's built for. If however you want to use a storage with filesystem semantics, then use something built for it. Stuff like s3fs-fuse, ZeroFS or JuiceFS are mixing things up which were intended for very different usecases.
And storage is never "unlimited", cloud storage isn't free so even there you will hit a limit at some point. And most if not all cloud providers also offers storage options for traditional filesystems (e.G. amazons efs).

How large is your complete data that you need "unlimited" storage? After all hdds with 8, 16, 12 and 24 TB in the consumer/prosumer/NAS/disc sector are available right now and while not really cheap will still be cheapter than a monthly cloud bill in the long run
 
ness1602 said:
Better than JuiceFS is of course CephFS, but i guess this is enough for you and probably your PVC ReadWriteMany problem.
Click to expand...

I love Ceph, and the improvements over the years by Proxmox to make it so easy to use are impressive.

Correct me if I'm wrong, but Ceph still doesn't 'back' to S3?
 
Johannes S said:
You didn't get my point: S3 is not a filesystem and any attempt to try to treat it as one is calling for trouble.



If you want to use S3 "natively" from K8S then use S3 for it, after all that's what it's built for. If however you want to use a storage with filesystem semantics, then use something built for it. Stuff like s3fs-fuse, ZeroFS or JuiceFS are mixing things up which were intended for very different usecases.
And storage is never "unlimited", cloud storage isn't free so even there you will hit a limit at some point. And most if not all cloud providers also offers storage options for traditional filesystems (e.G. amazons efs).

How large is your complete data that you need "unlimited" storage? After all hdds with 8, 16, 12 and 24 TB in the consumer/prosumer/NAS/disc sector are available right now and while not really cheap will still be cheapter than a monthly cloud bill in the long run
Click to expand...
Sorry If i wasn't clear.

JuiceFS doesn't present S3 as a file system, it has it's own FUSE filesystem that CAN be a cache for S3, but not a requirement. Juice's site lists its goals.

I'm not sure I'm using it outside its use cases; they list serving the JuiceFS filesystem via NFS as a supported design and it's working great. The caching is insane.

As for how much, of course nothing is unlimited but my needs are 10TB and growing and having that at home or on a VPS isn't practical. I'd rather have a JuiceFS VM running with < 50GB and 'cache' it to S3. So far it's working great.

I even have Proxmox mounting the NFS and backing up there, which is wonderful. I'll mount K8s to NFS there as well and be close to done.

Just found out this AM after wanting to test a DR scenario that JuiceFS automatically does hourly backups to S3 and has a restore option when you want to come back online. Ill test that tonight.

Appreciate all the advice, a great bunch of people here on the Proxmox Forums.
 
You must log in or register to reply here.
Top Bottom
Back