I dont think this is the arguable point. Id love to have proxmox gmbh provide every service I use in my enterprise, but that doesnt make it a value proposition for THEM. Perhaps you can convince them this is a pool they want to swim in- a good place to start is a feature request in their bugzilla, but if it were me you better have a good argument about what's in it for me. a storage solution is a big undertaking.
Shared Remote ZFS Storage
- Thread starter jtremblay
- Start date
-
- Tags
- storage storage design
I'm betting 90% of the code is PBS and Proxmox gmbh knows intimately which iSCSI target they expect....
Last edited:
The problem is the HA part. It does not make sense to run a HA cluster with a storage that is not HA itself. You're building in a SPOF and (sorry again) no sane person would want to run a HA cluster this way. This is a recipe for disaster. The hardware cost for the storage box is better spend in NVMe to build a CEPH cluster.
PBS is written in Rust and has nothing to do with iSCSI nor ZFS replication nor the code base of PVE, which is written in Perl. I don't understand why you're refering to it again and again.
The "server part" of the ZFS-over-iSCSI implementation is installing lio, targetcli and ssh. Once setup, there is nothing to configure. I don't need "a product" for this. The software behind this is already taken care of by the Debian maintainers.
In the enterprise this is not a good option. It is difficult enough staffing your IT with competent dependable people to begin with; the more you can farm out to function suppliers (eg, storage) the better. Ceph make sense when you have sufficient IT competence including ceph- There is a reason people fork over 6-7 figures for Nimbles, EMC AFAs, etc.
Only response I have for that is stay away from the casino
From what I understand, you just want Ceph. Blockbridge = Ceph, VMWare vSAN ~= Ceph. There is native support for iSCSI in Proxmox, there is native support for ZFS in Proxmox, you need to share it out, set up a container with Samba or NFS, a list of LinuxServer daemons is natively in Proxmox.
The whole point of Ceph in Proxmox is that you don’t need specialized IT staff to set it up. It’s dead simple, I am an advanced enterprise user and haven’t ever seen the need to tweak much of the Ceph except to enable Prometheus monitoring. There are some things you can do to improve performance, but that is only if you truly need to squeeze the bottom out. If the cost is to add another $15k server or hire a person, the choice is quickly made. If you can get 10% more performance out of several racks of hardware, it may be worth investing in an engineer.
If you don’t want to deal with the Ceph stuff, Canonical and others have professional services, and fairly sure the company behind Proxmox or one of its resellers can offer those as well - if it’s worth spending $2k to get a small performance improvement but native Ceph, with a simple 25 or 100G network, it’s really plug and play and the defaults work surprisingly well until you get to 100s of nodes with 1000s of disks.
The whole point of Ceph in Proxmox is that you don’t need specialized IT staff to set it up. It’s dead simple, I am an advanced enterprise user and haven’t ever seen the need to tweak much of the Ceph except to enable Prometheus monitoring. There are some things you can do to improve performance, but that is only if you truly need to squeeze the bottom out. If the cost is to add another $15k server or hire a person, the choice is quickly made. If you can get 10% more performance out of several racks of hardware, it may be worth investing in an engineer.
If you don’t want to deal with the Ceph stuff, Canonical and others have professional services, and fairly sure the company behind Proxmox or one of its resellers can offer those as well - if it’s worth spending $2k to get a small performance improvement but native Ceph, with a simple 25 or 100G network, it’s really plug and play and the defaults work surprisingly well until you get to 100s of nodes with 1000s of disks.
Last edited:
@guruevi, at Blockbridge, we do not utilize Ceph (or ZFS) in any capacity. Instead, we have developed a purpose-built storage stack that includes comprehensive implementations of SCSI, iSCSI, NVMe, and NVMe/TCP. Our design objectives differ significantly from Ceph's, as we focus on delivering low-latency/high-IOPS storage solutions tailored to MSPs, CSPs, and enterprises needing non-disruptive storage management & 24/7/365 critical support.
Some people overlook the significant investment required to build and support software. Developers, QA teams, release engineers, documentation, and testing infrastructure all come with costs. Moreover, when you're developing a product that is constantly evolving while also needing to maintain compatibility with another ever-changing product, the effort required for testing and support grows exponentially—it's an n^2 challenge that only becomes more complex over time.
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Some people overlook the significant investment required to build and support software. Developers, QA teams, release engineers, documentation, and testing infrastructure all come with costs. Moreover, when you're developing a product that is constantly evolving while also needing to maintain compatibility with another ever-changing product, the effort required for testing and support grows exponentially—it's an n^2 challenge that only becomes more complex over time.
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Sure, yet you can have a vendor that does all the work for you, as I understand the OP wants with a ZFS-over-iSCSI solution. So it does not matter if you pay a vendor for one or the other solution. You don't need competent people and if you want a non-HA storage for your HA cluster, you're already lacking them anyway.
LOL ! find one SMB hosting its own cloud with "a simple 25 or 100G network" have you even priced a 10gb \ rj45 switch? Using ceph in an SMB environment is cost prohibitive, it nearly demands 5 PVE Nodes, it will work on 2-3 but it's reliability goes to ..... and it still demands an isolated network layer. Again, if it was cost effective for VMWare to introduce "vSAN" then that same reasoning applied to an entry level PSS would also drive PVE, PBS , PMG subscriptions to these smaller businesses. With over 15 years of VMWare usage \ consultation, I can say that this is a common model:
3 ESXI node subscriptions
1 vSAN (or other block storage) subscription including hardware purchases for SAN\NAS
1 VEEAM subscription
I really don't see why Proxmox gmbh wouldn't see profits from the small investment in a PSS that just snaps into PVE.
The native support for iSCSI included with PVE is lacking , it doesn't include a native integration with the leading opensource storage provider (or even all of the most common target protocols) , i.e. TrueNAS Scale... My peers don't want to "roll their own" as this is the very cause for slow linux adoption in the SMB market as it increases the cost of owning a Linux system while decreasing the avenues of support... after all even Proxmox, a product with drop dead simplicity and reliability , doesn't have just one CSR at the helpdesk......
Still better than any PSS on a single machine, because it has fault tolerance. 3 nodes is totally fine for SMB and any node can fail leaving two working nodes and still redundancy and no dataloss. Even if this node fails, you will not have data loss. If you have a PSS and 3 nodes, you have one node more and not any machine can fail, just any of the compute nodes, yet if the storage node fails, you will have complete loss of operation. Even if you have PSS with ZFS replication, you will have data loss.
Use 3 nodes, a mesh network with 10, 25 or 100G (so without a switch) and NVMe. Very cheap, just sold such a system to a VMware-leaving customer and he is very happy with it. Connection to the outside world is currently via 1 GBE.
I really don't see how you would see profits in this. It's just a few commands and you'll have a working system that just need monitoring of the ZFS.
This has the same hardware requirements and costs as a proxmox cluster with ceph, and around 15k less in software license fees.
so all of these bugs are for an unnecessary solution ????
Make patches, not war!
Hide Search Description
Status: UNCONFIRMED, NEW, ASSIGNED, REOPENED, FIX PACKAGED, PATCH AVAILABLE, OUTDATED, POSTPONED, MORE INFO NEEDED, UNDECIDED Product: zfs Component: zfs Alias: zfs Summary: zfs Whiteboard: zfs Content: "zfs" Product: over Component: over Alias: over Summary: over Whiteboard: over Content: "over" Product: iscsi Component: iscsi Alias: iscsi Summary: iscsi Whiteboard: iscsi Content: "iscsi"
18 bugs found.
ID Product Comp Assignee▲ Status▲ Resolution Summary Changed
1448 pve Storage bugs NEW --- FreeNAS support for ZFS over iSCSI 13:47:25
1927 pve Storage bugs NEW --- support containers on ZFS over ISCSI storages 2023-12-11
2213 pve Storage bugs NEW --- ZFS over iSCSI doesn't work with Solaris 11.3 2019-05-17
3231 pve vzdump bugs NEW --- Backup sometimes hangs and blocks the I/O on the live machine 2024-11-25
3467 pve Storage bugs NEW --- Add sudo use to zfs/iscsi 2021-06-14
3685 pve Storage bugs NEW --- tgtd support for ZFS over iscsi 2021-10-19
4046 pve Storage bugs NEW --- PVE spams iSCSI target with SAI_GET_LBA_STATUS on moving storage from __ 2024-10-22
4665 common Document bugs NEW --- Glossary for either generated documentation or wiki (or both) 2023-04-13
5071 pve Storage bugs NEW --- ZFS over iSCSI - zpool not correctly referenced with iSCSI LIO plugin becasue /dev/<zpool> device not present anymore 2024-12-12
5299 pve Storage bugs NEW --- "write zeros = 1" while restoring backup to thin enabled ZFS over iSCSi storage 2024-03-13
5516 pve Storage bugs NEW --- ZFS over iSCSI storage not working with Debian 12 (Bookworm) target because of different zvol device paths 2024-08-02
960 pve Storage f.gruenbichler ASSI --- nas4free istgt path changed in new nas4free release 2016-05-03
2176 pve Backend bugs MORE --- Adding or removing iscsi disk crashes all VM's 2019-07-26
5024 pve vzdump bugs MORE --- zfs over iscsi storage unable to get disk information during backup 2023-11-06
5948 pve Kernel bugs MORE --- initrd.img missing cnic and bnx2x modules for bnx2 based nic/hba 2024-12-10
2059 pve Web UI bugs UNDE --- Can't do full clone of KVM template from shared storage to local storage on different node 2023-10-02
3662 pve Backend bugs UNDE --- TPM disfunctional with ZFS over iSCSI 2024-10-22
984 pve pve-comm bugs OUTD --- VM snapshot error (ZFS over iSCSI on NAS4Free) 2018-11-15
18 bugs found.
nesasary
Make patches, not war!
Hide Search Description
Status: UNCONFIRMED, NEW, ASSIGNED, REOPENED, FIX PACKAGED, PATCH AVAILABLE, OUTDATED, POSTPONED, MORE INFO NEEDED, UNDECIDED Product: zfs Component: zfs Alias: zfs Summary: zfs Whiteboard: zfs Content: "zfs" Product: over Component: over Alias: over Summary: over Whiteboard: over Content: "over" Product: iscsi Component: iscsi Alias: iscsi Summary: iscsi Whiteboard: iscsi Content: "iscsi"
18 bugs found.
ID Product Comp Assignee▲ Status▲ Resolution Summary Changed
1448 pve Storage bugs NEW --- FreeNAS support for ZFS over iSCSI 13:47:25
1927 pve Storage bugs NEW --- support containers on ZFS over ISCSI storages 2023-12-11
2213 pve Storage bugs NEW --- ZFS over iSCSI doesn't work with Solaris 11.3 2019-05-17
3231 pve vzdump bugs NEW --- Backup sometimes hangs and blocks the I/O on the live machine 2024-11-25
3467 pve Storage bugs NEW --- Add sudo use to zfs/iscsi 2021-06-14
3685 pve Storage bugs NEW --- tgtd support for ZFS over iscsi 2021-10-19
4046 pve Storage bugs NEW --- PVE spams iSCSI target with SAI_GET_LBA_STATUS on moving storage from __ 2024-10-22
4665 common Document bugs NEW --- Glossary for either generated documentation or wiki (or both) 2023-04-13
5071 pve Storage bugs NEW --- ZFS over iSCSI - zpool not correctly referenced with iSCSI LIO plugin becasue /dev/<zpool> device not present anymore 2024-12-12
5299 pve Storage bugs NEW --- "write zeros = 1" while restoring backup to thin enabled ZFS over iSCSi storage 2024-03-13
5516 pve Storage bugs NEW --- ZFS over iSCSI storage not working with Debian 12 (Bookworm) target because of different zvol device paths 2024-08-02
960 pve Storage f.gruenbichler ASSI --- nas4free istgt path changed in new nas4free release 2016-05-03
2176 pve Backend bugs MORE --- Adding or removing iscsi disk crashes all VM's 2019-07-26
5024 pve vzdump bugs MORE --- zfs over iscsi storage unable to get disk information during backup 2023-11-06
5948 pve Kernel bugs MORE --- initrd.img missing cnic and bnx2x modules for bnx2 based nic/hba 2024-12-10
2059 pve Web UI bugs UNDE --- Can't do full clone of KVM template from shared storage to local storage on different node 2023-10-02
3662 pve Backend bugs UNDE --- TPM disfunctional with ZFS over iSCSI 2024-10-22
984 pve pve-comm bugs OUTD --- VM snapshot error (ZFS over iSCSI on NAS4Free) 2018-11-15
18 bugs found.
nesasary
Oh, look, another "new member" stops by to tell a 20-year-old project how to do things.
@jt_telrite is the absence of a subscriber badge in your profile an oversight, or do you plan to make Proxmox 110 euro richer when they develop this mythical functionality that everyone needs?
explained by the Dunning Krueger effect. such is life...
@jt_telrite have you actually understood what you listed? these represent unclosed "bugs" in 20 YEARS totalling 18. 18! half of them are feature requests. others are wont fix/no longer relevent, not relevent to begin with, documentation related, or just not closed.
More to the point. Even if there WERE bugs specific to your request that have not been addressed doesnt mean they're relevant or NOT- just whether the devs considered them high enough importance to address. PVE does an admirable job presenting OPTIONS for deployment, not that ANY ONE OF THEM is actually important or equivalent in relevence to others.
You still need 3 or 4 machines with 8 or more large drives each ... $$$$$ (unless you don't use local zraid) , and a SAN and an external backup solution on it's own NAS (which would be another machine outside the cluster with PBS on NAS hardware). realistically, how often do you see hardware failures...
a QNAP with trueNAS scale is under $2k and you get iSCSI, SMB / CIFS and NFS shares.... https://nascompares.com/2023/01/06/truenas-scale-on-a-qnap-nas-installation-guide/ how would a PSS solution with access to Debian packages to "Roll your own" SAMBA/ NFS services not be valuable?????
Are you about to tell me that ZFS Raid is unstable, unreliable, irreparable?
pick a 2.5gbe switch
https://duckduckgo.com/?q=2.5+gbe+switch&t=brave&iar=shopping&iax=shopping&ia=shopping
to put a SAN behind the PVE and ...... CHEAP!
If you have PBS on second QNAP , you have enough redundancy to compensate for a dual channel or hyperconverged storage.
and if your going to "roll you own", an ubuntu samba server is well documented and can easily be a VM.....
again, realistically, how often do you see hardware failures...
oh look another stodgy old fart denies a good idea because it comes from someone outside his crony circle...
Dan Nicolae
Renowned Member
Mar 27, 2018
Add bookmark
#3
Minimum for Ceph is 3 nodes but not recomended for production.
You should use at least 6 nodes, 2 osd each and an enteprise ssd for bluestore db. Hardware, CPU 1 core for each OSD, 1GB RAM for each 1TB of OSD, 3 gigabit network cards, one for proxmox network, two for ceph network (bond).
Do not run VM on thesame server with Ceph / OSD's.
For the compute nodes, use minimum, 4 network cards. One for public (internet), one for Proxmox network, two for Ceph network (bond).
Renowned Member
Mar 27, 2018
Add bookmark
#3
Minimum for Ceph is 3 nodes but not recomended for production.
You should use at least 6 nodes, 2 osd each and an enteprise ssd for bluestore db. Hardware, CPU 1 core for each OSD, 1GB RAM for each 1TB of OSD, 3 gigabit network cards, one for proxmox network, two for ceph network (bond).
Do not run VM on thesame server with Ceph / OSD's.
For the compute nodes, use minimum, 4 network cards. One for public (internet), one for Proxmox network, two for Ceph network (bond).
It's disappointing to see this kind of disagreement in the forum 
Please be considerate.
The reality is that the requested functionality could indeed be valuable to some end users, particularly those with home labs. However, the costs associated with development, testing, maintenance, and ongoing support will outweigh any incremental revenue gains by many multiples.
Please understand that the PVE team is very thoughtful in prioritizing features that make the most sense for the platform. Given that they already have a cost-effective solution, I can't see this being a top priority. Most people paying for support need more enterprise features and reliability, not less...
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Please be considerate.The reality is that the requested functionality could indeed be valuable to some end users, particularly those with home labs. However, the costs associated with development, testing, maintenance, and ongoing support will outweigh any incremental revenue gains by many multiples.
Please understand that the PVE team is very thoughtful in prioritizing features that make the most sense for the platform. Given that they already have a cost-effective solution, I can't see this being a top priority. Most people paying for support need more enterprise features and reliability, not less...
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox