ZFS over iSCSI - Temporary Planned Outage

[EagleTG]

Hi everyone,

I have a two-node production Proxmox environment running ZFS over iSCSI to a FreeNAS/TrueNAS storage backend.

I am planning an outage to do some firmware upgrades to the FreeNAS/TrueNAS host hardware. Currently, the FreeNAS/TrueNAS environment is a single host with internal hardware redundancies, but without a full replicated/failover environment.

My plan is to migrate workloads to other storage to keep things online during the underlying storage system outage. However, being a long-time VMware user, I can't get out of my head some of the pain that we had with VMware and iSCSI relating to "all paths down". In scenarios where iSCSI connections fail, I have had occasions where VMware hosts will sometimes go into a mode where they are essentially inaccessible. They exert all efforts to attempt to reconnect to storage and become nearly unresponsive. On a few occasions, I've needed to contact VMware to get us out of the resulting mess that is created.

Does anyone have experience with Proxmox in such regard? Assuming there are no VM's or other guests running on the underlying storage environment, can I take that storage environment offline, have Proxmox hosts that were connected smoothly mark it as "offline", bring the storage environment back up, and have Proxmox gracefully re-attach and mark it back online?

If not, are there suggestions for prepping Promxox for the outage? Can I "temporarily" mark iSCSI storage as inactive or similar? My searching and Googling has so far not yielded a clear answer. Unfortunately, I don't have a similar Proxmox test environment for trying this out. My test deployments don't have the ZFS over iSCSI piece currently.

If worse comes to worse, I can plan a full outage of this environment, just trying to prevent the need to do so.

Thanks in advance!
-Todd

 

[bbgeek17]

You should have no issues with taking that storage down. Even if you have dependencies still active, only those should be affected.
It would be prudent to confirm that you have no connections to that storage. Use "pvesm list" and/or "iscsiadm" to confirm that everything is clean.

There is no option to mark the storage "temporary" inactive. You can just make a backup of /etc/pve/storage.conf and then edit the file to remove the blocks specific to iscsi storage. After you are done just swap the file back.

As long as you migrated all data away from the storage you should be fine. Keep in mind cloud-init drives, snapshots, etc.

Good luck

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[EagleTG]

Awesome, thanks BBGeek!! I'll report back once we perform the outage.

 

[EagleTG]

Just a quick update for anyone finding this thread, I decided to take the full outage just in case. I shut down all workloads and everything went perfectly. Upon booting everything up, I ran some testing with disconnecting offline workloads from storage and can confirm everything BBGeek said worked exactly as indicated. I did not see any issues similar to the "APD" situations that would plague VMware in certain similar situations.

Thanks again!