Playground time - virtual PBS with iSCSI Backend

Funny you brought this up. I just set this up earlier in the week to try out at home.

My setup under PVE 6.3-3:
  • VM1: TrueNAS Core
    • RAIDZ2 pool with a dedicated HBA and hard drives.
    • iSCSI share on a dataset with thin provisioning and compression enabled
  • VM2: Proxmox Backup Server
  • VM....: Various other VM, servers, gaming machines, etc.
So far it's been doing backups regularly in the week and working. I'd like to keep an eye on this setup and see if it works well for a longer period. It would be good to know if anyone else does this and whether or not it's a good idea.

Some high level steps of what I did:

Connect the iSCSI target in your PBS server
  1. Setup TrueNAS iSCSI share. I created one using the iSCSI wizard, but once setup it should be broadcasting your server for clients to discover.
  2. Proxmox Backup Server already has iscsi-tools command line installed on it. You can use it to discover and attach the iSCSI share to PBS. Before doing so edit /etc/iscsi/iscsid.conf so that it automatically connects the iSCSI node when PBS starts up.
    Code: 
    node.startup = automatic
  3. Run:
    Code: 
    iscsiadm --mode discovery --portal YOUR_SERVER_HOST --type sendtargets
    It should output a list of iSCSI targets setup on TrueNAS using IQN notation. For example:
    Code: 
    iqn.2021-04.com.example.myserver:myiscsi-drive
  4. Make note of the target name from the output. Login using:
    Code: 
    iscsiadm -m node --targetname=iqn.2021-04.com.example.myserver:myiscsi-drive --login
    This will login and attach the iSCSI to PBS. It also saves this config so if you setup Step #2 to "automatic" it will use this config to reconnect each startup.
  5. Use iscsiadm -m session -P 3to check the status of any iSCSI drives connected. It will also help you see what device file is associated with it. Mine was 'sdb'

    1619756031114.png

    You can also use lsblk to verify this
Mount the iSCSI drive as a datastore

This part I had troubles with, and actually still not sure if it's setup properly. I couldn't do the setup in the web UI so had to rely on CLI tools to format and setup the drive. Refer to the PBS documentation: https://pbs.proxmox.com/docs/

  1. Use fdisk /dev/sdx to partition / format your drive. Replace the device letter with yours.
  2. Format it using ext4 mkfs.ext4 /dev/sdx1. Again, replace the device letter and number with your own.
  3. Mount the device. I chose to put it in /mnt/proxmox-backup but perhaps there is a better place to put it. I noticed that creating a datastore using the web UI will create a folder in /mnt/datastore.

    Code: 
    mkdir /mnt/proxmox-backup
mount /dev/sdx1 /mnt/proxmox-backup
  4. You'll also want to update your /etc/fstab so this drive is mounted on startup. What is important is the _netdev flag, This will ensure that the network stack is ready to connect the iSCSI drive during boot up.
    Code: 
    /dev/sdx1 /mnt/proxmox-backup ext4 _netdev 0 0
  5. Finally create the PBS datastore using:
    Code: 
    proxmox-backup-manager datastore create backups /mnt/proxmox-backup

    1619755960614.png
You should then be able to see the datastore in PBS. I couldn't get the iSCSI drive to appear in the "Storage / Disks" and do the setup there.

Add the PBS server as storage in Proxmox VE 6.3+

1619755866586.png

Again, not sure if this is the right way of setting this up. But hopefully this can get some conversation going. I'd also be interested to know if people have pros and cons with actually utilizing this kind of setup. I don't have much space for physical servers so it really lets me leverage this main VM server I have. I'm curious about the differences of using ZFS or creating LVMs for these datastores.
 
  • Like
Reactions: eroux, Evertice, NicT and 13 others
zenopka said:
Code: 
mkdir /mnt/proxmox-backup
mount /dev/sdx1 /mnt/proxmox-backup
  1. You'll also want to update your /etc/fstab so this drive is mounted on startup. What is important is the _netdev flag, This will ensure that the network stack is ready to connect the iSCSI drive during boot up.
    Code: 
    /dev/sdx1 /mnt/proxmox-backup ext4 _netdev 0 0
Click to expand...
Hi All,

Just as a heads-up: In case you have several iscsi targets (happens to the best :-) ) please use blkid / uuid in /etc/fstab - that way you don't have effects in case the sd-devices get renumbered on reboot.

Tobias
 
  • Like
Reactions: NicT, ajeffco, scyto and 1 other person
Hey there @zenopka , I created an account explicitly to say 'thank you' for this post. To expound on this, your steps pointed me in the right direction for using an NFS Mount as a datastore in PBS!

Thank you so much for the insight!
 
  • Like
Reactions: spetrillo
I am having trouble mounting the storage to my PVE server. Could someone let me know how to do this in the GUI?
 
Very helpful.

But iscsi-tools are also in newest 3.3 not available. Installing with

Code: 
apt install open-iscsi

I've created the target as LVM module, increasing with additional LUNs is easier, then resizing the LUN.

Cheers.
 
  • Like
Reactions: richik75
Worked great for me. only issue i ran into is that the iscsi volume was 8TB and it was dos instead of GPT. fixed that and was able to create a 8tb partition and format it to ext4 and backups are flowing great atm.
Next step is to setup a second PBS for hard repo
 
  • Like
Reactions: Johannes S
I just built 2 different PBS w/iSCSI storage few weeks ago.
Both are in production, pushing a lot of data.

....
I built the first one as this post indicates, mounted in the PBS VM itself.
For this iSCSI mounted-in-the-OS version, I figured it would probably work better than the VM hosted by TrueNAS itself.
Yes the VM itself is more stable, but the performance is mid at best.
I'm unsatisfied with it and will probably decomm soon, going back to TrueNAS hosting the PBS VM.
....

For the second, very successful installation, I used iSCSI because i had an old iSCSI-only EMC device.
I did the the way the wiki lays out.
I used an ISCSi mount to the PVE host.
And you put an LVM on top of that as the datastore.

See this very long and explicit article simply called "Multipath".
https://pve.proxmox.com/wiki/Multipath#LVM_on_top_of_a_LUN

iSCSI mount in the PVE cluster

1746123809818.png


LVM mount on the iSCSI

1746123820730.png


Storage mounts on the host.
The iSCSI doesn't show a size, while the LVM does.
1746124773427.png

---
BTW ... I am not an iSCSI expert by any means. If anyone has tuning considerations for this LVM-on-LUN configuration, I would love to trade notes.
 
Last edited:
You must log in or register to reply here.
Top Bottom