I do use NexentaStor as well as OI (and illumian.org, the Debian-packaged illumos variant) but did not get the chance to use it in combination with Proxmox.
My overall experience with iSCSI vs. NFS was that albeit iSCSI might give you a tick more of performance, it's not always worth the time...