targetcli on zfs lockup?

G

godfather007

Renowned Member
Oct 17, 2008
94
4
73
48
Netherlands
Hi,

i'm abusing my PBS installation for some archival requirement.
Installed targetcli-fb and assigned a zvol through it to PVE.

Using this target directly from windows instead through PVE was working flawless.

But now the zvol is being in use after a reboot...

It seems the (PVE) LVM is mounted on PBS before it can be passed on to LIO.

lsblk looks like this (test-vm)

sda 8:0 0 32G 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 512M 0 part
└─sda3 8:3 0 31.5G 0 part
├─pbs-swap 252:0 0 3.8G 0 lvm [SWAP]
└─pbs-root 252:1 0 27.7G 0 lvm /
sr0 11:0 1 1024M 0 rom

zd0 230:0 0 60G 0 disk
└─TST-vm--101--disk--0 252:2 0 10G 0 lvm

vda 253:0 0 100G 0 disk
├─vda1 253:1 0 100G 0 part
└─vda9 253:9 0 8M 0 part



Any idea how i can overcome this?
 
Please show me your targetcli configuration and "zfs list". And also your Proxmox Client configuration for that storage "cat /etc/pve/storage.cfg".
 
Targetcli is empty now, because of the issue

Code: 
/backstores/block> cd /
/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 0]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 1]
  | o- iqn.2003-01.org.linux-iscsi.pbs.x8664:sn.94387e7b314e ............................................................. [TPGs: 1]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: 3]
  |     | o- iqn.1993-08.org.debian:01:126055dcb3f ................................................................ [Mapped LUNs: 0]
  |     | o- iqn.1993-08.org.debian:01:5260e4f5536 ................................................................ [Mapped LUNs: 0]
  |     | o- iqn.1993-08.org.debian:01:ee646172113e ............................................................... [Mapped LUNs: 0]
  |     o- luns .......................................................................................................... [LUNs: 0]
  |     o- portals .................................................................................................... [Portals: 1]
  |       o- 0.0.0.0:3260 ..................................................................................................... [OK]
  o- loopback ......................................................................................................... [Targets: 0]
  o- vhost ............................................................................................................ [Targets: 0]
  o- xen-pvscsi ....................................................................................................... [Targets: 0]
/>

but a backup json file exists with this content from when it was working:

Code: 
{
  "fabric_modules": [],
  "storage_objects": [
    {
      "alua_tpgs": [
        {
          "alua_access_state": 0,
          "alua_access_status": 0,
          "alua_access_type": 3,
          "alua_support_active_nonoptimized": 1,
          "alua_support_active_optimized": 1,
          "alua_support_offline": 1,
          "alua_support_standby": 1,
          "alua_support_transitioning": 1,
          "alua_support_unavailable": 1,
          "alua_write_metadata": 0,
          "implicit_trans_secs": 0,
          "name": "default_tg_pt_gp",
          "nonop_delay_msecs": 100,
          "preferred": 0,
          "tg_pt_gp_id": 0,
          "trans_delay_msecs": 0
        }
      ],
      "attributes": {
        "alua_support": 1,
        "block_size": 512,
        "emulate_3pc": 1,
        "emulate_caw": 1,
        "emulate_dpo": 1,
        "emulate_fua_read": 1,
        "emulate_fua_write": 1,
        "emulate_model_alias": 1,
        "emulate_pr": 1,
        "emulate_rest_reord": 0,
        "emulate_rsoc": 1,
        "emulate_tas": 1,
        "emulate_tpu": 0,
        "emulate_tpws": 0,
        "emulate_ua_intlck_ctrl": 0,
        "emulate_write_cache": 0,
        "enforce_pr_isids": 1,
        "force_pr_aptpl": 0,
        "is_nonrot": 1,
        "max_unmap_block_desc_count": 1,
        "max_unmap_lba_count": 524288,
        "max_write_same_len": 65535,
        "optimal_sectors": 32768,
        "pgr_support": 1,
        "pi_prot_format": 0,
        "pi_prot_type": 0,
        "pi_prot_verify": 0,
        "queue_depth": 128,
        "submit_type": 0,
        "unmap_granularity": 32,
        "unmap_granularity_alignment": 0,
        "unmap_zeroes_data": 0
      },
      "dev": "/dev/zvol/tst/iscsi",
      "name": "ISCSI",
      "plugin": "block",
      "readonly": false,
      "write_back": false,
      "wwn": "e04628fe-abb4-4fcf-a138-0aa4b1e99d47"
    }
  ],
  "targets": [
    {
      "fabric": "iscsi",
      "parameters": {
        "cmd_completion_affinity": "-1"
      },
      "tpgs": [
        {
          "attributes": {
            "authentication": 0,
            "cache_dynamic_acls": 0,
            "default_cmdsn_depth": 64,
            "default_erl": 0,
            "demo_mode_discovery": 1,
            "demo_mode_write_protect": 1,
            "fabric_prot_type": 0,
            "generate_node_acls": 0,
            "login_keys_workaround": 1,
            "login_timeout": 15,
            "prod_mode_write_protect": 0,
            "t10_pi": 0,
            "tpg_enabled_sendtargets": 1
          },
          "enable": true,
          "luns": [
            {
              "alias": "6e8a0858a0",
              "alua_tg_pt_gp_name": "default_tg_pt_gp",
              "index": 0,
              "storage_object": "/backstores/block/ISCSI"
            }
          ],
          "node_acls": [
            {
              "attributes": {
                "authentication": -1,
                "dataout_timeout": 3,
                "dataout_timeout_retries": 5,
                "default_erl": 0,
                "nopin_response_timeout": 30,
                "nopin_timeout": 15,
                "random_datain_pdu_offsets": 0,
                "random_datain_seq_offsets": 0,
                "random_r2t_offsets": 0
              },
              "mapped_luns": [
                {
                  "alias": "61b5bed6b6",
                  "index": 0,
                  "tpg_lun": 0,
                  "write_protect": false
                }
              ],
              "node_wwn": "iqn.1993-08.org.debian:01:ee646172113e"
            },
            {
              "attributes": {
                "authentication": -1,
                "dataout_timeout": 3,
                "dataout_timeout_retries": 5,
                "default_erl": 0,
                "nopin_response_timeout": 30,
                "nopin_timeout": 15,
                "random_datain_pdu_offsets": 0,
                "random_datain_seq_offsets": 0,
                "random_r2t_offsets": 0
              },
              "mapped_luns": [
                {
                  "alias": "77f658eac5",
                  "index": 0,
                  "tpg_lun": 0,
                  "write_protect": false
                }
              ],
              "node_wwn": "iqn.1993-08.org.debian:01:5260e4f5536"
            },
            {
              "attributes": {
                "authentication": -1,
                "dataout_timeout": 3,
                "dataout_timeout_retries": 5,
                "default_erl": 0,
                "nopin_response_timeout": 30,
                "nopin_timeout": 15,
                "random_datain_pdu_offsets": 0,
                "random_datain_seq_offsets": 0,
                "random_r2t_offsets": 0
              },
              "mapped_luns": [
                {
                  "alias": "304e1c80f9",
                  "index": 0,
                  "tpg_lun": 0,
                  "write_protect": false
                }
              ],
              "node_wwn": "iqn.1993-08.org.debian:01:126055dcb3f"
            }
          ],
          "parameters": {
            "AuthMethod": "CHAP,None",
            "DataDigest": "CRC32C,None",
            "DataPDUInOrder": "Yes",
            "DataSequenceInOrder": "Yes",
            "DefaultTime2Retain": "20",
            "DefaultTime2Wait": "2",
            "ErrorRecoveryLevel": "0",
            "FirstBurstLength": "65536",
            "HeaderDigest": "CRC32C,None",
            "IFMarkInt": "Reject",
            "IFMarker": "No",
            "ImmediateData": "Yes",
            "InitialR2T": "Yes",
            "MaxBurstLength": "262144",
            "MaxConnections": "1",
            "MaxOutstandingR2T": "1",
            "MaxRecvDataSegmentLength": "8192",
            "MaxXmitDataSegmentLength": "262144",
            "OFMarkInt": "Reject",
            "OFMarker": "No",
            "TargetAlias": "LIO Target"
          },
          "portals": [
            {
              "ip_address": "0.0.0.0",
              "iser": false,
              "offload": false,
              "port": 3260
            }
          ],
          "tag": 1
        }
      ],
      "wwn": "iqn.2003-01.org.linux-iscsi.pbs.x8664:sn.94387e7b314e"
    }
  ]
}

zfs list shows this:

tst is the pool
iscsi is the zvol on it.

Code: 
root@pbs:/etc/rtslib-fb-target/backup# zfs list
NAME        USED  AVAIL  REFER  MOUNTPOINT
tst        8.45G  88.0G  24.3M  /mnt/datastore/tst
tst/iscsi  8.43G  88.0G  8.43G  -
root@pbs:/etc/rtslib-fb-target/backup#

above is all on a PBS-test machine.
below is the storage.cfg on one of my tst-pve's:

Code: 
root@tprxu04:~# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content backup,vztmpl,iso

zfspool: local-zfs
        pool rpool/data
        content images,rootdir
        sparse 1

rbd: VMS
        content rootdir,images
        krbd 0
        pool VMS

pbs: PBSA-repl
        datastore PBSA-repl
        server pbsu01.utr.mgmt
        content backup
        fingerprint cc:cd:a4:c9:77:72:c4:1b:dd:db:c9:35:dd:97:a4:8d:2e:20:7f:dc:7b:3b:9e:3d:20:df:eb:02:76:e3:7f:52
        prune-backups keep-all=1
        username root@pam

pbs: PBSU01
        datastore PBSU01
        server pbsu01.utr.mgmt
        content backup
        fingerprint cc:cd:a4:c9:77:72:c4:1b:dd:db:c9:35:dd:97:a4:8d:2e:20:7f:dc:7b:3b:9e:3d:20:df:eb:02:76:e3:7f:52
        prune-backups keep-all=1
        username root@pam

iscsi: iSCSiRAW
        portal ops-tst-pbs.utr.mgmt
        target iqn.2003-01.org.linux-iscsi.pbs.x8664:sn.94387e7b314e
        content none

lvm: LVMonISCSI
        vgname TST
        base iSCSiRAW:0.0.0.scsi-36001405e04628feabb44fcfa1380aa4b
        content rootdir,images
        saferemove 0
        shared 1
 
Oh, dmesg shows this during boot:

Code: 
[   11.886889] Loading iSCSI transport class v2.0-870.
[   12.107469] Rounding down aligned max_sectors from 4294967295 to 4294967288
[   12.107551] db_root: cannot open: /etc/target
[   15.733787] rx_data returned 0, expecting 48.
[   15.733799] iSCSI Login negotiation failed.
[   16.055896] rx_data returned 0, expecting 48.
 
Targetcli is a little sensitive. According to the excerpt from your Targetcli, you are missing a few things: (vor example)

The block export:
Code: 
 o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 1]
 ALUA state: Active/optimized]
  | | o- volume4cephcluster .................. [/dev/zvol/ssd-pool/safe/targetcli/volume4cephcluster (75.0GiB) write-thru activated]
  | |   o- alua ................................................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]

And then i should look like this:
Code: 
 o- iscsi ............................................................................................................ [Targets: 1]
o- iqn.2024-12.org.linux-iscsi.backup.x8664:sn.965893387170 .......................................................... [TPGs: 1]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: 3]
  |     | o- iqn.2024-08.org.node01:01:b2b69707b91 ................................................................ [Mapped LUNs: 1]
  |     | | o- mapped_lun0 .................................................................... [lun0 block/volume4cephcluster (rw)]
  |     | o- iqn.2024-08.org.node02:01:31a01b811a6d ............................................................... [Mapped LUNs: 1]
  |     | | o- mapped_lun0 .................................................................... [lun0 block/volume4cephcluster (rw)]
  |     | o- iqn.2024-08.org.node03:01:b3c30e749e ................................................................. [Mapped LUNs: 1]
  |     |   o- mapped_lun0 .................................................................... [lun0 block/volume4cephcluster (rw)]
  |     o- luns .......................................................................................................... [LUNs: 1]
  |     | o- lun0 ............. [block/volume4cephcluster (/dev/zvol/ssd-pool/safe/targetcli/volume4cephcluster) (default_tg_pt_gp)]
  |     o- portals .................................................................................................... [Portals: 2]
  |       o- 172.16.10.16:3260 ................................................................................................ [OK]
  |       o- 192.168.10.20:3260 ............................................................................................... [OK

If the volume is not available when the targetcli starts, the configuration is simply discarded. For this reason, I have also removed targetcli from the systemd autostart and now only start it manually. The exported file system must be a volume. A dataset will not work as an ISCSI export.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom