SOLVED: ZVOL with LVM (for targetcli) gets lockd after reboot

G

godfather007

Renowned Member
Proxmox Subscriber
Oct 17, 2008
96
6
73
48
Netherlands
Hi,

i'm abusing my PBS installation for some archival requirement.
Installed targetcli-fb and assigned a zvol through it to PVE.

Using this target directly from windows instead through PVE was working flawless.

But now the zvol is being in use after a reboot...

It seems the (PVE) LVM is mounted on PBS before it can be passed on to LIO.

lsblk looks like this (test-vm)

sda 8:0 0 32G 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 512M 0 part
└─sda3 8:3 0 31.5G 0 part
├─pbs-swap 252:0 0 3.8G 0 lvm [SWAP]
└─pbs-root 252:1 0 27.7G 0 lvm /
sr0 11:0 1 1024M 0 rom

zd0 230:0 0 60G 0 disk
└─TST-vm--101--disk--0 252:2 0 10G 0 lvm

vda 253:0 0 100G 0 disk
├─vda1 253:1 0 100G 0 part
└─vda9 253:9 0 8M 0 part



Any idea how i can overcome this?
 
Please show me your targetcli configuration and "zfs list". And also your Proxmox Client configuration for that storage "cat /etc/pve/storage.cfg".
 
Targetcli is empty now, because of the issue

Code: 
/backstores/block> cd /
/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 0]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 1]
  | o- iqn.2003-01.org.linux-iscsi.pbs.x8664:sn.94387e7b314e ............................................................. [TPGs: 1]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: 3]
  |     | o- iqn.1993-08.org.debian:01:126055dcb3f ................................................................ [Mapped LUNs: 0]
  |     | o- iqn.1993-08.org.debian:01:5260e4f5536 ................................................................ [Mapped LUNs: 0]
  |     | o- iqn.1993-08.org.debian:01:ee646172113e ............................................................... [Mapped LUNs: 0]
  |     o- luns .......................................................................................................... [LUNs: 0]
  |     o- portals .................................................................................................... [Portals: 1]
  |       o- 0.0.0.0:3260 ..................................................................................................... [OK]
  o- loopback ......................................................................................................... [Targets: 0]
  o- vhost ............................................................................................................ [Targets: 0]
  o- xen-pvscsi ....................................................................................................... [Targets: 0]
/>

but a backup json file exists with this content from when it was working:

Code: 
{
  "fabric_modules": [],
  "storage_objects": [
    {
      "alua_tpgs": [
        {
          "alua_access_state": 0,
          "alua_access_status": 0,
          "alua_access_type": 3,
          "alua_support_active_nonoptimized": 1,
          "alua_support_active_optimized": 1,
          "alua_support_offline": 1,
          "alua_support_standby": 1,
          "alua_support_transitioning": 1,
          "alua_support_unavailable": 1,
          "alua_write_metadata": 0,
          "implicit_trans_secs": 0,
          "name": "default_tg_pt_gp",
          "nonop_delay_msecs": 100,
          "preferred": 0,
          "tg_pt_gp_id": 0,
          "trans_delay_msecs": 0
        }
      ],
      "attributes": {
        "alua_support": 1,
        "block_size": 512,
        "emulate_3pc": 1,
        "emulate_caw": 1,
        "emulate_dpo": 1,
        "emulate_fua_read": 1,
        "emulate_fua_write": 1,
        "emulate_model_alias": 1,
        "emulate_pr": 1,
        "emulate_rest_reord": 0,
        "emulate_rsoc": 1,
        "emulate_tas": 1,
        "emulate_tpu": 0,
        "emulate_tpws": 0,
        "emulate_ua_intlck_ctrl": 0,
        "emulate_write_cache": 0,
        "enforce_pr_isids": 1,
        "force_pr_aptpl": 0,
        "is_nonrot": 1,
        "max_unmap_block_desc_count": 1,
        "max_unmap_lba_count": 524288,
        "max_write_same_len": 65535,
        "optimal_sectors": 32768,
        "pgr_support": 1,
        "pi_prot_format": 0,
        "pi_prot_type": 0,
        "pi_prot_verify": 0,
        "queue_depth": 128,
        "submit_type": 0,
        "unmap_granularity": 32,
        "unmap_granularity_alignment": 0,
        "unmap_zeroes_data": 0
      },
      "dev": "/dev/zvol/tst/iscsi",
      "name": "ISCSI",
      "plugin": "block",
      "readonly": false,
      "write_back": false,
      "wwn": "e04628fe-abb4-4fcf-a138-0aa4b1e99d47"
    }
  ],
  "targets": [
    {
      "fabric": "iscsi",
      "parameters": {
        "cmd_completion_affinity": "-1"
      },
      "tpgs": [
        {
          "attributes": {
            "authentication": 0,
            "cache_dynamic_acls": 0,
            "default_cmdsn_depth": 64,
            "default_erl": 0,
            "demo_mode_discovery": 1,
            "demo_mode_write_protect": 1,
            "fabric_prot_type": 0,
            "generate_node_acls": 0,
            "login_keys_workaround": 1,
            "login_timeout": 15,
            "prod_mode_write_protect": 0,
            "t10_pi": 0,
            "tpg_enabled_sendtargets": 1
          },
          "enable": true,
          "luns": [
            {
              "alias": "6e8a0858a0",
              "alua_tg_pt_gp_name": "default_tg_pt_gp",
              "index": 0,
              "storage_object": "/backstores/block/ISCSI"
            }
          ],
          "node_acls": [
            {
              "attributes": {
                "authentication": -1,
                "dataout_timeout": 3,
                "dataout_timeout_retries": 5,
                "default_erl": 0,
                "nopin_response_timeout": 30,
                "nopin_timeout": 15,
                "random_datain_pdu_offsets": 0,
                "random_datain_seq_offsets": 0,
                "random_r2t_offsets": 0
              },
              "mapped_luns": [
                {
                  "alias": "61b5bed6b6",
                  "index": 0,
                  "tpg_lun": 0,
                  "write_protect": false
                }
              ],
              "node_wwn": "iqn.1993-08.org.debian:01:ee646172113e"
            },
            {
              "attributes": {
                "authentication": -1,
                "dataout_timeout": 3,
                "dataout_timeout_retries": 5,
                "default_erl": 0,
                "nopin_response_timeout": 30,
                "nopin_timeout": 15,
                "random_datain_pdu_offsets": 0,
                "random_datain_seq_offsets": 0,
                "random_r2t_offsets": 0
              },
              "mapped_luns": [
                {
                  "alias": "77f658eac5",
                  "index": 0,
                  "tpg_lun": 0,
                  "write_protect": false
                }
              ],
              "node_wwn": "iqn.1993-08.org.debian:01:5260e4f5536"
            },
            {
              "attributes": {
                "authentication": -1,
                "dataout_timeout": 3,
                "dataout_timeout_retries": 5,
                "default_erl": 0,
                "nopin_response_timeout": 30,
                "nopin_timeout": 15,
                "random_datain_pdu_offsets": 0,
                "random_datain_seq_offsets": 0,
                "random_r2t_offsets": 0
              },
              "mapped_luns": [
                {
                  "alias": "304e1c80f9",
                  "index": 0,
                  "tpg_lun": 0,
                  "write_protect": false
                }
              ],
              "node_wwn": "iqn.1993-08.org.debian:01:126055dcb3f"
            }
          ],
          "parameters": {
            "AuthMethod": "CHAP,None",
            "DataDigest": "CRC32C,None",
            "DataPDUInOrder": "Yes",
            "DataSequenceInOrder": "Yes",
            "DefaultTime2Retain": "20",
            "DefaultTime2Wait": "2",
            "ErrorRecoveryLevel": "0",
            "FirstBurstLength": "65536",
            "HeaderDigest": "CRC32C,None",
            "IFMarkInt": "Reject",
            "IFMarker": "No",
            "ImmediateData": "Yes",
            "InitialR2T": "Yes",
            "MaxBurstLength": "262144",
            "MaxConnections": "1",
            "MaxOutstandingR2T": "1",
            "MaxRecvDataSegmentLength": "8192",
            "MaxXmitDataSegmentLength": "262144",
            "OFMarkInt": "Reject",
            "OFMarker": "No",
            "TargetAlias": "LIO Target"
          },
          "portals": [
            {
              "ip_address": "0.0.0.0",
              "iser": false,
              "offload": false,
              "port": 3260
            }
          ],
          "tag": 1
        }
      ],
      "wwn": "iqn.2003-01.org.linux-iscsi.pbs.x8664:sn.94387e7b314e"
    }
  ]
}

zfs list shows this:

tst is the pool
iscsi is the zvol on it.

Code: 
root@pbs:/etc/rtslib-fb-target/backup# zfs list
NAME        USED  AVAIL  REFER  MOUNTPOINT
tst        8.45G  88.0G  24.3M  /mnt/datastore/tst
tst/iscsi  8.43G  88.0G  8.43G  -
root@pbs:/etc/rtslib-fb-target/backup#

above is all on a PBS-test machine.
below is the storage.cfg on one of my tst-pve's:

Code: 
root@tprxu04:~# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content backup,vztmpl,iso

zfspool: local-zfs
        pool rpool/data
        content images,rootdir
        sparse 1

rbd: VMS
        content rootdir,images
        krbd 0
        pool VMS

pbs: PBSA-repl
        datastore PBSA-repl
        server pbsu01.utr.mgmt
        content backup
        fingerprint cc:cd:a4:c9:77:72:c4:1b:dd:db:c9:35:dd:97:a4:8d:2e:20:7f:dc:7b:3b:9e:3d:20:df:eb:02:76:e3:7f:52
        prune-backups keep-all=1
        username root@pam

pbs: PBSU01
        datastore PBSU01
        server pbsu01.utr.mgmt
        content backup
        fingerprint cc:cd:a4:c9:77:72:c4:1b:dd:db:c9:35:dd:97:a4:8d:2e:20:7f:dc:7b:3b:9e:3d:20:df:eb:02:76:e3:7f:52
        prune-backups keep-all=1
        username root@pam

iscsi: iSCSiRAW
        portal ops-tst-pbs.utr.mgmt
        target iqn.2003-01.org.linux-iscsi.pbs.x8664:sn.94387e7b314e
        content none

lvm: LVMonISCSI
        vgname TST
        base iSCSiRAW:0.0.0.scsi-36001405e04628feabb44fcfa1380aa4b
        content rootdir,images
        saferemove 0
        shared 1
 
Oh, dmesg shows this during boot:

Code: 
[   11.886889] Loading iSCSI transport class v2.0-870.
[   12.107469] Rounding down aligned max_sectors from 4294967295 to 4294967288
[   12.107551] db_root: cannot open: /etc/target
[   15.733787] rx_data returned 0, expecting 48.
[   15.733799] iSCSI Login negotiation failed.
[   16.055896] rx_data returned 0, expecting 48.
 
Targetcli is a little sensitive. According to the excerpt from your Targetcli, you are missing a few things: (vor example)

The block export:
Code: 
 o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 1]
 ALUA state: Active/optimized]
  | | o- volume4cephcluster .................. [/dev/zvol/ssd-pool/safe/targetcli/volume4cephcluster (75.0GiB) write-thru activated]
  | |   o- alua ................................................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]

And then i should look like this:
Code: 
 o- iscsi ............................................................................................................ [Targets: 1]
o- iqn.2024-12.org.linux-iscsi.backup.x8664:sn.965893387170 .......................................................... [TPGs: 1]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: 3]
  |     | o- iqn.2024-08.org.node01:01:b2b69707b91 ................................................................ [Mapped LUNs: 1]
  |     | | o- mapped_lun0 .................................................................... [lun0 block/volume4cephcluster (rw)]
  |     | o- iqn.2024-08.org.node02:01:31a01b811a6d ............................................................... [Mapped LUNs: 1]
  |     | | o- mapped_lun0 .................................................................... [lun0 block/volume4cephcluster (rw)]
  |     | o- iqn.2024-08.org.node03:01:b3c30e749e ................................................................. [Mapped LUNs: 1]
  |     |   o- mapped_lun0 .................................................................... [lun0 block/volume4cephcluster (rw)]
  |     o- luns .......................................................................................................... [LUNs: 1]
  |     | o- lun0 ............. [block/volume4cephcluster (/dev/zvol/ssd-pool/safe/targetcli/volume4cephcluster) (default_tg_pt_gp)]
  |     o- portals .................................................................................................... [Portals: 2]
  |       o- 172.16.10.16:3260 ................................................................................................ [OK]
  |       o- 192.168.10.20:3260 ............................................................................................... [OK

If the volume is not available when the targetcli starts, the configuration is simply discarded. For this reason, I have also removed targetcli from the systemd autostart and now only start it manually. The exported file system must be a volume. A dataset will not work as an ISCSI export.
 
I'm not sure if we're on the same page..
Let me rephrase. Following is the targetcli from my (still functioning) production environment:

Code: 
/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 1]
  | | o- cons-archive-02 ......................................... [/dev/zvol/ZFSR600/cons-archive-02 (8.0TiB) write-thru activated]
  | |   o- alua ................................................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 1]
  | o- iqn.2003-01.org.linux-iscsi.pbsu01.x8664:sn.cb579c1bf64e .......................................................... [TPGs: 1]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: 3]
  |     | o- iqn.1993-08.org.debian:01:5ee3bf2413 ................................................................. [Mapped LUNs: 1]
  |     | | o- mapped_lun0 ....................................................................... [lun0 block/cons-archive-02 (rw)]
  |     | o- iqn.1993-08.org.debian:01:85c3c1362c40 ............................................................... [Mapped LUNs: 1]
  |     | | o- mapped_lun0 ....................................................................... [lun0 block/cons-archive-02 (rw)]
  |     | o- iqn.1993-08.org.debian:01:877c70dcaae ................................................................ [Mapped LUNs: 1]
  |     |   o- mapped_lun0 ....................................................................... [lun0 block/cons-archive-02 (rw)]
  |     o- luns .......................................................................................................... [LUNs: 1]
  |     | o- lun0 ................................... [block/cons-archive-02 (/dev/zvol/ZFSR600/cons-archive-02) (default_tg_pt_gp)]
  |     o- portals .................................................................................................... [Portals: 1]
  |       o- 10.80.11.192:3260 ................................................................................................ [OK]
  o- loopback ......................................................................................................... [Targets: 0]
  o- vhost ............................................................................................................ [Targets: 0]
  o- xen-pvscsi ....................................................................................................... [Targets: 0]

As soon as i reboot the host, the /dev/zvol/ZFSR600/cons-archive-02 gets discovered and, for some reason, because of the accessible LVM partition, it gets locked. When i try to reassign the /dev/zvol/ZFSR600/cons-archive-02, which is a clone of /dev/zvol/ZFSR600/cons-archive-01 targetcli responds with "device in use" ..
 
UNTESTED, use at your own risk:

I think that a LVM filter should work here... In /etc/lvm/lvm.conf, replace:

Code: 
devices {
     # added by pve-manager to avoid scanning ZFS zvols and Ceph rbds
     global_filter=["r|/dev/zd.*|","r|/dev/rbd.*|"]
}

with something like:

Code: 
devices {
     # added by pve-manager to avoid scanning ZFS zvols and Ceph rbds
     global_filter=["r|/dev/zd.*|","r|/dev/rbd.*|","r|/dev/zvol/ZFSR600/cons.*|"]
}

so the host will not activate the LVM that it may find in your iscsi target.
 
  • Like
Reactions: mariol and godfather007
You must log in or register to reply here.
Top Bottom