Where are my Proxmox files in a ZFS?

[ronejamesdo]

I apologize in advance for the ridiculously n00b question. I am not even sure it belongs here and not a ZFS forum. I have seen a couple of people ask this in other posts and gotten answers, but something about "it's a block thing" is not resonating with me. I watched an introduction to ZFS YouTube but it was all RAID concepts, which I get, and did not really get to this except to say that the OS is responsible for the file system and doesn't know about ZFS (like any RAID) (I think I remember him saying that anyway).

Pretending I'm dumb (-ahem-)... would anyone have the patience to sketch out how in ESXi I have a /vmfs/volumes/RAID/guest-name folder with .vmx and .vmdk and so on which are actually the VM. I can technically copy them to another machine and the .vmdk is the hard drive and the .vmx is some configs, and I can add them to another ESXi (of the sameish versions, etc)... but on (my) zfs Proxmox I have zfs -list which has the Name/Mount Point of my zfs storage, but has no files (at least from ls's perspective)? (I have added other folders for backup testing to that same zfs volume so I have Name/Mount Point my-zfs/backup-test and when I take a backup, those files are ls-able and seem like they might copy to another box (or survive and OS drive crash) without trouble.

There must be equivalents of the .vmdk somewhere in here... mustn't there?

Thank you (and if no one replies, I get it).

RJD

 

[Dunuin]

but something about "it's a block thing" is not resonating with me.

Virtual disks of VMs are block devices. No files and no filesystems. That is like having an partition on a harddisk on a Windows PC and asking yourself why you can't find that partition as a file in your explorer window.

I can technically copy them to another machine and the .vmdk is the hard drive and the .vmx is some configs, and I can add them to another ESXi (of the sameish versions, etc)... but on (my) zfs Proxmox I have zfs -list which has the Name/Mount Point of my zfs storage, but has no files (at least from ls's perspective)?

Because on ESXi those are files on a filesystem. ZFS uses zvols and those are more like partitions. On Windows, if you want to work with partitions you will also have to use fdisk to edit your partition table and not using dir/copy/move.

(I have added other folders for backup testing to that same zfs volume so I have Name/Mount Point my-zfs/backup-test and when I take a backup, those files are ls-able and seem like they might copy to another box (or survive and OS drive crash) without trouble.

Make sure you actually write them to your ZFS pool and not to an empty mountpoint where they then would end up on the root filesystem filling up your system disk. Please have a look at the ZFS basics and research for terms like "pool", "vdev", "zvol", "dataset". ZFS knows two things. Zvols and datasets. Datasets are filesystems and you can store files on them. Zvols are block devices you can partition and format. VMs are using zvols, LXCs will use datasets.

 

[ronejamesdo]

Is there a gparted or other type of internal or "sanctioned" tool for zvols? (Like if I wanted to copy an entire partition, could I do that from outside of the OS itself?)

Thanks.

 

[Kingneutron]

https://www.truenas.com/community/threads/zvol-cloning.61350/

FYI if you do a ' zfs list |grep vm ' you should see your zfs-based VM disks at the host level. Don't need to be root for that.

If you want to copy a VM disk, possibly your best bet is to do it in-VM the way you would with a physical system, using clonezilla or the like.

I have a utility script that should allow you to interact with a VM's zvol the way you normally would with a host-level disk, but ONLY do this with the VM powered off, AND only after you have made a backup of the VM.

I am not responsible for data loss. You will need to edit the script before running it.

https://github.com/kneutron/ansitest/blob/master/ZFS/mount-zvol-partitioned-as-disk.sh


Post back more on what you are trying to accomplish and why

 

[LnxBil]

Like if I wanted to copy an entire partition, could I do that from outside of the OS itself?

It's a file - like anything in any unix-based operating system - so you can just dd it. Why would you want to do that besides having a PVE backup?

FYI if you do a ' zfs list |grep vm ' you should see your zfs-based VM disks at the host level. Don't need to be root for that.

Normally, you need to be root for that unless you changed something.

https://github.com/kneutron/ansitest/blob/master/ZFS/mount-zvol-partitioned-as-disk.sh

Maybe look into kpartx, it's doing the heavy lifting and is just one command to setup the loopback devices.

 

[Kingneutron]

> Normally, you need to be root for that unless you changed something

Give it a try, works on Linux and OSX without having to be root for me. It's just a list/display command, not modifying anything.

 

[ronejamesdo]

It's a file - like anything in any unix-based operating system - so you can just dd it. Why would you want to do that besides having a PVE backup?


Normally, you need to be root for that unless you changed something.


Maybe look into kpartx, it's doing the heavy lifting and is just one command to setup the loopback devices.

 

[ronejamesdo]

I ended up at an answer with a combination of suggestions, being:

zfs get volsize BigDrive/vm-102-disk-0

zfs create -V 16G BigDrive/ddtestrjd
dd status=progress if=/dev/zvol/BigDrive/vm-102-disk-0 of=/dev/zvol/BigDrive/ddtestrjd

I see the volume in zfs (no mountpoint, like the vms). Refreshing the Proxmox site in my browser doesn't make it appear anywhere though. Any idea how I would make Proxmox see it?

Thanks.

(I realize this is not something I would normally do or hope to ever need to do, I just want to know for my own clarity on the nuts and bolts.). RJD

 

[bbgeek17]

I see the volume in zfs (no mountpoint, like the vms). Refreshing the Proxmox site in my browser doesn't make it appear anywhere though. Any idea how I would make Proxmox see it?

Proxmox expects to see certain naming for the disks it uses, ie vm-[id]-[index]. There is a way to force it to use custom names, but not in the general listings.

As you already realize, manual backend manipulation is not specific to Proxmox. Since you are using ZFS, you could have skipped a create and dd, and used ZFS native methods:
https://www.quernus.co.uk/2015/08/30/cloning-zvol-for-vm/

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[ronejamesdo]

Thank you. I did it this way instead:

zfs create -V 16G BigDrive/vm-106-disk-0
dd status=progress if=/dev/zvol/BigDrive/vm-102-disk-0 of=/dev/zvol/BigDrive/vm-106-disk-0

And now have:

BigDrive/vm-102-disk-0 17.3G 19.9T 11.0G -
BigDrive/vm-106-disk-0 17.3G 19.9T 11.0G -

But my Proxmox does not seem to see 106 anywhere. Is there another step?

 

[Dunuin]

Try qm rescan 106 and then attach the unattached disk that should be shown in the hardware tab of the VM with the VMID 106.

 

[ronejamesdo]

# qm rescan 106
400 too many arguments
qm disk rescan [OPTIONS]

# qm rescan
rescan volumes...

Neither of these seems to have changed anything in the GUI.

 

[ronejamesdo]

Wait, I take that back, 106 is in BigDrive's volumes now... now to mount it as VM somehow...

 

[bbgeek17]

Its qm disk rescan --vmid 106
As @Dunuin said, navigate to VM's hardware panel, locate the "unusedX" entry , and attach it to a controller.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[ronejamesdo]

What I mean to say is that there is no 106 VM. I dd copied vm-102-disk-0 into a vm-106-disk-0 that I'd created and I now see vm-102-disk-0 in BigDrive's "VM Disks", but it is not affiliated with any VM, I was hoping just to "add 106" point it at this disk, and have it all just work (again, only as test).

qm disk rescan --vmid 106

yields:

Configuration file 'nodes/moat/qemu-server/106.conf' does not exist

Would copying nodes/moat/qemu-server/102.conf to nodes/moat/qemu-server/106.conf be a terrible idea? (Or rather /etc/pve/qemu-server/?)

 

[bbgeek17]

What I mean to say is that there is no 106 VM. I dd copied vm-102-disk-0 into a vm-106-disk-0 that I'd created and I now see vm-102-disk-0 in BigDrive's "VM Disks", but it is not affiliated with any VM, I was hoping just to "add 106" point it at this disk, and have it all just work (again, only as test).

Frankly, it very unclear what you are doing and why. You have a VM102 disk, but its not affiliated with any VM? So you dont have VM102?
Nor do you have VM106?

Why dont you provide the following output as text encoded in CODE tags:
qm list
pvesm status
pvesm list BigDrive
qm config [VMID] (for each VM in question).

And please state your goal in concise clear form.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[ronejamesdo]

Sorry, what I am trying to do is see what would happen if for some reason I just had these disks and no Proxmox (or some copy of them).

Coming from ESXi, there was always a worst-case of /vmfs/volumes/the-disk/the-vm, which contained the actual file. In a worst case you could always take that folder, and add its contents to a same-version ESXi and it would do the right thing.

Since that's the way I'm used to thinking, that's the way I'm trying to emulate so that I can later understand why other ways are better with Proxmox (this is really just a learning exercise).

I do have a VM 102. I do not have a VM 106. I dd copied the zfs of vm-102... to vm-106... to see what a manual addition of something like this would be like.

So far I can now see that vm-106-disk-0 exists in the Proxmox box VM Disks menu. I am trying to get at how I would get it to become a VM.

 

[bbgeek17]

ESXi and it would do the right thing

It doesn't just "do the right thing". One would have to ensure the datastore is added, the VM is imported, the network is adjusted.

So far I can now see that vm-106-disk-0 exists in the Proxmox box VM Disks menu. I am trying to get at how I would get it to become a VM.

You dont make disk into VM. You would add a VM, then add a disk. If you backed up your VM config, you could restore that.

qm create 106
qm disk rescan --vmid 106
qm set 106 --scsi0 [storage]:[disk]

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[ronejamesdo]

Thank you that sorted me out.

If I'd have added the VM from the GUI (it was defaulting to 106) would it have noticed the vm-106-disk-0 that I'd copied and done good things or would it have wiped it (if you happen to know)?

For anyone who has any reason to attempt this in the future (including me), this is one working method:

zfs get volsize BigDrive/vm-102-disk-0

zfs create -V 16G BigDrive/vm-106-disk-0
dd status=progress if=/dev/zvol/BigDrive/vm-102-disk-0 of=/dev/zvol/BigDrive/vm-106-disk-0
qm create 106
qm disk rescan --vmid 106
qm set 106 --scsi0 BigDrive:vm-106-disk-0

Where you BigDrive is your volume and you wanted to copy your VM number 102 to the next number in line (which thus far did not exist) being 106 in the example.

Thanks everybody.

RJD

 

[bbgeek17]

If I'd have added the VM from the GUI (it was defaulting to 106) would it have noticed the vm-106-disk-0 that I'd copied and done good things or would it have wiped it (if you happen to know)?

I don't, never had to do something like this. I have reasonable expectations that nothing will be wiped out. It's easy to try - give it a shot and report it here for posterity.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox