scp file copy of large VMWare vmdk files to proxmox

[lmcandrew]

Hello,
Angry ESXi home user here, working to migrate as quickly as possible to Proxmox. I built my proxmox server and accepted all disk defaults during setup.
I have a 2TB disk and I can see that my / partition is 94 gb. and I literally can't see any other mounted file systems with sizable space. I understand the rest of the disk is taken up as lvm-disks

I am trying to copy my large (100GB) vmdk files onto the published file system and don't have anywhere to copy these files.

Can someone guide me how to find / build a large partition where I can scp my vmdk files for staging before running qm import disk?

I was successfully able to migrate a small vmdk by copying it into the /tmp directory and then cleaning it off after the qm import. No such luck with my large vms.

Anyone have suggestions for moving large vmdk files into this file system for transmogrification to proxmox?

	
	

		
root@proxmox241:/dev# df -kh
Filesystem            Size  Used Avail Use% Mounted on
udev                   16G     0   16G   0% /dev
tmpfs                 3.1G  2.0M  3.1G   1% /run
/dev/mapper/pve-root   94G   38G   52G  42% /
tmpfs                  16G   46M   16G   1% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
efivarfs              192K   99K   89K  53% /sys/firmware/efi/efivars
/dev/nvme0n1p2       1022M   12M 1011M   2% /boot/efi
/dev/fuse             128M   20K  128M   1% /etc/pve
tmpfs                 3.1G     0  3.1G   0% /run/user/0
	

 

[bbgeek17]

Anyone have suggestions for moving large vmdk files into this file system for transmogrification to proxmox?

One, easy, way would be to manually create a slice from lvm pool of appropriate size, ie 1TB. Format and mount it. SCP data to it, then import as raw block to LVM pool as managed by Proxmox. You can then discard the temporary mount/filesystem/slice.

Starting from step 4: https://www.linuxtechi.com/how-to-create-lvm-partition-in-linux/#Step_4_Create_LV_Logical_Volume

You can examine your LVM setup with : pvs, vgs, lvs

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[lmcandrew]

Absolutely fabulous, thank you for your fast and helpful response.

 

[alexskysilk]

another alternative: use sshfs to mount your remote vmware filestore on your proxmox host, and use qemu-img convert directly on the vmdk.

 

[lmcandrew]

holy cow, I love sshfs. It saves a huge amount of time by just copying and creating the disk directly with qm importdisk instead of copying it to a staging area and importing. Thanks!

 

[scuba1900]

I have a large external VM ( qcow2) I want to import into proxmox. The Proxmox disk that was used to install Proxmox is only 500GB and is not big enough.

All the other storage containers have been configured already and i do not want to destroy these. How can import/move this VM into proxmox. Currently it is on an external USB drive.

1) Copy from my remote machine using SCP , but to which location ? ( Can I SCP to a storage directory ( root /tmp is too small) ?)
2) Plug the usb drive directly to proxmox then ?

Thanks in advance

I guess the question is can I SCP or copy files into storage containers and then connect the VM with the file ?

 

[gfngfn256]

I guess the question is can I SCP or copy files into storage containers and then connect the VM with the file ?

If I understand you correctly this is what you want to do: Your VM, whose disk is to large for the "main install" disk should reside on one of your "storage containers" that do have room for it. I assume that these "storage containers" are correctly setup as Proxmox Storage of an images/ disk image content type.

If this is in fact what you want to do then:

1. Mount the external USB drive to a directory in PVE.
2. Create your VM in PVE.
3. In console/shell of PVE use qm disk import<vmid> <source> <storage> [OPTIONS] [1] passing the correct VMID required & the correct disk location from the above mountpoint and the correct storage location ID you chose. You can also pass the format of the disk to be used; options for this type may depend on your Storage type/s.
4. In PVE GUI, VM Hardware, choose the newly unused disk and Add it to the VM.
5. In PVE GUI, VM Options, check Boot Order/settings for the VM disk.
6. After everything is set - you can umount the USB drive.


[1] qm proxmox manual

 

[scuba1900]

If I understand you correctly this is what you want to do: Your VM, whose disk is to large for the "main install" disk should reside on one of your "storage containers" that do have room for it. I assume that these "storage containers" are correctly setup as Proxmox Storage of an images/ disk image content type.

If this is in fact what you want to do then:

1. Mount the external USB drive to a directory in PVE.
2. Create your VM in PVE.
3. In console/shell of PVE use qm disk import<vmid> <source> <storage> [OPTIONS] [1] passing the correct VMID required & the correct disk location from the above mountpoint and the correct storage location ID you chose. You can also pass the format of the disk to be used; options for this type may depend on your Storage type/s.
4. In PVE GUI, VM Hardware, choose the newly unused disk and Add it to the VM.
5. In PVE GUI, VM Options, check Boot Order/settings for the VM disk.
6. After everything is set - you can umount the USB drive.


[1] qm proxmox manual

Thanks , before reading this and what is currently being done, is that I am SCP from a remote machine into the storage container. The disk is 1 TB so it take a few hours to SCP. If my logic is right , I can then create a "new VM" and point this disk as the disk to use or i must still use the QM disk import command ?

 

[gfngfn256]

Thanks , before reading this and what is currently being done, is that I am SCP from a remote machine into the storage container. The disk is 1 TB so it take a few hours to SCP. If my logic is right , I can then create a "new VM" and point this disk as the disk to use or i must still use the QM disk import command ?

If you use the qm disk import method you'll need twice the storage size during the import, (original 1tb disk image copied by SCP + 1tb new imported disk image), you can still obviously (after import) delete the copied (by SCP) disk image.

Instead, what I would do in your current situation, is first create the new VM with a new disk (any size - doesn't matter), then once the VM is created, locate the VM's new disk location, & then use the mv command to move the copied (by SCP) from its location to the VM's new disk location. Then you'll issue :

qm rescan --vmid VMID

# replacing VMID with the actual VM ID number

Then go through steps 4 & 5 as above.

You can then also delete (in GUI) the new disk you created with the VM's creation.

I must point out, had you gone through the suggested procedure of mounting the external USB, you would have saved doing some of these tedious steps, and it can be assumed the copying procedure would be a whole lot faster.

 

[scuba1900]

Thanks for the patience and guidance. If I had know before I copied the file using SCP I would used your original guidance.
As you have shared in your last response I created a new disk any size, so I did and also the file ( VM) was copied from SCP so as now :

1) Disk that was copied via SCP is in location : /mnt/pve/WDC_4TB/M-PC-EAST.qcow2 ( 800 GB which I want to use )
2) Location of general hard drive for VM is at : /mnt/pve/WDC_4TB/images/108/vm-108-disk-0.qcow2 (30 GB or so )

You said to : & then use the mv command to move the copied (by SCP) from its location to the VM's new disk location.
How ?
I was reading the syntax and saw qm move_disk <vmid> <disk> <storage> but the disk I want to move does not have a vmid and also when the move is done there would be 2 disks in the /mnt/pve/WDC_4TB/images/108/ location, - the general disk created and the one I move with the command.

I would also have to delete the general drive using cli or gui ? or missed something

Thanks again

 

[gfngfn256]

No. You're first going to use the mv command in the following manner:

mv /mnt/pve/WDC_4TB/M-PC-EAST.qcow2 /mnt/pve/WDC_4TB/images/108/

This will move the disk file (800gb) to its correct VM image location. FYI you can have more than one disk for a VM.

After this you will continue, as above to attach this disk to the VM by entering (in PVE host shell):

qm rescan --vmid 108

Just make sure VM is not running.

Then go through steps 4 & 5 as above.

You can then also delete (in GUI) the new disk (30gb) you created with the VM's creation. You can do this by selecting in the left pane the VM, then in the next pane Hardware, then select in the next pane the 30gb drive (make sure you get the right one!), then press Detach & confirm Yes, then go down to the unused disk entry (normally the last entry) which should be the 30gb drive, & press the Remove button & confirm Yes
That should leave you with only your 800gb drive being used by the VM.

Proxmox has a learning curve, but everyone can learn it.

 

[scuba1900]

Thanks