spencerh's latest activity
-
S@spencerh Yeah, the netcat-dd is the last working branch. I just uploaded a few fixes to it based on the new codebase I'm working on. ./target/release/esxi-folder-fuse --test-fuse --use-fuse-streaming --esxi-host 10.20.30.40 --esxi-disk...
-
Sspencerh replied to the thread Import from ESXi Extremely Slow.Server running on ESX, client running on Proxmox: [root@my-esx:~] /usr/lib/vmware/vsan/bin/iperf3.copy -s -B 192.168.0.123 -p 8000 ----------------------------------------------------------- Server listening on 8000 (test #1)...
-
Sspencerh replied to the thread Import from ESXi Extremely Slow.I tend to agree; if you're doing these transfers over an open network speed probably isn't your primary concern anyway. It would also theoretically be pretty straightforward to inject an encryption/decryption step into the pipeline before/after...
-
SWhile I agree security is a good concern, in this case, is it? At least in our environment (and I know not everyone does it this way), we have all of our servers on the same L2 network dedicated to VM storage traffic, so none of it ever leaves...
-
Sspencerh replied to the thread Import from ESXi Extremely Slow.This looks awesome! I'll give this a test tomorrow and post the results.
-
SOkay... Progress. I've got the file transfer and conversion to work as expected as long as you want the output format in RAW. For our purposes, that's okay. We are using iSCSI, so no big deal. However, I'm having trouble getting the qcow2...
-
Sspencerh replied to the thread Import from ESXi Extremely Slow.There are also some additional niceties afforded by the import tool. It's definitely possible to migrate using shared storage of one kind or another but it involves more steps, both on the ESX side in the form of moving the VMs around to get them...
-
SSo, interesting finding. Trying to get the import process to process a stream of data vs chunks, so tried using qemu-img dd in doing so it goes from absolutely transferring as fast as possible to about 75-78MB/s. So there seems to be processing...
-
Sspencerh replied to the thread Import from ESXi Extremely Slow.That's much of the same issue we're having. 20 min for a 50GiB VM isn't the end of the world, but for some of our larger VMs there's no way we can sustain the downtime to move 2TiB at that rate. I know I've said this already, but I am deeply...
-
Sspencerh replied to the thread Import from ESXi Extremely Slow.I'm going to investigate how difficult this is, as since we've repurposed some of our ESX hosts into our Proxmox test environment we're already wired up to a few of the datastores off the SAN. I was just really trying to avoid straying from the...
-
Sspencerh replied to the thread Import from ESXi Extremely Slow.It's gigabit. I just did the math myself, dd just outputs total chunks and total time. The disk in question was 50 GiB and the dd process took 7 minutes and 53 seconds which gives an effective speed of around 0.85 Gb/s if my/Wolfram Alpha's math...
-
Sspencerh replied to the thread Import from ESXi Extremely Slow.Yeah, the 192.168.0.150 interface is the PVE side of the iPerf3 test I posted earlier. So interestingly it seems like I'm only able to pull ~0.9 Gb/s running dd and sending the results to null. [root@my-esx:~] time dd...
-
Sspencerh replied to the thread Import from ESXi Extremely Slow.OK just finished the nc test. It doubled throughput but it seems to still be a long shot from the theoretical network throughput we'd expect: root@pve:~# nc -l -p 8000 -s 192.168.0.150 | pv | dd bs=16M of=/rpool/data/testfile.vmdk 50.0GiB...
-
Sspencerh replied to the thread Import from ESXi Extremely Slow.Yeah I checked that on our end. Seems like it's disabled presently.
-
-
Sspencerh replied to the thread Import from ESXi Extremely Slow.I was also doing a bit of research this morning, and it looks like the throughput limitations of SSH from ESX and it seems that this is potentially an issue with resource limits applied to the busybox environment on the ESX side. Nothing...
-
Sspencerh replied to the thread Import from ESXi Extremely Slow.OK so I did a little bit of playing around with the 1.1.2 release this afternoon. Here's some questions and findings in no particular order: Does it fallback to HTTP whenever a password is provided? I don't seem to be able to figure out a way to...
-
-
Sspencerh replied to the thread Import from ESXi Extremely Slow.So weirdly I re-copied my test VM and it seemed to get exactly the same throughput. It's got 2 disks; a 1GiB and a 50Gib disk. With the original version, the VM transfers in ~20 minutes, and with the modified importer I see the exact same...
-
Sspencerh replied to the thread Import from ESXi Extremely Slow.Thanks, this is awesome! I'll give it a shot this afternoon and report back with my findings.
-
Sspencerh replied to the thread Import from ESXi Extremely Slow.I'm going to see if I can figure out how to get it to compile (I am also not a Rust developer) with that value changed to 8 and see if it has any impact on throughput. This is a great find, thanks for doing the legwork on this.
-
Sspencerh replied to the thread Import from ESXi Extremely Slow.Sorry forgot to mention this in my original post. No, I have no snapshots on the machine. We do have some other options, but I'll admit that at this point my curiosity has gotten the better of me and, options aside, I'd like to understand what's...
