Migration from Esxi to Proxmox issue

[mf1979]

Hello,

I have vmware esxi 7 on my physical server. I'm planning to change it Proxmox. But before that i need to get my VM from vmware.

I use qemu-mg convert without any issue. But my important VM has some important snapshots in Vmware system.

with snapshots there are more than on vmdk on vm folder and this is one of my vm host folder content in wmware;


-rw------- 1 root root 30.2G Mar 29 17:06 hana-r1-000001-sesparse.vmdk
-rw------- 1 root root 311 Mar 29 10:52 hana-r1-000001.vmdk
-rw------- 1 root root 175.4G Mar 30 12:40 hana-r1-000002-sesparse.vmdk
-rw------- 1 root root 318 Mar 29 19:18 hana-r1-000002.vmdk
-rw------- 1 root root 48.3G Apr 25 08:45 hana-r1-000004-sesparse.vmdk
-rw------- 1 root root 318 Apr 25 06:54 hana-r1-000004.vmdk
-rw------- 1 root root 275.6K Mar 29 10:52 hana-r1-Snapshot1.vmsn
-rw------- 1 root root 275.6K Mar 29 17:07 hana-r1-Snapshot2.vmsn
-rw------- 1 root root 275.6K Mar 30 12:49 hana-r1-Snapshot3.vmsn
-rw------- 1 root root 270.0G Mar 29 10:50 hana-r1-flat.vmdk
-rw------- 1 root root 264.5K Apr 25 08:45 hana-r1.nvram
-rw------- 1 root root 529 Mar 29 01:08 hana-r1.vmdk
-rw-r--r-- 1 root root 1.1K Mar 30 12:49 hana-r1.vmsd
-rwx------ 1 root root 3.0K Apr 25 08:45 hana-r1.vmx
-rw------- 1 root root 294.4K Apr 8 11:42 vmware-10.log
-rw------- 1 root root 208.1K Apr 25 05:52 vmware-11.log
-rw------- 1 root root 144.8K Apr 25 06:31 vmware-12.log
-rw------- 1 root root 189.7K Apr 25 08:45 vmware-13.log
-rw------- 1 root root 289.1K Apr 2 02:51 vmware-8.log
-rw------- 1 root root 327.0K Apr 4 11:58 vmware-9.log
-rw------- 1 root root 29.1K Apr 27 04:10 vmware.log
-rw------- 1 root root 108.0M Apr 25 06:09 vmx-hana-r1-0c597178abe3e25a955f6aaf701b501814c7d350-1.vswp
-rw------- 1 root root 108.0M Apr 4 11:59 vmx-hana-r1-974039424-1.vswp

There are 3 snapshots in this VM. I tried ;

qemu-img convert *.vmdk -O qcow2 /backup1/hana-r1.qcow2 -c

But result file hana-r1.qcow2 is not not working even not bootable in Proxmox VM.

1. How can i migrate this VM machine from vmware to proxmox with it's snaphosts?
2.What is best size reduced qemu-img command for this operation?
3. what this option in qemu-img;

-m 16 -W --> Can i use these option for this migration?

3. What is fastest qemu-img option for this migrate?

Thank you

 

[Dominic]

Hi!

Following the migration guide in the Proxmox VE wiki should at least give you something that boots.

To get the .vmdk files for qemu-img convert, did you

1. use the export function of ESXi or
2. copy all files from your ESXi storage to a storage that is available for your Proxmox VE host directly?

Trying the second version (using ESXi 6.7, so that might lead to some differences) qemu-img reports

mala# qemu-img convert -f vmdk pve-flat.vmdk -O qcow2 /mnt/pve/nfs/images/102/vm-102-disk-1.qcow2
qemu-img: Could not open 'pve-flat.vmdk': invalid VMDK image descriptor

What happens if you add -f vmdk to your commands, too?

Unfortunately, I am not an expert with ESXi and have not yet found a way to convert .vmdk files to .qcow2 that retains the information about the snapshots. What should work is restoring the different relevant snapshots in ESXi and exporting each of them. Then you could import those into Proxmox VE using the guide mentioned before. This would give you different virtual machines, however.

 

[mf1979]

Hello,

I tried export last night and convert ovf file with qm but not luck; got info from https://zachgrace.com/cheat_sheets/proxmox/ and https://www.itsfullofstars.de/2019/07/import-ova-as-proxmox-vm/

qm importovf 101 tempEsxciVM.ovf QnapNFS1 -format qcow2

I tried with widndows or Linux system but system no good result. system halts at booting stage " Botton from Hard Disk " but it stopped.

Then i tried convert vmdk from ovf file;

qm importdisk 100 testwindows.vmdk QnapNFS1 -format qcow2

This is worked but no internet connection because of virtio ethernet card. i changed it to e100 and internet is fine on disk VM now.

But i used same method for linux systems and it does not work. It stopped at Booting from Hard disk ... place.

1. How can we convert esxi VM with snaphosts to proxmox? Because there are more than 1 vmdk in snaphost VM.

2. What is m n and W parameters doing with qemu convert? Where should we need to use W parameter? I could not find really good document for these parameters.

3. How can we make -c (compress) option more more faster? I use it and it takes very long time and it does not use my whole cpu. How can we push -c option to use all over cpu power?

Thank you

 

[Dominic]

Hello!

This is worked but no internet connection because of virtio ethernet card. i changed it to e100 and internet is fine on disk VM now.

Using virtio network is one of the improvements you can get by appyling the Windows 10 guest best practices (noted at the end of the VMware section in the migration guide).

Unfortunately, I couldn't reproduce your problems with importovf or importdisk with Linux guests. Are you 100% sure that the export from ESXi has been successful for all disk images?

1. How can we convert esxi VM with snaphosts to proxmox? Because there are more than 1 vmdk in snaphost VM.

.ovf files contain information about which virtual disks (= .vmdk files) belong to an exported machine. Consequently, importovf should import all of them. I have not yet found a way to export guests from ESXi including their snapshots. Did you use the export function or just copy the files from your ESXi storage?

You can search for "Parameters to convert subcommand" in the qemu-img man page to get detailed information about the parameters to the convert command. It should, however, not be necessary to use this command.

 

[fortechitsolutions]

Footnote/comment in case it is of any use. RE: Snapshots in VMWare. A feature that is a source of much joy (ie, not-joy). I find they can be widely misunderstood in best use case / worst use case. Generally speaking, I would suggest that snapshots are things you want to have only on a temporary / transitory basis. They have impact on IO performance, and have repurcussions on long-term disk space use for underlying storage. Addition of multiple 'layers' of snapshots makes this more dramatic. Rolling back / removal of snapshots can cause 'IO spike' with performance impact.

If you create a snapshot, and leave it intact indefinitely, then disk space use for underlying data store will grow ~indefinitely as well (ie, beyond any 'limit' related to the VM Disk image size. Since you are perpetually adding writes / changes to a copy-on-write buffer. I've seen cases where a former-admin deployed a snapshot, left the company, and the snap was left active for >1 year, and the underlying VMware datastore runs out of disk space. Because nobody knows it is there, and nobody cares, until the datastore fills up and life is more exciting (ie, VM locks up until it has space to write.

So. My normal advice. (For what it is worth).

Use snapshots like a 'temporary rollback option' only. For example, you plan to do major patching tonight on server and are anxious it may break everything. So take a snapshot. Run the updates. Let them complete. Test the server. Validate things are good. Within 24 hours you make your decision, the updates were a success? or were they a failure.

If a success. Delete the snapshot, you no longer need to have ability to roll back to that old point in time.
If a fail. Roll back to that point in time using your snapshot, discard changes from the update / since the update. And release the snapshot, do not keep it.

Either way, you end up using the snapshot 'active' for ~24 hours or less.

Trying to keep your snapshots around as a 'long term roll back strategy' is dreadful disaster for performance, because it is complicated, has long term IO performance impact, and runs the risk of filling your underlying storage.

So if you want to have point-in-time rollback for your VM. Make a copy of the VM prior to executing the work. This copy is a 'cold bare standalone' copy and has no 'connection' (copy on write or otherwise) to the original. Move it to backup storage media so the disk space is a non issue. Keep as many of these as you like over time. Make one every day/week/month/whatever.

Just don't do this with 'snapshots' because life is gross otherwise, eventually.

So.

If you are following this model, you really should not be trying to migrate a VM and its snapshots. Because you really should not have a VM with which you prefer to preserve the snapshots. These are a symptom of problematic admin behaviour.

Instead, move away from using snapshots. If needed you have some admin work required to do so - ie - make a full copy of your whole mess on vmware, flip to a designated snapshot point in time, and make a backup of the VM as it stands in this state. Rinse and repeat as required. Keep these 'backup things' separate from 'the active VM'.

Migrating to proxmox, you can migrate things however you like. But don't think that migration of VM with its snapshots can/should/will work. Because it should not, and does not, by design. (IMHO).

Moving stuff from VMware to Proxmox is generally smooth and good process in my experience. And you should be much happier at the end of it all, having a ~more feature rich platform with less insane per-feature-incremental-pricing

Hope this is somewhat helpul, maybe.

Tim

 

[mf1979]

Footnote/comment in case it is of any use. RE: Snapshots in VMWare. A feature that is a source of much joy (ie, not-joy). I find they can be widely misunderstood in best use case / worst use case. Generally speaking, I would suggest that snapshots are things you want to have only on a temporary / transitory basis. They have impact on IO performance, and have repurcussions on long-term disk space use for underlying storage. Addition of multiple 'layers' of snapshots makes this more dramatic. Rolling back / removal of snapshots can cause 'IO spike' with performance impact.

If you create a snapshot, and leave it intact indefinitely, then disk space use for underlying data store will grow ~indefinitely as well (ie, beyond any 'limit' related to the VM Disk image size. Since you are perpetually adding writes / changes to a copy-on-write buffer. I've seen cases where a former-admin deployed a snapshot, left the company, and the snap was left active for >1 year, and the underlying VMware datastore runs out of disk space. Because nobody knows it is there, and nobody cares, until the datastore fills up and life is more exciting (ie, VM locks up until it has space to write.

So. My normal advice. (For what it is worth).

Use snapshots like a 'temporary rollback option' only. For example, you plan to do major patching tonight on server and are anxious it may break everything. So take a snapshot. Run the updates. Let them complete. Test the server. Validate things are good. Within 24 hours you make your decision, the updates were a success? or were they a failure.

If a success. Delete the snapshot, you no longer need to have ability to roll back to that old point in time.
If a fail. Roll back to that point in time using your snapshot, discard changes from the update / since the update. And release the snapshot, do not keep it.

Either way, you end up using the snapshot 'active' for ~24 hours or less.

Trying to keep your snapshots around as a 'long term roll back strategy' is dreadful disaster for performance, because it is complicated, has long term IO performance impact, and runs the risk of filling your underlying storage.

So if you want to have point-in-time rollback for your VM. Make a copy of the VM prior to executing the work. This copy is a 'cold bare standalone' copy and has no 'connection' (copy on write or otherwise) to the original. Move it to backup storage media so the disk space is a non issue. Keep as many of these as you like over time. Make one every day/week/month/whatever.

Just don't do this with 'snapshots' because life is gross otherwise, eventually.

So.

If you are following this model, you really should not be trying to migrate a VM and its snapshots. Because you really should not have a VM with which you prefer to preserve the snapshots. These are a symptom of problematic admin behaviour.

Instead, move away from using snapshots. If needed you have some admin work required to do so - ie - make a full copy of your whole mess on vmware, flip to a designated snapshot point in time, and make a backup of the VM as it stands in this state. Rinse and repeat as required. Keep these 'backup things' separate from 'the active VM'.

Migrating to proxmox, you can migrate things however you like. But don't think that migration of VM with its snapshots can/should/will work. Because it should not, and does not, by design. (IMHO).

Moving stuff from VMware to Proxmox is generally smooth and good process in my experience. And you should be much happier at the end of it all, having a ~more feature rich platform with less insane per-feature-incremental-pricing

Hope this is somewhat helpul, maybe.

Tim

That's totally true and it's very nice to read your description. How do you backup your VM in vmware.

What is best way to backup VM in proxmox in your real-time operations?

I already imported VM to proxmox without snaphots. But machines are very slow. I increased cpu and ram but did not change performance.

qm set 104 --bios ovmf
sed -i 's/scsi/ide/g' /etc/pve/qemu-server/104.conf

I used this two option because VM could not start without these options. It's Suse and redhat linux systems and got same issue.

Could you please tell me details VM migrate operation from VMware to proxmox. Please do not give me links. I read tons of documents. And it's not easy to find right way.

Please give me commands and details. I really appreciate .

 

[fortechitsolutions]

Hi,

In VMWare I normally use something like Veeam to do backups, either the free version is 'pretty good' or really if people are making decision to run vmware, they are probably not cost-averse so non-free veeam is better obviously. There are other tools similar to veeam which may be less expensive, but I have less experience with those so can't comment.

In proxmox. I normally use built-in VM Backup scheduler to run (nightly or weekly full VM Backup) - with snapshot and guest agent then it goes very smoothly normally. For "BIG" VMs clearly these can take a while; so nightly is less good in those cases.
My typical deploy to be honest is to do 2-level backup model
(a) VM level backups are less often (ie, weekly or monthly) using proxmox full VM Backup scheduler
(b) Inside-VM-Backups are done more often, ie, nightly or 2-4x daily depending on client needs and 'fault patterns' (ie, how often humans make mistakes?) - For windows VMs this might be partially as simple as using internal VSS Snaps on regular basis as 'first layer'. Normally I use something like BackupPC or a nice rsync script or similar - to do 'file level backup of changed data since last backup' - this allows more frequent backup of 'important data directories' but not all the drama of a full VM backup. Whatever tool works for you - is the main goal - there are many (!) ways to do this. The goal is just to do something which works, and which you understand. And can do recovery of data from easily if-when required.

I'm not certain why your VMs might be performing badly to be honest.

My experience with VMware migrate was pretty simple, but to be honest that is the vast minority of my use-case. Normally I am doing clean deploy VMs from scratch on proxmox, either containers (OVZ in past, now LXC) or KVM or both. Depending on client site and 'goodness of fit'.

Always with VMWare migrate, it had been important to make sure you remove traces of VMware - ie - VMWare tools guest agents etc. inside the VM.

Obviously you need to do VirtIO NIC and SCSI for optimal performance, otherwise your VMs will inherently have ~meh performance compared to what is possible on same hardware/box otherwise.

Make sure there are no red flags in dmesg / warnings / logs inside your guests - ie - unrecognized drivers, processes that are pounding/wasting CPU / any weird smoking guns that don't make sense. Check logs on proxmox as well for other hints.

Worst case you can deploy a clean reference system just to validate your core platform is as good as you think it is.

ie, do a clean Suse Linux Proxmox KVM VM install / just to establish "this is how good a clean install works on my platform". And see how much it differs from your VMware-migrated SUSE vm instance. Since it is a ~relatively trivial task to flip alive a new linux VM install from scratch on proxmox if all you want to do is spin it up and kick the wheels a bit.

These things should get you moving in the right direction. Hopefully.

Hope this helps a bit.


Tim

 

[mf1979]

Hello Tim,

Thank you very much giving all these advice

1. I tried to back VM on proxmox (Lzo or gzip) but it take too much long time. Snapshots are very nice and short and yes you're right they are taking too much space. Is there any way take snapshots and just backup of this snapshots? Because i have DB in VM and i cannot take online backup. It has to be offline and snapshots only solution for it. So i cannot stop server for a long time to take lzo or gzip backup.

There is a snapshot mode in Backup but i did not understand that mode. So it take first snapshot and then start to backup that snapshot version.

So this is my real example;

a. turn of DB in OS level. But VM is on.
b. Run backup with snaphosts mode and at the same time i cano turn on DB ? I mean i'm ok backup is not new i don't really care. Important thing is getting correct and stable backup of system.

I need to have FULL VM backup status. Just db backup is not what i am looking. Because OS installation configuration and software installation/configuration takes too much time. So i need to have FULL VM backup method.

Please show me little step bby step commands and examples if you don't mind.

2. After migration VM from esxi, it's not working even qm importdisk or importovf method. I have run these 2 commands;

qm set 104 --bios ovmf
sed -i 's/scsi/ide/g' /etc/pve/qemu-server/104.conf

and as you see they convert disk to ide ... after i migrate vm it shows scsi but it does not boot. It's starting but stopping at Booting section and does not move to forward. Whenever i convert settings to with two commands it works but slowly. How do you make convert operation with commands? and what is settings?

3. What do you think Ceph and Zfs? I use my local 2 ssd as directory and i have one qnap as a nfs storage. i want to test ceph but it looks little complicated. It's just for test. I'm happy with this configuration just want to try new storage option.

Thank you again.

 

[fortechitsolutions]

Hi,

Just a few points, in response to your questions. (briefly).

the 'backup mode' which uses 'snapshot' on proxmox, is very different beast from manually going in and 'taking a snapshot' of your VM.
The backup based upon snap - is a good approach to taking a copy of a VM without having to take it offline for duration of backup job. If you have the Qemu guest agent properly installed, it should be able to push a "quiescence" request to the guest OS and this should 'pause' DB or at least encourate it to flush active buffers so that you don't get a 'crash-consistent db backup' but rather a 'good' backup. But really it is critical you test to be sure how good the backup works.

ie, you should not have to 'stop' your guest VM to take a backup. So it is OK if the backup job takes ~hours - because you don't have hours of outage, maybe you have a few seconds for the snapshot tasks to do their thing early / and late in the backup job.

really you should test this, key steps are:
(a) install guest agent, it is not present by default, read the wiki and get this in place on your VM
(b) do a 'test backup with snapshot mode' in proxmox. It is ok if it takes hours. really.
(c) do a 'test restore' of your backup, validate how well it works, if your DB is ok, etc.
remember backups are only good if you fully validate the whole process (Backup, restore, ok!)

Otherwise. If you want 'more often backups' within your VM. Which I still recommend is a good idea.
(a) for example, if using mysql, have a scheduled job to dump mysql DB data to disk at regular scheduled time (ie, 1am) and the harvest this 'raw cold dump file' using offline backup method (ie, back-agent-based or similar). Or learn how to use your SQL transaction logs / backup recovery process. Or something. But do not rely 100% on 'vm level backups' to save your skin in case of a major server crash. Assuming the DB/server is important, you need to do better. / have more options at your disposal.

if using MS_SQL there are built-in tools for non-free editions of SQL server. that can do DB Dump to cold-export. Or you can use free version of tools like "SQL Backup and FTP" which are actually pretty decent if you want to harvest one MS_SQL DB and dump a cold backup to disk. This is just one example toolt to illustrate the point.

end of the day. You need to have a well documented, well understood, reliably working backup process. VM-level backups done by proxmox can be definitely part of the solution, but they really should not be the only aspect to the solution.

There is no formal solution to do 'diff/delta' based proxmox level VM backups. The assumption is that you use inexpensive storage tank (ie, USB drive, or NFS Filer with inexpensive disk) and thus you really don't care about 'wasting space' in backups, because - the space is ~free compared to the value of the backups.

If you are really on shoestring budget, and need a way to diff/delta the proxmox integrated backups. There is a 3rd-party patch available which is 100% outside of official proxmox scope / support / being blessed as a valid tool etc. Use at own risk, etc etc. I do not recommend it for production use / unless you have tested it fully and are comfortable with it, and accept that it could in fact break everything and destroy your backups I won't even give the link to the tool. But as a hint, if you google for "proxmox differential backup" you may start to find a few bread crumbs that lead you towads this add-on. note this add-on adds more 'drama' to your proxmox because any time you want to upgrade proxmox (yes, any time!) - then you have to (uninstall the patch) / (do updates) / reinstall the newest version of the patch which is available for that new version of proxmox. ie, it is not entirely without some 'integration pain'. So your miles may vary about value vs pain.

Otherwise. On your other topics.

regarding your VM migation from ESX to proxmox.
This process is documented in the wiki.
If your guest is Windows based, it might be you failed to do the 'mergeide.reg' trick. prior to exporting out of ESX. Have a look at the wiki docs if this does not ring a bell.

if the copy-convert-migrate still fails to work. I have seen in some reluctant cases, that you can migrate in such cases, if you wish, by using Clonezilla. But it is slightly more work. But not all that much. And maybe not a bad thing to learn how to use a good free tool like clonezilla as part of the deal.

Otherwise. In terms of Ceph and ZFS.

I'm highly motivated to keep my proxmox installs 'simple and stable' as top priority. Just because of the way I use proxmox (ie, I deploy it at client sites, I am not full time at any one site). Maybe if it was me full time managing a stable of proxmox, more traditional "I am the IT guy at <Company> I would be less risk averse. But. Given that I hate dealing with unnecessary drama, I like boring stable smooth operations. I would far rather use "less sexy, exciting, innovative solutions" any day - if it meant 'less drama, fewer midnight trips to the server room, less panic phone calls from client because server blew up again'.

So things like Ceph and ZFS fall more into the category of "stuff that just is not worth using for my deployments". And stuff like "Local storage, NFS storage" are - nice and reliable. Very easily understood. Not a lot of complexity in terms of tuning, workload / performance drama. Exposure to risk in terms of points of failure. Exposure to risk in terms of 'complex, harder to upgrade and maintain'.

I am less keen on ZFS, despite it being a 'good' product/tool conceptually. Because of (A) some bad-old-day experience with ZFS when it was still only an early product on solaris. and after losing data spontaneously, you tend to feel a bit burned. this long (!!) pre-dates zfs existing on linux or bsd. I realize it has changed a lot since then. But generally - (B) it still is 'more complicated than basic raid solution for local storage for proxmox IMHO'. And while it might be 'cool' to do stuff like snapshots, dedup, compression, nifty cache volumes, etc etc. It quickly becomes 'more complex'. Do I have enough ram? CPU? good sized cache? fast enough cache? multiple cache for read vs write? why is my ZFS performance slow for certain workloads? Golly gosh so many fun things to debug! Life is not nearly short enough, I want more complexity! yes! that is fun!
(no it is not. at least not for me.)

So. Ceph is kind of more of the same, in terms of complexity, in my humble opinion. Potentially, I think ceph is a great fit for a 'big serious proxmox deployment' if you have a specific set of use case, for example
-- 10 nodes proxmox cluster
-- large number of VMs required to be in service
-- nice to have hypercoverged storage / compute virtualization cluster stack
-- obviously, you want fast (10gig ) interconnect, and multiple interconnect for your cluster (ie, proxmox-management vs "VM network bridge" vs storage) - not just everything running on one interface off each box. ceph is possible on 1gig but I am pretty sure it is not worth the effort in terms of (performance expected and complexity required).
-- significant storage needs, ie, probably on the order of ~no less than 20+ Tb to be sensible, once you get this many nodes and you need a few disks per node for ceph / and a few ceph nodes doing specific roles in terms of operational status and redundancy. Ceph wants to have a decent sized pool of 'spindles' or at least parallel storage targets (ie, many SSD drives/SATA drives is much better than fewer). Arguably you could deploy with lots of tiny drives but I am not sure that makes a whole lot of sense

maybe if I was again a full time IT guy working for one BigCorp and we had such a project, maybe I would be massively keen on Ceph, and I would deploy and manage a few proxmox-ceph clusters. And that might be fun and exciting.

(MESSAGE CONTINUES BELOW - too wordy!)

 

[fortechitsolutions]

(AND NOW THE SAGA CONCLUDES)
...
(tongue in cheek)

But in contrast, I tend to have - more / multiple smaller client sites, who have fairly modest needs, and the storage and VM requirements are not a great fit at any single site. So it just does not make sense. I spec'ced out a few years ago a proxmox-ceph build for one client, and it ?might? be possible to do a 'tiny' ceph deploy with 3-4 nodes, but even then I think it is a lot more complexity and generally speaking I'm just happier to do something simple such as

-- 2-3 proxmox nodes in small cluster, if you need that much VM capacity
-- shared "SAN" storage if you really want to go that way. (Intel modular server was quite nifty for this). Or a traditional fibre SAN with dual-port access to single controller let you do a 'pretty easy 2-node proxmox cluster with a single shared SAN storage array'. But of course you have a single point of failure here. depending on your preference towards HA hardware etc.
-- and really what is just simpler, is to use an NFS Filer as your shared storage tank. Because over the years of using proxmox, I have found NFS is really 'far better than it seems that it ought to be' as a storage target. Even on boring gigabit ethernet. And now that 10gig is commodity, this is more easy to do on 'cheap builds' even. Even using a not terribly exciting NFS filer device like a Synology or QNAP NAS - works absurdly well, it turns out. Not just as your 'backup tank' storage but as primary VM storage.

obviously IOPS on gig ether NFS are not the same as IOPS on a local hardware raid with SSD backed disk pool. But. It is pretty good. and fine for workloads where - IOPS requirements are 'modest'.

really - some sites are so small, all the need is one box. Local storage. NFS or USB tank for backup storage. That is it. really. And it works. Very. Very. nicely.

and my general experience with 'reliability and failure' - is that - I am virtually always happy to deploy proxmox with local storage. and not fuss about 'oh golly , what if my server goes offline, gasp, what about my perfect HA architecture and 100% uptime assurance"? The reality is that server hardware does not actually fail that often, unless it was deployed by an idiot who forgot to setup raid, or buy a server with redundant power supplies. because generally speaking, once raid and power redundancy are dealt with - your biggest source of risk for server crashes has more to do with human error (ooops! I kicked over the UPS, ha ha, everything lost power, outage, oops!)
or maybe "oops - spilled the coffee!! ha ha could not see that coming"
or maybe "oops, power company had an outage in a storm, our UPS ran out of battery, ah well better luck next time!".
or maybe .. you get the idea.

ie, clearly it is up to you / and your business context to make informed decisions about risk and reliabiltiy. and how to meet your targets / goals.

but I find that generally, making stuff (more complex) so as to achieve (higher HA fault tolerance)
often results in
(more complexity and confusion and higher rate of human error)
which ultimately defeats the purposes of trying to make the platform more stable / better uptime - in the first place.

ie, be really sure you know what you are getting into, if you decide to use 'fancier features and configs' such as ZFS, Ceph, HA Cluster, etc etc.

ie, playing with Ceph and ZFS (or other nifty new fun exciting features) - on test boxes for learning, is never bad thing. Deciding when to use it for production is another thing.

(ie, keep your production gear and your test boxes - separate. as in 100% separate. test box can be just an old PC, or a few old PCs, etc etc. But don't get tempted to 'play test' on production, unless you are ok with idea of spending a night of unpaid overtime doing a business critical data restore "fun-evening" just because you went 'oops' and didn't quite understand sometihng until it was a tiny bit too late. ie, making mistakes is part of learning, but best do so on a place where you can say, "ah wasn't that fun, clean install and start over!". Instead of "oh sh*t sh*t sh*t sh*t !!!"



Tim

 

[mf1979]

Hi Tim,

Thank you for all of your explanation and it was fun to read all of them

1. I installed same OS and DB on proxmox form scratch from zero and it's more more faster than vm convert one. so probably sme disk or vm configuration is missing. I tried with Suse and Redhat 8 and got same result. Newly install on Proxmox is more more faster than exported from esxi one. So if they were Windows , then 'mergeide.reg' trick. but these are linux systems.

2. What i found is for backup, i shutdown server and started Backup to NFS share and it was good actually. If i take backup local storage (SSD)

SSD LZO
INFO: transferred 214748 MB in 181 seconds (1186 MB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 3.71GB

But when i backup it to NFS share;

Qnap LZO vzdump-qemu-110-2020_05_02-01_34_55.vma.lzo
INFO: transferred 214748 MB in 4379 seconds (49 MB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 3.77GB

I don't know why NFS share speed cannot get over 50 MB/s. It has two 1 gig ethernet. When i try to copy file it's more more faster but this backup method have special trick or some other issue maybe.

 

[mf1979]

3. I will try to backup VM when it's online and i will restore it to another proxmox box and i will let you know if DB is alive I don't want to backup just DB. It has own free backup/restore but as i told you before, in my case, i have to get full VM backup because there are;

a. OS installation / configuration steps
b.DB installation / configuration steps
c. Software installation / configuration steps

And these are taking too much time and i don't want to do it all.

4.I copied backup (lzo and gzip both) to another proxmox and restored there. It's perfect and restore without any issues( except disable iso from cd So what i will, i will shutdown VM and take backup. If something happened then i will restore this backup

5. Ceph --> I don't have production system. So i thought i could play around this but as you said it requires too many disk / switch and other requirements.

Even for proxmox i don't use it's lvm and lvm thin storage. I installed Proxmox on 16gb USB plugged my server and i use My 2 TB and 1 TB local SSD as ext4 type storage and QNAP NFS as you said for simple and stable way

LVM is not bad but too many step to configure and then exten blah blah. I have more thna enought space in my QNAP so i don't care about lvm.

I need to find why backup speed is just 50 mb on nfs share. maybe some proxmox limitation but why it took mmore more faster on ssd storage.

6. I'm looking for think disk configuration in proxmox. Is it only availabe for lvm thin? Why qcow2 does not have it?

I want to give 200gb disk but in os level i want to see it grows as much as i us it. Maybe i will never use 200 gb but why it keeps 200 gb file in my disk.

What is you solution for thin disk file ?

Thank you Tim

 

[mf1979]

And why we cannot run multiple backup operation in same time. It can run only one backup operation in same time.

Others are waiting with this info;

INFO: trying to get global lock - waiting...

 

[fortechitsolutions]

Hi,

glad my reply was some entertainment value

so, a few high level thoughts.
-- gig ether, it is not unusual to see less-than-perfect performance. 100-120Mb/sec is kind of 'best case scenario theoretical maximum' and getting around 50Mb/s is about what I might expect normally for a VM Backup over gig ether. Note that even with multiple gig interfaces, generally you cannot get a single IO stream to be faster than single gig pipe. ie, LACP trunking lets you manage 2 or more gig connections as 'one logical connection' but you never get more than 1gig for a 'single transfer session' - rather you can run more in parallel and use the available pool of 1gig pipes efficiently. If you have a multi-threaded transfer protocol, you might be able to get >1gig throughput, but generally most protocols do not operate that way. So your NFS Backup is going to be capped at 1gig throughput, and that is about all you can hope to get.

-- it might end up that if you wish to do the fastest possible solution, you will backup first to local SSD; and then copy over to NFS remote as second step. I believe there are some nice 'backup hook' options if you wish that can allow you to automate this. Or can just do via script / cron etc.

I'm guessing (not sure) that proxmox has built-in rate-limiter to constrain number of parallel backups that run concurrently; possibly has some logic built-in based on CPU or storage; or maybe just defaults to 'one' - I am not sure - maybe other(s) can comment / or you can dig through forum a bit for this topic.

If you are not using LVM with proxmox storage, I think you will have lost ability for 'snapshot' backup of local filesystem for backups. because proxmox is leveraging LVM to provide that. So that is one of the benefits of using the vanilla 'appliance install configs' with LVM present - because this function 'just works'.

It is good if you have established that - your VMs can run faster if you do clean installs; but as you say, frustrating if the migrated ones are just not as fast. Possibly there is still some crud present inside the VM / traces from Vmware / some other optimal elements. Possibly the cleanest path is just clean install / migrate / be done with it.


Tim

 

[mf1979]

HI Tim,

Thank you for help.

1. Is there any maul lzo or gzip backup command we can run at OS level and then we can run more than one backup in same time

2. how do you use copy/past function in SPICE or No VNC screen?

 

[fortechitsolutions]

Hi, for backups, take a look here,
https://pve.proxmox.com/wiki/Backup_and_Restore

there is a lot of detail how to trigger backup events via CLI. Various command-line options you can designate. Controls on limits, preferred compression, target storage, etc.


For copy-paste in the console. I am not sure to be honest. Generally I only use console when native mode access is not accessible, ie

Linux VMs - I access via SSH typically directly
Windows VMs - I access via RDP typically

ie for me, console is mostly reserved for "VM did not boot, need console access to debug".


T

 

[mf1979]

Thank you Tim.

I got really good knowledge from you.