Proxmox VM file transfer slowing down suddenly!
- Thread starter haleeth
- Start date
Not enough information to go on, but I would guess: don't use QLC SSDs.
Last edited:
Also noted that, If I give around 2-3 min between each transfer, the transfer speed is increased for the next 1 or 2 transfers, might be cache?
That behaviour sounds a lot like what happens with QLC based SSDs when they run out of SLC cache.
They go super, super slow until they've managed to internally process the data.
That being said, maybe it's something else after all.
Would you be ok to run
That should give us a bunch of useful info for understanding how your storage is set up.
They go super, super slow until they've managed to internally process the data.
That being said, maybe it's something else after all.
Would you be ok to run
lsblk -o tran,name,type,size,vendor,model,label,rota,phy-sec on your Proxmox host and paste the output here?That should give us a bunch of useful info for understanding how your storage is set up.
That behaviour sounds a lot like what happens with QLC based SSDs when they run out of SLC cache.
They go super, super slow until they've managed to internally process the data.
That being said, maybe it's something else after all.
Would you be ok to runlsblk -o tran,name,type,size,vendor,model,label,rota,phy-secon your Proxmox host and paste the output here?
That should give us a bunch of useful info for understanding how your storage is set up.
TRAN NAME TYPE SIZE VENDOR MODEL LABEL ROTA PHY-SEC loop0 loop 74.2M 0 512 loop1 loop 103.4M 0 512 loop2 loop 38.8M 0 512 sda disk 1.7T QEMU QEMU HARDDISK 0 512 ├─sda1 part 1M 0 512 ├─sda2 part 2G 0 512 └─sda3 part 1.7T 0 512 └─ubuntu--vg-ubuntu--lv lvm 1.7T 0 512ata sr0 rom 1024M QEMU QEMU DVD-ROM 1 512Sorry, I misread itThanks @haleeth. That looks like it was run inside a virtual machine, rather than on the Proxmox host.
Would you be ok to do it on the Proxmox host instead?
If I'm misunderstanding something though, let me know. Wouldn't be the first time.
Bash:
TRAN NAME TYPE SIZE VENDOR MODEL LABEL ROTA PHY-SEC
sata sda disk 1.9T ATA WDS2TB2G0A-00JH30 0 512
├─sda1 part 1007K 0 512
├─sda2 part 1G 0 512
└─sda3 part 1.9T 0 512
├─pve-swap lvm 8G 0 512
├─pve-root lvm 96G 0 512
├─pve-data_tmeta lvm 15.9G 0 512
│ └─pve-data-tpool lvm 1.7T 0 512
│ ├─pve-data lvm 1.7T 0 512
│ ├─pve-vm--101--disk--0 lvm 1.7T 0 512
│ ├─pve-vm--102--disk--0 lvm 4M 0 512
│ ├─pve-vm--102--disk--1 lvm 60G 0 512
│ ├─pve-vm--102--disk--2 lvm 4M 0 512
│ └─pve-vm--100--disk--0 lvm 8G 0 512
└─pve-data_tdata lvm 1.7T 0 512
└─pve-data-tpool lvm 1.7T 0 512
├─pve-data lvm 1.7T 0 512
├─pve-vm--101--disk--0 lvm 1.7T 0 512
├─pve-vm--102--disk--0 lvm 4M 0 512
├─pve-vm--102--disk--1 lvm 60G 0 512
├─pve-vm--102--disk--2 lvm 4M 0 512
└─pve-vm--100--disk--0 lvm 8G 0 512Ahhh. That model of drive "WDS2TB2G0A-00JH30" is the problem. That's one of the "QLC" drives (a type of flash) that people have been mentioning in the previous posts.
Western Digital is super dodgy about putting that info in front of customers where it'd be useful (before they buy).
As an example, this is the official Western Digital information page for that whole range of drives:
https://www.westerndigital.com/products/internal-drives/wd-green-sata-2-5-ssd
Notice there's not a single mention of "QLC" on it anywhere, even though it's a QLC drive?
Unfortunately, that's pretty typical of Western Digital.
They know that if they mentioned QLC on that page, they would have less sales. So they don't mention it.
Hmmm, while those are a bit better than QLC drives, they're not fantastic either.
Western Digital is super dodgy about putting that info in front of customers where it'd be useful (before they buy).
As an example, this is the official Western Digital information page for that whole range of drives:
https://www.westerndigital.com/products/internal-drives/wd-green-sata-2-5-ssd
Notice there's not a single mention of "QLC" on it anywhere, even though it's a QLC drive?
Unfortunately, that's pretty typical of Western Digital.
They know that if they mentioned QLC on that page, they would have less sales. So they don't mention it.
Hmmm, while those are a bit better than QLC drives, they're not fantastic either.
Last edited:
@haleeth With the computer you're using, does it have a free PCIe slot?
I'm wondering if adding in a SAS card (example here) might be an option for you?
If you could do that, plus get a hold of some SAS drives (example, example, example) then your disk transfers would go a lot faster without all of the pausing.
I'm wondering if adding in a SAS card (example here) might be an option for you?
If you could do that, plus get a hold of some SAS drives (example, example, example) then your disk transfers would go a lot faster without all of the pausing.
You are not suggesting to replace a consumer SSD with more consumer SSDs and using ZFS on it which is NOT recommended?!
Ahhh. That model of drive "WDS2TB2G0A-00JH30" is the problem. That's one of the "QLC" drives (a type of flash) that people have been mentioning in the previous posts.
Western Digital is super dodgy about putting that info in front of customers where it'd be useful (before they buy).
As an example, this is the official Western Digital information page for that whole range of drives:
https://www.westerndigital.com/products/internal-drives/wd-green-sata-2-5-ssd
Notice there's not a single mention of "QLC" on it anywhere, even though it's a QLC drive?
Unfortunately, that's pretty typical of Western Digital.
They know that if they mentioned QLC on that page, they would have less sales. So they don't mention it.
Hmmm, while those are a bit better than QLC drives, they're not fantastic either.
Thanks, but how does one check if it is a QLC or not when buying an SSD, and can I overcome this issue by switching to a traditional HDD?
@haleeth With the computer you're using, does it have a free PCIe slot?
I'm wondering if adding in a SAS card (example here) might be an option for you?
If you could do that, plus get a hold of some SAS drives (example, example, example) then your disk transfers would go a lot faster without all of the pausing.
Unfortunately, it's not possible since I am using a tiny PC as my home server! but I am planning to upgrade very soon!
Traditional hard drives are good for being a cheap way to store larger amounts of data, but their maximum speed on their very best day is still pretty slow.
That being said, they don't have the same kind of pausing issues that QLC drives do. Instead, they're just slow for everything.
For some stuff (maybe watching a movie) that's completely fine though. For other things (like copying a movie between disks) they're also not great.
Research before buying.
Before buying a new SSD, find out if it's a consumer grade of SSD (like your WD Green one, and like the Crucial MX 500) or an enterprise grade of SSD. And if it's a consumer grade of SSD, find out for sure what kind of flash it is using first. If you can't find that info, ask people that will know. You're welcome to ask here on this forum too, we're into that kind of thing.
Bear in mind that enterprise grade SSDs generally don't use SATA for their physical interface, although some exist which do (example). The Samsung PM893's have a good reputation btw, and would work in your existing tiny PC.
The downside is that enterprise SSDs tend to be more expensive. Sometimes a lot more.
Since you're planning on upgrading though, then maybe look at getting a PC that has room for adding a PCIe card + some SAS drives? (examples in previous comments
)
Last edited:
DC600M from Kingston are 'pretty cheap' and nice to use with ZFS too. We have a few of them but bought just around 6 months ago. So I can't say anything about the endurance yet.
Awesome. I was looking for another example to point towards. That'll do.
They seem like they'd be a decent option, as their endurance is reasonable (1 DWPD for 5 years, or 1.66 DWPD for 3 years):
Thanks, My connection speed is 1 Gbps, but noted that one of my friends is using traditional HDDs for a home server, and still getting speeds around 100+MB/s, is that what you are referring to as slow or even slower because I am not quite sure about it, but the speed around 100MB/s is fair enough for meTraditional hard drives are good for being a cheap way to store larger amounts of data, but their maximum speed on their very best day is still pretty slow.
That being said, they don't have the same kind of pausing issues that QLC drives do. Instead, they're just slow for everything.
For some stuff (maybe watching a movie) that's completely fine though. For other things (like copying a movie between disks) they're also not great.
Research before buying.
Before buying a new SSD, find out if it's a consumer grade of SSD (like your WD Green one, and like the Crucial MX 500) or an enterprise grade of SSD. And if it's a consumer grade of SSD, find out for sure what kind of flash it is using first. If you can't find that info, ask people that will know. You're welcome to ask here on this forum too, we're into that kind of thing.
Bear in mind that enterprise grade SSDs generally don't use SATA for their physical interface, although some exist which do (example). The Samsung PM893's have a good reputation btw, and would work in your existing tiny PC.
Noted
The downside is that enterprise SSDs tend to be more expensive. Sometimes a lot more.
Since you're planning on upgrading though, then maybe look at getting a PC that has room for adding a PCIe card + some SAS drives? (examples in previous comments
Yes, that's the idea
It's kind of complicated.
HDD's can get decent speeds (100+MB/s) if they're moving sequential data around (ie large files).
But they really suck if you have to move lots of very small files around. Hard drives are terrible at random access (ie small file handling) when compared to any ssd.
As an example, I recently had to copy about 400,000 small files (between 1k and 4k each), along with 400,000 directories (1 per file) from a 4 disk RAIDZ system with traditional hard drives to an ssd on the same computer.
It took over 12 hours.
I then copied the same files and directories from that ssd to a different ssd in the same computer, and it finished in under 20 minutes.
If you're mostly dealing with sequential access (ie large files), then you could be ok. If you're using lots of small files though... they're not a good first choice.
It's kind of complicated.
HDD's can get decent speeds (100+MB/s) if they're moving sequential data around (ie large files).
But they really suck if you have to move lots of very small files around. Hard drives are terrible at random access (ie small file handling) when compared to any ssd.
As an example, I recently had to copy about 400,000 small files (between 1k and 4k each), along with 400,000 directories (1 per file) from a 4 disk RAIDZ system with traditional hard drives to an ssd on the same computer.
It took over 12 hours.
I then copied the same files and directories from that ssd to a different ssd in the same computer, and it finished in under 20 minutes.
If you're mostly dealing with sequential access (ie large files), then you could be ok. If you're using lots of small files though... they're not a good first choice.
Thanks for the assistance. it was very helpful going forward