[SOLVED] Proxmox installation choices for storage

[acti67]

Hello,

I am planning a migration of our infrastructure to Proxmox asap. I am having some difficulty choosing how to store VMs and application's data. So I am writing this post to have some advices from your part. What I would like is the possibility of having a cluster, as online/redondant as possible. (Applications: file shared storage for applications, databases replication, front web, memory cache, ...).

These are my 4 servers configuration:
* Intel Xeon 8c/ 16t / RAM 128 GB / 2x4TB SATA + 2x1TB NVME (localisation: datacenter A)
* Intel Xeon 8c/ 16t / RAM 128 GB / 2x4TB SATA + 2x1TB NVME (localisation: datacenter A)
* Intel Xeon 4c/ 8t / RAM 32 GB / 2x2TB SATA (localisation: datacenter B)
* Intel Xeon 4c/ 8t / RAM 32 GB / 2x2TB SATA (localisation: datacenter C)

Every servers have a public and a private network (different gigabit interfaces). There is no hardware raid for storage.

The question today is: what would be the best choice (EXT4/Linstor-DRBD, ZFS, ...) for a cluster with these servers?

Thank you for your help

 

[Dominic]

Hi!

Creating a cluster across multiple datacenters is not recommended, as the underlying corosync network needs a really low latency network. You could take a look at storage replication with ZFS.

We ship our own Ceph packages. Those are also included in the subscription agreement. Generally, Ceph is tightly integrated into Proxmox VE and thus a good shared storage solution for many use cases. Shared storage across multiple datacenters might, however, be problematic.

For local storage ZFS gives a lot of enterprise-level features to protect your data, among them a sort of software RAID. You can find a list in our reference documentation.

 

[acti67]

Hi,

I have some questions to continue the topic. I would like to explore each solutions pros and cons according to my case.

Is ZFS adapted for our smallest servers (32GB of RAM) with VMs?

For Ceph option, I think our number of disks inside our different servers are limited for a strong infrastructure. Are these servers suitable for Ceph clustering (without taking in account the network latency)?

What do you thing and what are your feedbacks about DRBD / Linstor (here: datacenter A sync mode and async for others)? Maybe do you have other idea for my architecture?

Is there a good way to have a HA cluster inside our datacenter A and backup (replication or others) inside B and D?

Thank you for your experience sharing

Best regards

 

[Dominic]

Hi,

as rule of thumb, one could say 8GB for ZFS generally + 1GB per TB of disk storage. So 32GB is probably not a bad start. Still, it depends a lot on how much RAM your VMs will need exactly.

About Ceph, ideally you would have all NVMEs of equal size and a separate small disk for Proxmox VE. Also, Ceph memory requirements might be even higher (Ceph monitors... require memory) . Please give the relevant section in the documentation a quick look.

What do you thing and what are your feedbacks about DRBD / Linstor

DRDB is not integrated into Proxmox VE from our side. As far as I know they have some sort of plugin for Proxmox VE? Anyway, I have never used it myself and unfortunately cannot give you a competent answer to this question.

Is there a good way to have a HA cluster inside our datacenter A and backup (replication or others) inside B and D?

Something like that would be my recommendation. Having a third (ideally equal) node in datacenter A would allow you to create a serious cluster with working quorum and high availability. You can try the HA simulator, to get an idea of this. If possible, you connect those 3 nodes with multiple 10 Gbps or higher networks, for redundant storage and cluster networks. A full mesh network might be an idea for 3 nodes. Networks with 10 Gbps upwards usually also give you the necessary low latency.

You could then install Proxmox Backup Server (PBS)

• on your Proxmox VE nodes and
• in your datacenters B and C (or you drop one of those in favor of the new datacenter A machine)

This way, you can create backups locally fast. This results in little downtime for your VMs. Then you sync the backups to your datacenters B and C. Doesn't matter that much, if this takes some time. Syncing is a built-in feature of PBS. PBS officially is still beta, but I would say "mature" beta.

Are those servers rented? Can you maybe get cheap evaluation servers?

 

[acti67]

as rule of thumb, one could say 8GB for ZFS generally + 1GB per TB of disk storage. So 32GB is probably not a bad start. Still, it depends a lot on how much RAM your VMs will need exactly.

Ok, that's what I read about ZFS requirements for RAM.

About Ceph, ideally you would have all NVMEs of equal size and a separate small disk for Proxmox VE. Also, Ceph memory requirements might be even higher (Ceph monitors... require memory) . Please give the relevant section in the documentation a quick look.

Ceph seems to be not the best choice with my servers configurations, in term of storages.

DRDB is not integrated into Proxmox VE from our side. As far as I know they have some sort of plugin for Proxmox VE? Anyway, I have never used it myself and unfortunately cannot give you a competent answer to this question.

I tried Linstor / DRBD between the two nodes in DC A and with the DRBD-Proxy for long distance in other DC. But I am not satisfy by the result's performance despite the availability of HA in this configuration. It would be great for the redondancy of our VM with data files storage for web applications.

You could then install Proxmox Backup Server (PBS)

• on your Proxmox VE nodes and
• in your datacenters B and C (or you drop one of those in favor of the new datacenter A machine)

This way, you can create backups locally fast. This results in little downtime for your VMs. Then you sync the backups to your datacenters B and C. Doesn't matter that much, if this takes some time. Syncing is a built-in feature of PBS. PBS officially is still beta, but I would say "mature" beta.

Are those servers rented? Can you maybe get cheap evaluation servers?

At this time, I cannot contractually change my servers. I can get a VPS with a proxmox to make a quorum in DC A, but is there a good idea?

So in summary :
* Install all my servers with ZFS RAID1 on SATA disks
* Add a cache with NVME disks for ZFS despite of the among of RAM? Use NVME disks for VM storage that need speed as a seconday data pool?
* Make a cluster only in DC A and standalone in other DCs
But at this time, I don't have a redondant and HA storage for my web applications data...

 

[Dominic]

* Add a cache with NVME disks for ZFS despite of the among of RAM? Use NVME disks for VM storage that need speed as a seconday data pool?

There is no general rule for that. Depends again a lot on what resources your VMs require in reality.

I can get a VPS with a proxmox to make a quorum in DC A, but is there a good idea?

Assuming that you get reasonable latency between the VPS and your other two nodes, this could solve this

But at this time, I don't have a redondant and HA storage for my web applications data...

problem. Using so called QDevices is not ideal, but might be a temporary solution if you want to get a really cheap VPS. If you make the VPS a "real" third node, you can also set rules for HA so that your powerful node is preferred.

 

[acti67]

Thank you for all these relevant information.

I am currently trying ZFS in a clustered environment (DC A). It seems to be a good option for my use case.

But I see in the list of processes that some "pveproxy worker (shutdown)" appears after some working hours:

www-data 18884  0.0  0.0 365540 127408 ?       S    00:00   0:05 pveproxy worker (shutdown)
www-data 18885  0.0  0.1 373436 135224 ?       S    00:00   0:05 pveproxy worker (shutdown)
www-data 18886  0.0  0.0 361164 123208 ?       S    00:00   0:05 pveproxy worker (shutdown)
www-data  2976  0.0  0.0 365040 127348 ?       S    00:54   0:04 pveproxy worker (shutdown)
www-data 23638  0.0  0.0 360828 124280 ?       S    01:29   0:04 pveproxy worker (shutdown)
www-data  4022  0.0  0.0 361452 123904 ?       S    19:20   0:00 pveproxy worker (shutdown)

This is an example from syslog:

Oct 15 19:20:26 venus pveproxy[4022]: got inotify poll request in wrong process - disabling inotify
Oct 15 01:29:56 venus pveproxy[23638]: got inotify poll request in wrong process - disabling inotify

Where are they come from? And why? They stay like zombies.

All the best

 

[Dominic]

Can you please post

systemctl status pveproxy
journalctl -u pveproxy

 

[acti67]

Yes of course. I have restarted the servers last evening, but nothing changed (errors still persist).

$ systemctl status pveproxy

root@venus:~# systemctl status pveproxy
● pveproxy.service - PVE API Proxy Server
   Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-10-15 21:29:26 CEST; 12h ago
  Process: 2207 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
  Process: 2209 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
  Process: 18959 ExecReload=/usr/bin/pveproxy restart (code=exited, status=0/SUCCESS)
Main PID: 2213 (pveproxy)
    Tasks: 7 (limit: 4915)
   Memory: 354.6M
   CGroup: /system.slice/pveproxy.service
           ├─ 2213 pveproxy
           ├─ 3781 pveproxy worker
           ├─12413 pveproxy worker (shutdown)
           ├─17286 pveproxy worker (shutdown)
           ├─20749 pveproxy worker
           ├─28193 pveproxy worker
           └─28532 pveproxy worker (shutdown)

Oct 16 09:26:54 venus pveproxy[2213]: worker 28193 started
Oct 16 09:26:55 venus pveproxy[28192]: worker exit
Oct 16 09:43:26 venus pveproxy[3459]: worker exit
Oct 16 09:43:26 venus pveproxy[2213]: worker 3459 finished
Oct 16 09:43:26 venus pveproxy[2213]: starting 1 worker(s)
Oct 16 09:43:26 venus pveproxy[2213]: worker 20749 started
Oct 16 09:53:57 venus pveproxy[27374]: worker exit
Oct 16 09:53:57 venus pveproxy[2213]: worker 27374 finished
Oct 16 09:53:57 venus pveproxy[2213]: starting 1 worker(s)
Oct 16 09:53:57 venus pveproxy[2213]: worker 3781 started

$ journalctl -u pveproxy

-- Logs begin at Thu 2020-10-15 21:29:15 CEST, end at Fri 2020-10-16 10:10:14 CEST. --
Oct 15 21:29:25 venus systemd[1]: Starting PVE API Proxy Server...
Oct 15 21:29:26 venus pveproxy[2209]: Using '/etc/pve/local/pveproxy-ssl.pem' as certificate for the web interface.
Oct 15 21:29:26 venus pveproxy[2213]: starting server
Oct 15 21:29:26 venus pveproxy[2213]: starting 3 worker(s)
Oct 15 21:29:26 venus pveproxy[2213]: worker 2214 started
Oct 15 21:29:26 venus pveproxy[2213]: worker 2215 started
Oct 15 21:29:26 venus pveproxy[2213]: worker 2216 started
Oct 15 21:29:26 venus systemd[1]: Started PVE API Proxy Server.
Oct 15 22:00:00 venus pveproxy[2213]: worker 2216 finished
Oct 15 22:00:00 venus pveproxy[2213]: starting 1 worker(s)
Oct 15 22:00:00 venus pveproxy[2213]: worker 12414 started
Oct 15 22:00:01 venus pveproxy[12413]: got inotify poll request in wrong process - disabling inotify
Oct 15 22:06:59 venus pveproxy[2213]: worker 2215 finished
Oct 15 22:06:59 venus pveproxy[2213]: starting 1 worker(s)
Oct 15 22:06:59 venus pveproxy[2213]: worker 17287 started
Oct 15 22:07:01 venus pveproxy[17286]: got inotify poll request in wrong process - disabling inotify
Oct 15 22:10:13 venus pveproxy[2214]: Clearing outdated entries from certificate cache
Oct 15 22:10:13 venus pveproxy[12414]: Clearing outdated entries from certificate cache
Oct 15 22:10:31 venus pveproxy[17287]: Clearing outdated entries from certificate cache
Oct 15 22:19:43 venus pveproxy[2213]: worker 2214 finished
Oct 15 22:19:43 venus pveproxy[2213]: starting 1 worker(s)
Oct 15 22:19:43 venus pveproxy[2213]: worker 28533 started
Oct 15 22:19:46 venus pveproxy[28532]: got inotify poll request in wrong process - disabling inotify
Oct 15 22:21:05 venus pveproxy[2213]: worker 12414 finished
Oct 15 22:21:05 venus pveproxy[2213]: starting 1 worker(s)
Oct 15 22:21:05 venus pveproxy[2213]: worker 24787 started
Oct 15 22:21:06 venus pveproxy[28533]: Clearing outdated entries from certificate cache
Oct 15 22:21:10 venus pveproxy[24786]: got inotify poll request in wrong process - disabling inotify
Oct 15 22:21:44 venus pveproxy[24787]: Clearing outdated entries from certificate cache
Oct 15 22:21:56 venus pveproxy[24786]: worker exit
Oct 15 22:32:00 venus pveproxy[2213]: worker 17287 finished
Oct 15 22:32:00 venus pveproxy[2213]: starting 1 worker(s)
Oct 15 22:32:00 venus pveproxy[2213]: worker 31157 started
Oct 15 22:32:04 venus pveproxy[31150]: got inotify poll request in wrong process - disabling inotify
Oct 15 22:32:15 venus pveproxy[28533]: proxy detected vanished client connection
Oct 15 22:32:16 venus pveproxy[31150]: worker exit
Oct 15 22:32:17 venus pveproxy[31157]: Clearing outdated entries from certificate cache
Oct 15 22:51:45 venus pveproxy[28533]: Clearing outdated entries from certificate cache
Oct 15 22:52:22 venus pveproxy[24787]: Clearing outdated entries from certificate cache
Oct 15 22:57:18 venus pveproxy[24787]: worker exit
Oct 15 22:57:18 venus pveproxy[2213]: worker 24787 finished
Oct 15 22:57:18 venus pveproxy[2213]: starting 1 worker(s)
Oct 15 22:57:18 venus pveproxy[2213]: worker 2593 started
Oct 15 23:01:04 venus pveproxy[2593]: Clearing outdated entries from certificate cache
Oct 15 23:02:23 venus pveproxy[31157]: Clearing outdated entries from certificate cache
Oct 15 23:05:24 venus pveproxy[28533]: worker exit
Oct 15 23:05:24 venus pveproxy[2213]: worker 28533 finished
Oct 15 23:05:24 venus pveproxy[2213]: starting 1 worker(s)
Oct 15 23:05:24 venus pveproxy[2213]: worker 13872 started
Oct 15 23:06:56 venus pveproxy[13872]: Clearing outdated entries from certificate cache
Oct 15 23:32:51 venus pveproxy[31157]: Clearing outdated entries from certificate cache
Oct 15 23:33:55 venus pveproxy[2593]: Clearing outdated entries from certificate cache
Oct 15 23:37:03 venus pveproxy[13872]: Clearing outdated entries from certificate cache
Oct 15 23:44:09 venus pveproxy[13872]: worker exit
Oct 15 23:44:09 venus pveproxy[31157]: worker exit
Oct 15 23:44:09 venus pveproxy[2213]: worker 13872 finished
Oct 15 23:44:09 venus pveproxy[2213]: worker 31157 finished
Oct 15 23:44:09 venus pveproxy[2213]: starting 2 worker(s)
Oct 15 23:44:09 venus pveproxy[2213]: worker 17973 started
Oct 15 23:44:09 venus pveproxy[2213]: worker 17974 started
Oct 15 23:44:45 venus pveproxy[17974]: Clearing outdated entries from certificate cache
Oct 15 23:45:05 venus pveproxy[17973]: Clearing outdated entries from certificate cache
Oct 15 23:50:35 venus pveproxy[2593]: worker exit
Oct 15 23:50:35 venus pveproxy[2213]: worker 2593 finished
Oct 15 23:50:35 venus pveproxy[2213]: starting 1 worker(s)
Oct 15 23:50:35 venus pveproxy[2213]: worker 26958 started
Oct 15 23:52:11 venus pveproxy[26958]: Clearing outdated entries from certificate cache
Oct 16 00:00:00 venus systemd[1]: Reloading PVE API Proxy Server.
Oct 16 00:00:01 venus pveproxy[18959]: send HUP to 2213
Oct 16 00:00:01 venus pveproxy[2213]: received signal HUP
Oct 16 00:00:01 venus pveproxy[2213]: server closing
Oct 16 00:00:01 venus pveproxy[2213]: server shutdown (restart)
Oct 16 00:00:01 venus systemd[1]: Reloaded PVE API Proxy Server.
Oct 16 00:00:01 venus pveproxy[2213]: Using '/etc/pve/local/pveproxy-ssl.pem' as certificate for the web interface.
Oct 16 00:00:01 venus pveproxy[2213]: restarting server
Oct 16 00:00:01 venus pveproxy[2213]: starting 3 worker(s)
Oct 16 00:00:01 venus pveproxy[2213]: worker 19070 started
Oct 16 00:00:01 venus pveproxy[2213]: worker 19071 started
Oct 16 00:00:01 venus pveproxy[2213]: worker 19072 started
Oct 16 00:00:06 venus pveproxy[26958]: worker exit
Oct 16 00:00:06 venus pveproxy[17974]: worker exit
Oct 16 00:00:06 venus pveproxy[2213]: worker 26958 finished
Oct 16 00:00:06 venus pveproxy[2213]: worker 17974 finished
Oct 16 00:00:06 venus pveproxy[2213]: worker 17973 finished
Oct 16 00:00:07 venus pveproxy[19828]: worker exit
Oct 16 00:39:46 venus pveproxy[19072]: Clearing outdated entries from certificate cache
Oct 16 00:39:46 venus pveproxy[19071]: Clearing outdated entries from certificate cache
Oct 16 00:57:20 venus pveproxy[19071]: worker exit
Oct 16 00:57:20 venus pveproxy[2213]: worker 19071 finished
Oct 16 00:57:20 venus pveproxy[2213]: starting 1 worker(s)
Oct 16 00:57:20 venus pveproxy[2213]: worker 10991 started
Oct 16 00:58:42 venus pveproxy[19070]: worker exit
Oct 16 00:58:42 venus pveproxy[2213]: worker 19070 finished
Oct 16 00:58:42 venus pveproxy[2213]: starting 1 worker(s)
Oct 16 00:58:42 venus pveproxy[2213]: worker 12810 started
Oct 16 00:59:05 venus pveproxy[19072]: worker exit
Oct 16 00:59:05 venus pveproxy[2213]: worker 19072 finished
Oct 16 00:59:05 venus pveproxy[2213]: starting 1 worker(s)
Oct 16 00:59:05 venus pveproxy[2213]: worker 13500 started
Oct 16 02:12:37 venus pveproxy[10991]: worker exit
Oct 16 02:12:37 venus pveproxy[2213]: worker 10991 finished
Oct 16 02:12:37 venus pveproxy[2213]: starting 1 worker(s)
Oct 16 02:12:37 venus pveproxy[2213]: worker 26426 started
Oct 16 02:21:01 venus pveproxy[13500]: worker exit
Oct 16 02:21:01 venus pveproxy[2213]: worker 13500 finished
Oct 16 02:21:01 venus pveproxy[2213]: starting 1 worker(s)
Oct 16 02:21:01 venus pveproxy[2213]: worker 6661 started
Oct 16 02:30:33 venus pveproxy[12810]: worker exit
Oct 16 02:30:33 venus pveproxy[2213]: worker 12810 finished
Oct 16 02:30:33 venus pveproxy[2213]: starting 1 worker(s)
Oct 16 02:30:33 venus pveproxy[2213]: worker 21180 started
Oct 16 03:21:57 venus pveproxy[6661]: worker exit
Oct 16 03:21:57 venus pveproxy[2213]: worker 6661 finished
Oct 16 03:21:57 venus pveproxy[2213]: starting 1 worker(s)
Oct 16 03:21:57 venus pveproxy[2213]: worker 1130 started
Oct 16 03:51:52 venus pveproxy[26426]: worker exit
Oct 16 03:51:52 venus pveproxy[2213]: worker 26426 finished
Oct 16 03:51:52 venus pveproxy[2213]: starting 1 worker(s)
Oct 16 03:51:52 venus pveproxy[2213]: worker 13555 started
Oct 16 03:58:09 venus pveproxy[21180]: worker exit
Oct 16 03:58:09 venus pveproxy[2213]: worker 21180 finished
Oct 16 03:58:09 venus pveproxy[2213]: starting 1 worker(s)
Oct 16 03:58:09 venus pveproxy[2213]: worker 23261 started
Oct 16 04:42:40 venus pveproxy[1130]: worker exit
Oct 16 04:42:40 venus pveproxy[2213]: worker 1130 finished
Oct 16 04:42:40 venus pveproxy[2213]: starting 1 worker(s)
Oct 16 04:42:40 venus pveproxy[2213]: worker 24600 started
Oct 16 05:12:51 venus pveproxy[23261]: worker exit
Oct 16 05:12:51 venus pveproxy[2213]: worker 23261 finished
Oct 16 05:12:51 venus pveproxy[2213]: starting 1 worker(s)
Oct 16 05:12:51 venus pveproxy[2213]: worker 5052 started
Oct 16 05:19:03 venus pveproxy[13555]: worker exit
Oct 16 05:19:03 venus pveproxy[2213]: worker 13555 finished
Oct 16 05:19:03 venus pveproxy[2213]: starting 1 worker(s)
Oct 16 05:19:03 venus pveproxy[2213]: worker 14282 started
Oct 16 06:11:45 venus pveproxy[14282]: worker exit
Oct 16 06:11:45 venus pveproxy[2213]: worker 14282 finished
Oct 16 06:11:45 venus pveproxy[2213]: starting 1 worker(s)
Oct 16 06:11:45 venus pveproxy[2213]: worker 28989 started
Oct 16 06:31:58 venus pveproxy[24600]: worker exit
Oct 16 06:31:58 venus pveproxy[2213]: worker 24600 finished
Oct 16 06:31:58 venus pveproxy[2213]: starting 1 worker(s)
Oct 16 06:31:58 venus pveproxy[2213]: worker 26708 started
Oct 16 06:59:15 venus pveproxy[5052]: worker exit
Oct 16 06:59:15 venus pveproxy[2213]: worker 5052 finished
Oct 16 06:59:15 venus pveproxy[2213]: starting 1 worker(s)
Oct 16 06:59:15 venus pveproxy[2213]: worker 3277 started
Oct 16 07:22:58 venus pveproxy[28989]: worker exit
Oct 16 07:22:58 venus pveproxy[2213]: worker 28989 finished
Oct 16 07:22:58 venus pveproxy[2213]: starting 1 worker(s)
Oct 16 07:22:58 venus pveproxy[2213]: worker 5833 started
[...]

$ ps faux

[...]
root      2149  0.2  0.0 300392 86560 ?        Ss   Oct15   1:49 pve-firewall
root      2151  0.1  0.0 297972 85424 ?        Ss   Oct15   0:53 pvestatd
root      2202  0.0  0.0 350504 120424 ?       Ss   Oct15   0:00 pvedaemon
root     26206  0.0  0.0 358756 127804 ?       S    05:48   0:02  \_ pvedaemon worker
root      8199  0.0  0.0 358764 127480 ?       S    06:19   0:02  \_ pvedaemon worker
root     30577  0.0  0.0 358728 127344 ?       S    09:28   0:00  \_ pvedaemon worker
root      2210  0.0  0.0 331412 96004 ?        Ss   Oct15   0:02 pve-ha-crm
www-data  2213  0.0  0.1 352028 136260 ?       Ss   Oct15   0:00 pveproxy
www-data 28193  0.0  0.0 360816 131380 ?       S    09:26   0:01  \_ pveproxy worker
www-data 20749  0.0  0.0 360672 131408 ?       S    09:43   0:01  \_ pveproxy worker
www-data  3781  0.1  0.0 360840 131544 ?       S    09:53   0:01  \_ pveproxy worker
www-data  2225  0.0  0.0  67692 55772 ?        Ss   Oct15   0:00 spiceproxy
www-data 19067  0.0  0.0  67944 51976 ?        S    00:00   0:00  \_ spiceproxy worker
root      2227  0.0  0.0 331008 95712 ?        Ss   Oct15   0:07 pve-ha-lrm
www-data 12413  0.0  0.0 365428 128688 ?       S    Oct15   0:03 pveproxy worker (shutdown)
www-data 17286  0.0  0.0 365256 128452 ?       S    Oct15   0:03 pveproxy worker (shutdown)
www-data 28532  0.0  0.0 365788 129380 ?       S    Oct15   0:03 pveproxy worker (shutdown)
root     19075  0.0  0.0  86172  1952 ?        Ssl  00:00   0:02 /usr/sbin/pvefw-logger

Edit: This is the latest PVE Enterprise release up-to-date.

Thank you for your help

 

[Dominic]

You can try to renew the certificates of your cluster using

pvecm updatecerts

man page https://pve.proxmox.com/pve-docs/pvecm.1.html

 

[acti67]

Servers are fresh installed next to your advises. We use LetsEncrypt and certs are up-to-date.
Do we still renew them?

We have the same mistake on all our Proxmox setup.

 

[acti67]

The problem seems to be solved with your solution, thank you