Privacy of multiple users on one Proxmox machine

[esi_y]

What an interesting discussion. But the answer is really quite simple. The VMs are as safe from the party in possession of the hardware than e.g. EC2 instance on AWS is from a court warrant.

Only data that already got there as encrypted (e.g. from a client outside of the realm) are safe insofar as the encryption and the chosen key is well-chosen. All the other topics like encryption between two endpoints or LUKS full-drive encryption, let alone just plain access control do not change anything about physical access to the data and the processes producing them. Anything encrypted inside the VMs which used keys also stored inside the VMs could of course be decrypted with the use of the same keys. LUKS keys are in RAM and could be dumped.

There's really not much to it, actually.

 

[proxenjoy]

What an interesting discussion. But the answer is really quite simple. The VMs are as safe from the party in possession of the hardware than e.g. EC2 instance on AWS is from a court warrant.

Only data that already got there as encrypted (e.g. from a client outside of the realm) are safe insofar as the encryption and the chosen key is well-chosen. All the other topics like encryption between two endpoints or LUKS full-drive encryption, let alone just plain access control do not change anything about physical access to the data and the processes producing them. Anything encrypted inside the VMs which used keys also stored inside the VMs could of course be decrypted with the use of the same keys. LUKS keys are in RAM and could be dumped.

There's really not much to it, actually.

Thank all of you guys meyergru, gfngfn256 for your patience to explain the concepts. I really learned a lot. Yeahh, it looks simple now for me, but I first needed to learn why it is simple. So this means a proper encryption key that cannot be brute forced. Sure if quantum computers come around it won't be safe anymore maybe, but then I am sure we figure out something else again to fight against it. I will stay a cat and mouse game anyways.

How, I look at it now is that all my friends private data should be encrypted on my friends device before it is send to my server. In that case he can store his private Nextcloud files, a Joplin or a Matrix chat server on my hardware because of end-to-end encryption.

Now, regarding applications with public data it really doesn't matter if someone can inspect that in my opinion. The reason being it is already public anyways.

 

[esi_y]

if your application has end to end encryption you are good to go!! That's it basically

Actually, you are not, often. The term is completely overused and sometimes even abused. This is e.g. fixed to huge extend by now, but it may simply mean more are waiting to be found out:

https://matrix.org/blog/2022/09/28/...-encryption-vulns-in-matrix-sdks-and-clients/

https://nebuchadnezzar-megolm.github.io/static/paper.pdf

Everything with fast development (and no specialised security staff) suffers from this, perpetually.

 

[gfngfn256]

In that case he can store his private Nextcloud files, a Joplin or a Matrix chat server on my hardware because of end-to-end encryption.

Please note I haven't checked all of these application's modes of incorporating so-called "end-to-end encryption", but just looking at Joplin's I see the Master Keys are saved to the database, so bear in mind; that is on YOUR server. Looking on their site, I see they claim that:

Master Keys​

The master keys are used to encrypt and decrypt data. They can be generated from the Encryption Service and are saved to the database. They are themselves encrypted via a user password using a strong encryption method.

So I guess it is down to that "strong encryption method" (gives me the shivers!).

What I fail to understand, if I were your friend, why not use the Joplin cloud instead of your server. I must be honest, I would rather my data is out there anonymously, than on my friend's PC who knows who I am, and isn't guaranteed to be online as much as a Cloud company. Add to that, HW failure that is more likely at your end than Joplin's. Also if you decide "you don't like him anymore" - poof, he's lost all his data.
Is he just trying to save 28 Euros a year?

 

[esi_y]

So I guess it is down to that "strong encryption method" (gives me the shivers!).

FWIW It's using AES 256bit in CCM mode, i.e. stream cipher. Great for implementation errors to be found later on though.

I would rather my data is out there anonymously, than on my friend's PC who knows who I am, and isn't guaranteed to be online as much as a Cloud company. Add to that, HW failure that is more likely at your end than Joplin's.

They are also most likely to get their "cloud" targeted, because the target is known. Note I am not advocating security through obscurity, but when there will be e.g. a Bitwarden breach you can bet the first affected will be those using their infrastructure, even if it was just protocol error. And the disclosure would come months later only. At least you see logs on your own infrastructure.

Also if you decide "you don't like him anymore" - poof, he's lost all his data.

Surely he has backups.



Anyhow, I will be honest, I was like this once too, concerned seriously about security as a college freshman. The more you understand these, the more you find out the so-called security is mostly illusory. Especially if something is "best in class" or such. Nothing with active development is truly secure. Within industry it is then mostly about compliance.

 

[proxenjoy]

FWIW It's using AES 256bit in CCM mode, i.e. stream cipher. Great for implementation errors to be found later on though.




They are also most likely to get their "cloud" targeted, because the target is known. Note I am not advocating security through obscurity, but when there will be e.g. a Bitwarden breach you can bet the first affected will be those using their infrastructure, even if it was just protocol error. And the disclosure would come months later only. At least you see logs on your own infrastructure.



Surely he has backups.



Anyhow, I will be honest, I was like this once too, concerned seriously about security as a college freshman. The more you understand these, the more you find out the so-called security is mostly illusory. Especially if something is "best in class" or such. Nothing with active development is truly secure. Within industry it is then mostly about compliance.

Thank you for your reply esi_y,

Yeahhhh, the problem that your friend will delete your data I am not so worried about neither and we have backups as you said. Sure, like with Matrix it is not 100% full proof, because people can always discover vulnerabilities, but that's the same with all the centralised applications. The likelihood for centralised applications to be a target for hacking is also a lot bigger than for the average Joe with his small server at home. (as you mentioned with the Bitwarden example).

It is more that we both want to back up each other's data. So we both already have our own server in our own house and we want to use each other's server as a backup to form a HA cluster. My friend is just two houses away from me so we will just pull a cable from my house to his s.t we can have HA low latency both on our home networks.

The only thing I am worried about is to make sure that personal data is kept private (not that my friend would intentionally try to attack me, but just for an experimental point of view ). Now, I think with an application like Nextcloud which is end to end encrypted this should be possible. That is pictures, images, calendars, Nextcloud(Notes) and all the other applications Nextcloud offers. So you create a nextcloud VM on your server and a replication of that VM on your friends server.

Hopefully, you tell me that that will work haha

 

[gfngfn256]

The likelihood for centralised applications to be a target for hacking is also a lot bigger than for the average Joe with his small server at home.

True. But the general security hardening of the average Joe's home server doesn't come anywhere near to the enterprise solutions. I must be honest here, looking at the OP's post - I'm not sure how much he knows about security hardening at all.

It is more that we both want to back up each other's data. So we both already have our own server in our own house and we want to use each other's server as a backup to form a HA cluster.

So just setup a PBS instance on both servers, & use encryption. I wouldn't make an HA cluster for that. HA stands for High Availability, that is the ability to automatically run a single instance of a VM/s on the available server at the time. Is that really what you want?

Another point an HA cluster requires a minimum of 3 nodes (or 2 & a Qdevice).

Next, making a cluster with your neighbor, means both of you having access to all of your DATA. Why not invite others on the street to join?

 

[esi_y]

My friend is just two houses away from me so we will just pull a cable from my house to his s.t we can have HA low latency both on our home networks.

What kind of connection and routing will that have?

But the general security hardening of the average Joe's home server doesn't come anywhere near to the enterprise solutions.

But we may give a piece of advice on this forum to have the OP e.g. run this entirely within VPN only accessed segment.

security hardening at all.

So that this is not really a problem.

Hopefully, you tell me that that will work haha

I am not sure what you are attempting to achieve still. Maybe after you answer ...

HA stands for High Availability, that is the ability to automatically run a single instance of a VM/s on the available server at the time. Is that really what you want?

... then it would make more sense to maybe just recommend a simple encrypted backup "across the street."

Why not invite others on the street to join?

I used to be the most grumpy on this forum at times, I am glad I found a match.

 

[proxenjoy]

True. But the general security hardening of the average Joe's home server doesn't come anywhere near to the enterprise solutions. I must be honest here, looking at the OP's post - I'm not sure how much he knows about security hardening at all.


So just setup a PBS instance on both servers, & use encryption. I wouldn't make an HA cluster for that. HA stands for High Availability, that is the ability to automatically run a single instance of a VM/s on the available server at the time. Is that really what you want?

Another point an HA cluster requires a minimum of 3 nodes (or 2 & a Qdevice).

Next, making a cluster with your neighbor, means both of you having access to all of your DATA. Why not invite others on the street to join?

Yes my understanding on security hardening is very limited, otherwise I would not be asking these questions .

But yess you are right you need 3 nodes ideally. However, I would like a system that when the servers at my house are down for reason x (e.g fire, power outage, hardware failure), that the end-to-end encrypted application e.g Nextcloud fails over to the server of my neighbor. The only thing I could find for that is HA cluster or rsync with the VM on my neighbors server for instance. I tried PBS with encryption already but it takes way to long for the VM to be up and running again, plus this needs to be done manually, i.e PBS does not automatically spin up another VM of the latest backup and changes DNS on my neighbors server if my server is down.

Or is there a way to get a PBS server to automatically start a VM from the latest backup when the server from which the backup is made is down? Maybe a script that if it does not get a ping back it will start a VM from the latest backup ?

"Next, making a cluster with your neighbor, means both of you having access to all of your DATA"

Is there not a possibility to create users with different permissions ? same as you create a user that can only have access to a particular pool of resources and cannot interact nor see the VM's of the owner of the Promox box ?

 

[gfngfn256]

Is there not a possibility to create users with different permissions ?

Yes there is - but there is still a root/administrator who will have cluster wide accessibility. So in your case, assuming you & your neighbor will have root/administrator rights, I'm afraid what I said still stands. Maybe your friend or you is prepared to only be a guest on their own server?

 

[proxenjoy]

What kind of connection and routing will that have?

But we may give a piece of advice on this forum to have the OP e.g. run this entirely within VPN only accessed segment.
So that this is not really a problem.

I am not sure what you are attempting to achieve still. Maybe after you answer ...
... then it would make more sense to maybe just recommend a simple encrypted backup "across the street."

I used to be the most grumpy on this forum at times, I am glad I found a match.

Yeahhh I am just trying to learn guys, I am also new to this field, so I am sorry for my lack of understanding.

Regarding your connection and routing question I still need to think about it. But I would think to just plug cable from one of my NIC's to one of his NIC's server. Then associate the NIC to the VM's you want HA for. Again good question that I don't really have an answer for

 

[gfngfn256]

Regarding your connection and routing question I still need to think about it. But I would think to just plug cable from one of my NIC's to one of his NIC's server. Then associate the NIC to the VM's you want HA for.

I think I'm getting more frustrated than you. Your ideas, while being very "interesting", aren't in fact based on reality.

I'm going to make a suggestion - IDK how long you've been using Proxmox so far (you joined this forum yesterday), but what I would do in your case, is first experiment, test & learn the whole PVE experience for at least a few weeks; create VMs, LXCs & clusters ALL AT HOME IN YOUR OWN ENVIRONMENT & NW. You'll learn a lot. Then move on to inviting others in.

I don't mean to be degrading - I'm just telling you what I have done (as well as many others) & would do in your present position. You will learn a lot & it will be most enjoyable.

Good luck.

 

[esi_y]

I think I'm getting more frustrated than you. Your ideas, while being very "interesting", aren't in fact based on reality.

I'm going to make a suggestion - IDK how long you've been using Proxmox so far (you joined this forum yesterday),

I learned over time on the forum that it's my responsibility as a party voluntarily replying to assess the skillset, etc. of the asking party. They can't really tell us what they do not know (yet) because they do not know it.

but what I would do in your case, is first experiment, test & learn the whole PVE experience for at least a few weeks; create VMs, LXCs & clusters ALL AT HOME IN YOUR OWN ENVIRONMENT & NW.

+1 for this one, when you go testing things, first do it locally. I would also suggest to start with HA completely OFF. You can go check logs later and see how the cluster was doing (e.g. it was losing quroum, etc., during migration and such).

You'll learn a lot.

After you get familiar with the non-HA bits and pieces, then read the HA stack part, especially fencing:
https://pve.proxmox.com/wiki/High_Availability#ha_manager_fencing

Then when you check back your logs, you will realise that had you been running that setup as HA, you would be getting reboots all those times you lost quorum.

Then move on to inviting others in.

Feel free to ask here, but at this point you are brainstorming a bit of everything, not having specific-enough questions for a straightforward answer. People mean well here, but if you ask a question and in turn re-iterate something in the thread they had supposedly already answered, confirming we are not on the same page, it really needs to take a step back (before follow up questions to make sense).

To give you the context, also where I believe @gfngfn256 is coming from ...

Regarding your connection and routing question I still need to think about it. But I would think to just plug cable from one of my NIC's to one of his NIC's server. Then associate the NIC to the VM's you want HA for. Again good question that I don't really have an answer for

You reminded me I have yet to follow up on another thread here from earlier today:

https://forum.proxmox.com/threads/6-node-ha-cluster-split-brain.152081/#post-689437

That's 6 nodes setup with HA where the 3+3 are connected with LACP + MLAG across redundantly over 100G OS2 fibre link and ... it's not providing High Availability, at all despite having set up 2 rings on corosync:
https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_redundancy

Once you start getting your head around those things, it's easier to answer.

Regarding the use cases you mentioned.

Have you considered simply replicating data (as offsite backup) to each other? It's a good start. Even there you have options if you want to e.g. run some tool like restic or zfs send | receive over that dedicated link, etc. If you cross connect it, you will have questions about routing here yet, no problem.

The other thing, from architecture point of view, maybe instead of e.g. Nextcloud, you are better off with a solution like Resilio Sync (used to have BitTorrent in the name):
https://www.resilio.com/individuals/

It literally allows you to make it decentralised and some of the "nodes" are encrypted, i.e. data replicated to there are already sent encrypted, keys do stay on nodes you choose. This works fine for environments like public cloud too.

For really private data, simply put them e.g. into VeraCrypt or LUKS container? Also instead of having anything open to the internet, consider putting all the VMs behind a wireguard or IPSec connection.

That's probably enough brainstorming to start thinking now?

 

[proxenjoy]

I learned over time on the forum that it's my responsibility as a party voluntarily replying to assess the skillset, etc. of the asking party. They can't really tell us what they do not know (yet) because they do not know it.



+1 for this one, when you go testing things, first do it locally. I would also suggest to start with HA completely OFF. You can go check logs later and see how the cluster was doing (e.g. it was losing quroum, etc., during migration and such).



After you get familiar with the non-HA bits and pieces, then read the HA stack part, especially fencing:
https://pve.proxmox.com/wiki/High_Availability#ha_manager_fencing

Then when you check back your logs, you will realise that had you been running that setup as HA, you would be getting reboots all those times you lost quorum.



Feel free to ask here, but at this point you are brainstorming a bit of everything, not having specific-enough questions for a straightforward answer. People mean well here, but if you ask a question and in turn re-iterate something in the thread they had supposedly already answered, confirming we are not on the same page, it really needs to take a step back (before follow up questions to make sense).

To give you the context, also where I believe @gfngfn256 is coming from ...



You reminded me I have yet to follow up on another thread here from earlier today:

https://forum.proxmox.com/threads/6-node-ha-cluster-split-brain.152081/#post-689437

That's 6 nodes setup with HA where the 3+3 are connected with LACP + MLAG across redundantly over 100G OS2 fibre link and ... it's not providing High Availability, at all despite having set up 2 rings on corosync:
https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_redundancy

Once you start getting your head around those things, it's easier to answer.

Regarding the use cases you mentioned.

Have you considered simply replicating data (as offsite backup) to each other? It's a good start. Even there you have options if you want to e.g. run some tool like restic or zfs send | receive over that dedicated link, etc. If you cross connect it, you will have questions about routing here yet, no problem.

The other thing, from architecture point of view, maybe instead of e.g. Nextcloud, you are better off with a solution like Resilio Sync (used to have BitTorrent in the name):
https://www.resilio.com/individuals/

It literally allows you to make it decentralised and some of the "nodes" are encrypted, i.e. data replicated to there are already sent encrypted, keys do stay on nodes you choose. This works fine for environments like public cloud too.

For really private data, simply put them e.g. into VeraCrypt or LUKS container? Also instead of having anything open to the internet, consider putting all the VMs behind a wireguard or IPSec connection.

That's probably enough brainstorming to start thinking now?

Thank you for all your replies guys! I will definitely look into all of that and start experimenting myself!

Thank you for your suggestion with Resilio Sync. I had a look at it but my goal is to only use self hosted open source applications and not proprietary ones, but thanks anyways.

Surely, I will come back to here one day after trying out haha! thanks cheers!

 

[esi_y]

Thank you for your suggestion with Resilio Sync. I had a look at it but my goal is to only use self hosted open source applications and not proprietary ones, but thanks anyways.

Sorry, I have not realised this - last time I remembered it there were sources available. Anyhow it is self hosted, but quick search (as is often the case) gives an OS alternative:
https://syncthing.net/

But some features won't be as mature:
https://docs.syncthing.net/users/untrusted.html

 

[proxenjoy]

Sorry, I have not realised this - last time I remembered it there were sources available. Anyhow it is self hosted, but quick search (as is often the case) gives an OS alternative:
https://syncthing.net/

But some features won't be as mature:
https://docs.syncthing.net/users/untrusted.html

Don't be sorry hahah, yeahh syncthing is awesome, but that won't work for the average user. You want open source, self hosted applications that look as nice or even nicer then the for proprietary ones. Therefore, Nextcloud gives amazing GUI's and Apps which synchthing does not. For instance, for photos you have this https://apps.nextcloud.com/apps/memories. That's why I am referring to Nextcloud the whole time.

 

[esi_y]

Don't be sorry hahah, yeahh syncthing is awesome, but that won't work for the average user. You want open source, self hosted applications that look as nice or even nicer then the for proprietary ones. Therefore, Nextcloud gives amazing GUI's and Apps which synchthing does not. For instance, for photos you have this https://apps.nextcloud.com/apps/memories. That's why I am referring to Nextcloud the whole time.

I am not really into this home-thing much, probably becoming a dinosaur contempt with photos being just files. As such, worse yet, I would have my own (generation's) stereotypes. E.g. not expose anything PHP. I know there were other cool apps for photo gallery (I would not find them now), but as always, things come and go, especially with open source. For files alone, reliability, security ... even rsync is cool.

I would definitely advise putting not just behind reverse proxy, but also within a VPN (not just the PVE host, but also the Nextcloud VM).

 

[proxenjoy]

I am not really into this home-thing much, probably becoming a dinosaur contempt with photos being just files. As such, worse yet, I would have my own (generation's) stereotypes. E.g. not expose anything PHP. I know there were other cool apps for photo gallery (I would not find them now), but as always, things come and go, especially with open source. For files alone, reliability, security ... even rsync is cool.

I would definitely advise putting not just behind reverse proxy, but also within a VPN (not just the PVE host, but also the Nextcloud VM).

yeahhh, I do think aesthetics is really important for people to use open source self hosted alternatives

I thought as well to use rsync ! Just create one Nextcloud VM with rsync + and then connect it to the other Nextcloud VM on my friends server via a VPN. Then when my server is down, I can just change DNS to the server of my friend until I have my own server up and running again to change it back.