[SOLVED] Running three web server guests on private network, one public IP

A

adrian_vg

Active Member
Mar 8, 2020
74
26
28
Sweden
Hello all!

Got my first Proxmox host running and am now planning to create three separate web servers; one will serve a homepage, second will run Nextcloud and the third will run a picture gallery.

Previously I've used my router's forwarding rules to route all web traffic to a single server running the homepage, Nextcloud and picture gallery as virtual hosts.
This was kind of a PITA for various reasons, and I now want to separate these three services to three separate guests.

I also have a domain name registered:

https://adrianvg.com --> homepage virtual host
https://adrianvg.com/nextcloud --> Nextcloud virtual host
https://adrianvg.com/gallery --> gallery virtual host


What I've read on the web is to use a different port for each of the servers, like eg here; https://forum.proxmox.com/threads/multiple-vps-behind-one-public-ip.1681/post-9177.

10.10.10.3:10991 nextcloud
10.10.10.4:10992 homepage
10.10.10.5:10993 gallery

And this is where my problem starts.
How do I distinguish what calls to eg port 10991 should go to nextcloud when all incoming calls originate from port 443?


Incoming calls would be something like this:
Service nameSource IPPort rangeLocal IPLocal portProtocol
https*.*.*.*44310.10.10.310991TCP
https*.*.*.*44310.10.10.410992TCP
https*.*.*.*44310.10.10.510993TCP

Is there a better way to do this on the Proxmox host instead?
Maybe port forward all external incoming calls to port 443 to the Proxmox host, which would run apache, which would redirect https://adrianvg.com/nextcloud to 10.10.10.3:10991? All incoming to https://adrianvg.com/ to 10.10.10.4:10992 and so on?

When using apache as a redirecter, if at all possible, maybe I won't need different ports?

I can't quite wrap myself around how to set this up. What's the common way to set this up? What's easiest?

Hints and suggestions are greatly appreciated!
Thanks in advance.





My general network layout can be seen in https://forum.proxmox.com/threads/m...ss-proxmox-guests-from-lan.66733/#post-299871.

My current proxmox host /etc/network/interfaces/:

root@dragonborn:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto eno1
#"real" ip address on skynet-tng.internal
iface eno1 inet static
address 192.168.0.9
netmask 255.255.255.0
gateway 192.168.0.1

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

auto vmbr0
#private sub network
iface vmbr0 inet static
address 10.10.10.1
netmask 255.255.255.0
bridge_ports none
bridge_stp off
bridge_fd 0

post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE


//AvG
 
Last edited:
Didn't know I did, but thanks for the confirmation on what to look for and the keywords "reverse proxy". :)

I've only worked with apache since forever or so, never touched nginx really.
Is nginx a high treshold to get a handle on?

//AvG
 
I'm finding quite some info about using reverse proxy with Proxmox and guest containers.
Is this info the same if I use vms instead?
 
I've never used containers, but I can tell you about my setup.

All 443 requests to my domain/IP are routed to my nginx server's IP, which is a guest VM on Proxmox. Nginx will then parse the traffic and route it based on subdomain. Some of the downstream machines are also guests on the VM, some not. I am able to proxy to any IP/port combo that the nginx server can reach. You may also need to lookup specific reverse proxy configurations for your services.
 
Maybe HAproxy as a vm is a solution. According to the interweb it's commonly used as a way to "sort" traffic to various web-services and basically ideal for use with Proxmox when only one public IP is available.

I'm slowly getting where I wan't to be... :)
 
mccoy1996 said:
I've never used containers, but I can tell you about my setup.

All 443 requests to my domain/IP are routed to my nginx server's IP, which is a guest VM on Proxmox. Nginx will then parse the traffic and route it based on subdomain. Some of the downstream machines are also guests on the VM, some not. I am able to proxy to any IP/port combo that the nginx server can reach. You may also need to lookup specific reverse proxy configurations for your services.
Click to expand...

Thanks! Didn't see your new post till just now.
You posted something similar to my own thoughts, except you're using nginx.

I'll experiment a bit with both haproxy and nginx and see which suits me best.

This is getting interesting! :)
 
mccoy1996 said:
I've never used containers, but I can tell you about my setup.

All 443 requests to my domain/IP are routed to my nginx server's IP, which is a guest VM on Proxmox. Nginx will then parse the traffic and route it based on subdomain. Some of the downstream machines are also guests on the VM, some not. I am able to proxy to any IP/port combo that the nginx server can reach. You may also need to lookup specific reverse proxy configurations for your services.
Click to expand...

Hello mccoy1996!

I can't get the redirection to work when passing all incoming traffic on port 443 on my nginx-vm to eg the nexcloud-vm.

Would it be possible to have an example of your conf-files and how you did this?

I've looked at eg https://www.liquidweb.com/kb/redirecting-urls-using-nginx/ for examples but, it's a no-go and can't figure out what I'm doing wrong.

//AvG
 
An update.

Installed haproxy-vm on my proxmox host, and got it to work, more or less, on my internal network.

Router 192.168.0.1
haproxy 10.10.10.7
nextcloud 10.10.10.3
Office computer 192.168.0.100

The office computer can access the nextcloud server on 10.10.10.7:80/nextcloud which is good.

On the router I set all incoming traffic to port 80 and 443 to be forwarded to 10.10.10.7 and anticipated that it should be redirected to the nextcloud server. It didn't.

I am currently maxing out my google-fu in order to get forward on this, ie to have external traffic properly directed to 10.10.10.3 by way of 10.10.10.7.

In the meantime any hints are quite welcome.
Thank in advance!





Below is the mostly plain vanilla haproxy-config with some additions for my particular network setup.

/etc/haproxy/haproxy.cfg:

global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon

# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private

# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
# An alternative list with additional directives can be obtained from
# https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3

defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http


frontend Local_Server
bind 10.10.10.7:80
mode http
default_backend My_Web_Servers

backend My_Web_Servers
mode http
balance roundrobin
option forwardfor
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
option httpchk HEAD / HTTP/1.1rnHost:localhost
server nextcloud 10.10.10.3:80
#server web2.local 192.168.1.102:80
#server web3.local 192.168.1.103:80
 
Another update.
Started anew as I couldn't make neither haproxy nor nginx reverse proxying properly, in other words - I opted to use apache instead which I kinda know "good enough"

Anyway, I got my main web site up and running and internally (ie I browse to the the actual webserver on the 192-net) the main site looks as expected.
However, when access it from outside with ddns name, it looks like something from the early days of the interweb with all kinds of links etc.

Since it looks okay internally, I can't but help feeling the reverse proxy does something to the page/s served, in transition.
As far as I know I got php and all needed modules installed on both the reverse proxy as well as on the homepage server.

The ugly, external look:
Screenshot_20200322_201800.png

When it should look like this:
Screenshot_20200322_201833.png

Not sure if this is a proxmox issue or just something with the reverse proxy setup?

This is the reverse proxy-conf on the apache-server redirecting to the homepage webserver.
<VirtualHost *:80>
ServerName homepage.com
ServerAdmin webmaster@localhost
ErrorLog ${APACHE_LOG_DIR}/error-joomla.log
CustomLog ${APACHE_LOG_DIR}/access-joomla.log combined
ServerName srbu.se
#ProxyHTMLStripComments on
#ProxyRequests off
#SetOutputFilter proxy-html
#ProxyHTMLDoctype XHTML

SSLProxyEngine off
ProxyPreserveHost On
ProxyPass / http://192.168.0.5/
ProxyPassReverse / http://192.168.0.5/
</VirtualHost>

<VirtualHost *:443>
ErrorLog ${APACHE_LOG_DIR}/error-joomla-ssl.log
CustomLog ${APACHE_LOG_DIR}/access-joomla.log combined
ServerName homepage.com
# ProxyHTMLStripComments on
# ProxyRequests off
# SetOutputFilter proxy-html
# ProxyHTMLDoctype XHTML
SSLProxyEngine on
ProxyPreserveHost On

SSLEngine On
SSLCertificateFile /etc/letsencrypt/live/srbu.se/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/srbu.se/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/srbu.se/chain.pem

ProxyPass / http://192.168.0.5/
ProxyPassReverse / http://192.168.0.5/
</VirtualHost>


And this is the homepage webserver, serving the pages seen in the pic.
<VirtualHost *:80>
ServerAdmin webmaster@localhost
ServerName homepage.com
DocumentRoot /var/www/html/joomla
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

<VirtualHost *:443>
ErrorLog ${APACHE_LOG_DIR}/error-nextcloud-ssl.log
CustomLog ${APACHE_LOG_DIR}/access-nextcloud-ssl.log combined
ServerName homepage.com
DocumentRoot /var/www/html/joomla

SSLEngine On
SSLCertificateFile /etc/letsencrypt/live/srbu.se/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/srbu.se/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/srbu.se/chain.pem
</VirtualHost>



Am I missing something here, that makes the externally served page look weird??
Thanks for any pointers.
 
We are using Apache for reverse proxy too, something similar to what @adrian_vg posted. Also the apache is doing the SSL termination with letsencrpt. We added extra protection by requesting passwords or limiting by IP certain paths ex: /wp-admin.

As an brute force protection, we use csf that is monitoring the Apache logs. If you accessed more then 5 times per minute /login.php for example you will have your IP blocked for one day.
 
Hi all!
Since March I've done some major overhaul and tried to fix stuff and learn more abut reverse proxying.
I've come so far as having my Joomla site up SSL:ed and all on proxmox.
All calls on ports 80 and 443 from the internet are port forwarded in the router to a vm running Joomla.

This Joomla vm has a vhost block with reverse proxy setting pointing to my Nextcloud vm.

Accessing my Joomla site using my registered domain example.com works fine.

The problem begins when I try to access my Nextcloud instance.

Surfing from my internal network to cloud.example.com works, so I guess the reverse proxy does its thing, but it shows me the internal ip 192.168.0.10 in the address field.

Accessing cloud.example.com from the internet, does not work.

I'm not sure what to do about this or what, if anything, I've effed up during my experimenting...
Any pointers are greatly appreciated.
Below are the main site conf's I use.

Can anybody help me out? Thanks!

* example.com is an obfuscation of the real domain of course. :)

I've added a subdomain "cloud" with my domain registrar, as an addition to my main site example.com.
I set it as a CNAME with cloud.example.com pointing back to example.com, hoping I did that right.

Also, why does it it all work for my main joomla site at example.com, but not cloud.example.com?
The apache vhost conf files are almost identical.



joomla.conf on example.com includes a block for cloud.example.com.

<virtualhost cloud.example.com>
#SSLProxyEngine On # Tested with and without, I don't see anything happen in either case
ProxyPreserveHost On
ProxyRequests off
ProxyPass / https://cloud.srbu.se/
ProxyPassReverse / https://cloud.srbu.se/
</virtualhost>



Nextcloud.conf on cloud.example.com has the normal stuff with ssl etc.

<VirtualHost cloud.example.com:80>
ServerAdmin webmaster@example.com
ServerName cloud.example.com
DocumentRoot /var/www/nextcloud
Redirect permanent / https://cloud.example.com
ErrorLog /var/log/apache2/nextcloud-80-error.log
CustomLog /var/log/apache2/nexcloud-80.log common
#RewriteEngine on
#RewriteCond %{SERVER_NAME} =cloud.example.com
#RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

<virtualhost cloud.example.com:443>
ServerAdmin webmaster@example.com
ServerName cloud.example.com
DocumentRoot /var/www/nextcloud
SSLEngine on
SSLProxyEngine on
# SSLProxyVerify none
# SSLProxyCheckPeerCN off
# SSLProxyCheckPeerName off
ProxyPreserveHost On
ProxyRequests off
# SetOutputFilter INFLATE;proxy-html;DEFLATE;
# ProxyHTMLInterp On
# ProxyHTMLExtended On
# ProxyHTMLURLMap (.*)192.168..10(.*) https://cloud.example.com$2 [Rin]
# ProxyVia On
# <Proxy *>
# Require host cloud.example.com
# </Proxy>
Header always set Strict-Transport-Security "max-age=63072000"
# RequestHeader set X-Forwarded-Port "443"
RequestHeader set X-Forwarded-Host cloud.example.com
# RemoteIPHeader X-Forwarded-For
RequestHeader set X-Forwarded-Proto "https"
<IfModule mod_headers.c>
Header always set Strict-Transport-Security "max-age=15552000; includeSubDomains"
</IfModule>
ErrorLog /var/log/apache2/nextcloud-443-error.log
CustomLog /var/log/apache2/nexcloud-443.log common
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA38>
SSLHonorCipherOrder on
<Directory "/var/www/cgi-bin">
SSLOptions +StdEnvVars
</Directory>
SSLCertificateFile /etc/ssl/2020-11-07/certificate.crt
SSLCertificateKeyFile /etc/ssl/2020-11-07/private.key
SSLCertificateChainFile /etc/ssl/2020-11-07/ca_bundle.crt
#Include /etc/letsencrypt/options-ssl-apache.conf
RewriteEngine On
RewriteRule ^/\.well-known/carddav https://%{SERVER_NAME}/remote.php/dav/ [R=301,L]
RewriteRule ^/\.well-known/caldav https://%{SERVER_NAME}/remote.php/dav/ [R=301,L]
</virtualhost>
 
Last edited:
Took almost a year but I now have a working solution.
Since I wandered into docker a while ago, it was just a question of time until I discovered this thing:

https://github.com/jc21/nginx-proxy-manager

Long story short, making this work with my already present SSL:ed web services was a no-brainer.
I now on Proxmox have a docker vm running nginx, forwarding all traffic correctly between the homepage and the cloud-storage vm-servers.

Next in line is to have a look at why this thing works and my built-from-scratch reverse proxy-solution doesn't.
That is however a different story and possibly not for this particular forum.

Thanks all that have commented and hinted on this matter!
 
bobmc said:
Yeah this looks like a great way of managing nginx - shame it's docker only. Would be awesome if someone had the skills to convert it to a lxc template :)
Click to expand...

I do believe I saw a guide using this with a Proxmox container flash past me while I researched docker proxy managers.
I'll see if I can find it again.

In the meantime, this is the pretty project guide page for jc12's ngninx proxy manager:
https://nginxproxymanager.com/setup/
 
Greetings @adrian_vg :

A few thoughts for you ...

Container: ubuntu 20.04
vCPU: 1
RAM: 256MB works, but you can scale up as in the increase in traffic demands

NOTES
1. LXC container to keep it really light and have the flex to dynamically scale up vCPU/RAM as needed
2. Ubuntu 20.04 as it's dirt simple
3. # apt install haproxy
4. If all of your sites are SSL (assuming they would be) and you're using https://letsencrypt.org/ for free certs on each container/vm (which I do), you'll be using layer (L4) load balancing with TCP.
5. You will not be terminating the SSL sessions on HAProxy, but 'switching' them via their FQDN's and terminating SSL at the servers instead.
6. This configuration allows you to configure your DNS for numerous (lot's) of FQDN's to a single IP address.

Regarding leveraging HAProxy and a sample configuration to accomplish the objective, I have setup a simple container and have the following configuration defined for HAProxy:
Code: 
root@haproxy:/etc/haproxy# more haproxy.cfg
global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
#       ca-base /etc/ssl/certs
#       crt-base /etc/ssl/private

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
#        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CH
ACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
#        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
#        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

# Single VIP
frontend ft_ssl_vip
  bind 192.168.1.2:443
  mode tcp

  tcp-request inspect-delay 5s
  tcp-request content accept if { req_ssl_hello_type 1 }

  default_backend bk_ssl_default

# Using SNI to make routing decision
backend bk_ssl_default
  mode tcp

  acl app01 req_ssl_sni -i app01.domain.com
  acl app02 req_ssl_sni -i app02.domain.com
  acl app03 req_ssl_sni -i app03.domain.com

  use-server app01 if app01
  use-server app02 if app02
  use-server app03 if app03

  option ssl-hello-chk
  server app01 192.168.1.10:443 check
  server app02 192.168.1.11:443 check
  server app03 192.168.1.12:443 check

This configuration works like a charm. Hope it proves helpful to you.

Andy
 
Last edited:
  • Like
Reactions: cymesa, Linux for Work and bobmc
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back