[TUTORIAL] PVE proxy with kubernetes nginx ingress

[r.jochum]

Today I moved my Proxmox behind an Kubernetes NGINX Ingress, I did that because I want a CORS, a Let's Encrypt Cert with a single Public IP and port 443.

The Service YAML

apiVersion: v1
kind: Service
metadata:
  annotations:
    field.cattle.io/ipAddresses: '["10.167.160.10"]'
  name: pve01
  namespace: my-pve-namespace
spec:
  clusterIP: None
  ports:
  - name: pve
    port: 8006
    protocol: TCP
    targetPort: 8006
  sessionAffinity: None
  type: ClusterIP

And the ingress yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    ingress.kubernetes.io/configuration-snippet: "proxy_set_header Host $http_host;\nproxy_set_header
      X-Real-IP $remote_addr;\nproxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      \     \nproxy_http_version 1.1;\nproxy_set_header Upgrade $http_upgrade;\nproxy_set_header
      Connection $connection_upgrade;      "
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/cors-allow-headers: Authorization
    nginx.ingress.kubernetes.io/cors-allow-methods: PUT, GET, POST, OPTIONS, DELETE
    nginx.ingress.kubernetes.io/cors-allow-origin: "*"
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/server-snippet: |-
      client_max_body_size 100G;
      server_tokens off;
      proxy_hide_header X-Powered-By;
  generation: 2
  managedFields:
  name: pve01
  namespace: my-pve-namespace
spec:
  rules:
  - host: pve01.mydomain.com
    http:
      paths:
      - backend:
          serviceName: pve01
          servicePort: 8006
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - pve01.mydomain.com
    secretName: pve01-mycomain-com-tls

 

[r.jochum]

Now I also added a NGINX TCP LoadBalancer for Spice Port 3128

See [0] for the doku on how to that, see also [1] if you have troubles with it.

0: https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/
1: https://medium.com/@ahmedwaleedmali...n-kubernetes-using-nginx-ingress-9b8fd639c576

 

[enmanuelmoreira]

It works for me with a little modification:

---

apiVersion: v1
kind: Service
metadata:
  name: pve01
  namespace: networking
  annotations:
    field.cattle.io/ipAddresses: '["192.168.10.100"]'
spec:
  type: ClusterIP
  clusterIP: None
  ports:
    - name: pve-port
      port: 8006
      protocol: TCP
      targetPort: 8006
  sessionAffinity: None
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: pve01
  namespace: networking
  annotations:
    ingress.kubernetes.io/configuration-snippet: "proxy_set_header Host $http_host;\nproxy_set_header
      X-Real-IP $remote_addr;\nproxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      \     \nproxy_http_version 1.1;\nproxy_set_header Upgrade $http_upgrade;\nproxy_set_header
      Connection $connection_upgrade;      "
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/cors-allow-headers: Authorization
    nginx.ingress.kubernetes.io/cors-allow-methods: PUT, GET, POST, OPTIONS, DELETE
    nginx.ingress.kubernetes.io/cors-allow-origin: "*"
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/server-snippet: |-
      client_max_body_size 100G;
      server_tokens off;
      proxy_hide_header X-Powered-By;
  generation: 2
  managedFields:
spec:
  rules:
    - host: &host pve01.domain.local
      http:
        paths:
          - backend:
              service:
                name: pve01
                port:
                  number: 8006
            path: /
            pathType: ImplementationSpecific
  tls:
    - hosts:
        - *host
      secretName: *host
---

k3s: 1.25.11
nginx ingress controller: 4.7.1
cert-manager: 1.12.2

Thank you!

 

[enmanuelmoreira]

Updated 01-15-2024

---

apiVersion: v1
kind: Service
metadata:
  name: pve01
  namespace: networking
  annotations:
    field.cattle.io/ipAddresses: '["192.168.10.100"]'
spec:
  type: ClusterIP
  clusterIP: None
  ports:
    - name: pve-port
      port: 8006
      protocol: TCP
      targetPort: 8006
  sessionAffinity: None
---
apiVersion: v1
kind: Endpoints
metadata:
  name: pve01
subsets:
  - addresses:
      - ip: "192.168.1.100"
    ports:
      - name: https
        port: 8006
        protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: pve01
  namespace: networking
  annotations:
    ingress.kubernetes.io/configuration-snippet: "proxy_set_header Host $http_host;\nproxy_set_header
      X-Real-IP $remote_addr;\nproxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      \     \nproxy_http_version 1.1;\nproxy_set_header Upgrade $http_upgrade;\nproxy_set_header
      Connection $connection_upgrade;      "
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/cors-allow-headers: Authorization
    nginx.ingress.kubernetes.io/cors-allow-methods: PUT, GET, POST, OPTIONS, DELETE
    nginx.ingress.kubernetes.io/cors-allow-origin: "*"
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/server-snippet: |-
      client_max_body_size 100G;
      server_tokens off;
      proxy_hide_header X-Powered-By;
  generation: 2
  managedFields:
spec:
  rules:
    - host: &host pve01.domain.local
      http:
        paths:
          - backend:
              service:
                name: pve01
                port:
                  number: 8006
            path: /
            pathType: ImplementationSpecific
  tls:
    - hosts:
        - *host
      secretName: *host
---

k3s: 1.28.5
nginx ingress controller: 4.9.0
cert-manager: 1.13.3