[SOLVED] Strange problem with Proxmox/PBS connection

V

villain666

Renowned Member
Oct 4, 2012
37
5
73
Hi

I setup two node test cluster with qdevice on my work linux station
install debian 10 and proxmox 6.4 over it. Run mirror on zfs, setup 2 vms (1 vm to 1 node), run replication from one to other)

next big step - upgrade to proxmox 7

first node - first migrate to ifupdown2 (pve6.4 version) and upgrade to bulseye and pve7
second node - upgrade to bulseye and pve7 and next migrate to ifupdown (pve7 version)

mostly work fine, cluster, replication, migration, ha...

next step - setup PBS on first node
first node - works fine
pvesm scan pbs 192.168.122.231 root@pam --password xxxxxxxxxxxxxxx --fingerprint "D4:BC:7A:AF:35:52:B7:26:DF:E3:43:F5:E4:60:69:FA:93:2C:34:DE:15:EC:4D:D3:36:2D:EB:CA:3B:B5:CC:9E" ┌────────┬─────────┐ │ store │ comment │ ╞════════╪═════════╡ │ backup │ │ └────────┴─────────┘

but on second node
pvesm scan pbs 192.168.122.231 root@pam --password xxxxxxxxxxxx --fingerprint "44:15:2a:59:24:86:7b:ad:d4:6d:c7:6a:b7:03:15:dc:e9:62:aa:f9:51:63:d9:44:44:b3:f3:26:0c:74:d6:f0" error fetching datastores - 500 Can't connect to 192.168.122.231:8007

I try run tcpdump on first node and no see connection from second node to 8007 port

How to debug this problem with connection to PBS from this node?
iptables, ip route, etc - looks good

i`m totally confused...
 
pveversion -v proxmox-ve: 7.0-2 (running kernel: 5.11.22-2-pve) pve-manager: 7.0-10 (running version: 7.0-10/d2f465d3) pve-kernel-5.11: 7.0-5 pve-kernel-helper: 7.0-5 pve-kernel-5.11.22-2-pve: 5.11.22-4 ceph-fuse: 14.2.21-1 corosync: 3.1.2-pve2 criu: 3.15-1+pve-1 glusterfs-client: 9.2-1 ifupdown2: 3.1.0-1+pmx2 libjs-extjs: 7.0.0-1 libknet1: 1.21-pve1 libproxmox-acme-perl: 1.2.0 libproxmox-backup-qemu0: 1.2.0-1 libpve-access-control: 7.0-4 libpve-apiclient-perl: 3.2-1 libpve-common-perl: 7.0-5 libpve-guest-common-perl: 4.0-2 libpve-http-server-perl: 4.0-2 libpve-storage-perl: 7.0-9 libspice-server1: 0.14.3-2.1 lvm2: 2.03.11-2.1 lxc-pve: 4.0.9-4 lxcfs: 4.0.8-pve2 novnc-pve: 1.2.0-3 proxmox-backup-client: 2.0.5-2 proxmox-backup-file-restore: 2.0.5-2 proxmox-mini-journalreader: 1.2-1 proxmox-widget-toolkit: 3.3-5 pve-cluster: 7.0-3 pve-container: 4.0-8 pve-docs: 7.0-5 pve-edk2-firmware: 3.20200531-1 pve-firewall: 4.2-2 pve-firmware: 3.2-4 pve-ha-manager: 3.3-1 pve-i18n: 2.4-1 pve-qemu-kvm: 6.0.0-2 pve-xtermjs: 4.12.0-1 qemu-server: 7.0-10 smartmontools: 7.2-pve2 spiceterm: 3.2-2 vncterm: 1.7-1 zfsutils-linux: 2.0.5-pve1 root@proxmox232:~#
 
hi,

villain666 said:
but on second node
pvesm scan pbs 192.168.122.231 root@pam --password xxxxxxxxxxxx --fingerprint "44:15:2a:59:24:86:7b:ad:d4:6d:c7:6a:b7:03:15:dc:e9:62:aa:f9:51:63:d9:44:44:b3:f3:26:0c:74:d6:f0" error fetching datastores - 500 Can't connect to 192.168.122.231:8007

I try run tcpdump on first node and no see connection from second node to 8007 port

How to debug this problem with connection to PBS from this node?
iptables, ip route, etc - looks good
Click to expand...
does ping work both ways? can you ping node2 from pbs and vice versa?

do you have a firewall setup on node1? maybe that's preventing the backup client from reaching it.
 
yes
.231
ping 192.168.122.232 PING 192.168.122.232 (192.168.122.232) 56(84) bytes of data. 64 bytes from 192.168.122.232: icmp_seq=1 ttl=64 time=0.155 ms 64 bytes from 192.168.122.232: icmp_seq=2 ttl=64 time=0.298 ms ^C --- 192.168.122.232 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1010ms rtt min/avg/max/mdev = 0.155/0.226/0.298/0.071 ms pvecm nodes Membership information ---------------------- Nodeid Votes Qdevice Name 1 1 A,V,NMW proxmox231 (local) 2 1 A,V,NMW proxmox232 0 1 Qdevice

.232
ping -4 192.168.122.231 PING 192.168.122.231 (192.168.122.231) 56(84) bytes of data. 64 bytes from 192.168.122.231: icmp_seq=1 ttl=64 time=0.274 ms 64 bytes from 192.168.122.231: icmp_seq=2 ttl=64 time=0.303 ms ^C --- 192.168.122.231 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1010ms rtt min/avg/max/mdev = 0.274/0.288/0.303/0.014 ms telnet 192.168.122.231 8007 Trying 192.168.122.231... Connected to 192.168.122.231. Escape character is '^]'. fdgsfdlgkfdsglksfdlgk Connection closed by foreign host.

I can telnet from second node .232 to first node .231 to port 8007, but pvesm dont connect
and tcpdump don`t see any trffic
 
oguz said:
do you have a firewall setup on node1? maybe that's preventing the backup client from reaching it.
Click to expand...
you haven't answered this, what are the firewall rules on your node1 where PBS is running?
villain666 said:
I can telnet from second node .232 to first node .231 to port 8007, but pvesm dont connect
and tcpdump don`t see any trffic
Click to expand...
how do you run tcpdump? are you sniffing the correct interface?

try like this:
Code: 
tcpdump -i vmbr0 'port 8006' -vv
 
1. No, no one firewall rules (on both node)
proxmox231
Code: 
iptables -xvnL
Chain INPUT (policy ACCEPT 22410989 packets, 13521272392 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 5353 packets, 1858795 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 17351163 packets, 6345010945 bytes)
    pkts      bytes target     prot opt in     out     source               destination


iptables -xvnL -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         


iptables -xvnL -t mangle
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination

proxmox232

Code: 
iptables -xvnL
Chain INPUT (policy ACCEPT 18917898 packets, 6248813049 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 14828622 packets, 12354277123 bytes)
    pkts      bytes target     prot opt in     out     source               destination   

iptables -xvnL -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         


iptables -xvnL -t mangle
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination

interfaces on node2 (proxmox232) (ipv6 disable on this host)

Code: 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
    link/ether 3c:7c:3f:d9:92:ec brd ff:ff:ff:ff:ff:ff
3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3c:7c:3f:d9:92:ec brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.232/24 scope global vmbr0
       valid_lft forever preferred_lft forever
4: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether e6:ea:7d:b0:78:a1 brd ff:ff:ff:ff:ff:ff

I think you mean 8007 port (where pbs listen) for tcpdump

first screenshot
tcpdump run on node2 - proxmox232 (no one packet out 8-( to network)

second screenshot - i try telnet - works fine
 

Attachments

  • screenshot-2021-07-23_15-51-20.png
    screenshot-2021-07-23_15-51-20.png
    222.8 KB · Views: 14
  • screenshot-2021-07-23_15-55-49.png
    screenshot-2021-07-23_15-55-49.png
    198.5 KB · Views: 13
in your first post you attempt to scan the same IP (192.168.122.231), but provide two different fingerprints. the second scan likely fails because the fingerprint is wrong?
 
no no no (my mistake when copying, but in this case does not matter)

if wrong fingerprint - other error message
for example
first node (proxmox231)
Code: 
pvesm scan pbs 192.168.122.231 root@pam --password xxxxxxxxxxxxxxxx --fingerprint "D4:BC:7A:AF:35:52:B7:26:DF:E3:43:F5:E4:60:69:FA:93:2C:34:DE:15:EC:4D:D3:36:2D:EB:CA:3B:B5:CC:9E"
┌────────┬─────────┐
│ store  │ comment │
╞════════╪═════════╡
│ backup │         │
└────────┴─────────┘

pvesm scan pbs 192.168.122.231 root@pam --password xxxxxxxxxx --fingerprint "D4:BC:7A:AF:35:52:B7:26:DF:E3:43:F5:E4:60:69:FA:93:2C:34:DE:15:EC:4D:D3:36:2D:EB:CA:3B:B5:CC:AA"
error fetching datastores - fingerprint 'D4:BC:7A:AF:35:52:B7:26:DF:E3:43:F5:E4:60:69:FA:93:2C:34:DE:15:EC:4D:D3:36:2D:EB:CA:3B:B5:CC:9E' not verified, abort!

second node - proxmox232
Code: 
pvesm scan pbs 192.168.122.231 root@pam --password xxxxxxxxxxxx --fingerprint "D4:BC:7A:AF:35:52:B7:26:DF:E3:43:F5:E4:60:69:FA:93:2C:34:DE:15:EC:4D:D3:36:2D:EB:CA:3B:B5:CC:9E"
error fetching datastores - 500 Can't connect to 192.168.122.231:8007
 
can you try connecting with the plain 'proxmox-backup-client' while having tcpdump -i any port 8007 running on both nodes?
 
proxmox-backup-client works

Code: 
proxmox-backup-client login --repository root@pam@192.168.122.231:backup
Password for "root@pam": ***************
fingerprint: d4:bc:7a:af:35:52:b7:26:df:e3:43:f5:e4:60:69:fa:93:2c:34:de:15:ec:4d:d3:36:2d:eb:ca:3b:b5:cc:9e
Are you sure you want to continue connecting? (y/n): y
fingerprint: d4:bc:7a:af:35:52:b7:26:df:e3:43:f5:e4:60:69:fa:93:2c:34:de:15:ec:4d:d3:36:2d:eb:ca:3b:b5:cc:9e
Are you sure you want to continue connecting? (y/n): y
Code: 
proxmox-backup-client backup pve.pxar:/etc/
Starting backup: host/proxmox232/2021-07-26T16:04:15Z
Client name: proxmox232
Starting backup protocol: Mon Jul 26 21:04:15 2021
No previous manifest available.
Upload directory '/etc/' to 'root@pam@192.168.122.231:8007:backup' as pve.pxar.didx
skipping mount point: "pve"
pve.pxar: had to backup 2.16 MiB of 2.16 MiB (compressed 499.93 KiB) in 0.10s
pve.pxar: average backup speed: 22.49 MiB/s
Uploaded backup catalog (29.08 KiB)
Duration: 0.12s
End Time: Mon Jul 26 21:04:15 2021
 
and if you configure a PBS storage via node1, is it then active on node2 (e.g., check pvesm status)?
 
pveversion -v does that show any difference between the nodes?
 
could you post the output form both nodes?
 
proxmox231
Code: 
proxmox-ve: 7.0-2 (running kernel: 5.11.22-3-pve)
pve-manager: 7.0-10 (running version: 7.0-10/d2f465d3)
pve-kernel-5.11: 7.0-6
pve-kernel-helper: 7.0-6
pve-kernel-5.11.22-3-pve: 5.11.22-6
pve-kernel-5.11.22-2-pve: 5.11.22-4
ceph-fuse: 14.2.21-1
corosync: 3.1.2-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.21-pve1
libproxmox-acme-perl: 1.2.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-5
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-9
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.7-1
proxmox-backup-file-restore: 2.0.7-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.0-8
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-2
pve-firmware: 3.2-4
pve-ha-manager: 3.3-1
pve-i18n: 2.4-1
pve-qemu-kvm: 6.0.0-2
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-11
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1

proxmox232
Code: 
proxmox-ve: 7.0-2 (running kernel: 5.11.22-3-pve)
pve-manager: 7.0-10 (running version: 7.0-10/d2f465d3)
pve-kernel-5.11: 7.0-6
pve-kernel-helper: 7.0-6
pve-kernel-5.11.22-3-pve: 5.11.22-6
pve-kernel-5.11.22-2-pve: 5.11.22-4
ceph-fuse: 14.2.21-1
corosync: 3.1.2-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.21-pve1
libproxmox-acme-perl: 1.2.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-5
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-9
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.7-1
proxmox-backup-file-restore: 2.0.7-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.0-8
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-2
pve-firmware: 3.2-4
pve-ha-manager: 3.3-1
pve-i18n: 2.4-1
pve-qemu-kvm: 6.0.0-2
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-11
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1
 
upgrade path for this nodes different

proxmox231 (setup debian10, setup pve 6.4, upgrade ifupdown -> ifupdown2, upgrade 6.4 -> 7.0, setup pbs)
proxmox232 (setup debian10, pve6.4, upgrade 6.4 -> 7.0, upgrade ifupdown -> ifupdown2)
 
Last edited:
since PBS client itself seems to work, let's try the perl-based API client that is used for the scan and activation check:

Code: 
use strict;
use warnings;
use Data::Dumper;
use PVE::APIClient::LWP;

my $params = {
    username => 'root@pam'
    password => 'CHANGEME',
};

my $server = '192.168.122.231';

my $conn = PVE::APIClient::LWP->new(
    %$params,
    host => $server,
    port => 8007,
    timeout => 7,
    cookie_name => 'PBSAuthCookie',
    manual_verification => 1,
    ssl_opts => { verify_hostname => 0 },
);

my $response = $conn->get('/api2/json/admin/datastore', {});
print Dumper($response), "\n";

on proxmox 232, put this into a file (with CHANGEME replaced by your password), make it executable (chmod +x FILE) and then run it (./FILE). it should either print an error, or a list of datastores as json.
 
add
print Dumper($conn);
after constructor

Code: 
$VAR1 = bless( {
                 'username' => 'root@pam',
                 'manual_verification' => 1,
                 'port' => 8007,
                 'host' => '192.168.122.231',
                 'timeout' => 7,
                 'useragent' => bless( {
                                         'no_proxy' => [],
                                         'use_eval' => 1,
                                         'requests_redirectable' => [
                                                                      'GET',
                                                                      'HEAD'
                                                                    ],
                                         'conn_cache' => bless( {
                                                                  'cc_limit_total' => 50,
                                                                  'cc_conns' => []
                                                                }, 'LWP::ConnCache' ),
                                         'max_size' => undef,
                                         'def_headers' => bless( {
                                                                   'user-agent' => 'libwww-perl/6.52',
                                                                   'accept-encoding' => 'gzip'
                                                                 }, 'HTTP::Headers' ),
                                         'ssl_opts' => {
                                                         'SSL_verify_mode' => 1,
                                                         'verify_hostname' => 0,
                                                         'SSL_verify_callback' => sub { "DUMMY" }
                                                       },
                                         'proxy' => {},
                                         'local_address' => undef,
                                         'protocols_allowed' => [
                                                                  'http',
                                                                  'https'
                                                                ],
                                         'send_te' => 1,
                                         'max_redirect' => 7,
                                         'show_progress' => undef,
                                         'handlers' => {
                                                         'response_header' => bless( [
                                                                                       {
                                                                                         'callback' => sub { "DUMMY" },
                                                                                         'm_media_type' => 'html',
                                                                                         'line' => '/usr/share/perl5/LWP/UserAgent.pm:768',
                                                                                         'owner' => 'LWP::UserAgent::parse_head'
                                                                                       }
                                                                                     ], 'HTTP::Config' )
                                                       },
                                         'protocols_forbidden' => undef,
                                         'timeout' => 7
                                       }, 'LWP::UserAgent' ),
                 'fingerprint' => {
                                    'cache' => {},
                                    'last_unknown' => undef
                                  },
                 'protocol' => 'https',
                 'password' => 'xxxxxxxxxxxxxxxx',
                 'cookie_name' => 'PBSAuthCookie',
                 'register_fingerprint_cb' => undef
               }, 'PVE::APIClient::LWP' );
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back