Search results

I checked systemctl list-units on all hosts and don't see anything that does not belong, you mentioned directory, what should I look for there? Where does that GUI pannel pull its information?

Not sure if it is because I am in the middle of rebalancing, but when I restart OSDs I get a bunch of "active+undersized+degraded+remapped+backfill_wait:" PGs, so I stoped restarting them.

All say: Host: virt01 Address: Unknown

I upgraded ceph on my 21 node cluster from 14.2.15 to 14.2.20 and restarted all services except OSDs. I am using dual 40 gig ethernet and I was seeing about 1.8 GB/s on rebalancing, but now I am seeing less than 100 MB/s. CephFS has dropped to an embarrassing 61.5 MB/s with fio. Jobs: 1 (f=1)...

When I go to my dashboard I see two virt01: in ceph Monitors, Managers and Meta Data Servers, I would like to get rid of the one with the ?, but not sure how this window is populated. My ceph.conf looks normal as far as I can tell: [global] auth_client_required = none...

Ran into the same issue, got hacked in the process, having all your firewalls die is NOT a good thing.

This is very annoying, I have over 100 VMs spamming my logs. I really don't want to restart them all.

The best I have seen is 23.6 Gbits/sec with iperf3. Setup is E5-2690V2 with ConnectX-3 MCX354A-FCAT QSFP DAC into Cisco Nexus 3132Q 9000 Byte MTU. - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 13.7 GBytes...

I use two Cisco Nexus 3131Q off eBay, I got them for $400 each, but it looks like now the lowest price is about $700 each. Still not a bad price for 40 Gig 32 QSFP ports...

Thank you very much for your reply, so it sounds like bonding two 40 gbit/s interfaces and not using RDMA is my best bet.

I am upgrading a 16 node cluster that has 2 NVMe drives and 3 SATA drives used for ceph. My network cards are Mellanox MCX354A-FCBT and have 2 QSFP ports that can be configured as Infiniband or Ethernet. My question is how best should I utilize the two ports. My options are: 1) LACP into VPC...

Yep, I was able to fix my system with rmmod mlx4_ib and systemctl restart networking

You rock!

Running latest proxmox, everyting looks like its working, but I noticed that on all 16 hosts pvesr.service is starting every minute. syslog:Dec 28 18:35:00 virt0 systemd[1]: pvesr.service: Succeeded. syslog:Dec 28 18:36:00 virt0 systemd[1]: pvesr.service: Succeeded. syslog:Dec 28 18:37:00 virt0...

Yes, that is true, but my SATA drives don't support them either. I find it hard to believe that the fact that they are not Pro makes them slower than spinning disks.

I am running the latest version of proxmox on a 16 node 40 gbe cluster. Each node has 2 Samsung 960 EVO 250GB NVMe SSDs and 3 Hitachi 2 TB 7200 RPM Ultrastar disks. I am using bluestore for all disks with two crush rules, one for fast nvme and slow for hdd. I have tested bandwidth between all...

Yes, thanks!

You rock my world! I pulled 1 of each of the two links and everything is fast again! So, what should I change on the cisco or proxmox side to get both links?

switch# show port-channel load-balance Port Channel Load-Balancing Configuration: System: source-dest-ip Port Channel Load-Balancing Addresses Used Per-Protocol: Non-IP: source-dest-mac IP: source-dest-ip I think the biggest issue is the 0 cur/MB/s root@virt0:~# rados -p fast bench 120...

allow-vmbr0 bond0 iface bond0 inet manual ovs_bonds enp129s0 enp129s0d1 ovs_type OVSBond ovs_bridge vmbr0 ovs_options lacp=active bond_mode=balance-tcp mtu 9000 jumbo frames tested with ping -s 9000 between all 12 nodes root@virt11:~# ovs-appctl bond/show bond0 ---- bond0...