Ceph Issue

bit2gb · Oct 31, 2019

I have 4 nodes with latest version of proxmox:

Kernel Version

Linux 5.0.18-1-pve #1 SMP PVE 5.0.18-2 (Fri, 2 Aug 2019 14:51:00 +0200)

PVE Manager Version

pve-manager/6.0-5/f8a710d7

On one of them from 1 week (so after 1 month after i upgrade all nodes) all win vms run very very slow at simple commands like open file explorer or total commander.
Into syslog i see:

mon.pvem2@3(peon).paxos(paxos updating c 5976311..5976890) lease_expire from mon.0 v2:10.14.6.162:3300/0 is 6.853901 seconds in the past; mons are probably laggy (or possibly clocks are too skewed)
Oct 31 17:00:51 pvem2 ceph-mon[1335]: 2019-10-31 17:00:51.852 7fcacfa03700 -1 mon.pvem2@3(peon).paxos(paxos updating c 5976311..5976891) lease_expire from mon.0 v2:10.14.6.162:3300/0 is 6.853172 seconds in the past; mons are probably laggy (or possibly clocks are too skewed)
Oct 31 17:00:52 pvem2 ceph-mon[1335]: 2019-10-31 17:00:52.456 7fcad2208700 -1 mon.pvem2@3(peon) e4 get_health_metrics reporting 1 slow ops, oldest is log(1 entries from seq 1 at 2019-10-31 16:07:43.269851)
Oct 31 17:00:53 pvem2 ceph-mon[1335]: 2019-10-31 17:00:53.520 7fcacfa03700 -1 mon.pvem2@3(peon).paxos(paxos updating c 5976311..5976892) lease_expire from mon.0 v2:10.14.6.162:3300/0 is 6.853575 seconds in the past; mons are probably laggy (or possibly clocks are too skewed)
Oct 31 17:00:55 pvem2 ceph-mon[1335]: 2019-10-31 17:00:55.508 7fcacfa03700 -1 mon.pvem2@3(peon).paxos(paxos updating c 5976311..5976893) lease_expire from mon.0 v2:10.14.6.162:3300/0 is 6.852882 seconds in the past; mons are probably laggy (or possibly clocks are too skewed)
Oct 31 17:00:56 pvem2 ceph-mon[1335]: 2019-10-31 17:00:56.540 7fcacfa03700 -1 mon.pvem2@3(peon).paxos(paxos updating c 5976311..5976894) lease_expire from mon.0 v2:10.14.6.162:3300/0 is 6.853588 seconds in the past; mons are probably laggy (or possibly clocks are too skewed)
Oct 31 17:00:57 pvem2 ceph-mon[1335]: 2019-10-31 17:00:57.456 7fcad2208700 -1 mon.pvem2@3(peon) e4 get_health_metrics reporting 1 slow ops, oldest is log(1 entries from seq 1 at 2019-10-31 16:07:43.269851)

What is the problem here?
I have 256 GB of ram and 32 cores

proxmox-ve: 6.0-2 (running kernel: 5.0.18-1-pve)
pve-manager: 6.0-5 (running version: 6.0-5/f8a710d7)
pve-kernel-5.0: 6.0-6
pve-kernel-helper: 6.0-6
pve-kernel-4.15: 5.4-7
pve-kernel-5.0.18-1-pve: 5.0.18-2
pve-kernel-4.15.18-19-pve: 4.15.18-45
pve-kernel-4.15.17-1-pve: 4.15.17-9
ceph: 14.2.1-pve2
ceph-fuse: 14.2.1-pve2
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.10-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-3
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-7
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-61
lxcfs: 3.0.3-pve60
novnc-pve: 1.0.0-60
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-5
pve-cluster: 6.0-4
pve-container: 3.0-5
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-6
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-2
pve-qemu-kvm: 4.0.0-5
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-7
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.1-pve1

Thanks,
Alin.

Alwin · Oct 31, 2019

mon.pvem2@3(peon).paxos(paxos updating c 5976311..5976890) lease_expire from mon.0 v2:10.14.6.162:3300/0 is 6.853901 seconds in the past; mons are probably laggy (or possibly clocks are too skewed)

Check that all servers have the same time synchronized timedatectl and maybe restart them if the time is good.

bit2gb · Nov 1, 2019

Hi,

Is same output for all of them

root@pvem2:~# timedatectl
Local time: Fri 2019-11-01 10:15:15 EET
Universal time: Fri 2019-11-01 08:15:15 UTC
RTC time: Fri 2019-11-01 08:15:16
Time zone: Europe/Bucharest (EET, +0200)
System clock synchronized: no
NTP service: active
RTC in local TZ: no

root@pvem2:~# timedatectl
Local time: Fri 2019-11-01 10:15:15 EET
Universal time: Fri 2019-11-01 08:15:15 UTC
RTC time: Fri 2019-11-01 08:15:16
Time zone: Europe/Bucharest (EET, +0200)
System clock synchronized: no
NTP service: active
RTC in local TZ: no

Alwin · Nov 4, 2019

System clock synchronized: no

The time is not synchronized wit an NTP.
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#_time_synchronization

bit2gb · Nov 6, 2019

And can be this the cause of slownes on that node?

bit2gb · Nov 8, 2019

Some update?

Alwin · Nov 8, 2019

bit2gb said:
And can be this the cause of slownes on that node?

Yes.

bit2gb · Nov 11, 2019

Ok And what i can do? because on other servers default config works fine.

Thanks

Alwin · Nov 11, 2019

bit2gb said:
Ok And what i can do? because on other servers default config works fine.

For Ceph it doesn't matter if the time is correct. It just needs to be the same for the whole cluster. Use a local time server and let all cluster nodes sync from this local source.
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#_time_synchronization

Once they are all in sync the "slowness" should disappear. The log messages (first post) should also vanish.

bit2gb · Nov 12, 2019

I set same ntp server for all of them but slowness still exist only on that node with only 1 vm on 256gb ram and ssd intel so it's not from hardware.
VM have same tamplate like on other nodes and works fine for a period of time.
What i can do?

Thanks

Alwin · Nov 18, 2019

What does timedatectl show on all the nodes in the cluster?

bit2gb · Nov 19, 2019

root@pvem2:~# timedatectl
Local time: Tue 2019-11-19 10:33:48 EET
Universal time: Tue 2019-11-19 08:33:48 UTC
RTC time: Tue 2019-11-19 08:34:09
Time zone: Europe/Bucharest (EET, +0200)
System clock synchronized: no
NTP service: active
RTC in local TZ: no
root@pvem2:~#

Same output on all of them

Alwin · Nov 19, 2019

bit2gb said:
System clock synchronized: no

The time is still not synced. Did you restart the systemd-timesyncd.service?

Code:

root@pve6ceph01:~# timedatectl
               Local time: Tue 2019-11-19 10:29:13 CET
           Universal time: Tue 2019-11-19 09:29:13 UTC
                 RTC time: Tue 2019-11-19 09:29:14
                Time zone: Europe/Vienna (CET, +0100)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

This is an example of my cluster.

bit2gb · Nov 20, 2019

the problem is i cant set on yes System clock sync.

i tried:
systemctl restart systemd-timesyncd.service
timedatectl set-ntp true
systemctl restart systemd-timesyncd

content from timesyncd.conf:

[Time]
NTP=0.ro.pool.ntp.org 1.ro.pool.ntp.org 2.ro.pool.ntp.org 3.ro.pool.ntp.org
FallbackNTP=0.debian.pool.ntp.org 1.debian.pool.ntp.org 2.debian.pool.ntp.org 3.debian.pool.ntp.org
RootDistanceMaxSec=5
PollIntervalMinSec=32
PollIntervalMaxSec=2048

How can i enable it?

Thanks

Alwin · Nov 20, 2019

bit2gb said:
NTP=0.ro.pool.ntp.org

Configure only one, so all nodes sync from the same NTP. Please also use a local NTP, as the ntp pool will use different servers to get the time. So the time will drift more/less every time.

What is the service status showing, systemctl status systemd-timesyncd.service -l?

bit2gb · Nov 22, 2019

● systemd-timesyncd.service - Network Time Synchronization
Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
Drop-In: /lib/systemd/system/systemd-timesyncd.service.d
└─disable-with-time-daemon.conf
Active: inactive (dead) since Wed 2019-11-20 15:25:31 EET; 1 day 20h ago
Condition: start condition failed at Fri 2019-11-22 11:29:31 EET; 2s ago
└─ ConditionFileIsExecutable=!/usr/sbin/ntpd was not met
Docs: man:systemd-timesyncd.service(8)
Main PID: 3079765 (code=exited, status=0/SUCCESS)
Status: "Shutting down..."

Nov 20 15:25:31 pvem1 systemd[1]: Stopping Network Time Synchronization...
Nov 20 15:25:31 pvem1 systemd[1]: systemd-timesyncd.service: Succeeded.
Nov 20 15:25:31 pvem1 systemd[1]: Stopped Network Time Synchronization.
Nov 20 15:26:43 pvem1 systemd[1]: Condition check resulted in Network Time Synchronization being skipped.
Nov 20 15:27:13 pvem1 systemd[1]: Condition check resulted in Network Time Synchronization being skipped.
Nov 20 15:27:33 pvem1 systemd[1]: Condition check resulted in Network Time Synchronization being skipped.
Nov 20 15:27:36 pvem1 systemd[1]: Condition check resulted in Network Time Synchronization being skipped.
Nov 22 11:29:27 pvem1 systemd[1]: Condition check resulted in Network Time Synchronization being skipped.
Nov 22 11:29:30 pvem1 systemd[1]: Condition check resulted in Network Time Synchronization being skipped.
Nov 22 11:29:31 pvem1 systemd[1]: Condition check resulted in Network Time Synchronization being skipped.

Alwin · Nov 22, 2019

bit2gb said:
└─ ConditionFileIsExecutable=!/usr/sbin/ntpd was not met

You are running another ntp daemon. Either configure that daemon or remove it.

bit2gb · Dec 3, 2019

How i can do that?

Search

Search

Ceph Issue

bit2gb

Well-Known Member

Alwin

Proxmox Retired Staff

bit2gb

Well-Known Member

Alwin

Proxmox Retired Staff

bit2gb

Well-Known Member

bit2gb

Well-Known Member

Alwin

Proxmox Retired Staff

bit2gb

Well-Known Member

Alwin

Proxmox Retired Staff

bit2gb

Well-Known Member

Alwin

Proxmox Retired Staff

bit2gb

Well-Known Member

Alwin

Proxmox Retired Staff

bit2gb

Well-Known Member

Alwin

Proxmox Retired Staff

bit2gb

Well-Known Member

Alwin

Proxmox Retired Staff

bit2gb

Well-Known Member