PVE Metrics with Graphite

jpop · Sep 10, 2022

Hi All,

I created a small container to run Graphite for collecting PVE stats. Everything runs, except I am getting an Operation Not Permitted when sending metrics to graphite.

Bash:

root@toxuthat:~# systemctl status pvestatd.service
● pvestatd.service - PVE Status Daemon
     Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2022-09-10 07:53:08 PDT; 50s ago
    Process: 840581 ExecStart=/usr/bin/pvestatd start (code=exited, status=0/SUCCESS)
   Main PID: 840582 (pvestatd)
      Tasks: 1 (limit: 154385)
     Memory: 80.3M
        CPU: 1.223s
     CGroup: /system.slice/pvestatd.service
             └─840582 pvestatd

Sep 10 07:53:38 toxuthat pvestatd[840582]: lxc status update error: metrics send error 'graphite': failed to send metrics: Operation not permitted
Sep 10 07:53:38 toxuthat pvestatd[840582]: metrics send error 'graphite': failed to send metrics: Operation not permitted
Sep 10 07:53:48 toxuthat pvestatd[840582]: node status update error: metrics send error 'graphite': failed to send metrics: Operation not permitted
Sep 10 07:53:48 toxuthat pvestatd[840582]: qemu status update error: metrics send error 'graphite': failed to send metrics: Operation not permitted
Sep 10 07:53:48 toxuthat pvestatd[840582]: lxc status update error: metrics send error 'graphite': failed to send metrics: Operation not permitted
Sep 10 07:53:49 toxuthat pvestatd[840582]: metrics send error 'graphite': failed to send metrics: Operation not permitted
Sep 10 07:53:58 toxuthat pvestatd[840582]: node status update error: metrics send error 'graphite': failed to send metrics: Operation not permitted
Sep 10 07:53:58 toxuthat pvestatd[840582]: qemu status update error: metrics send error 'graphite': failed to send metrics: Operation not permitted
Sep 10 07:53:58 toxuthat pvestatd[840582]: lxc status update error: metrics send error 'graphite': failed to send metrics: Operation not permitted
Sep 10 07:53:58 toxuthat pvestatd[840582]: metrics send error 'graphite': failed to send metrics: Operation not permitted

Here is my status.cfg:

Bash:

graphite:
  server 192.168.10.79
  path proxmox
  port 2003

Here are my versions:

Bash:

proxmox-ve: 7.2-1 (running kernel: 5.15.39-4-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-9
pve-kernel-helper: 7.2-9
pve-kernel-5.15.39-4-pve: 5.15.39-4
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-8
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.5-1
proxmox-backup-file-restore: 2.2.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-2
pve-docs: 7.2-2
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-5
pve-firmware: 3.5-1
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.0.0-2
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-4
smartmontools: 7.2-pve3

Is there something I am missing? What other logs could I check to troubleshoot?

Regards,
Jpop

jpop · Sep 10, 2022

jpop said:

Hi All,

I created a small container to run Graphite for collecting PVE stats. Everything runs, except I am getting an Operation Not Permitted when sending metrics to graphite.

Bash:

root@toxuthat:~# systemctl status pvestatd.service
● pvestatd.service - PVE Status Daemon
     Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2022-09-10 07:53:08 PDT; 50s ago
    Process: 840581 ExecStart=/usr/bin/pvestatd start (code=exited, status=0/SUCCESS)
   Main PID: 840582 (pvestatd)
      Tasks: 1 (limit: 154385)
     Memory: 80.3M
        CPU: 1.223s
     CGroup: /system.slice/pvestatd.service
             └─840582 pvestatd

Sep 10 07:53:38 toxuthat pvestatd[840582]: lxc status update error: metrics send error 'graphite': failed to send metrics: Operation not permitted
Sep 10 07:53:38 toxuthat pvestatd[840582]: metrics send error 'graphite': failed to send metrics: Operation not permitted
Sep 10 07:53:48 toxuthat pvestatd[840582]: node status update error: metrics send error 'graphite': failed to send metrics: Operation not permitted
Sep 10 07:53:48 toxuthat pvestatd[840582]: qemu status update error: metrics send error 'graphite': failed to send metrics: Operation not permitted
Sep 10 07:53:48 toxuthat pvestatd[840582]: lxc status update error: metrics send error 'graphite': failed to send metrics: Operation not permitted
Sep 10 07:53:49 toxuthat pvestatd[840582]: metrics send error 'graphite': failed to send metrics: Operation not permitted
Sep 10 07:53:58 toxuthat pvestatd[840582]: node status update error: metrics send error 'graphite': failed to send metrics: Operation not permitted
Sep 10 07:53:58 toxuthat pvestatd[840582]: qemu status update error: metrics send error 'graphite': failed to send metrics: Operation not permitted
Sep 10 07:53:58 toxuthat pvestatd[840582]: lxc status update error: metrics send error 'graphite': failed to send metrics: Operation not permitted
Sep 10 07:53:58 toxuthat pvestatd[840582]: metrics send error 'graphite': failed to send metrics: Operation not permitted

Here is my status.cfg:

Bash:

graphite:
  server 192.168.10.79
  path proxmox
  port 2003

Here are my versions:

Bash:

proxmox-ve: 7.2-1 (running kernel: 5.15.39-4-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-9
pve-kernel-helper: 7.2-9
pve-kernel-5.15.39-4-pve: 5.15.39-4
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-8
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.5-1
proxmox-backup-file-restore: 2.2.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-2
pve-docs: 7.2-2
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-5
pve-firmware: 3.5-1
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.0.0-2
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-4
smartmontools: 7.2-pve3

Is there something I am missing? What other logs could I check to troubleshoot?

Regards,
Jpop

Never mind. It was my firewall. Talk about biting me where it hurts.

nada · Nov 8, 2022

hi Jpop
Error may be caused by carbon. Is it running at your graphite CT ?
you may have something like this

root@graphite:~# /opt/graphite/bin/carbon-cache.py start

root@graphite:~# pstree
systemd-+-accounts-daemon---2*[{accounts-daemon}]
|-3*[agetty]
|-apache2-+-2*[apache2---26*[{apache2}]]
| `-5*[apache2---7*[{apache2}]]
|-carbon-cache.py---2*[{carbon-cache.py}]
|-collectd---11*[{collectd}]
|-cron
|-dbus-daemon
|-master-+-pickup
| `-qmgr
|-memcached---9*[{memcached}]
|-networkd-dispat---{networkd-dispat}
|-rsyslogd---2*[{rsyslogd}]
|-sshd---sshd---bash---pstree
|-systemd---(sd-pam)
|-systemd-journal
|-systemd-logind
|-systemd-network
|-systemd-resolve
`-uuidd

root@graphite:~# la /opt/graphite/storage/whisper/proxmox/
total 19
drwxr-xr-x 6 root root 6 Nov 7 17:45 .
drwxr-xr-x 5 root root 5 Nov 7 17:45 ..
drwxr-xr-x 44 root root 44 Nov 7 18:15 lxc
drwxr-xr-x 7 root root 7 Nov 7 17:54 nodes
drwxr-xr-x 15 root root 15 Nov 7 18:02 qemu
drwxr-xr-x 7 root root 7 Nov 7 17:45 storages

Search

Search

PVE Metrics with Graphite

jpop

New Member

jpop

New Member

nada

Member