[SOLVED] Node crashes and resets without log entry + RRD update error

[def1ant]

Hey there,

im not sure if the two issues are related - if not, i will change the thread title accordingly!

So my little proxmox homeserver keeps crashing after a migration to a different motherboard without leaving any traces in the syslog why.
It only happens if you start a larger file transfer, otherwise the node will run stable 24/7.

I'm running Virtual Environment 6.1-8 on a AMD e-350 based embedded board. This installation of proxmox was originally installed on an intel server board, wich was replaced due to high power consumption.
I was pleasently surprised that everything (except the network card, wich was to be expected) worked fine one the new board, including my 5 hard disk ZFS array.
So i just plugged the system ssd and my 5 hard disks to the new board and started it.
This may very well be the reason for my problems, and the safest way would be a fresh install of proxmox. But if possible, i'd like to avoid that.

So the only thing the syslog is offering is:

Apr 26 13:12:09 pve pmxcfs[863]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/pve: mmaping file '/var/lib/rrdcached/db/pve2-node/pve': Invalid argument
Apr 26 13:12:09 pve pmxcfs[863]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local: mmaping file '/var/lib/rrdcached/db/pve2-storage/pve/local': Invalid argument
Apr 26 13:12:09 pve pmxcfs[863]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local-lvm: mmaping file '/var/lib/rrdcached/db/pve2-storage/pve/local-lvm': Invalid argument
Apr 26 13:12:19 pve pmxcfs[863]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/pve: mmaping file '/var/lib/rrdcached/db/pve2-node/pve': Invalid argument
Apr 26 13:12:19 pve pmxcfs[863]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local-lvm: mmaping file '/var/lib/rrdcached/db/pve2-storage/pve/local-lvm': Invalid argument
Apr 26 13:12:19 pve pmxcfs[863]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local: mmaping file '/var/lib/rrdcached/db/pve2-storage/pve/local': Invalid argument
Apr 26 13:12:30 pve pmxcfs[863]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/pve: mmaping file '/var/lib/rrdcached/db/pve2-node/pve': Invalid argument
Apr 26 13:12:30 pve pmxcfs[863]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local: mmaping file '/var/lib/rrdcached/db/pve2-storage/pve/local': Invalid argument
Apr 26 13:12:30 pve pmxcfs[863]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local-lvm: mmaping file '/var/lib/rrdcached/db/pve2-storage/pve/local-lvm': Invalid argument
Apr 26 13:12:39 pve pmxcfs[863]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/pve: mmaping file '/var/lib/rrdcached/db/pve2-node/pve': Invalid argument
Apr 26 13:12:39 pve pmxcfs[863]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local-lvm: mmaping file '/var/lib/rrdcached/db/pve2-storage/pve/local-lvm': Invalid argument
Apr 26 13:12:39 pve pmxcfs[863]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local: mmaping file '/var/lib/rrdcached/db/pve2-storage/pve/local': Invalid argument
Apr 26 13:12:49 pve pmxcfs[863]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/pve: mmaping file '/var/lib/rrdcached/db/pve2-node/pve': Invalid argument
Apr 26 13:12:50 pve pmxcfs[863]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local-lvm: mmaping file '/var/lib/rrdcached/db/pve2-storage/pve/local-lvm': Invalid argument
Apr 26 13:12:50 pve pmxcfs[863]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local: mmaping file '/var/lib/rrdcached/db/pve2-storage/pve/local': Invalid argument
Apr 26 13:12:59 pve pmxcfs[863]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/pve: mmaping file '/var/lib/rrdcached/db/pve2-node/pve': Invalid argument
Apr 26 13:12:59 pve pmxcfs[863]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local: mmaping file '/var/lib/rrdcached/db/pve2-storage/pve/local': Invalid argument
Apr 26 13:12:59 pve pmxcfs[863]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/pve/local-lvm: mmaping file '/var/lib/rrdcached/db/pve2-storage/pve/local-lvm': Invalid argument

This is spammed constantly. The graphs on the summary page appear to be working though.
I've tried restarting rddcached and flushed using "rddcached -P FLUSH", no change.

The files its looking for seem to exist, but are empty...?


On proxmox i am running a couple of vms. The only one constantly active is openmediavault.

The system runs stable, until you start a larger file transfer to the storage ZFS.
Then the whole node hardresets after two minutes of transfer or so, without any log entries.

Currently the node has 4gb of ram, 3 of wich are allocated to the omv vm. As it looked like the system would crash as soon as the ram fully fills up i ran a memorytest86 over night - no errors detected.


Any help would be greatly apreciated!
Nice sunday to everyone.

Cheers
Eric

 

[oguz]

hi,

The system runs stable, until you start a larger file transfer to the storage ZFS.

i'd guess you're running out of memory. zfs uses a lot of memory alone, your system has 4G and 3 is allocated to VM. PVE also needs some amount of ram to function.

your choices are basically upgrading your ram or using a different storage instead of zfs.

 

[def1ant]

Thanks for your reply!
That was also my working theory atm - more ram is currently on the way.
I will update if the additional ram helped.

Cheers

 

[def1ant]

@oguz After upgrading the RAM to 8gb the system runs stable again.
Thank you very much!

 

[oguz]

you're welcome, you can mark the thread solved so others can know what to expect!