Search results

Wir müssen an den Servern hardwareseitige Änderungen vornehmen, die das erforderlich machen.

Vielen Dank schon mal für den Hinweis. Wenn ich mir die Ceph Doku diesbzgl. so ansehe, sollte man in diesem Fall also so vorgehen: 1) Hinzufügen neuer Mons mit den neuen IPs 2) Entfernen der alten Mons 3) Anpassen der ceph.conf, damit die clients die neuen mons finden Über die Proxmox GUI geht...

Hallo zusammen, wir betreiben einen Proxmox VE (8.1.4) / Ceph (18.2.1) Cluster bestehend aus 3 Nodes. Die aktuelle Ceph Config sieht so aus: [global] auth_client_required = none auth_cluster_required = none auth_service_required = none cephx_sign_messages = false...

@Max Carrara Thx a lot for this finding in the docs. Now it gets clearer why there is some space "FREE" but "No space left". As can be read, in the meantime we were able to initially free enough space to make GC Job run again (which is still running at the time of writing).

This PBS is our Backup Target for all productive / critical VMs of a connected productive PVE Cluster. So as we urgently need it running again, and no one seems to have an idea we did the following in the meantime: -> Moved away ~30-40 of the oldest directories in ".chunks" to another FS and so...

I just noticed this: root@pbs:/mnt/datastore/backup# df -i Filesystem Inodes IUsed IFree IUse% Mounted on backup 6828558 6828558 0 100% /mnt/datastore/backup Could this be an explanation? I'm wondering if / that ZFS really also has inodes like ext4...

Nobody with an idea?

Nope: root@pbs:~# zfs list -t all NAME USED AVAIL REFER MOUNTPOINT backup 14.4T 0B 14.4T /mnt/datastore/backup root@pbs:~#

root@pbs:~# zfs list NAME USED AVAIL REFER MOUNTPOINT backup 14.4T 0B 14.4T /mnt/datastore/backup root@pbs:~# zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT backup 14.5T 14.4T 125G - - 89% 99% 1.00x...

After googling around, another thing we tried was this: -> copy an old dir under .chunks to another FS -> "cat /dev/null >..." to all files in that dir -> rm all files in that dir Didn't help either, FS remains 100% full... as a plus what we noticed and is even more confusing: Before this step...

Hi everybody, at a PBS the backup storage, which is a ZFS, is completely full, so the GC Job fails: In the Server all physical ports are used so we can't simply add more HDDs for extending the pool. In theory we just need some few megs of space so the GC Job can do it's work.. we even tried...

So for finishing this thread, here the current situation: First of all, we never really found out, what was going wrong in the Proxmox/Ceph Cluster. What we did till today is this: Took an older HP Proliant Server, installed Proxmox on it Migrated all VMs and their data to this single Server...

Sure, here it is: # fio --ioengine=libaio --filename=/dev/nvme0n1 --direct=1 --sync=1 --rw=randwrite --bs=4K --numjobs=1 --iodepth=1 --runtime=600 --time_based --name=fio fio: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.25 Starting...

Of course: # fio --ioengine=libaio --filename=/dev/nvme0n1 --direct=1 --sync=1 --rw=randwrite --bs=4K --numjobs=1 --iodepth=1 --runtime=60 --time_based --name=fio fio: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.25 Starting 1...

We are going to in the next days, waiting for arrival... There is no newer one, we already checked that a while ago during investigations That would be sooo nice :S Thx

Yes, we already postet this early: I would be completely with you. But as we assumes sth. like that too, we freed the NVMEs, removed /wiped them from ceph and - at the moment - just added the 2TB ones back to ceph. So IMHO they are now completely empty, but the fio tests are as bad as...

Another question: What about network latencies? We found this: https://www.cubewerk.de/2020/10/23/ceph-performance-guide-2020-for-ssd-nvme/ There one can read: What can affect the overall performance of a ceph-cluster? Slow network (latency!), bad/slow disks, lack of CPU-cycles. [...] ping -c...

Yep, fingers crossed... It's really frustrating, as we are searching for the root cause for some weeks now and didn't really get hands on it till today... and so many contradictory facts / situations... o_O

Yes, it is:

Nope, as described in the first post it is not just one homeserver but a productive 3 - Node Proxmox / Ceph cluster, running ~30 VMs on it partly used by customers, so reinstalling from scratch would really be "suboptimal" The Cluster is running for a longer time now with this configuration and...