Recent content by ilia987

  1. I

    [SOLVED] upgrade issue 8.0.2 - > 8.3.0, node not usable

    found a workaround that fixed it, updated on first post
  2. I

    [SOLVED] upgrade issue 8.0.2 - > 8.3.0, node not usable

    i prefer not to touch working storage's (the cluster currently used our company) systemctl status pvestatd shows enabled and running
  3. I

    [SOLVED] upgrade issue 8.0.2 - > 8.3.0, node not usable

    Ceph working on all other nodes, and the ladt mount(docs) in this picture is nfs based , hosted on qnap and it is accessible via cmd
  4. I

    [SOLVED] upgrade issue 8.0.2 - > 8.3.0, node not usable

    no change, all networks are marked as down, going inside network mounted storage (that marked as down) via cmd is correct, all file are there and accessible
  5. I

    [SOLVED] upgrade issue 8.0.2 - > 8.3.0, node not usable

    yes df is working, and return correct values, and mount|grep nfs works as well
  6. I

    [SOLVED] upgrade issue 8.0.2 - > 8.3.0, node not usable

    I have a cluster of around 15 nodes. Today i successfully upgraded pbs from 3.0 to 3.3.2. (reboot and full test of backup\restore worked ) Then i Tried to upgrade a single node (from the GUI). all went normal and system worked after upgrade. i Initiated a reboot, to make sure everything...
  7. I

    chronyd issues

    i came across it , did not help. But i think i found a reason, still not sure if it caused it. all Proxmox nodes have chrony services and should sync by it. i have two more dedicated servers DC, DC backup and the backup had a time issue, after it fixed most of the issues resolved But i dont...
  8. I

    chronyd issues

    I have a cluster of around 20 nodes, Proxmox 8.0.4 +ceph (on 4 of them) all have the same configuration of chronyd. and most of the time everything works. recently (in the past month) i started to have issues with time sync. usually it comes after a power failure, some servers are out of...
  9. I

    Proxmox VE Ceph Benchmark 2023/12 - Fast SSDs and network speeds in a Proxmox VE Ceph Reef cluster

    does a benchmark exist with more then 3 nodes for example in the area of 10 nodes? does the throughput scales accordingly for multi client usage?
  10. I

    [SOLVED] Ceph hang in Degraded data redundancy

    Update - full shut down of all ceph nodes solved the issue (one by one did not help)
  11. I

    [SOLVED] Ceph hang in Degraded data redundancy

    There is another change i noticed today: pgs scrub issue. *till now the systems are running and responsive but i dont think is is healy
  12. I

    [SOLVED] Ceph hang in Degraded data redundancy

    ceph osd pool ls detail pool 2 'ceph-lxc' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 227165 lfor 0/136355/136651 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps_queue...
  13. I

    [SOLVED] Ceph hang in Degraded data redundancy

    ceph -s cluster: id: 8ebca482-f985-4e74-9ff8-35e03a1af15e health: HEALTH_WARN Degraded data redundancy: 1608/62722158 objects degraded (0.003%), 28 pgs degraded, 22 pgs undersized services: mon: 3 daemons, quorum pve-srv2,pve-srv3,pve-srv4 (age 2d) mgr...
  14. I

    [SOLVED] Ceph hang in Degraded data redundancy

    flow: 1 servers had reboot due to power maintenance, 2 (after the reboot) i noticed one server had bad clock sync - fixing the issue and another reboot solved it) the 3. after time sync fixed cluster started to load and rebalance, 4 it hang at error state (data looks ok and everything stable and...
  15. I

    Shutdown of the Hyper-Converged Cluster (CEPH)

    We have a setup of around 30 servers, 4 of them with ceph storage, Unfortunately we have many power outages in our building and the backup battery does not last for long periods , casing entire cluster crash, (server, switches, storages) Most of the time the entire cluster turn up when the...