proxmox 6.2 ceph issue

R

Romainp

Active Member
Jan 23, 2018
11
0
41
51
Hi all! I really hope someome could guide me with this issue because I do not know cepth much and I am not sure about the next steps..
So I have a 3 nodes cluster, same hardware. ceph was working without any issue until a reboot I had to do because of a network issue. Now on a node, ceph is down, most of the osd are down/out. I have tried to make those osd "in" again but cannot start them anyway. If I do this, some become "out" again.

Here are some logs

Code: 
root@SOCSMI-SRV3:/var/log/ceph# ceph health detail
HEALTH_WARN 24 osds down; 1 host (24 osds) down; Degraded data redundancy: 4595689/13787067 objects degraded (33.333%), 2560 pgs degraded, 2560 pgs undersized
OSD_DOWN 24 osds down
    osd.48 (root=default,host=SOCSMI-SRV3) is down
    osd.49 (root=default,host=SOCSMI-SRV3) is down
    osd.50 (root=default,host=SOCSMI-SRV3) is down
    osd.51 (root=default,host=SOCSMI-SRV3) is down
    osd.52 (root=default,host=SOCSMI-SRV3) is down
    osd.53 (root=default,host=SOCSMI-SRV3) is down
    osd.54 (root=default,host=SOCSMI-SRV3) is down
    osd.55 (root=default,host=SOCSMI-SRV3) is down
    osd.56 (root=default,host=SOCSMI-SRV3) is down
    osd.57 (root=default,host=SOCSMI-SRV3) is down
    osd.58 (root=default,host=SOCSMI-SRV3) is down
    osd.59 (root=default,host=SOCSMI-SRV3) is down
    osd.60 (root=default,host=SOCSMI-SRV3) is down
    osd.61 (root=default,host=SOCSMI-SRV3) is down
    osd.62 (root=default,host=SOCSMI-SRV3) is down
    osd.63 (root=default,host=SOCSMI-SRV3) is down
    osd.64 (root=default,host=SOCSMI-SRV3) is down
    osd.65 (root=default,host=SOCSMI-SRV3) is down
    osd.66 (root=default,host=SOCSMI-SRV3) is down
    osd.67 (root=default,host=SOCSMI-SRV3) is down
    osd.68 (root=default,host=SOCSMI-SRV3) is down
    osd.69 (root=default,host=SOCSMI-SRV3) is down
    osd.70 (root=default,host=SOCSMI-SRV3) is down
    osd.71 (root=default,host=SOCSMI-SRV3) is down
OSD_HOST_DOWN 1 host (24 osds) down
    host SOCSMI-SRV3 (root=default) (24 osds) is down
PG_DEGRADED Degraded data redundancy: 4595689/13787067 objects degraded (33.333%), 2560 pgs degraded, 2560 pgs undersized
    pg 1.7cd is active+undersized+degraded, acting [35,23]
    pg 1.7ce is stuck undersized for 73002.137990, current state active+undersized+degraded, last acting [6,37]
    pg 1.7cf is stuck undersized for 73003.125764, current state active+undersized+degraded, last acting [0,38]
    pg 1.7d0 is stuck undersized for 73003.115012, current state active+undersized+degraded, last acting [0,32]
    pg 1.7d1 is stuck undersized for 73002.137108, current state active+undersized+degraded, last acting [34,20]
    pg 1.7d2 is stuck undersized for 73002.120444, current state active+undersized+degraded, last acting [4,36]
    pg 1.7d3 is stuck undersized for 73018.159866, current state active+undersized+degraded, last acting [27,7]
    pg 1.7d4 is stuck undersized for 73002.141011, current state active+undersized+degraded, last acting [28,0]
    pg 1.7d5 is stuck undersized for 73002.143990, current state active+undersized+degraded, last acting [19,32]
    pg 1.7d6 is stuck undersized for 73018.171505, current state active+undersized+degraded, last acting [10,25]
    pg 1.7d7 is stuck undersized for 73003.132391, current state active+undersized+degraded, last acting [17,46]
    pg 1.7d8 is stuck undersized for 73003.129911, current state active+undersized+degraded, last acting [17,42]
    pg 1.7d9 is stuck undersized for 73018.167376, current state active+undersized+degraded, last acting [4,45]
    pg 1.7da is stuck undersized for 73018.151950, current state active+undersized+degraded, last acting [15,44]
    pg 1.7db is stuck undersized for 73018.183700, current state active+undersized+degraded, last acting [24,23]
    pg 1.7dc is stuck undersized for 73003.113184, current state active+undersized+degraded, last acting [35,13]
    pg 1.7dd is stuck undersized for 73003.136707, current state active+undersized+degraded, last acting [31,19]
    pg 1.7de is stuck undersized for 73002.143750, current state active+undersized+degraded, last acting [2,38]
    pg 1.7df is stuck undersized for 73002.134902, current state active+undersized+degraded, last acting [11,26]
    pg 1.7e0 is stuck undersized for 73002.119510, current state active+undersized+degraded, last acting [22,47]
    pg 1.7e1 is stuck undersized for 73003.130654, current state active+undersized+degraded, last acting [10,29]
    pg 1.7e2 is stuck undersized for 73003.134833, current state active+undersized+degraded, last acting [46,22]
    pg 1.7e3 is stuck undersized for 73018.153941, current state active+undersized+degraded, last acting [19,39]
    pg 1.7e4 is stuck undersized for 73002.141248, current state active+undersized+degraded, last acting [33,17]
    pg 1.7e5 is stuck undersized for 73003.130155, current state active+undersized+degraded, last acting [39,1]
    pg 1.7e6 is stuck undersized for 73002.125134, current state active+undersized+degraded, last acting [18,34]
    pg 1.7e7 is stuck undersized for 73018.163222, current state active+undersized+degraded, last acting [35,23]
    pg 1.7e8 is stuck undersized for 73018.169650, current state active+undersized+degraded, last acting [40,19]
    pg 1.7e9 is stuck undersized for 73003.133874, current state active+undersized+degraded, last acting [23,28]
    pg 1.7ea is stuck undersized for 73002.138131, current state active+undersized+degraded, last acting [37,4]
    pg 1.7eb is stuck undersized for 73003.122079, current state active+undersized+degraded, last acting [46,20]
    pg 1.7ec is stuck undersized for 73003.127444, current state active+undersized+degraded, last acting [20,31]
    pg 1.7ed is stuck undersized for 73002.121792, current state active+undersized+degraded, last acting [10,37]
    pg 1.7ee is stuck undersized for 73003.128326, current state active+undersized+degraded, last acting [40,13]
    pg 1.7ef is stuck undersized for 73018.175634, current state active+undersized+degraded, last acting [46,5]
    pg 1.7f0 is stuck undersized for 73002.123425, current state active+undersized+degraded, last acting [17,28]
    pg 1.7f1 is stuck undersized for 73002.130478, current state active+undersized+degraded, last acting [30,19]
    pg 1.7f2 is stuck undersized for 73003.142126, current state active+undersized+degraded, last acting [36,14]
    pg 1.7f3 is stuck undersized for 73002.136261, current state active+undersized+degraded, last acting [34,1]
    pg 1.7f4 is stuck undersized for 73003.147187, current state active+undersized+degraded, last acting [16,30]
    pg 1.7f5 is stuck undersized for 73018.165342, current state active+undersized+degraded, last acting [33,22]
    pg 1.7f6 is stuck undersized for 73002.137469, current state active+undersized+degraded, last acting [7,27]
    pg 1.7f7 is stuck undersized for 73002.133176, current state active+undersized+degraded, last acting [40,18]
    pg 1.7f8 is stuck undersized for 73018.167834, current state active+undersized+degraded, last acting [17,46]
    pg 1.7f9 is stuck undersized for 73002.122180, current state active+undersized+degraded, last acting [10,39]
    pg 1.7fa is stuck undersized for 73003.142536, current state active+undersized+degraded, last acting [9,24]
    pg 1.7fb is stuck undersized for 73002.137872, current state active+undersized+degraded, last acting [7,35]
    pg 1.7fc is stuck undersized for 73003.121659, current state active+undersized+degraded, last acting [30,23]
    pg 1.7fd is stuck undersized for 73003.116221, current state active+undersized+degraded, last acting [11,41]
    pg 1.7fe is stuck undersized for 73018.125948, current state active+undersized+degraded, last acting [13,38]
    pg 1.7ff is stuck undersized for 73002.133519, current state active+undersized+degraded, last acting [8,32]

I have moved all my vm to the other node, so I don't care (I guess) deleting/recreating if it's needed but I am not sure what to do at this level. Thanks for any help or advices you can give :)
R.
 
More logs in case it's needed

Code: 
root@SOCSMI-SRV3:/var/log/ceph# ceph pg 1.7cd query
{
    "state": "active+undersized+degraded",
    "snap_trimq": "[]",
    "snap_trimq_len": 0,
    "epoch": 3413,
    "up": [
        35,
        23
    ],
    "acting": [
        35,
        23
    ],
    "acting_recovery_backfill": [
        "23",
        "35"
    ],
    "info": {
        "pgid": "1.7cd",
        "last_update": "3346'2435868",
        "last_complete": "3346'2435868",
        "log_tail": "3189'2432816",
        "last_user_version": 2435868,
        "last_backfill": "MAX",
        "last_backfill_bitwise": 0,
        "purged_snaps": [],
        "history": {
            "epoch_created": 292,
            "epoch_pool_created": 292,
            "last_epoch_started": 3209,
            "last_interval_started": 3208,
            "last_epoch_clean": 3200,
            "last_interval_clean": 3199,
            "last_epoch_split": 0,
            "last_epoch_marked_full": 0,
            "same_up_since": 3208,
            "same_interval_since": 3208,
            "same_primary_since": 3098,
            "last_scrub": "3206'2435609",
            "last_scrub_stamp": "2022-08-02 05:41:23.998567",
            "last_deep_scrub": "3206'2435609",
            "last_deep_scrub_stamp": "2022-08-02 05:41:23.998567",
            "last_clean_scrub_stamp": "2022-08-02 05:41:23.998567"
        },
        "stats": {
            "version": "3346'2435868",
            "reported_seq": "3515899",
            "reported_epoch": "3413",
            "state": "active+undersized+degraded",
            "last_fresh": "2022-08-03 09:32:19.665334",
            "last_change": "2022-08-02 13:16:03.404719",
            "last_active": "2022-08-03 09:32:19.665334",
            "last_peered": "2022-08-03 09:32:19.665334",
            "last_clean": "2022-08-02 13:15:47.353098",
            "last_became_active": "2022-08-02 13:16:03.404719",
            "last_became_peered": "2022-08-02 13:16:03.404719",
            "last_unstale": "2022-08-03 09:32:19.665334",
            "last_undegraded": "2022-08-02 13:16:03.381574",
            "last_fullsized": "2022-08-02 13:16:03.381523",
            "mapping_epoch": 3208,
            "log_start": "3189'2432816",
            "ondisk_log_start": "3189'2432816",
            "created": 292,
            "last_epoch_clean": 3200,
            "parent": "0.0",
            "parent_split_bits": 0,
            "last_scrub": "3206'2435609",
            "last_scrub_stamp": "2022-08-02 05:41:23.998567",
            "last_deep_scrub": "3206'2435609",
            "last_deep_scrub_stamp": "2022-08-02 05:41:23.998567",
            "last_clean_scrub_stamp": "2022-08-02 05:41:23.998567",
            "log_size": 3052,
            "ondisk_log_size": 3052,
            "stats_invalid": false,
            "dirty_stats_invalid": false,
            "omap_stats_invalid": false,
            "hitset_stats_invalid": false,
            "hitset_bytes_stats_invalid": false,
            "pin_stats_invalid": false,
            "manifest_stats_invalid": true,
            "snaptrimq_len": 0,
            "stat_sum": {
                "num_bytes": 7793781760,
                "num_objects": 2142,
                "num_object_clones": 575,
                "num_object_copies": 6426,
                "num_objects_missing_on_primary": 0,
                "num_objects_missing": 0,
                "num_objects_degraded": 2142,
                "num_objects_misplaced": 0,
                "num_objects_unfound": 0,
                "num_objects_dirty": 2142,
                "num_whiteouts": 0,
                "num_read": 1075651,
                "num_read_kb": 118887853,
                "num_write": 2387680,
                "num_write_kb": 252537140,
                "num_scrub_errors": 0,
                "num_shallow_scrub_errors": 0,
                "num_deep_scrub_errors": 0,
                "num_objects_recovered": 997,
                "num_bytes_recovered": 4171870208,
                "num_keys_recovered": 0,
                "num_objects_omap": 0,
                "num_objects_hit_set_archive": 0,
                "num_bytes_hit_set_archive": 0,
                "num_flush": 0,
                "num_flush_kb": 0,
                "num_evict": 0,
                "num_evict_kb": 0,
                "num_promote": 0,
                "num_flush_mode_high": 0,
                "num_flush_mode_low": 0,
                "num_evict_mode_some": 0,
                "num_evict_mode_full": 0,
                "num_objects_pinned": 0,
                "num_legacy_snapsets": 0,
                "num_large_omap_objects": 0,
                "num_objects_manifest": 0,
                "num_omap_bytes": 0,
                "num_omap_keys": 0,
                "num_objects_repaired": 0
            },
            "up": [
                35,
                23
            ],
            "acting": [
                35,
                23
            ],
            "avail_no_missing": [
                "35",
                "23"
            ],
            "object_location_counts": [
                {
                    "shards": "23,35",
                    "objects": 2142
                }
            ],
            "blocked_by": [],
            "up_primary": 35,
            "acting_primary": 35,
            "purged_snaps": []
        },
        "empty": 0,
        "dne": 0,
        "incomplete": 0,
        "last_epoch_started": 3209,
        "hit_set_history": {
            "current_last_update": "0'0",
            "history": []
        }
    },
    "peer_info": [
        {
            "peer": "23",
            "pgid": "1.7cd",
            "last_update": "3346'2435868",
            "last_complete": "3346'2435868",
            "log_tail": "3189'2432616",
            "last_user_version": 2435655,
            "last_backfill": "MAX",
            "last_backfill_bitwise": 0,
            "purged_snaps": [],
            "history": {
                "epoch_created": 292,
                "epoch_pool_created": 292,
                "last_epoch_started": 3209,
                "last_interval_started": 3208,
                "last_epoch_clean": 3200,
                "last_interval_clean": 3199,
                "last_epoch_split": 0,
                "last_epoch_marked_full": 0,
                "same_up_since": 3208,
                "same_interval_since": 3208,
                "same_primary_since": 3098,
                "last_scrub": "3206'2435609",
                "last_scrub_stamp": "2022-08-02 05:41:23.998567",
                "last_deep_scrub": "3206'2435609",
                "last_deep_scrub_stamp": "2022-08-02 05:41:23.998567",
                "last_clean_scrub_stamp": "2022-08-02 05:41:23.998567"
            },
            "stats": {
                "version": "3206'2435647",
                "reported_seq": "3515444",
                "reported_epoch": "3206",
                "state": "active+clean",
                "last_fresh": "2022-08-02 12:49:45.067548",
                "last_change": "2022-08-02 05:41:23.998602",
                "last_active": "2022-08-02 12:49:45.067548",
                "last_peered": "2022-08-02 12:49:45.067548",
                "last_clean": "2022-08-02 12:49:45.067548",
                "last_became_active": "2022-07-28 01:57:17.865900",
                "last_became_peered": "2022-07-28 01:57:17.865900",
                "last_unstale": "2022-08-02 12:49:45.067548",
                "last_undegraded": "2022-08-02 12:49:45.067548",
                "last_fullsized": "2022-08-02 12:49:45.067548",
                "mapping_epoch": 3208,
                "log_start": "3189'2432616",
                "ondisk_log_start": "3189'2432616",
                "created": 292,
                "last_epoch_clean": 3200,
                "parent": "0.0",
                "parent_split_bits": 0,
                "last_scrub": "3206'2435609",
                "last_scrub_stamp": "2022-08-02 05:41:23.998567",
                "last_deep_scrub": "3206'2435609",
                "last_deep_scrub_stamp": "2022-08-02 05:41:23.998567",
                "last_clean_scrub_stamp": "2022-08-02 05:41:23.998567",
                "log_size": 3031,
                "ondisk_log_size": 3031,
                "stats_invalid": false,
                "dirty_stats_invalid": false,
                "omap_stats_invalid": false,
                "hitset_stats_invalid": false,
                "hitset_bytes_stats_invalid": false,
                "pin_stats_invalid": false,
                "manifest_stats_invalid": true,
                "snaptrimq_len": 0,
                "stat_sum": {
                    "num_bytes": 7793781760,
                    "num_objects": 2142,
                    "num_object_clones": 575,
                    "num_object_copies": 6426,
                    "num_objects_missing_on_primary": 0,
                    "num_objects_missing": 0,
                    "num_objects_degraded": 0,
                    "num_objects_misplaced": 0,
                    "num_objects_unfound": 0,
                    "num_objects_dirty": 2142,
                    "num_whiteouts": 0,
                    "num_read": 1075630,
                    "num_read_kb": 118885705,
                    "num_write": 2387467,
                    "num_write_kb": 252459172,
                    "num_scrub_errors": 0,
                    "num_shallow_scrub_errors": 0,
                    "num_deep_scrub_errors": 0,
                    "num_objects_recovered": 997,
                    "num_bytes_recovered": 4171870208,
                    "num_keys_recovered": 0,
                    "num_objects_omap": 0,
                    "num_objects_hit_set_archive": 0,
                    "num_bytes_hit_set_archive": 0,
                    "num_flush": 0,
                    "num_flush_kb": 0,
                    "num_evict": 0,
                    "num_evict_kb": 0,
                    "num_promote": 0,
                    "num_flush_mode_high": 0,
                    "num_flush_mode_low": 0,
                    "num_evict_mode_some": 0,
                    "num_evict_mode_full": 0,
                    "num_objects_pinned": 0,
                    "num_legacy_snapsets": 0,
                    "num_large_omap_objects": 0,
                    "num_objects_manifest": 0,
                    "num_omap_bytes": 0,
                    "num_omap_keys": 0,
                    "num_objects_repaired": 0
                },
                "up": [
                    35,
                    23
                ],
                "acting": [
                    35,
                    23
                ],
                "avail_no_missing": [],
                "object_location_counts": [],
                "blocked_by": [],
                "up_primary": 35,
                "acting_primary": 35,
                "purged_snaps": []
            },
            "empty": 0,
            "dne": 0,
            "incomplete": 0,
            "last_epoch_started": 3209,
            "hit_set_history": {
                "current_last_update": "0'0",
                "history": []
            }
        }
    ],
    "recovery_state": [
        {
            "name": "Started/Primary/Active",
            "enter_time": "2022-08-02 13:16:03.381556",
            "might_have_unfound": [],
            "recovery_progress": {
                "backfill_targets": [],
                "waiting_on_backfill": [],
                "last_backfill_started": "MIN",
                "backfill_info": {
                    "begin": "MIN",
                    "end": "MIN",
                    "objects": []
                },
                "peer_backfill_info": [],
                "backfills_in_flight": [],
                "recovering": [],
                "pg_backend": {
                    "pull_from_peer": [],
                    "pushing": []
                }
            },
            "scrub": {
                "scrubber.epoch_start": "3199",
                "scrubber.active": false,
                "scrubber.state": "INACTIVE",
                "scrubber.start": "MIN",
                "scrubber.end": "MIN",
                "scrubber.max_end": "MIN",
                "scrubber.subset_last_update": "0'0",
                "scrubber.deep": false,
                "scrubber.waiting_on_whom": []
            }
        },
        {
            "name": "Started",
            "enter_time": "2022-08-02 13:15:48.344180"
        }
    ],
    "agent_state": {}
}
 
Code: 
root@SOCSMI-SRV3:/var/log/ceph# ceph osd tree
ID CLASS WEIGHT    TYPE NAME            STATUS REWEIGHT PRI-AFF
-1       117.88550 root default
-3        39.29517     host SOCSMI-SRV1
 0   hdd   1.63730         osd.0            up  1.00000 1.00000
 1   hdd   1.63730         osd.1            up  1.00000 1.00000
 2   hdd   1.63730         osd.2            up  1.00000 1.00000
 3   hdd   1.63730         osd.3            up  1.00000 1.00000
 4   hdd   1.63730         osd.4            up  1.00000 1.00000
 5   hdd   1.63730         osd.5            up  1.00000 1.00000
 6   hdd   1.63730         osd.6            up  1.00000 1.00000
 7   hdd   1.63730         osd.7            up  1.00000 1.00000
 8   hdd   1.63730         osd.8            up  1.00000 1.00000
 9   hdd   1.63730         osd.9            up  1.00000 1.00000
10   hdd   1.63730         osd.10           up  1.00000 1.00000
11   hdd   1.63730         osd.11           up  1.00000 1.00000
12   hdd   1.63730         osd.12           up  1.00000 1.00000
13   hdd   1.63730         osd.13           up  1.00000 1.00000
14   hdd   1.63730         osd.14           up  1.00000 1.00000
15   hdd   1.63730         osd.15           up  1.00000 1.00000
16   hdd   1.63730         osd.16           up  1.00000 1.00000
17   hdd   1.63730         osd.17           up  1.00000 1.00000
18   hdd   1.63730         osd.18           up  1.00000 1.00000
19   hdd   1.63730         osd.19           up  1.00000 1.00000
20   hdd   1.63730         osd.20           up  1.00000 1.00000
21   hdd   1.63730         osd.21           up  1.00000 1.00000
22   hdd   1.63730         osd.22           up  1.00000 1.00000
23   hdd   1.63730         osd.23           up  1.00000 1.00000
-5        39.29517     host SOCSMI-SRV2
24   hdd   1.63730         osd.24           up  1.00000 1.00000
25   hdd   1.63730         osd.25           up  1.00000 1.00000
26   hdd   1.63730         osd.26           up  1.00000 1.00000
27   hdd   1.63730         osd.27           up  1.00000 1.00000
28   hdd   1.63730         osd.28           up  1.00000 1.00000
29   hdd   1.63730         osd.29           up  1.00000 1.00000
30   hdd   1.63730         osd.30           up  1.00000 1.00000
31   hdd   1.63730         osd.31           up  1.00000 1.00000
32   hdd   1.63730         osd.32           up  1.00000 1.00000
33   hdd   1.63730         osd.33           up  1.00000 1.00000
34   hdd   1.63730         osd.34           up  1.00000 1.00000
35   hdd   1.63730         osd.35           up  1.00000 1.00000
36   hdd   1.63730         osd.36           up  1.00000 1.00000
37   hdd   1.63730         osd.37           up  1.00000 1.00000
38   hdd   1.63730         osd.38           up  1.00000 1.00000
39   hdd   1.63730         osd.39           up  1.00000 1.00000
40   hdd   1.63730         osd.40           up  1.00000 1.00000
41   hdd   1.63730         osd.41           up  1.00000 1.00000
42   hdd   1.63730         osd.42           up  1.00000 1.00000
43   hdd   1.63730         osd.43           up  1.00000 1.00000
44   hdd   1.63730         osd.44           up  1.00000 1.00000
45   hdd   1.63730         osd.45           up  1.00000 1.00000
46   hdd   1.63730         osd.46           up  1.00000 1.00000
47   hdd   1.63730         osd.47           up  1.00000 1.00000
-7        39.29517     host SOCSMI-SRV3
48   hdd   1.63730         osd.48         down  1.00000 1.00000
49   hdd   1.63730         osd.49         down  1.00000 1.00000
50   hdd   1.63730         osd.50         down        0 1.00000
51   hdd   1.63730         osd.51         down        0 1.00000
52   hdd   1.63730         osd.52         down        0 1.00000
53   hdd   1.63730         osd.53         down        0 1.00000
54   hdd   1.63730         osd.54         down        0 1.00000
55   hdd   1.63730         osd.55         down        0 1.00000
56   hdd   1.63730         osd.56         down        0 1.00000
57   hdd   1.63730         osd.57         down        0 1.00000
58   hdd   1.63730         osd.58         down        0 1.00000
59   hdd   1.63730         osd.59         down        0 1.00000
60   hdd   1.63730         osd.60         down        0 1.00000
61   hdd   1.63730         osd.61         down        0 1.00000
62   hdd   1.63730         osd.62         down        0 1.00000
63   hdd   1.63730         osd.63         down        0 1.00000
64   hdd   1.63730         osd.64         down        0 1.00000
65   hdd   1.63730         osd.65         down        0 1.00000
66   hdd   1.63730         osd.66         down        0 1.00000
67   hdd   1.63730         osd.67         down        0 1.00000
68   hdd   1.63730         osd.68         down        0 1.00000
69   hdd   1.63730         osd.69         down  1.00000 1.00000
70   hdd   1.63730         osd.70         down  1.00000 1.00000
71   hdd   1.63730         osd.71         down  1.00000 1.00000
 
Code: 
root@SOCSMI-SRV3:/var/log/ceph# ceph osd df
ID CLASS WEIGHT  REWEIGHT SIZE    RAW USE DATA    OMAP    META    AVAIL    %USE  VAR  PGS STATUS
 0   hdd 1.63730  1.00000 1.6 TiB 604 GiB 602 GiB 115 MiB 1.5 GiB  1.0 TiB 36.02 0.92 100     up
 1   hdd 1.63730  1.00000 1.6 TiB 644 GiB 642 GiB  87 MiB 1.6 GiB  1.0 TiB 38.41 0.98 100     up
 2   hdd 1.63730  1.00000 1.6 TiB 636 GiB 634 GiB 101 MiB 1.6 GiB  1.0 TiB 37.94 0.97 104     up
 3   hdd 1.63730  1.00000 1.6 TiB 648 GiB 646 GiB 127 MiB 1.6 GiB  1.0 TiB 38.66 0.99 104     up
 4   hdd 1.63730  1.00000 1.6 TiB 654 GiB 653 GiB 100 MiB 1.7 GiB 1022 GiB 39.03 1.00 106     up
 5   hdd 1.63730  1.00000 1.6 TiB 626 GiB 625 GiB  96 MiB 1.6 GiB  1.0 TiB 37.35 0.96 100     up
 6   hdd 1.63730  1.00000 1.6 TiB 771 GiB 769 GiB 135 MiB 1.8 GiB  906 GiB 45.97 1.18 126     up
 7   hdd 1.63730  1.00000 1.6 TiB 680 GiB 678 GiB  95 MiB 1.6 GiB  997 GiB 40.54 1.04 107     up
 8   hdd 1.63730  1.00000 1.6 TiB 621 GiB 619 GiB  86 MiB 1.5 GiB  1.0 TiB 37.02 0.95 100     up
 9   hdd 1.63730  1.00000 1.6 TiB 695 GiB 694 GiB 137 MiB 1.7 GiB  981 GiB 41.48 1.06 112     up
10   hdd 1.63730  1.00000 1.6 TiB 763 GiB 761 GiB 122 MiB 1.7 GiB  913 GiB 45.52 1.16 123     up
11   hdd 1.63730  1.00000 1.6 TiB 583 GiB 581 GiB 116 MiB 1.4 GiB  1.1 TiB 34.77 0.89  96     up
12   hdd 1.63730  1.00000 1.6 TiB 681 GiB 679 GiB 130 MiB 1.6 GiB  996 GiB 40.62 1.04 111     up
13   hdd 1.63730  1.00000 1.6 TiB 657 GiB 655 GiB 108 MiB 1.7 GiB 1020 GiB 39.17 1.00 109     up
14   hdd 1.63730  1.00000 1.6 TiB 555 GiB 553 GiB  80 MiB 1.4 GiB  1.1 TiB 33.08 0.85  89     up
15   hdd 1.63730  1.00000 1.6 TiB 598 GiB 596 GiB 126 MiB 1.5 GiB  1.1 TiB 35.65 0.91  99     up
16   hdd 1.63730  1.00000 1.6 TiB 614 GiB 612 GiB  93 MiB 1.5 GiB  1.0 TiB 36.61 0.94  98     up
17   hdd 1.63730  1.00000 1.6 TiB 623 GiB 621 GiB 120 MiB 1.5 GiB  1.0 TiB 37.16 0.95 103     up
18   hdd 1.63730  1.00000 1.6 TiB 688 GiB 686 GiB 134 MiB 1.6 GiB  988 GiB 41.04 1.05 111     up
19   hdd 1.63730  1.00000 1.6 TiB 636 GiB 635 GiB 109 MiB 1.7 GiB  1.0 TiB 37.95 0.97 107     up
20   hdd 1.63730  1.00000 1.6 TiB 702 GiB 700 GiB  98 MiB 1.6 GiB  975 GiB 41.85 1.07 118     up
21   hdd 1.63730  1.00000 1.6 TiB 708 GiB 706 GiB 109 MiB 1.6 GiB  969 GiB 42.22 1.08 116     up
22   hdd 1.63730  1.00000 1.6 TiB 702 GiB 700 GiB  86 MiB 1.9 GiB  974 GiB 41.88 1.07 113     up
23   hdd 1.63730  1.00000 1.6 TiB 643 GiB 642 GiB 130 MiB 1.6 GiB  1.0 TiB 38.37 0.98 108     up
24   hdd 1.63730  1.00000 1.6 TiB 669 GiB 667 GiB  98 MiB 1.6 GiB 1008 GiB 39.90 1.02 109     up
25   hdd 1.63730  1.00000 1.6 TiB 695 GiB 693 GiB  85 MiB 2.3 GiB  981 GiB 41.48 1.06 113     up
26   hdd 1.63730  1.00000 1.6 TiB 684 GiB 682 GiB 124 MiB 1.6 GiB  993 GiB 40.78 1.04 109     up
27   hdd 1.63730  1.00000 1.6 TiB 690 GiB 689 GiB 131 MiB 1.6 GiB  986 GiB 41.17 1.05 112     up
28   hdd 1.63730  1.00000 1.6 TiB 537 GiB 535 GiB 102 MiB 1.4 GiB  1.1 TiB 32.02 0.82  92     up
29   hdd 1.63730  1.00000 1.6 TiB 534 GiB 532 GiB  96 MiB 1.5 GiB  1.1 TiB 31.83 0.81  90     up
30   hdd 1.63730  1.00000 1.6 TiB 704 GiB 702 GiB 108 MiB 1.8 GiB  973 GiB 41.96 1.07 117     up
31   hdd 1.63730  1.00000 1.6 TiB 704 GiB 702 GiB 129 MiB 1.7 GiB  973 GiB 41.99 1.07 112     up
32   hdd 1.63730  1.00000 1.6 TiB 663 GiB 662 GiB  94 MiB 1.6 GiB 1013 GiB 39.57 1.01 109     up
33   hdd 1.63730  1.00000 1.6 TiB 627 GiB 625 GiB 107 MiB 1.5 GiB  1.0 TiB 37.39 0.96 102     up
34   hdd 1.63730  1.00000 1.6 TiB 627 GiB 626 GiB  87 MiB 1.6 GiB  1.0 TiB 37.43 0.96 100     up
35   hdd 1.63730  1.00000 1.6 TiB 727 GiB 725 GiB 103 MiB 1.7 GiB  949 GiB 43.37 1.11 117     up
36   hdd 1.63730  1.00000 1.6 TiB 733 GiB 731 GiB 106 MiB 1.9 GiB  944 GiB 43.70 1.12 115     up
37   hdd 1.63730  1.00000 1.6 TiB 716 GiB 714 GiB 120 MiB 1.9 GiB  961 GiB 42.68 1.09 118     up
38   hdd 1.63730  1.00000 1.6 TiB 538 GiB 537 GiB 108 MiB 1.4 GiB  1.1 TiB 32.09 0.82  88     up
39   hdd 1.63730  1.00000 1.6 TiB 817 GiB 815 GiB 137 MiB 2.0 GiB  859 GiB 48.74 1.25 127     up
40   hdd 1.63730  1.00000 1.6 TiB 753 GiB 751 GiB 142 MiB 1.8 GiB  924 GiB 44.91 1.15 125     up
41   hdd 1.63730  1.00000 1.6 TiB 583 GiB 582 GiB  96 MiB 1.5 GiB  1.1 TiB 34.80 0.89  93     up
42   hdd 1.63730  1.00000 1.6 TiB 568 GiB 566 GiB 119 MiB 1.4 GiB  1.1 TiB 33.88 0.87  97     up
43   hdd 1.63730  1.00000 1.6 TiB 635 GiB 633 GiB 136 MiB 1.6 GiB  1.0 TiB 37.84 0.97 103     up
44   hdd 1.63730  1.00000 1.6 TiB 598 GiB 596 GiB 102 MiB 1.6 GiB  1.1 TiB 35.66 0.91  97     up
45   hdd 1.63730  1.00000 1.6 TiB 696 GiB 694 GiB 139 MiB 1.7 GiB  980 GiB 41.52 1.06 114     up
46   hdd 1.63730  1.00000 1.6 TiB 732 GiB 730 GiB 145 MiB 1.7 GiB  945 GiB 43.64 1.12 118     up
47   hdd 1.63730  1.00000 1.6 TiB 504 GiB 502 GiB  71 MiB 1.3 GiB  1.1 TiB 30.04 0.77  83     up
48   hdd 1.63730  1.00000     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
49   hdd 1.63730  1.00000     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
50   hdd 1.63730        0     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
51   hdd 1.63730        0     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
52   hdd 1.63730        0     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
53   hdd 1.63730        0     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
54   hdd 1.63730        0     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
55   hdd 1.63730        0     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
56   hdd 1.63730        0     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
57   hdd 1.63730        0     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
58   hdd 1.63730        0     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
59   hdd 1.63730        0     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
60   hdd 1.63730        0     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
61   hdd 1.63730        0     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
62   hdd 1.63730        0     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
63   hdd 1.63730        0     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
64   hdd 1.63730        0     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
65   hdd 1.63730        0     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
66   hdd 1.63730        0     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
67   hdd 1.63730        0     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
68   hdd 1.63730        0     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
69   hdd 1.63730  1.00000     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
70   hdd 1.63730  1.00000     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
71   hdd 1.63730  1.00000     0 B     0 B     0 B     0 B     0 B      0 B     0    0   0   down
                    TOTAL  79 TiB  31 TiB  31 TiB 5.2 GiB  78 GiB   48 TiB 39.10
MIN/MAX VAR: 0/1.25  STDDEV: 12.59
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back