Hi, I have a new cluster of 4 nodes, 3 of them have ceph. Code: root@pve3:~# pveversion -v proxmox-ve: 5.1-25 (running kernel: 4.13.4-1-pve) pve-manager: 5.1-35 (running version: 5.1-35/722cc488) pve-kernel-4.13.4-1-pve: 4.13.4-25 libpve-http-server-perl: 2.0-6 lvm2: 2.02.168-pve6 corosync: 2.4.2-pve3 libqb0: 1.0.1-1 pve-cluster: 5.0-15 qemu-server: 5.0-17 pve-firmware: 2.0-3 libpve-common-perl: 5.0-20 libpve-guest-common-perl: 2.0-13 libpve-access-control: 5.0-7 libpve-storage-perl: 5.0-16 pve-libspice-server1: 0.12.8-3 vncterm: 1.5-2 pve-docs: 5.1-12 pve-qemu-kvm: 2.9.1-2 pve-container: 2.0-17 pve-firewall: 3.0-3 pve-ha-manager: 2.0-3 ksm-control-daemon: 1.2-2 glusterfs-client: 3.8.8-1 lxc-pve: 2.1.0-2 lxcfs: 2.0.7-pve4 criu: 2.11.1-1~bpo90 novnc-pve: 0.6-4 smartmontools: 6.5+svn4324-1 zfsutils-linux: 0.7.2-pve1~bpo90 openvswitch-switch: 2.7.0-2 ceph: 12.2.1-pve3 Code: ceph -v ceph version 12.2.1 (1a629971a9bcaaae99e5539a3a43f800a297f267) luminous (stable) I'm using all SSD single pool. Bluestore, no rocks-db or WAL. The "journal" or whatever it's called now is 100MB a disk. ceph-cache is enabled. Cache per VM set at no-cache. I transferred 4 VMs from VMware vsphere over and testing them. The machines are doing nothing. I mean, they do have CentOS 6 on them and apache but nobody communicates with them. I'm doing automatic backup every night to an NFS share. Now, I noticed that even though the machines are just sitting there - the raw usage of ceph is constantly growing. This is the output of "ceph df detail": Code: GLOBAL: SIZE AVAIL RAW USED %RAW USED OBJECTS 8941G 8888G 54546M 0.60 3960 POOLS: NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED VMpool 1 N/A N/A 14337M 0.17 2812G 3960 3960 490k 525k 43013M So the pool usage with replica 3 is 43013 MB, which is fine and it grows very slowly, i.e. several MB a day. But the "RAW USED" 54546M in the GLOBAL section grows much faster - about 1GB/day. If I run fstrim on the VMs - it helps a little ( 5-20MBs in total). So what's with the 11GB difference between the GLOBAL and POOL usage? How is the GLOBAL usage calculated? And the most important - why does it grow by itself? If I transfer all of my 50 VMs over, and there are about 20 VMs with 100GB-800GB - what are the consequences? Code: ceph -s cluster: id: a1ba7570-38aa-4410-9318-92f3788ef7ef health: HEALTH_OK services: mon: 3 daemons, quorum pve1,pve2,pve3 mgr: pve3(active), standbys: pve2, pve1 osd: 12 osds: 12 up, 12 in data: pools: 1 pools, 1024 pgs objects: 3960 objects, 14337 MB usage: 54546 MB used, 8888 GB / 8941 GB avail pgs: 1024 active+clean io: client: 1364 B/s wr, 0 op/s rd, 0 op/s wr Code: ceph -w cluster: id: a1ba7570-38aa-4410-9318-92f3788ef7ef health: HEALTH_OK services: mon: 3 daemons, quorum pve1,pve2,pve3 mgr: pve3(active), standbys: pve2, pve1 osd: 12 osds: 12 up, 12 in data: pools: 1 pools, 1024 pgs objects: 3960 objects, 14337 MB usage: 54569 MB used, 8888 GB / 8941 GB avail pgs: 1024 active+clean io: client: 1023 B/s wr, 0 op/s rd, 0 op/s wr Please assist, I must know what I'm getting into before I go on. Thanks!