Storage question, im stumped after hours of research :(! Shrink local and increase Directory storage

R

redactedhosting

New Member
Apr 20, 2024
22
3
3
Hi all, im trying to reduce my local storage and re assign it to my "agents" storage that i use for all of my LXCs & VMs. Im short, i have 6ish TB in my node1 local, and i am looking at 3.6TB Agents storage in directory type.

My question is how can i cut local in half, and move the remaining 3tb to 3.6TB Agents storage. Screenshots attached. Thank you for any advice on this.
lsblk && lvs:

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 20G 0 loop
loop1 7:1 0 32G 0 loop
loop2 7:2 0 100G 0 loop
sda 8:0 0 6.5T 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 1G 0 part
└─sda3 8:3 0 6.5T 0 part
├─pve-swap 252:0 0 8G 0 lvm [SWAP]
└─pve-root 252:1 0 6.5T 0 lvm /
sdb 8:16 0 3.3T 0 disk
└─sdb1 8:17 0 3.3T 0 part /mnt/pve/agents
sr0 11:0 1 1024M 0 rom
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root pve -wi-ao---- <6.54t
swap pve -wi-ao---- 8.00g
 

Attachments

  • storage4.PNG
    storage4.PNG
    14.6 KB · Views: 12
  • storage3.PNG
    storage3.PNG
    32 KB · Views: 8
  • storage2.PNG
    storage2.PNG
    32.2 KB · Views: 10
  • storage1.PNG
    storage1.PNG
    22.5 KB · Views: 8
Hi, I admit I don't quite understand your question. What exactly you want to achieve.
Are you trying to use a part of sda in the storage of type "directory" in sdb? I don't know if it's possible without reformatting sdb first. From your post I understand that in sdb you already have got data (VM and LXC images).

Please post the result of pvesm status

One could reformat sdb to be another PV (physical volume) for LVM and add it into "pve" VG. But sdb contains data, doesn't it?...

Then shrink pve-root and use the regained space for a new LV (logical volume) for VMs and LXCs.

And there are even more questions. E.g. do you want to keep "directory" type storage? Or use LVM-thin? They have different features and possibilities. See the links below.

You should also consider what happens when one of the disks have failed.

Have you already studied https://pve.proxmox.com/pve-docs/chapter-pvesm.html and https://pve.proxmox.com/wiki/Storage ? Maybe you have, as you have written "stumped after hours of research" ;)

But first of all, have backups! :)
 
Hi,
what you can try is:
  1. reduce size of LV pve-root (lvreduce)
  2. create a partition sda4 with the freed up space, type LVM (sgdisk)
  3. create a PV of this partition sda4 (pvecreate) and a new Volume Group (vgcreate).
  4. create a new LV (lvcreate) for the agents.
  5. mount the new LV
  6. copy the data from /mnt/pve/agents to the new LV
  7. remove the agents storage from PVE and if required umount it
  8. wipe the disk sdb, initialize as gpt and for LVM with sgdisk
  9. create a PV of sdb (pvecreate)
  10. add the PV to the agents Volume Group (vgextend)
  11. extend the LV of the agents (lvextend)
  12. add the LV as directory to PVE storage
Having a backup is a good idea nevertheless.
 
Onslow said:
Hi, I admit I don't quite understand your question. What exactly you want to achieve.
Are you trying to use a part of sda in the storage of type "directory" in sdb? I don't know if it's possible without reformatting sdb first. From your post I understand that in sdb you already have got data (VM and LXC images).

Please post the result of pvesm status

One could reformat sdb to be another PV (physical volume) for LVM and add it into "pve" VG. But sdb contains data, doesn't it?...

Then shrink pve-root and use the regained space for a new LV (logical volume) for VMs and LXCs.

And there are even more questions. E.g. do you want to keep "directory" type storage? Or use LVM-thin? They have different features and possibilities. See the links below.

You should also consider what happens when one of the disks have failed.

Have you already studied https://pve.proxmox.com/pve-docs/chapter-pvesm.html and https://pve.proxmox.com/wiki/Storage ? Maybe you have, as you have written "stumped after hours of research" ;)

But first of all, have backups! :)
Click to expand...
Results from pvesm status:
Name Type Status Total Used Available %
agents dir active 3512847472 489273184 3023574288 13.93%
external dir disabled 0 0 0 N/A
local dir active 6910509448 454070716 6174572148 6.57%
snowball dir disabled 0 0 0 N/A

I guess in summary, im trying to shrink sda3 by 3Tb and move that 3Tb to sdb1. My current sever is running 6 drives all on hotswappable storage with a RAID setup. But i cant recall, i might of set the default partitions to the simpliist form when installing Proxmox thus local and agents.
 
fba said:
Hi,
what you can try is:
  1. reduce size of LV pve-root (lvreduce)
  2. create a partition sda4 with the freed up space, type LVM (sgdisk)
  3. create a PV of this partition sda4 (pvecreate) and a new Volume Group (vgcreate).
  4. create a new LV (lvcreate) for the agents.
  5. mount the new LV
  6. copy the data from /mnt/pve/agents to the new LV
  7. remove the agents storage from PVE and if required umount it
  8. wipe the disk sdb, initialize as gpt and for LVM with sgdisk
  9. create a PV of sdb (pvecreate)
  10. add the PV to the agents Volume Group (vgextend)
  11. extend the LV of the agents (lvextend)
  12. add the LV as directory to PVE storage
Having a backup is a good idea nevertheless.
Click to expand...
For creating a backup of the whole node1, how would you recommend i do that? Im pretty familiar with backups inside of Prox but doing it outside would be similar to a system image, correct? im reading about proxmox backup server right now, do you recommend using that?
 
redactedhosting said:
For creating a backup of the whole node1, how would you recommend i do that? Im pretty familiar with backups inside of Prox but doing it outside would be similar to a system image, correct?
Click to expand...
Not really. The recommended way is making backups of particular VMs and LXCs.

And you can make a backup of the hypervisor's system itself, especially if you made some customizations after the installation. Especially the contents of /etc and possibly /root because your history files (e.g. .bash_history ) can help to remind you what you have done in the system.
And copy ISO images if they aren't easily accessible currently.

redactedhosting said:
im reading about proxmox backup server right now, do you recommend using that?
Click to expand...
Yes, I recommend it. Proxmox Backup Server is an efficient and good product.
 
Onslow said:
No problem. Please post the outputs of the following commands:
df -hT
pvs
vgs
Click to expand...
Results from the commands:

Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 63G 0 63G 0% /dev
tmpfs tmpfs 13G 3.9M 13G 1% /run
/dev/mapper/pve-root ext4 6.5T 434G 5.8T 7% /
tmpfs tmpfs 63G 66M 63G 1% /dev/shm
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
efivarfs efivarfs 64K 30K 30K 51% /sys/firmware/efi/efivars
/dev/sdb1 xfs 3.3T 467G 2.9T 14% /mnt/pve/agents
/dev/fuse fuse 128M 44K 128M 1% /etc/pve
tmpfs tmpfs 13G 0 13G 0% /run/user/0
PV VG Fmt Attr PSize PFree
/dev/sda3 pve lvm2 a-- <6.55t 0
VG #PV #LV #SN Attr VSize VFree
pve 1 2 0 wz--n- <6.55t 0
 
Onslow said:
Not really. The recommended way is making backups of particular VMs and LXCs.

And you can make a backup of the hypervisor's system itself, especially if you made some customizations after the installation. Especially the contents of /etc and possibly /root because your history files (e.g. .bash_history ) can help to remind you what you have done in the system.
And copy ISO images if they aren't easily accessible currently.


Yes, I recommend it. Proxmox Backup Server is an efficient and good product.
Click to expand...
Awesome, ok I have a backup of all of the vms/LXcs on both node 1 (the one we want to change) and node 2. Now ill make a backup of the hypervisor itself. As far as the commands you have given me to run, im a little confused on so from what i understand is we are shrinking sda3 (root) creating sda4 ... 3.5Tb ... creating a group and then then im creating a LV storage block for *agents*? after that we are mounting it ? moving all of the data to the 3.5TB removing *Agents* directory storage, and then recreating it with an extra 3.5TB, resulting in 6Tb?

Sorry for the run on questions, just trying to understand the process so if it dosent go as smoothly i can understand where i am stumped. Thank you for the help and i look forward to your in depth explination : )
 
Yeah, that's the way. A Volume Group of a LVM can span over multiple partitions/disks (sda4 and sdb in this case).
At the moment you have a single Volume Group called pve only with only sda3 as Physical Volume and a separate disk /dev/sdb.
So there are two options: the way I described, leading to two Volume Groups, one for PVE and the other one for the agents.
The other option would be to use only a single Volume Group (pve) by adding the disk /dev/sdb to it. Same steps as described above, just leave out second half of step 3.
 
Onslow, thank you for the insight, i did not think of that and am now looking into how to convert it into ext4, Has anyone used gparted? i heard that might be safer to boot into and try this before i gum up it anymore lol.
 
If I'm correctly following the plan suggested by fba, you won't be converting the current /mnt/pve/agents into ext4 or anything. The filesystem type will be defined after point 4 in the procedure. I.e. when you create a new filesystem in the regained part of sda. You will be able to choose ext4 or xfs or whatever.
The current /mnt/pve/agents will be destroyed.

And you'll need some "live CD" to shrink the current /dev/mapper/pve-root filesystem and only after shrinking the filesystem, it will be possible to shrink the pve-root LV.
As such "live CD" the Gparted or SystemRescue will be OK, I think.
 
  • Like
Reactions: fba
One quick question onslow and fba, would migrating the agents to node 2 + backing them up before reformating agents etc be enough for this procedure? For example, in my thinking process, it would make sense, migrate everything off of node 1 to node 2 + backup and storage so my web applications and everything stay active. Then complete these procedures, i should then be able to migrate back everything that was moved to node 2, correct? (hopefully without too much complexity)
 
The shrinking requires the LV to be not mounted. This is as @Onslow mentioned possible with a live disk. The Proxmox Install iso is ok for that, just select Advanced Options and Debug Mode.

For preventing service interruption moving vm/lxc to another mode is a valid option.
Are we talking about a cluster or two independent nodes?
 
fba said:
The shrinking requires the LV to be not mounted. This is as @Onslow mentioned possible with a live disk. The Proxmox Install iso is ok for that, just select Advanced Options and Debug Mode.

For preventing service interruption moving vm/lxc to another mode is a valid option.
Are we talking about a cluster or two independent nodes?
Click to expand...
This is 2 nodes, connected via the clustering function in proxmox, I can accsess the first node if the 2nd is offline but if the main node1 is offline, node 2 2fa gltiches out sadly :/ do you think this will affect the procedure?
 
Nice for learning but is the solution really wanted as with a combined sda+sdb lvm your possibility to fail is twice after - when sda fail sdb is useless or if sdb fail sda is useless. Better to save data from small disk, reinstall there pve, move pve config from big to small pve inst and after make a data disk from the big disk.
 
Intresting, I never thought of that, how could one move an entire root/pve install to another storage without a complete re install re configure of the current system?
 
If you ever work just with directory storage (which a like) why not do a debian advanced inst where it's possible to install without lvm at all and after upgrade to pve ?! Otherwise to your question you would eg boot a live linux but again has to hack with the lvm volumes and different filesystem sizes ... all these block overhead knowleadge and command stuff isn't necessary if having just files where just the cmd's cp and/or mv are needed ...
:)
 
Thank you everyone for the brainstorming. After live booting into GParted, it looks like i won't be able to set up my storage like I would like without doing a complete tech refresh, as Walter suggested. So I think my current plan is to get a third node, maybe a thinkcenter micro pc or any micro pc to act as a third node (this will house all the data and keep everything online with the other node. Then I will proceed with a complete tech refresh of node1, update it to the latest version 9, and set the storage to how i think it would work well as which is 3tb on root and 6.5tb on agents. After that ill waterfall the method to the node 2 and node 3 doing the same.
Thank you again to everyone's brainstorming. I feel like this is just the multiple birds one stone approach. :)
 
  • Like
Reactions: waltar
You must log in or register to reply here.
Top Bottom