New Cluster and FreeNAS Appliance - Setup recommendations needed

[madjeff]

OK, let's start from the top. Been using ProxMox with great results at several locations. One of the locations is growing and we will be expanding the existing single server to a cluster config with attached NAS. I've been researching the forums and the web and am not wrapping my head around exactly how to do a few things, so I need some guidance as I move forward.

The current environment is a single Dell R610 Dual-quad w/64Gb RAM. 2 - 73Gb drives in RAID1 for system, 4 - 500Gb drives in RAID5 for VM storage. It's been a rock-solid performer and we are currently running 10-15 VMs (KVM), mostly Ubuntu 10.04 VMs with a Windows box or two thrown in. Disk images are in RAW format and stored locally on LVM on the RAID5 array.

The new environment consists of 2 more identical Dell R610's and a new FreeNAS appliance from iX Systems. If you are curious, this is the box. Everything is connected via iSCSI-aware Gbit switches, and performance to/from the NAS is smoking fast.

The cluster is created and synced, and I have iSCSI targets setup and working on the NAS. I connected the targets in ProxMox and then created an LVM group to incorporate the NAS storage into the cluster. So far so good.

The next thing I want to do is start migrating the locally-stored VMs off the master to the new nodes, moving the raw disk images in the process to the NAS storage. From here things get a little hazy for me and I need some guidance.

Initially I just tried to migrate a VM to the new node. I was under the impression you could migrate a KVM-based VM from one node to another and thought I would test it. The VM migrated to node2, but the disk image never copied. After reading more I realize the disk images have to be stored on the NAS for a direct migration to happen, is that correct?

My current goal at this point is to move the local-backed VM disk images to the NAS so I can enable migration of the VMs between cluster nodes as needed.

So, how do I proceed? I need to migrate some of these VM's but am not sure how to accomplish it, and my searching is not coming up with much. And of course, I'm under the gun as I need to get some of these moved by tomorrow. Any help or guidance or just pointing me to a good procedure would be greatly appreciated.

 

[jekader]

The VM migrated to node2, but the disk image never copied.

strange - it should! I also experimented with migrating KVM guests (my cluster nodes haver no common storage) and the raw image got copied from one node to another. How can a vm start on another node without having it's HDD image locally? Nodes don't mount each other's hard drives.

 

[udo]

...
So, how do I proceed? I need to migrate some of these VM's but am not sure how to accomplish it, and my searching is not coming up with much. And of course, I'm under the gun as I need to get some of these moved by tomorrow. Any help or guidance or just pointing me to a good procedure would be greatly appreciated.

Hi,
the recommendet way is: shutdown vm, backup (with all disk), qmrestore (with the switch --storage iscsivg).
Then your VM run on shared storage.

You can do that by hand: create for the VM a disk on the iscsi-san with the same size. shutdown vm. copy diskfile (you must have raw format): dd if=/var/lib/vz/images/112/ vm-112-disk-1.raw of=/dev/iscsivg/vm-112-disk-1 bs=1024k
after dd is finish, you change /etc/qemu-server/112.conf (in this example), that the first disk is the disk in the san (remove the local disk). start vm!

look for the right diskname with lvdisplay

Udo

 

[madjeff]

strange - it should! I also experimented with migrating KVM guests (my cluster nodes haver no common storage) and the raw image got copied from one node to another. How can a vm start on another node without having it's HDD image locally? Nodes don't mount each other's hard drives.

So this should work correctly then? Mine fails every time if I tried to copy a VM that has a disk image located on local (not shared) storage. I wonder what is causing the disk copy to fail? The logs don't give much hint other than that it failed.

 

[madjeff]

Hi,
the recommendet way is: shutdown vm, backup (with all disk), qmrestore (with the switch --storage iscsivg).
Then your VM run on shared storage.

I'll have to try that. To be honest I haven't done a lot of experimenting with the backup and qmrestore functions, usually just copy/clone images via dd.

You can do that by hand: create for the VM a disk on the iscsi-san with the same size. shutdown vm. copy diskfile (you must have raw format): dd if=/var/lib/vz/images/112/ vm-112-disk-1.raw of=/dev/iscsivg/vm-112-disk-1 bs=1024k
after dd is finish, you change /etc/qemu-server/112.conf (in this example), that the first disk is the disk in the san (remove the local disk). start vm!

look for the right diskname with lvdisplay

Udo

I tried to do a manual copy before but have not had much luck. I do the usual, create a VM, create a new disk, the dd copy from old to new like so:

dd if=/dev/vmstore/vm-101-disk-1 of=/dev/NasVMStore/vm-101-disk-1

Here's a portion of the lvdisplay:

pmox01:~# lvdisplay
  --- Logical volume ---
  LV Name                /dev/NasVMStore/vm-117-disk-1
  VG Name                NasVMStore
  LV UUID                BnVngU-IRBT-y1GJ-puRf-mKko-9nAV-OYgloI
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                8.00 GB
  Current LE             2048
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:18
   
  --- Logical volume ---
  LV Name                /dev/NasVMStore/vm-101-disk-1
  VG Name                NasVMStore
  LV UUID                Wt8Gl6-j9j5-F9Zv-JA92-83Ka-ygJt-NmynJ1
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                8.00 GB
  Current LE             2048
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:19
   
  --- Logical volume ---
  LV Name                /dev/vmstore/vm-101-disk-1
  VG Name                vmstore
  LV UUID                rYgTBc-p0PP-vJL6-UTl5-2egT-3fwG-mGPOvR
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                8.00 GB
  Current LE             2048
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:3

vmstore is the lv on the original server. NasVMStore is the new shared iscsi target lv. I can create new disks on the share just fine, but when I try and copy an existing I get failures.

mox01:~# dd if=/dev/vmstore/vm-101-disk-1 of=/dev/NasVMStore/vm-101-disk-1
dd: writing to `/dev/NasVMStore/vm-101-disk-1': No space left on device
18857+0 records in
18856+0 records out
9654272 bytes (9.7 MB) copied, 0.0910875 s, 106 MB/s
pmox01:~#

There's plenty of space on the nas (currently that iscsi target has 500Gb free) and I can create disks directly on the san no problem. The 101 disk is 8Gb.

Any other info needed I can provide.

 

[willie]

Hi first excuse my lame english

I recently came across a similar situation

I did it cloning disk with clonezilla.

I added a second HD (in hardware tab) from the shared storage. Rebooted the VM with the clonezilla iso loaded in the CDROM and cloned first (local) HD on the second (shared). After this, unloaded the iso. Disabled the local disk (hardware tab), changed the boot disk to the second (shared) in the options tab. And it works, its easier to do than to explain.

In addition if anything goes wrong just disable the new HD and re-enable the old and you have the VM online again. This way I migrated several machines a night.

Hope it helps

 

[udo]

...

mox01:~# dd if=/dev/vmstore/vm-101-disk-1 of=/dev/NasVMStore/vm-101-disk-1
dd: writing to `/dev/NasVMStore/vm-101-disk-1': No space left on device
18857+0 records in
18856+0 records out
9654272 bytes (9.7 MB) copied, 0.0910875 s, 106 MB/s
pmox01:~#

There's plenty of space on the nas (currently that iscsi target has 500Gb free) and I can create disks directly on the san no problem. The 101 disk is 8Gb.

Any other info needed I can provide.

Hi,
there must be something wrong with the nas. It's looks for me, that you write (to the cache) and then the job fails, because you have no write permissions to the nas?
But strange that you can create an logical volume....

Any error messages in the log (on the proxmox ve host, and nas)?

Udo

 

[madjeff]

I've triple-checked permissions on the NAS, the problem is not there. Nothing in the FreeNAS logs either. Looking under syslog tab in ProxMox web doesn't show anything out of the ordinary either.

So, I decided to take a step back. I deleted the test VMs I created on the second and third nodes via the web interface. I then went in to delete the NAS-located LVM Storage (named Nas_VM_Storage) and iscsi target (named iscsi_VM_Storage) I created previously in Proxmox, but when I click the dropdown and choose "Delete" I get the follow error popup:

Error: command '/sbin/vgchange -aln NasVMStore' failed with exit code 5

I can't delete it.

 

[tom]

remove outdated storage definition in /etc/pve/storage.cfg

 

[madjeff]

Also, I just want to make sure I have this clear in my head. Will ProxMox migrate a KVM with a raw disk located locally on that node, to another node? Will it copy that raw disk image to the second node local storage? Unfortunately I'm still unclear on this, and if it's suppose to work but is not then maybe the issue lies elsewhere in my config. I'm going to create a quick small vm and test again moving from one node to another, without involving the NAS device at all.

 

[udo]

I've triple-checked permissions on the NAS, the problem is not there. Nothing in the FreeNAS logs either. Looking under syslog tab in ProxMox web doesn't show anything out of the ordinary either.

So, I decided to take a step back. I deleted the test VMs I created on the second and third nodes via the web interface. I then went in to delete the NAS-located LVM Storage (named Nas_VM_Storage) and iscsi target (named iscsi_VM_Storage) I created previously in Proxmox, but when I click the dropdown and choose "Delete" I get the follow error popup:

Error: command '/sbin/vgchange -aln NasVMStore' failed with exit code 5

I can't delete it.

Hi,
try on the console:

lvdisplay
vgdisplay
pvdisplay

are the vg allways there? Open LVs? You can use "lvchange -f -a n NAME" and the same with vgchange (or try vgremove). The last is pvremove.

After that you can delete the storage entry in /etc/pve/storage.cfg

Udo

 

[madjeff]

remove outdated storage definition in /etc/pve/storage.cfg

Thanks! That fixed that issue.

 

[udo]

Also, I just want to make sure I have this clear in my head. Will ProxMox migrate a KVM with a raw disk located locally on that node, to another node? Will it copy that raw disk image to the second node local storage? Unfortunately I'm still unclear on this, and if it's suppose to work but is not then maybe the issue lies elsewhere in my config. I'm going to create a quick small vm and test again moving from one node to another, without involving the NAS device at all.

Hi,
only offline! For online migrations (which rock!) you need shared storage - e.g. the disk content are'nt copied.

Udo

 

[madjeff]

Hi,
try on the console:

lvdisplay
vgdisplay
pvdisplay

This is all on the master node, after I've removed the NAS storage entries.

lvdisplay:

pmox01:~# lvdisplay
  --- Logical volume ---
  LV Name                /dev/vmstore/vm-101-disk-1
  VG Name                vmstore
  LV UUID                rYgTBc-p0PP-vJL6-UTl5-2egT-3fwG-mGPOvR
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                8.00 GB
  Current LE             2048
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:3
   
  --- Logical volume ---
  LV Name                /dev/vmstore/vm-106-disk-1
  VG Name                vmstore
  LV UUID                oK5HxK-I53y-EwTd-pOe5-qkN6-5FdY-GoDBYU
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                8.00 GB
  Current LE             2048
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:4
   
  --- Logical volume ---
  LV Name                /dev/vmstore/vm-105-disk-1
  VG Name                vmstore
  LV UUID                IAdsVw-VNWq-bKEe-l63S-6lVW-YSXL-lCZRMN
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                20.00 GB
  Current LE             5120
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:5
   
  --- Logical volume ---
  LV Name                /dev/vmstore/vm-107-disk-1
  VG Name                vmstore
  LV UUID                f1P3uO-2gb4-GhAg-BDpL-NG8h-lQ9T-wNz20C
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                32.00 GB
  Current LE             8192
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:6
   
  --- Logical volume ---
  LV Name                /dev/vmstore/vm-108-disk-1
  VG Name                vmstore
  LV UUID                ku2T5R-1L3E-PTNB-23up-Ndez-ixPw-6Poy6u
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                150.00 GB
  Current LE             38400
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:7
   
  --- Logical volume ---
  LV Name                /dev/vmstore/vm-109-disk-1
  VG Name                vmstore
  LV UUID                iGKGD1-ILGU-nw0n-4O33-EBHS-sk7b-y7W2kv
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                150.00 GB
  Current LE             38400
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:8
   
  --- Logical volume ---
  LV Name                /dev/vmstore/vm-110-disk-1
  VG Name                vmstore
  LV UUID                It4QQa-qMpD-LNvU-6jqA-hDOf-Maem-eCLmwp
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                150.00 GB
  Current LE             38400
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:9
   
  --- Logical volume ---
  LV Name                /dev/vmstore/vm-111-disk-1
  VG Name                vmstore
  LV UUID                aHj08P-nKaI-vPQZ-2yOR-6IcD-k86l-Z88Clk
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                150.00 GB
  Current LE             38400
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:10
   
  --- Logical volume ---
  LV Name                /dev/vmstore/vm-114-disk-1
  VG Name                vmstore
  LV UUID                cTSBK8-ENEI-PwTH-kGBE-tbPe-3s87-S7iyxy
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                8.00 GB
  Current LE             2048
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:12
   
  --- Logical volume ---
  LV Name                /dev/vmstore/vm-115-disk-1
  VG Name                vmstore
  LV UUID                5Vy1sH-I9tH-uqXK-pP2S-jPAi-NDyg-OvwBbG
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                8.00 GB
  Current LE             2048
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:13
   
  --- Logical volume ---
  LV Name                /dev/vmstore/vm-112-disk-1
  VG Name                vmstore
  LV UUID                q0nDyL-RdKH-Po2z-3nQq-het4-761M-LdkBDC
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                8.00 GB
  Current LE             2048
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:14
   
  --- Logical volume ---
  LV Name                /dev/vmstore/vm-116-disk-1
  VG Name                vmstore
  LV UUID                2JjA4h-l9MF-eN5w-Jcis-Ad5r-b0Tj-aIhOX2
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                8.00 GB
  Current LE             2048
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:15
   
  --- Logical volume ---
  LV Name                /dev/pve/swap
  VG Name                pve
  LV UUID                bdK0sS-r2JS-AMRC-OgF9-L6aV-R6In-wqSf2g
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                8.38 GB
  Current LE             2144
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:0
   
  --- Logical volume ---
  LV Name                /dev/pve/root
  VG Name                pve
  LV UUID                ytgrRJ-jV8I-F4mX-l7Mr-axzt-4qkp-XZb2oW
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                16.75 GB
  Current LE             4288
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:1
   
  --- Logical volume ---
  LV Name                /dev/pve/data
  VG Name                pve
  LV UUID                U5YT48-EhV5-s580-YkKG-q1GM-OC67-xxC8uq
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                38.12 GB
  Current LE             9760
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           251:2
   
pmox01:~#

vgdisplay:

pmox01:~# vgdisplay
  --- Volume group ---
  VG Name               vmstore
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  746
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                12
  Open LV               11
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               1.36 TB
  PE Size               4.00 MB
  Total PE              357311
  Alloc PE / Size       179200 / 700.00 GB
  Free  PE / Size       178111 / 695.75 GB
  VG UUID               7VtMnS-BBX1-U5XU-jDBx-oCNp-F5RS-YXqUjn
   
  --- Volume group ---
  VG Name               pve
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               67.25 GB
  PE Size               4.00 MB
  Total PE              17215
  Alloc PE / Size       16192 / 63.25 GB
  Free  PE / Size       1023 / 4.00 GB
  VG UUID               da3yyr-RE45-Gnua-rfuc-orAu-2VZt-LCRCpf

 

[madjeff]

pvdisplay:

pmox01:~# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sdb
  VG Name               vmstore
  PV Size               1.36 TB / not usable 4.00 MB
  Allocatable           yes 
  PE Size (KByte)       4096
  Total PE              357311
  Free PE               178111
  Allocated PE          179200
  PV UUID               zZLIos-x4Pg-eZXQ-6oFs-C4DN-CeyF-2e4srk
   
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               pve
  PV Size               67.25 GB / not usable 2.41 MB
  Allocatable           yes 
  PE Size (KByte)       4096
  Total PE              17215
  Free PE               1023
  Allocated PE          16192
  PV UUID               zfCN7V-e5x2-wT1F-347G-veyg-7GU6-p1lafT

VM 101 disk is not connected to any VM at the moment, it was the first one I tried to migrate, and I would like to copy that over to a new VM once I get this figured out. The rest are all connected to live VMs.

 

[madjeff]

Hi,
only offline! For online migrations (which rock!) you need shared storage - e.g. the disk content are'nt copied.

Udo

OK, so what I'm hearing is offline migration is this; ProxMox copies the VM config info, but the copying of the actual disk image from one node to another is a manual process via backup or copying disk to disk. Am I stating that correctly? =)

Sorry for my slow thinking process today, been sick all week and the brain's still a little foggy. =)

 

[udo]

OK, so what I'm hearing is offline migration is this; ProxMox copies the VM config info, but the copying of the actual disk image from one node to another is a manual process via backup or copying disk to disk. Am I stating that correctly? =)

Sorry for my slow thinking process today, been sick all week and the brain's still a little foggy. =)

Hi,
no - in a cluster proxmox copy the vm-disk from the local storage (pve-data: /var/lib/vz) to the other cluster member (and the config).
Just try that with an small VM (~4G).

In the other post you write, that the other disks on the NAS are in use?! You don't dd to an active lv? This can be the explanation why you can't write...

Udo

 

[madjeff]

Well, it looks like it is now working. I recreated the iscsi target and LVM storage and then created a new VM on the master node with the disk on the NAS. Everything created ok, then I did the previous dd copy of my original vm-101 disk I had tried to copy previously to the nas and it copied perfectly. I then did a online migrate to node3 and it worked perfectly again.

Not sure what was different this time, but it worked. Next I'll test a local node to local node move with a small VM and see what happens.

 

[madjeff]

Hi,
no - in a cluster proxmox copy the vm-disk from the local storage (pve-data: /var/lib/vz) to the other cluster member (and the config).
Just try that with an small VM (~4G).

In the other post you write, that the other disks on the NAS are in use?! You don't dd to an active lv? This can be the explanation why you can't write...

Udo

OK, failed again.

I created a small test VM (vm-103) on the master node with a 4Gb RAW disk stored on the local LVM (Storage) and tried doing a migrate from node1 to node2:

Feb 18 14:35:23 pmox01 pvedaemon[17135]: migrating VM 103 from node 1 (localhost) to node 2 (10.10.10.103)
Feb 18 14:35:24 pmox01 qmigrate[17136]: starting migration of VM 103 to host '10.10.10.103'
Feb 18 14:35:24 pmox01 qmigrate[17136]: copying disk images
Feb 18 14:35:24 pmox01 qmigrate[17136]: starting VM on remote host '10.10.10.103'
Feb 18 14:35:24 pmox01 qmigrate[17136]: VM 103 stopping
Feb 18 14:35:25 pmox01 kernel: vmbr0: port 11(tap103i0d0) entering forwarding state
Feb 18 14:35:25 pmox01 kernel: vmbr0: port 11(tap103i0d0) entering disabled state
Feb 18 14:35:25 pmox01 qmigrate[17136]: online migrate failure - command '/usr/bin/ssh -c blowfish -o BatchMode=yes root@10.10.10.103 /usr/sbin/qm --skiplock start 103 --incoming tcp' failed with exit code 2
Feb 18 14:35:26 pmox01 qmigrate[17136]: migration finished with problems (duration 00:00:03)
Feb 18 14:35:26 pmox01 pvedaemon[17135]: VM 103 migration failed -

The web interface shows VM-103 on Node2, and if I click on it and look at the hardware tab I can see a hard disk listed, but the path/name is the same as the original node. In other words, the Volume ID is the same as it was on node1: Storage:vm-103-disk-1. But there is no LVM volume on node2 names Storage like there is on node1, just the standard pve local storage. Does that explanation make sense? =)

 

[dietmar]

I created a small test VM (vm-103) on the master node with a 4Gb RAW disk stored on the local LVM (Storage) and tried doing a migrate from node1 to node2:

Please can you post the VM configuration. And the storage.cfg too?