after removing an iscsi storage definition I can't connect to the web ui

K

kawarmc

New Member
Jun 15, 2009
28
3
3
Hello to all.
First of all, thanks to all the developers for the great job. This is a fantastic piece of software.

I've had a problem today I am not able to solve. Long time ago I defined an iscsi connection to be used as a backup storage for the VMs.

I never used that storage so today I decided to remove it. There were no files on it at all, actually, it was going to be used as a mounted directory but never created an FS on it.

After removing the iscsi defition from the web ui, the connection to apache got frozen and after several minutes it gave me an error 500 about perl not responding.

If I connect to the cli using ssh, when I issue the command "qm list", nothing is shown and I have to Ctrl-C to quit from qm.

I've restarted pvetunnel, pvedaemon and pvemirror with no luck at all.

I can't restart the host as it contains about 20 VMs I can't stop, and I'm not able to migrate them to other nodes on the cluster.

Could you give me some advice on what to do?

Thanks to all.
 
Now, I can't even connect to the web UI. It keeps telling me that my username or password are incorrect.

I know for sure that if I reboot the machine everything will come back online again, but i can't do that.
 
kawarmc said:
Now, I can't even connect to the web UI. It keeps telling me that my username or password are incorrect.

I know for sure that if I reboot the machine everything will come back online again, but i can't do that.
Click to expand...
Hi,
use ssh to connect to the server. The gui from pve 1.x had trouble if one defined storage don't work. Perhaps you have remove the wrong storage?
You wrote, you don't have an filesystem on it. You are sure, that you don't use this storage for lvm?
Look with pvdisplay, vgdisplay and lvdisplay.

Udo
 
The storage was not used at all. It wasn't on any pv or anything, I double checked that before removing it.

But, as you said, I tried vgdisplay and pvdisplay and both commands got frozen. I couldn't even terminate them with Ctl-C.

I'm out of ideas...
 
kawarmc said:
The storage was not used at all. It wasn't on any pv or anything, I double checked that before removing it.

But, as you said, I tried vgdisplay and pvdisplay and both commands got frozen. I couldn't even terminate them with Ctl-C.

I'm out of ideas...
Click to expand...
Hi,
then you should see something in messages or syslog (/var/log/)

Udo
 
I've read both logfiles several times, and I thought there was nothing important until I saw the following kernel error:

Jul 8 10:57:58 esx1 kernel: INFO: task vgs:26015 blocked for more than 120 seconds.
Jul 8 10:57:58 esx1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 8 10:57:58 esx1 kernel: vgs D ffff88103328f000 0 26015 21774 0x00000000
Jul 8 10:57:58 esx1 kernel: ffff881035915000 0000000000000086 ffff88102f26a340 ffff88102f26a340
Jul 8 10:57:58 esx1 kernel: 0000000000000002 ffff881031fa4960 000000000000fa40 ffff88063aa13fd8
Jul 8 10:57:58 esx1 kernel: 0000000000016940 0000000000016940 ffff88103328f000 ffff88103328f2f8
Jul 8 10:57:58 esx1 kernel: Call Trace:
Jul 8 10:57:58 esx1 kernel: [<ffffffff81169d22>] ? blk_peek_request+0x18b/0x19f
Jul 8 10:57:58 esx1 kernel: [<ffffffff811692ec>] ? blk_insert_cloned_request+0x5e/0x67
Jul 8 10:57:58 esx1 kernel: [<ffffffff812482ec>] ? dm_table_unplug_all+0x4b/0xb4
Jul 8 10:57:58 esx1 kernel: [<ffffffff81247f62>] ? dm_request_fn+0x181/0x1a3
Jul 8 10:57:58 esx1 kernel: [<ffffffff8131478d>] ? io_schedule+0x9b/0xfc
Jul 8 10:57:58 esx1 kernel: [<ffffffff81117906>] ? __blockdev_direct_IO+0x910/0xa60
Jul 8 10:57:58 esx1 kernel: [<ffffffff8117ca1f>] ? kobject_get+0x12/0x17
Jul 8 10:57:58 esx1 kernel: [<ffffffff81115d77>] ? blkdev_direct_IO+0x45/0x4a
Jul 8 10:57:58 esx1 kernel: [<ffffffff8111508f>] ? blkdev_get_blocks+0x0/0x8b
Jul 8 10:57:58 esx1 kernel: [<ffffffff810b82a0>] ? generic_file_aio_read+0xf6/0x538
Jul 8 10:57:58 esx1 kernel: [<ffffffff812103b7>] ? get_device+0x14/0x1a
Jul 8 10:57:58 esx1 kernel: [<ffffffff81245bc1>] ? dm_blk_open+0x10/0x5f
Jul 8 10:57:58 esx1 kernel: [<ffffffffa035622a>] ? sim_systemcall+0x51/0x263 [simfs]
Jul 8 10:57:58 esx1 kernel: [<ffffffff811164df>] ? blkdev_open+0x0/0x96
Jul 8 10:57:58 esx1 kernel: [<ffffffff81116546>] ? blkdev_open+0x67/0x96
Jul 8 10:57:58 esx1 kernel: [<ffffffff810f15fd>] ? do_sync_read+0xce/0x113
Jul 8 10:57:58 esx1 kernel: [<ffffffff810f4b73>] ? cp_new_stat+0xe9/0xfc
Jul 8 10:57:58 esx1 kernel: [<ffffffff81066a16>] ? autoremove_wake_function+0x0/0x2e
Jul 8 10:57:58 esx1 kernel: [<ffffffff81115322>] ? block_ioctl+0x38/0x3c
Jul 8 10:57:58 esx1 kernel: [<ffffffff810fd532>] ? vfs_ioctl+0x21/0x6c
Jul 8 10:57:58 esx1 kernel: [<ffffffff810fda80>] ? do_vfs_ioctl+0x48d/0x4cb
Jul 8 10:57:58 esx1 kernel: [<ffffffff810f214e>] ? vfs_read+0xa6/0xff
Jul 8 10:57:58 esx1 kernel: [<ffffffff810f22c1>] ? sys_read+0x49/0xc4
Jul 8 10:57:58 esx1 kernel: [<ffffffff81010c12>] ? system_call_fastpath+0x16/0x1b


It seems that part of the LVM code on the kernel broke. I guess I'll finally have to reboot the machine...
 
You must log in or register to reply here.
Top Bottom