New ZFS plugin causes a crushing web interface on NexentaStor

johnplv · Oct 31, 2013

After adding the disk to the virtual machine using the new ZFS plugin,
when i try to view a list of ZVOL on NexentaStor it's causes a crushing NMV

Code:

NMV Issue Report
------------------------------------------------------------------------
Title: NMV Internal Error: <class 'NZA.NZAObjectNotFound'>
Date:  Thu Oct 31 10:53:07 2013


Request: GET /data/scsitarget/zvols/ HTTP/1.1
Headers:
Accept = text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding = gzip,deflate,sdch
Accept-Language = ru,en;q=0.8
Connection = Keep-Alive
Content-Length = 
Content-Type = 
Cookie = ys-jobviewer_expanded=b%3A0; nmv_session=07c605fe3e1d2f684ef0e11c66cce9d9929aa8b2; session_id=d634c7d3042da1c312ccec42c79f92aef47e5681
Host = 127.0.0.1:4000
Referer = [URL]https://*******:2000/data/scsitarget/mappings/[/URL]
Remote-Addr = 127.0.0.1
User-Agent = Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 YaBrowser/13.10.1500.8765 Safari/537.36


Cookies:
nmv_session = 07c605fe3e1d2f684ef0e11c66cce9d9929aa8b2
session_id = d634c7d3042da1c312ccec42c79f92aef47e5681
ys-jobviewer_expanded = b%3A0


Params:
<empty>


Details:
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.5/cherrypy/_cphttptools.py", line 121, in _run
    self.main()
  File "/usr/lib/pymodules/python2.5/cherrypy/_cphttptools.py", line 273, in main
    body = page_handler(*virtual_path, **self.params)
  File "/usr/lib/pymodules/python2.5/turbogears/identity/conditions.py", line 244, in _wrapper
    return fn(*args, **kw)
  File "<string>", line 3, in index
  File "/usr/lib/pymodules/python2.5/turbogears/controllers.py", line 360, in expose
    *args, **kw)
  File "<string>", line 5, in run_with_transaction
  File "/usr/lib/pymodules/python2.5/turbogears/database.py", line 363, in so_rwt
    retval = func(*args, **kw)
  File "<string>", line 5, in _expose
  File "/usr/lib/pymodules/python2.5/turbogears/controllers.py", line 373, in <lambda>
    mapping, fragment, args, kw)))
  File "/usr/lib/pymodules/python2.5/turbogears/controllers.py", line 410, in _execute_func
    output = errorhandling.try_call(func, *args, **kw)
  File "/usr/lib/pymodules/python2.5/turbogears/errorhandling.py", line 77, in try_call
    return func(self, *args, **kw)
  File "<string>", line 3, in index
  File "/usr/lib/python2.5/site-packages/nmv/workarounds/__init__.py", line 105, in expose
    output = func(*args, **kwargs)
  File "/var/lib/nza/plugins/nmv/nmv-comstar/controllers/zvols.py", line 401, in index
  File "<string>", line 3, in __call__
  File "/usr/lib/pymodules/python2.5/turbogears/controllers.py", line 355, in expose
    *args, **kw)
  File "<string>", line 5, in _expose
  File "/usr/lib/pymodules/python2.5/turbogears/controllers.py", line 373, in <lambda>
    mapping, fragment, args, kw)))
  File "/usr/lib/pymodules/python2.5/turbogears/controllers.py", line 420, in _execute_func
    return _process_output(output, template, format, content_type, mapping, fragment)
  File "<string>", line 3, in _process_output
  File "/usr/lib/python2.5/site-packages/nmv/workarounds/__init__.py", line 234, in _process_output
    return func(output, *args, **kwargs)
  File "/usr/lib/pymodules/python2.5/turbogears/controllers.py", line 88, in _process_output
    fragment=fragment)
  File "/usr/bin/nmv.py", line 229, in render
    return _render(*args, **kwargs)
  File "/usr/lib/pymodules/python2.5/turbogears/view/base.py", line 159, in render
    return engine.render(**kw)
  File "/usr/lib/pymodules/python2.5/turbokid/kidsupport.py", line 220, in render
    output=output, format=format)
  File "/usr/lib/pymodules/python2.5/kid/__init__.py", line 301, in serialize
    raise_template_error(module=self.__module__)
  File "/usr/lib/pymodules/python2.5/kid/__init__.py", line 299, in serialize
    return serializer.serialize(self, encoding, fragment, format)
  File "/usr/lib/pymodules/python2.5/kid/serialization.py", line 107, in serialize
    text = ''.join(self.generate(stream, encoding, fragment, format))
  File "/usr/lib/pymodules/python2.5/kid/serialization.py", line 629, in generate
    for ev, item in self.apply_filters(stream, format):
  File "/usr/lib/pymodules/python2.5/kid/serialization.py", line 165, in format_stream
    for ev, item in stream:
  File "/usr/lib/pymodules/python2.5/kid/parser.py", line 221, in _coalesce
    for ev, item in stream:
  File "/usr/lib/pymodules/python2.5/kid/serialization.py", line 477, in inject_meta_tags
    for ev, item in stream:
  File "/usr/lib/pymodules/python2.5/kid/parser.py", line 179, in _track
    for p in stream:
  File "/usr/lib/pymodules/python2.5/kid/filter.py", line 32, in apply_matches
    item = stream.expand()
  File "/usr/lib/pymodules/python2.5/kid/parser.py", line 108, in expand
    for ev, item in self._iter:
  File "/usr/lib/pymodules/python2.5/kid/parser.py", line 179, in _track
    for p in stream:
  File "/usr/lib/pymodules/python2.5/kid/parser.py", line 221, in _coalesce
    for ev, item in stream:
  File "/var/lib/nza/plugins/nmv/nmv-comstar/templates/master.py", line 110, in _pull
  File "/usr/lib/pymodules/python2.5/turbogears/widgets/base.py", line 229, in __call__
    return self.display(*args, **params)
  File "/usr/lib/pymodules/python2.5/turbogears/widgets/meta.py", line 128, in lockwidget
    output = self.__class__.display(self, *args, **kw)
  File "/usr/lib/pymodules/python2.5/turbogears/widgets/base.py", line 389, in display
    return super(CompoundWidget, self).display(value, **params)
  File "/usr/lib/pymodules/python2.5/turbogears/widgets/base.py", line 296, in display
    return transform(params, self.template_c)
  File "/usr/lib/pymodules/python2.5/turbokid/kidsupport.py", line 234, in transform
    return kid.ElementStream(template.transform()).expand()
  File "/usr/lib/pymodules/python2.5/kid/parser.py", line 108, in expand
    for ev, item in self._iter:
  File "/usr/lib/pymodules/python2.5/kid/parser.py", line 179, in _track
    for p in stream:
  File "/usr/lib/pymodules/python2.5/kid/parser.py", line 179, in _track
    for p in stream:
  File "/usr/lib/pymodules/python2.5/kid/filter.py", line 26, in apply_matches
    for ev, item in stream:
  File "/usr/lib/pymodules/python2.5/kid/parser.py", line 179, in _track
    for p in stream:
  File "/usr/lib/pymodules/python2.5/kid/parser.py", line 221, in _coalesce
    for ev, item in stream:
  File "<string>", line 100, in _pull
  File "/var/lib/nza/plugins/nmv/nmv-comstar/widgets/base.py", line 282, in _display_column
  File "/var/lib/nza/plugins/nmv/nmv-comstar/controllers/zvols.py", line 302, in <lambda>
  File "/var/lib/nza/plugins/nmv/nmv-comstar/controllers/zvols.py", line 46, in is_zvol_shared
  File "/usr/lib/python2.5/site-packages/NZA/__init__.py", line 1109, in __call__
    return self.conn.call(self.path, self.suff, self.name, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/NZA/__init__.py", line 1247, in call
    return (pool or super(NMSLocalConnection, self).call)(path, suff, name, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/NZA/__init__.py", line 1269, in call
    return (conn and conn.call or call)(path, suff, name, *args, **kwargs)
  File "<string>", line 3, in call
  File "/usr/bin/nmv.py", line 277, in call
    return call(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/NZA/__init__.py", line 1055, in call
    raise nza_exc(*e.args)
[B]NZAObjectNotFound: Unable to locate the specified zvol 'datapool/vm-1101-disk-2'[/B]
[B]Error location in template file '/var/lib/nza/plugins/nmv/nmv-comstar/templates/master.kid'[/B]
[B]on line 46 between columns 12 and 43:[/B]
            <div>${widget(value, **params)} ...

but everything else continues to work, just does not work web interface on page "data/scsitarget/mappings/"
Now I get around this problem by removing the LUN mapping through the console (sbdadm delete-lu) and adding it again via the web interface.
In the old NexentaStor plugin all worked fine.
PS: NexentaStor ver. 3.1.3.5

mir · Oct 31, 2013

You have remembered to do this: "Nexenta: rm /root/.bash_profile. To avoid to go in nmc console by default"
Read more here: http://pve.proxmox.com/wiki/Storage:_ZFS

johnplv · Oct 31, 2013

this was done, otherwise the ZFS plugin might not work at all

mir · Oct 31, 2013

I have no clue. Is this perhaps a bug in 3.1.3.5? Latest available is 3.1.5.

3.1.4 lists among other things this enhancement:
13466 Updated map manager for Comstar / HA deployments.
(http://info.nexenta.com/rs/nexenta/images/doc_3.1_release_notes_3.1.4.pdf)

Maybe this is what you are experiencing because nobody has tested on 3.1.3?

johnplv · Oct 31, 2013

It's so sad

. Version 3.1.5 was just released and I think there is still a lot of people with version 3.1.3

mir · Oct 31, 2013

Well, version 3.1.4 was released more than 1½ years ago and I guess, but can give no certainty, that it will work on 3.1.4.

spirit · Nov 4, 2013

Hi,

I don't have problem on my side with 3.1.3.5,I'll try to do more tests.

(The only nmv problem I haved had is sometimes when I rename a volume (when creating a base volume), because nexenta api don't support it and sometime don't see in his own cache the new volumename).

I'll keep you in touch.

(I'll try to test also 3.1.5)

spirit · Nov 4, 2013

Mir,

I can reproduce the problem (same error) with the last version of zfsplugin from pve-storage git.

It's work fine with initial commited version of zfsplugin.

mir · Nov 4, 2013

I will have to study the diff

spirit · Nov 4, 2013

mir said:
I will have to study the diff

I'll try to debug too.
(Note that I can start perfectly the vm, and lun is accessible)

using delete-lu seem to fix the problem in nexenta gui, so it's seem to be related to lun and not the zvol itself.

spirit · Nov 4, 2013

Not sure, but before

Code:

 sub zfs_create_lu {
    my ($scfg, $zvol) = @_;
    .....
     my $digest = md5_hex($zvol);

now

Code:

sub zfs_create_lu {
    my ($scfg, $zvol) = @_;
    .....
    my $guid = zfs_request($scfg, undef, 'create_lu', "$base/$scfg->{pool}/$zvol");
  ...
}
sub run_lun_command {
    my ($scfg, $timeout, $method, @params) = @_;
   .....

   my $digest = md5_hex($params[0]);
...

}

So, now the digest is not computed from $zvol but from $base/$scfg->{pool}/$zvol.

mir · Nov 4, 2013

That sounds correct. I will create a patch solving the problem later today. Stay tuned

mir · Nov 4, 2013

A patch is available on the developer list.

johnplv · Nov 5, 2013

thank you very much.
where can I get this patch?

PS: Can I more ask you to add options for LUN mapping: Initiator Group, Target Group?

mir · Nov 5, 2013

johnplv said:
thank you very much.
where can I get this patch?

Please wait because tests has proofed that it seems to be either a bug in Nexenta or because Nexenta is picky about GUID.
Both solution requires another patch which we are currently working on. Hopefully will we have a solution later today.

johnplv said:
PS: Can I more ask you to add options for LUN mapping: Initiator Group, Target Group?

This is on my todo list. But new features will first be added after the code has stabilized.

Search

Search

New ZFS plugin causes a crushing web interface on NexentaStor

johnplv

Renowned Member

mir

Famous Member

johnplv

Renowned Member

mir

Famous Member

johnplv

Renowned Member

mir

Famous Member

spirit

Distinguished Member

spirit

Distinguished Member

mir

Famous Member

spirit

Distinguished Member

spirit

Distinguished Member

mir

Famous Member

mir

Famous Member

johnplv

Renowned Member

mir

Famous Member