Various errors and tracebacks found

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Various errors and tracebacks found

jcalcote
Hi Yaguang,

I've been documenting the VSM/Ceph functional interface for our internal QA team - just light documentation to help them understand what they need to test. I've run into a few errors that I thought you might like to know about:

1. When I tried to remove an OSD from the cluster, it hung for a long time, and then finally indicated an error with barber pole animation in the status field of the cluster server list screen. I found the following entries in the vsm-scheduler.log:

vsm-scheduler.log:2015-11-03 20:04:31    ERROR [vsm.openstack.common.rpc.amqp] Exception during message handling
vsm-scheduler.log:ProcessExecutionError_Remote: Unexpected error while running command.
vsm-scheduler.log:Stderr: '2015-11-03 20:04:30.967548 7fc4f597f700  0 monclient(hunting): authenticate timed out after 300\n2015-11-03 20:04:30.967605 7fc4f597f700  0 librados: client.admin authentication error (110) Connection timed out\nError connecting to cluster: TimedOut\n'
vsm-scheduler.log:ProcessExecutionError: Unexpected error while running command.
vsm-scheduler.log:Stderr: '2015-11-03 20:04:30.967548 7fc4f597f700  0 monclient(hunting): authenticate timed out after 300\n2015-11-03 20:04:30.967605 7fc4f597f700  0 librados: client.admin authentication error (110) Connection timed out\nError connecting to cluster: TimedOut\n'
vsm-scheduler.log:2015-11-03 20:12:19    ERROR [vsm.openstack.common.rpc.amqp] Exception during message handling
vsm-scheduler.log:ProcessExecutionError_Remote: Unexpected error while running command.
vsm-scheduler.log:Stderr: '2015-11-03 20:12:19.257376 7ff9a26ae700  0 monclient(hunting): authenticate timed out after 300\n2015-11-03 20:12:19.257452 7ff9a26ae700  0 librados: client.admin authentication error (110) Connection timed out\nError connecting to cluster: TimedOut\n'
vsm-scheduler.log:ProcessExecutionError: Unexpected error while running command.
vsm-scheduler.log:Stderr: '2015-11-03 20:12:19.257376 7ff9a26ae700  0 monclient(hunting): authenticate timed out after 300\n2015-11-03 20:12:19.257452 7ff9a26ae700  0 librados: client.admin authentication error (110) Connection timed out\nError connecting to cluster: TimedOut\n'
vsm-scheduler.log:2015-11-03 20:12:19    ERROR [vsm.openstack.common.rpc.common] Returning exception Unexpected error while running command.
vsm-scheduler.log:Stderr: '2015-11-03 20:12:19.257376 7ff9a26ae700  0 monclient(hunting): authenticate timed out after 300\n2015-11-03 20:12:19.257452 7ff9a26ae700  0 librados: client.admin authentication error (110) Connection timed out\nError connecting to cluster: TimedOut\n'
vsm-scheduler.log:ProcessExecutionError: Unexpected error while running command.

(Note, the above output comes from an igrep of the log directory for "error", so the messages are not necessarily contiguous.)

2. Every time I've tried to use the "Reset Server Status" button in the Action field in that same screen, I get a trace back in the browser (I have horizon debug enabled - I imagine, if I didn't have debug enabled, I'd just see a 500 server error). The following is the stack trace:

Environment:


Request Method: POST
Request URL: https://vsm-controller/dashboard/vsm/storageservermgmt/

Django Version: 1.6.1
Python Version: 2.7.6
Installed Applications:
['vsm_dashboard',
 'django.contrib.contenttypes',
 'django.contrib.auth',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'compressor',
 'horizon',
 'vsm_dashboard.dashboards.vsm',
 'openstack_auth',
 'vsm_dashboard.dashboards.ifdash']
Installed Middleware:
('django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'horizon.middleware.HorizonMiddleware',
 'django.middleware.doc.XViewMiddleware',
 'django.middleware.locale.LocaleMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware')


Traceback:
File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py" in get_response
  112.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/lib/python2.7/dist-packages/horizon/decorators.py" in dec
  36.         return view_func(request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/horizon/decorators.py" in dec
  52.             return view_func(request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/horizon/decorators.py" in dec
  36.         return view_func(request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/django/views/generic/base.py" in view
  69.             return self.dispatch(request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/django/views/generic/base.py" in dispatch
  87.         return handler(request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/horizon/tables/views.py" in post
  221.         return self.get(request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/horizon/tables/views.py" in get
  157.         handled = self.construct_tables()
File "/usr/lib/python2.7/dist-packages/horizon/tables/views.py" in construct_tables
  148.             handled = self.handle_table(table)
File "/usr/lib/python2.7/dist-packages/horizon/tables/views.py" in handle_table
  124.         handled = self._tables[name].maybe_handle()
File "/usr/lib/python2.7/dist-packages/horizon/tables/base.py" in maybe_handle
  1607.                 return self.take_action(action_name, obj_id)
File "/usr/lib/python2.7/dist-packages/horizon/tables/base.py" in take_action
  1449.                 response = action.multiple(self, self.request, obj_ids)
File "/usr/lib/python2.7/dist-packages/horizon/tables/actions.py" in multiple
  302.                 return self.handle(data_table, request, object_ids)
File "/usr/lib/python2.7/dist-packages/horizon/tables/actions.py" in handle
  810.                 exceptions.handle(request, ignore=ignore)
File "/usr/lib/python2.7/dist-packages/horizon/exceptions.py" in handle
  334.     six.reraise(exc_type, exc_value, exc_traceback)
File "/usr/lib/python2.7/dist-packages/horizon/tables/actions.py" in handle
  794.                 self.action(request, datum_id)
File "/usr/share/vsm-dashboard/vsm_dashboard/wsgi/../../vsm_dashboard/dashboards/vsm/storageservermgmt/tables.py" in action
  91.         vsmapi.servers.reset_status(request, [obj_id])

Exception Type: AttributeError at /vsm/storageservermgmt/
Exception Value: 'module' object has no attribute 'servers'

I can reproduce this one every time, so if you need more information, just ask.

These issues are seen on a clean deployment (using vms with virtual drives as OSDs) of the 2.0 release tarball.

Thanks in advance for any insight you might have -
John
Reply | Threaded
Open this post in threaded view
|

RE: Various errors and tracebacks found

ywang19
Administrator

Hi John,

 

1.       Only from the vsm-scheduler can’t identify what’s the problem, Could you provide “ceph -s ” info and log files under /var/log/vsm log from controller node and agents node, and /var/log/ceph on vsm agents node?

2.       From the error information of email, it is a bug of vsm reset_status of server manage. Could you fill one JIRA ticket?

 

-yaguang

 

From: jcalcote [via vsm-discuss] [mailto:ml-node+[hidden email]]
Sent: Wednesday, November 04, 2015 11:01 AM
To: Wang, Yaguang
Subject: Various errors and tracebacks found

 

Hi Yaguang,

I've been documenting the VSM/Ceph functional interface for our internal QA team - just light documentation to help them understand what they need to test. I've run into a few errors that I thought you might like to know about:

1. When I tried to remove an OSD from the cluster, it hung for a long time, and then finally indicated an error with barber pole animation in the status field of the cluster server list screen. I found the following entries in the vsm-scheduler.log:

 
vsm-scheduler.log:2015-11-03 20:04:31    ERROR [vsm.openstack.common.rpc.amqp] Exception during message handling
vsm-scheduler.log:ProcessExecutionError_Remote: Unexpected error while running command.
vsm-scheduler.log:Stderr: '2015-11-03 20:04:30.967548 7fc4f597f700  0 monclient(hunting): authenticate timed out after 300\n2015-11-03 20:04:30.967605 7fc4f597f700  0 librados: client.admin authentication error (110) Connection timed out\nError connecting to cluster: TimedOut\n'
vsm-scheduler.log:ProcessExecutionError: Unexpected error while running command.
vsm-scheduler.log:Stderr: '2015-11-03 20:04:30.967548 7fc4f597f700  0 monclient(hunting): authenticate timed out after 300\n2015-11-03 20:04:30.967605 7fc4f597f700  0 librados: client.admin authentication error (110) Connection timed out\nError connecting to cluster: TimedOut\n'
vsm-scheduler.log:2015-11-03 20:12:19    ERROR [vsm.openstack.common.rpc.amqp] Exception during message handling
vsm-scheduler.log:ProcessExecutionError_Remote: Unexpected error while running command.
vsm-scheduler.log:Stderr: '2015-11-03 20:12:19.257376 7ff9a26ae700  0 monclient(hunting): authenticate timed out after 300\n2015-11-03 20:12:19.257452 7ff9a26ae700  0 librados: client.admin authentication error (110) Connection timed out\nError connecting to cluster: TimedOut\n'
vsm-scheduler.log:ProcessExecutionError: Unexpected error while running command.
vsm-scheduler.log:Stderr: '2015-11-03 20:12:19.257376 7ff9a26ae700  0 monclient(hunting): authenticate timed out after 300\n2015-11-03 20:12:19.257452 7ff9a26ae700  0 librados: client.admin authentication error (110) Connection timed out\nError connecting to cluster: TimedOut\n'
vsm-scheduler.log:2015-11-03 20:12:19    ERROR [vsm.openstack.common.rpc.common] Returning exception Unexpected error while running command.
vsm-scheduler.log:Stderr: '2015-11-03 20:12:19.257376 7ff9a26ae700  0 monclient(hunting): authenticate timed out after 300\n2015-11-03 20:12:19.257452 7ff9a26ae700  0 librados: client.admin authentication error (110) Connection timed out\nError connecting to cluster: TimedOut\n'
vsm-scheduler.log:ProcessExecutionError: Unexpected error while running command.


(Note, the above output comes from an igrep of the log directory for "error", so the messages are not necessarily contiguous.)

2. Every time I've tried to use the "Reset Server Status" button in the Action field in that same screen, I get a trace back in the browser (I have horizon debug enabled - I imagine, if I didn't have debug enabled, I'd just see a 500 server error). The following is the stack trace:

 
Environment:
 
 
Request Method: POST
Request URL: https://vsm-controller/dashboard/vsm/storageservermgmt/
 
Django Version: 1.6.1
Python Version: 2.7.6
Installed Applications:
['vsm_dashboard',
 'django.contrib.contenttypes',
 'django.contrib.auth',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'compressor',
 'horizon',
 'vsm_dashboard.dashboards.vsm',
 'openstack_auth',
 'vsm_dashboard.dashboards.ifdash']
Installed Middleware:
('django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'horizon.middleware.HorizonMiddleware',
 'django.middleware.doc.XViewMiddleware',
 'django.middleware.locale.LocaleMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware')
 
 
Traceback:
File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py" in get_response
  112.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/lib/python2.7/dist-packages/horizon/decorators.py" in dec
  36.         return view_func(request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/horizon/decorators.py" in dec
  52.             return view_func(request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/horizon/decorators.py" in dec
  36.         return view_func(request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/django/views/generic/base.py" in view
  69.             return self.dispatch(request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/django/views/generic/base.py" in dispatch
  87.         return handler(request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/horizon/tables/views.py" in post
  221.         return self.get(request, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/horizon/tables/views.py" in get
  157.         handled = self.construct_tables()
File "/usr/lib/python2.7/dist-packages/horizon/tables/views.py" in construct_tables
  148.             handled = self.handle_table(table)
File "/usr/lib/python2.7/dist-packages/horizon/tables/views.py" in handle_table
  124.         handled = self._tables[name].maybe_handle()
File "/usr/lib/python2.7/dist-packages/horizon/tables/base.py" in maybe_handle
  1607.                 return self.take_action(action_name, obj_id)
File "/usr/lib/python2.7/dist-packages/horizon/tables/base.py" in take_action
  1449.                 response = action.multiple(self, self.request, obj_ids)
File "/usr/lib/python2.7/dist-packages/horizon/tables/actions.py" in multiple
  302.                 return self.handle(data_table, request, object_ids)
File "/usr/lib/python2.7/dist-packages/horizon/tables/actions.py" in handle
  810.                 exceptions.handle(request, ignore=ignore)
File "/usr/lib/python2.7/dist-packages/horizon/exceptions.py" in handle
  334.     six.reraise(exc_type, exc_value, exc_traceback)
File "/usr/lib/python2.7/dist-packages/horizon/tables/actions.py" in handle
  794.                 self.action(request, datum_id)
File "/usr/share/vsm-dashboard/vsm_dashboard/wsgi/../../vsm_dashboard/dashboards/vsm/storageservermgmt/tables.py" in action
  91.         vsmapi.servers.reset_status(request, [obj_id])
 
Exception Type: AttributeError at /vsm/storageservermgmt/
Exception Value: 'module' object has no attribute 'servers'


I can reproduce this one every time, so if you need more information, just ask.

These issues are seen on a clean deployment (using vms with virtual drives as OSDs) of the 2.0 release tarball.

Thanks in advance for any insight you might have -
John


If you reply to this email, your message will be added to the discussion below:

http://vsm-discuss.33411.n7.nabble.com/Various-errors-and-tracebacks-found-tp234.html

To start a new topic under vsm-discuss, email [hidden email]
To unsubscribe from vsm-discuss, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

RE: Various errors and tracebacks found

jcalcote
I entered https://01.org/jira/browse/VSM-379 for the reset status button issue. I'll try to reproduce the other error and respond here with the information you requested.

John
Reply | Threaded
Open this post in threaded view
|

RE: Various errors and tracebacks found

ywang19
Administrator

Ok. will look at the issue.

 

-yaguang

 

From: jcalcote [via vsm-discuss] [mailto:ml-node+[hidden email]]
Sent: Thursday, November 05, 2015 1:12 AM
To: Wang, Yaguang
Subject: RE: Various errors and tracebacks found

 

I entered https://01.org/jira/browse/VSM-379 for the reset status button issue. I'll try to reproduce the other error and respond here with the information you requested.

John


If you reply to this email, your message will be added to the discussion below:

http://vsm-discuss.33411.n7.nabble.com/Various-errors-and-tracebacks-found-tp234p237.html

To start a new topic under vsm-discuss, email [hidden email]
To unsubscribe from vsm-discuss, click here.
NAML