i'm having some trouble with an openstack testlab(one node setup) i am currently using. i deployed it with ansible, there was no problem at all till i added an additional compute node. Before i rebootet the server openstack showed no error but after the reboot, i get following errors when i execute nova service-list
ERROR (ClientException): Unexpected API Error. Please report this at and attach the Nova API log if possible. <type 'exceptions.ValueError'> (HTTP 500) (Request-ID: req-5dbb3a59-5783-4e5a-b6cd-f7fa5bc8db29) nova-api log shows this:
2015-11-30 12:00:44.972 3661 DEBUG nova.osapi_compute.wsgi.server [-] (3661) accepted ('::ffff:192.168.10.162', 46282, 0, 0) server /usr/lib/python2.7/dist-packages/eventlet/wsgi.py:826 2015-11-30 12:00:44.976 3661 DEBUG keystoneclient.session [-] REQ: curl -g -i -X GET liberty.$my_domain:35357/v3/auth/tokens -H "X-Subject-Token: {SHA1}a3568885d6c05c121f5cbccd6e19137345bf0593" -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}201d648c5b922dee3988a17d63e517dde7163824" _http_log_request /usr/lib/python2.7/dist-packages/keystoneclient/session.py:198 2015-11-30 12:00:45.021 3661 DEBUG keystoneclient.session [-] RESP: [200] Content-Length: 6723 X-Subject-Token: {SHA1}a3568885d6c05c121f5cbccd6e19137345bf0593 Vary: X-Auth-Token Keep-Alive: timeout=5, max=100 Server: Apache/2.4.7 (Ubuntu) Connection: Keep-Alive Date: Mon, 30 Nov 2015 11:00:44 GMT x-openstack-request-id: req-6404e2ea-67e3-44f0-8cd9-995bcea1cd25 Content-Type: application/json X-Distribution: Ubuntu RESP BODY: {"token": {"methods": ["password"], "roles": [{"id": "210649a5ce144f0c906e4f70628ec6ad", "name": "admin"}], "expires_at": "2015-11-30T12:00:44.958292Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "306aae4bab5a44308fd2ff3a8c05fc31", "name": "admin"}, "catalog": "<removed>", "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "2325518556b74f4cb29ba61e7718063a", "name": "admin"}, "audit_ids": ["RsXfP4BvTcStIyHYefcs4Q"], "issued_at": "2015-11-30T11:00:44.958325Z"}} _http_log_response /usr/lib/python2.7/dist-packages/keystoneclient/session.py:215 2015-11-30 12:00:45.026 3661 DEBUG nova.api.openstack.wsgi [req-2aa83bc8-4f1a-4c2c-9b8c-0b9460538699 2325518556b74f4cb29ba61e7718063a 306aae4bab5a44308fd2ff3a8c05fc31 - - -] Calling method '<bound method VersionsController.show of <nova.api.openstack.compute.versionsV21.VersionsController object at 0x7fdbee1eadd0>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:792 2015-11-30 12:00:45.030 3661 INFO nova.osapi_compute.wsgi.server [req-2aa83bc8-4f1a-4c2c-9b8c-0b9460538699 2325518556b74f4cb29ba61e7718063a 306aae4bab5a44308fd2ff3a8c05fc31 - - -] ::ffff:192.168.10.162 "GET /v2/ HTTP/1.1" status: 200 len: 597 time: 0.0556979 2015-11-30 12:00:45.234 3661 DEBUG nova.api.openstack.wsgi [req-c2969f62-f3e7-450a-9cef-080904a4e3df 2325518556b74f4cb29ba61e7718063a 306aae4bab5a44308fd2ff3a8c05fc31 - - -] Calling method '<bound method ServiceController.index of <nova.api.openstack.compute.services.ServiceController object at 0x7fdbee26b350>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:792 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions [req-c2969f62-f3e7-450a-9cef-080904a4e3df 2325518556b74f4cb29ba61e7718063a 306aae4bab5a44308fd2ff3a8c05fc31 - - -] Unexpected exception in API method 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/services.py", line 188, in index 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions _services = self._get_services_list(req) 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/services.py", line 80, in _get_services_list 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions _services = self._get_services(req) 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/services.py", line 44, in _get_services 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions _services = self.host_api.service_get_all(context, set_zones=True) 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3492, in service_get_all 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions set_zones=set_zones) 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 171, in wrapper 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs) 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/objects/service.py", line 318, in get_all 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions db_services) 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 367, in obj_make_list 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions **extra_args) 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/objects/service.py", line 169, in _from_db_object 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions service[key] = db_service[key] 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 702, in __setitem__ 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions setattr(self, name, value) 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 71, in setter 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions field_value = field.coerce(self, name, value) 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/fields.py", line 187, in coerce 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions return self._null(obj, attr) 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/fields.py", line 165, in _null 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions raise ValueError(_("Field `%s' cannot be None") % attr) 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions ValueError: Field `forced_down' cannot be None 2015-11-30 12:00:45.261 3661 ERROR nova.api.openstack.extensions 2015-11-30 12:00:45.265 3661 INFO nova.api.openstack.wsgi [req-c2969f62-f3e7-450a-9cef-080904a4e3df 2325518556b74f4cb29ba61e7718063a 306aae4bab5a44308fd2ff3a8c05fc31 - - -] HTTP exception thrown: Unexpected API Error. Please report this at and attach the Nova API log if possible. <type 'exceptions.ValueError'> 2015-11-30 12:00:45.266 3661 DEBUG nova.api.openstack.wsgi [req-c2969f62-f3e7-450a-9cef-080904a4e3df 2325518556b74f4cb29ba61e7718063a 306aae4bab5a44308fd2ff3a8c05fc31 - - -] Returning 500 to user: Unexpected API Error. Please report this at and attach the Nova API log if possible. <type 'exceptions.ValueError'> __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1175 2015-11-30 12:00:45.270 3661 INFO nova.osapi_compute.wsgi.server [req-c2969f62-f3e7-450a-9cef-080904a4e3df 2325518556b74f4cb29ba61e7718063a 306aae4bab5a44308fd2ff3a8c05fc31 - - -] ::ffff:192.168.10.162 "GET /v2/306aae4bab5a44308fd2ff3a8c05fc31/os-services HTTP/1.1" status: 500 len: 425 time: 0.0442750 my /etc/nova/nova.conf looks like this
DEFAULT use_ipv6 = True my_ip = $IP rpc_backend = rabbit enabled_apis=osapi_compute,metadata verbose = True debug = True log_dir = /var/log/nova state_path = /var/lib/nova s3_host = $IP ec2_host = $IP ec2_dmz_host = $IP api_paste_config = /etc/nova/api-paste.ini resume_guests_state_on_host_boot = True default_ephemeral_format = ext4 force_config_drive = True force_dhcp_release = True osapi_compute_listen = :: osapi_compute_listen_port = 8774 Scheduler scheduler_driver = nova.scheduler.filter_scheduler.FilterScheduler Auth auth_strategy = keystone keystone_ec2_url = $IP:5000/v2.0/ec2tokens VNC configuration - Dual-Stacked novnc_enabled = True novncproxy_host = :: novncproxy_port = 6080 Cinder volume_api_class = nova.volume.cinder.API NETWORK - NEUTRON network_api_class = nova.network.neutronv2.api.API security_group_api = neutron linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver firewall_driver = nova.virt.firewall.NoopFirewallDriver neutron url = $IP:9696 auth_url = $IP:35357 auth_plugin = password project_domain_id = default user_domain_id = default region_name = RegionOne project_name = service username = neutron password = $Password service_metadata_proxy = True metadata_proxy_shared_secret = UltraMeta13 SPICE configuration - Dual-Stacked spice enabled = True html5proxy_host = :: html5proxy_base_url = $IP:6082/spice_auto.html keymap = en-us database connection = mysql+pymysql://novaUser:$password@$IP/nova oslo_messaging_rabbit rabbit_host = $IP rabbit_userid = openstack rabbit_password = $Pasword oslo_concurrency lock_path = /var/lock/nova keystone_authtoken auth_uri = $IP:5000 auth_url = $IP:35357 auth_plugin = password project_domain_id = default user_domain_id = default project_name = service username = nova password = $Password 1 Answer
We ran into this issue when upgrading from kilo to liberty. We did the following fix:
On the nova database we ran:
select id,host,deleted,forced_down from services ; This resulted in some services with a forced_down value of NULL.
To fix we ran:
update services set forced_down=0 where id in (list of ids from earlier query); 
