ERROR nova.conductor.manager RemoteError: Remote error: MessagingTimeout Timed out waiting for a reply to message ID 220e4dfcff3d4c8197503bc123d98c2b
2021-02-25 11:03:01.751 3204 ERROR nova.conductor.manager [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch\n new_args[argname] = self.serializer.deserialize_entity(ctxt, arg)\n', u' File "/usr/lib/python2.7/site-packages/nova/rpc.py", line 135, in deserialize_entity\n return self._base.deserialize_entity(context, entity)\n', u' File "/usr/lib/python2.7/site-packages/nova/objects/base.py", line 242, in deserialize_entity\n entity = self._process_object(context, entity)\n', u' File "/usr/lib/python2.7/site-packages/nova/objects/base.py", line 204, in _process_object\n context, objprim, version_manifest)\n', u' File "/usr/lib/python2.7/site-packages/nova/conductor/api.py", line 38, in object_backport_versions\n object_versions)\n', u' File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 250, in object_backport_versions\n object_versions=object_versions)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 181, in call\n transport_options=self.transport_options)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 129, in _send\n transport_options=transport_options)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 646, in send\n transport_options=transport_options)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 634, in _send\n call_monitor_timeout)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 523, in wait\n message = self.waiters.get(msg_id, timeout=timeout)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 401, in get\n \'to message ID %s\' % msg_id)\n', u'MessagingTimeout: Timed out waiting for a reply to message ID 220e4dfcff3d4c8197503bc123d98c2b\n'].
2021-02-25 11:03:01.751 3204 ERROR nova.conductor.manager
2021-02-25 11:03:01.964 3203 WARNING oslo_messaging._drivers.amqpdriver [req-757391e3-e4ef-4e7f-881e-c4e3172f15a6 349102449f1d48d09a3f7ad138323ea3 e41dcda96cdf42aebac1ca48b689cfff - default default] reply_6bb32da76a234e479d6aae4939ffb4c2 doesn't exists, drop reply to a1f08fc61d8940f5af4caab8a8ccbb0d: AMQPDestinationNotFound: exchange reply_6bb32da76a234e479d6aae4939ffb4c2 doesn't exists
2021-02-25 11:03:01.965 3203 INFO oslo_messaging._drivers.amqpdriver [req-757391e3-e4ef-4e7f-881e-c4e3172f15a6 349102449f1d48d09a3f7ad138323ea3 e41dcda96cdf42aebac1ca48b689cfff - default default] The reply a1f08fc61d8940f5af4caab8a8ccbb0d cannot be sent reply_6bb32da76a234e479d6aae4939ffb4c2 reply queue don't exist after 60 sec abandoning...
2021-02-25 11:03:02.195 3204 WARNING nova.scheduler.utils [req-757391e3-e4ef-4e7f-881e-c4e3172f15a6 349102449f1d48d09a3f7ad138323ea3 e41dcda96cdf42aebac1ca48b689cfff - default default] Failed to compute_task_build_instances: Remote error: MessagingTimeout Timed out waiting for a reply to message ID 220e4dfcff3d4c8197503bc123d98c2b
[u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch\n new_args[argname] = self.serializer.deserialize_entity(ctxt, arg)\n', u' File "/usr/lib/python2.7/site-packages/nova/rpc.py", line 135, in deserialize_entity\n return self._base.deserialize_entity(context, entity)\n', u' File "/usr/lib/python2.7/site-packages/nova/objects/base.py", line 242, in deserialize_entity\n entity = self._process_object(context, entity)\n', u' File "/usr/lib/python2.7/site-packages/nova/objects/base.py", line 204, in _process_object\n context, objprim, version_manifest)\n', u' File "/usr/lib/python2.7/site-packages/nova/conductor/api.py", line 38, in object_backport_versions\n object_versions)\n', u' File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 250, in object_backport_versions\n object_versions=object_versions)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 181, in call\n transport_options=self.transport_options)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 129, in _send\n transport_options=transport_options)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 646, in send\n transport_options=transport_options)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 634, in _send\n call_monitor_timeout)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 523, in wait\n message = self.waiters.get(msg_id, timeout=timeout)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 401, in get\n \'to message ID %s\' % msg_id)\n', u'MessagingTimeout: Timed out waiting for a reply to message ID 220e4dfcff3d4c8197503bc123d98c2b\n'].: RemoteError: Remote error: MessagingTimeout Timed out waiting for a reply to message ID 220e4dfcff3d4c8197503bc123d98c2b
2021-02-25 11:03:02.195 3204 WARNING nova.scheduler.utils [req-757391e3-e4ef-4e7f-881e-c4e3172f15a6 349102449f1d48d09a3f7ad138323ea3 e41dcda96cdf42aebac1ca48b689cfff - default default] [instance: c54c338c-8b90-4327-92e5-06befcefb4c5] Setting instance to ERROR state.: RemoteError: Remote error: MessagingTimeout Timed out waiting for a reply to message ID 220e4dfcff3d4c8197503bc123d98c2b
2021-02-25 11:04:02.922 3203 ERROR oslo.messaging._drivers.impl_rabbit [req-32c0505f-ec95-4e14-bf43-c50d03a03422 - - - - -] [db8ae567-9f96-4197-a3f9-9c588b18c03d] AMQP server on controller:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out
2021-02-25 11:04:03.929 3203 INFO oslo.messaging._drivers.impl_rabbit [req-32c0505f-ec95-4e14-bf43-c50d03a03422 - - - - -] [db8ae567-9f96-4197-a3f9-9c588b18c03d] Reconnected to AMQP server on controller:5672 via [amqp] client with port 53480.
2021-02-25 11:04:28.974 3203 ERROR oslo.messaging._drivers.impl_rabbit [req-b79c8d9e-2b5c-4041-b824-20ea905c905e - - - - -] [0e8a18f6-ed2f-48e8-afc7-ce095475ae33] AMQP server on controller:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out
2021-02-25 11:04:29.982 3203 INFO oslo.messaging._drivers.impl_rabbit [req-b79c8d9e-2b5c-4041-b824-20ea905c905e - - - - -] [0e8a18f6-ed2f-48e8-afc7-ce095475ae33] Reconnected to AMQP server on controller:5672 via [amqp] client with port 53484.
2021-02-25 11:05:55.716 3203 ERROR oslo.messaging._drivers.impl_rabbit [req-adb6f4d9-3306-4026-9979-659e69585d12 - - - - -] [0026edf2-9b6d-4a36-83bd-647d3b70deed] AMQP server on controller:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: IOError: Server unexpectedly closed connection
2021-02-25 11:05:56.724 3203 INFO oslo.messaging._drivers.impl_rabbit [req-adb6f4d9-3306-4026-9979-659e69585d12 - - - - -] [0026edf2-9b6d-4a36-83bd-647d3b70deed] Reconnected to AMQP server on controller:5672 via [amqp] client with port 53486
2021-02-25 11:03:01.751 3204 ERROR nova.conductor.manager [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch\n new_args[argname] = self.serializer.deserialize_entity(ctxt, arg)\n', u' File "/usr/lib/python2.7/site-packages/nova/rpc.py", line 135, in deserialize_entity\n return self._base.deserialize_entity(context, entity)\n', u' File "/usr/lib/python2.7/site-packages/nova/objects/base.py", line 242, in deserialize_entity\n entity = self._process_object(context, entity)\n', u' File "/usr/lib/python2.7/site-packages/nova/objects/base.py", line 204, in _process_object\n context, objprim, version_manifest)\n', u' File "/usr/lib/python2.7/site-packages/nova/conductor/api.py", line 38, in object_backport_versions\n object_versions)\n', u' File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 250, in object_backport_versions\n object_versions=object_versions)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 181, in call\n transport_options=self.transport_options)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 129, in _send\n transport_options=transport_options)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 646, in send\n transport_options=transport_options)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 634, in _send\n call_monitor_timeout)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 523, in wait\n message = self.waiters.get(msg_id, timeout=timeout)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 401, in get\n \'to message ID %s\' % msg_id)\n', u'MessagingTimeout: Timed out waiting for a reply to message ID 220e4dfcff3d4c8197503bc123d98c2b\n'].
2021-02-25 11:03:01.751 3204 ERROR nova.conductor.manager
2021-02-25 11:03:01.964 3203 WARNING oslo_messaging._drivers.amqpdriver [req-757391e3-e4ef-4e7f-881e-c4e3172f15a6 349102449f1d48d09a3f7ad138323ea3 e41dcda96cdf42aebac1ca48b689cfff - default default] reply_6bb32da76a234e479d6aae4939ffb4c2 doesn't exists, drop reply to a1f08fc61d8940f5af4caab8a8ccbb0d: AMQPDestinationNotFound: exchange reply_6bb32da76a234e479d6aae4939ffb4c2 doesn't exists
2021-02-25 11:03:01.965 3203 INFO oslo_messaging._drivers.amqpdriver [req-757391e3-e4ef-4e7f-881e-c4e3172f15a6 349102449f1d48d09a3f7ad138323ea3 e41dcda96cdf42aebac1ca48b689cfff - default default] The reply a1f08fc61d8940f5af4caab8a8ccbb0d cannot be sent reply_6bb32da76a234e479d6aae4939ffb4c2 reply queue don't exist after 60 sec abandoning...
2021-02-25 11:03:02.195 3204 WARNING nova.scheduler.utils [req-757391e3-e4ef-4e7f-881e-c4e3172f15a6 349102449f1d48d09a3f7ad138323ea3 e41dcda96cdf42aebac1ca48b689cfff - default default] Failed to compute_task_build_instances: Remote error: MessagingTimeout Timed out waiting for a reply to message ID 220e4dfcff3d4c8197503bc123d98c2b
[u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch\n new_args[argname] = self.serializer.deserialize_entity(ctxt, arg)\n', u' File "/usr/lib/python2.7/site-packages/nova/rpc.py", line 135, in deserialize_entity\n return self._base.deserialize_entity(context, entity)\n', u' File "/usr/lib/python2.7/site-packages/nova/objects/base.py", line 242, in deserialize_entity\n entity = self._process_object(context, entity)\n', u' File "/usr/lib/python2.7/site-packages/nova/objects/base.py", line 204, in _process_object\n context, objprim, version_manifest)\n', u' File "/usr/lib/python2.7/site-packages/nova/conductor/api.py", line 38, in object_backport_versions\n object_versions)\n', u' File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 250, in object_backport_versions\n object_versions=object_versions)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 181, in call\n transport_options=self.transport_options)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 129, in _send\n transport_options=transport_options)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 646, in send\n transport_options=transport_options)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 634, in _send\n call_monitor_timeout)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 523, in wait\n message = self.waiters.get(msg_id, timeout=timeout)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 401, in get\n \'to message ID %s\' % msg_id)\n', u'MessagingTimeout: Timed out waiting for a reply to message ID 220e4dfcff3d4c8197503bc123d98c2b\n'].: RemoteError: Remote error: MessagingTimeout Timed out waiting for a reply to message ID 220e4dfcff3d4c8197503bc123d98c2b
2021-02-25 11:03:02.195 3204 WARNING nova.scheduler.utils [req-757391e3-e4ef-4e7f-881e-c4e3172f15a6 349102449f1d48d09a3f7ad138323ea3 e41dcda96cdf42aebac1ca48b689cfff - default default] [instance: c54c338c-8b90-4327-92e5-06befcefb4c5] Setting instance to ERROR state.: RemoteError: Remote error: MessagingTimeout Timed out waiting for a reply to message ID 220e4dfcff3d4c8197503bc123d98c2b
2021-02-25 11:04:02.922 3203 ERROR oslo.messaging._drivers.impl_rabbit [req-32c0505f-ec95-4e14-bf43-c50d03a03422 - - - - -] [db8ae567-9f96-4197-a3f9-9c588b18c03d] AMQP server on controller:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out
2021-02-25 11:04:03.929 3203 INFO oslo.messaging._drivers.impl_rabbit [req-32c0505f-ec95-4e14-bf43-c50d03a03422 - - - - -] [db8ae567-9f96-4197-a3f9-9c588b18c03d] Reconnected to AMQP server on controller:5672 via [amqp] client with port 53480.
2021-02-25 11:04:28.974 3203 ERROR oslo.messaging._drivers.impl_rabbit [req-b79c8d9e-2b5c-4041-b824-20ea905c905e - - - - -] [0e8a18f6-ed2f-48e8-afc7-ce095475ae33] AMQP server on controller:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out
2021-02-25 11:04:29.982 3203 INFO oslo.messaging._drivers.impl_rabbit [req-b79c8d9e-2b5c-4041-b824-20ea905c905e - - - - -] [0e8a18f6-ed2f-48e8-afc7-ce095475ae33] Reconnected to AMQP server on controller:5672 via [amqp] client with port 53484.
2021-02-25 11:05:55.716 3203 ERROR oslo.messaging._drivers.impl_rabbit [req-adb6f4d9-3306-4026-9979-659e69585d12 - - - - -] [0026edf2-9b6d-4a36-83bd-647d3b70deed] AMQP server on controller:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: IOError: Server unexpectedly closed connection
2021-02-25 11:05:56.724 3203 INFO oslo.messaging._drivers.impl_rabbit [req-adb6f4d9-3306-4026-9979-659e69585d12 - - - - -] [0026edf2-9b6d-4a36-83bd-647d3b70deed] Reconnected to AMQP server on controller:5672 via [amqp] client with port 53486
