Catch ConnectionError with Django-Haystack and ElasticSearch
I am currently using django-haystack and elasticsearch in a project and everything works as expected when elasticsearch is running.
Stack settings:
HAYSTACK_CONNECTIONS = {
'default': {
'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
'URL': 'http://127.0.0.1:9200/',
'INDEX_NAME': 'haystack',
},
}
HAYSTACK_SIGNAL_PROCESSOR = 'haystack.signals.RealtimeSignalProcessor'
I am using RealtimeSignalProcessor to update the index in real time.
The problem comes when elasticsearch is not working as trying to add / update any object gives us the following error:
ConnectionError(('Connection aborted.', error(111, 'Connection refused'))) caused by: ProtocolError(('Connection aborted.', error(111, 'Connection refused')))
Is there a way to catch / manage this error?
This would be useful in a production environment so that users can add / update objects without crashing when elasticsearch is down.
Thanks in advance.
source to share
I suggest you subclass ElasticSearchBackend
and wrap the methods update
, remove
and clear
around a decorator that catches exceptions. This way you keep elasticsearch functions, but you can override their behavior.
I use to wrap them with a decorator, soundless:
def mute_error(f):
def error_wrapper(*args, **kwargs):
try:
return f(*args, **kwargs)
except:
print('Connection Error')
return error_wrapper
Then add it to your project, configure HAYSTACK_BACKEND
:
HAYSTACK_CONNECTIONS = {
'default': {
'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
'URL': 'http://127.0.0.1:9200/',
'INDEX_NAME': 'haystack',
},
'robust_elasticsearch':{
'ENGINE': 'YOURAPP.backend.RobustElasticSearchEngine',
'URL': 'http://127.0.0.1:9200/',
'INDEX_NAME': 'haystack',
}
}
Take a look at the django-haystack documentation. You must also subclass BaseEngine , this is not described correctly.
Here is the code:
from django.utils.decorators import method_decorator
from haystack.backends.elasticsearch_backend import ElasticsearchSearchBackend, ElasticsearchSearchEngine
from haystack.backends import BaseEngine
from haystack.backends import log_query
from urllib3.exceptions import ProtocolError, ConnectionError
class RobustElasticSearchBackend(ElasticsearchSearchBackend):
"""A robust backend that doesn't crash when no connection is available"""
def mute_error(f):
def error_wrapper(self, *args, **kwargs):
try:
return f(self, *args, **kwargs)
except TransportError:
self.log.warn('Connection Error: elasticsearch communication error')
return error_wrapper
def __init__(self, connectionalias, **options):
super(RobustElasticSearchBackend, self).__init__(connectionalias, **options)
@mute_error
def update(self, indexer, iterable, commit=True):
super(RobustElasticSearchBackend, self).update(indexer, iterable, commit)
@mute_error
def remove(self, obj, commit=True):
super(RobustElasticSearchBackend, self).remove(obj, commit)
@mute_error
def clear(self, models=[], commit=True):
super(RobustElasticSearchBackend, self).clear(models, commit)
class RobustElasticSearchEngine(ElasticsearchSearchEngine):
backend = RobustElasticSearchBackend
We are just overriding the engine, not the SearchQuery subclass, since the one provided by the default elasticsearch class is enough for us.
source to share
To make this work for my setup I had to
import elasticsearch
and edit:
def mute_error(f):
def error_wrapper(self, *args, **kwargs):
try:
return f(self, *args, **kwargs)
except elasticsearch.TransportError:
self.log.warn('Connection Error: elasticsearch communication error')
return error_wrapper
source to share