Elasticsearch

Backing up elasticsearch database into a file

There is an opensource npm library that help to export and import elasticsearch index. Here is their tool elasticsearch-dump page where you can find more information how to do that.

# Backup index data to a file:
elasticdump \
  --input=http://production.es.com:9200/my_index \
  --output=/data/my_index_mapping.json \
  --type=mapping
elasticdump \
  --input=http://production.es.com:9200/my_index \
  --output=/data/my_index.json \
  --type=data

Useful commands

Test Elasticsearch is working curl -i http://production.es.com:9200/?pretty=true

Overall cluster health check curl -i http://production.es.com:9200/_cluster/health?pretty=true

Shard level health check curl -i http://production.es.com:9200/_cat/shards?v

View ES indices curl http://production.es.com:9200/_cat/indices?v

Common issues

Indexes set to readonly

If your indexes get set to readonly and can't be reindexed due to this, run the below. This is usually due to an OOM event or hitting the storage limit imposed by elastic by default - this should be 90% disk use. Where you enter the index name, this can be a wildcard e.g. magento2_* to perform on all indexes beginning with magento2_

curl -X PUT "http://production.es.com:9200/[INDEX_NAME]/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "index.blocks.read_only_allow_delete": null
}'

OOM events

If the Elasticsearch service keeps dying and the error logs mention outofmemoryevent and says java heap size, you can increase the Java heap size via the /etc/elasticsearch/jvm.options file. By default this is set to 1g. This value should not be more than 50% of the physical RAM. If you edit this file and change the below:

Version 5

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms4g
-Xmx4g

Version 7

For version 7 we change this be creating a new config file in /etc/elasticsearch/jvm.options.d/. this must end with .options.

e.g.

sudo vim /etc/elasticsearch/jvm.options.d/custom.options

# JVM Total Heap Space
-Xms2g
-Xmx2g

Errors relating to failed shards

If any errors are showing in the elasticsearch logs stating that shards have failed you can run the cluster healthcheck (likely to say RED and have many unassigned shards) and then the shard level healthcheck command above. If these show a large amount of unassigned shards then run the following:

curl -s -XPOST "http://production.es.com:9200/_cluster/reroute?retry_failed" | jq '
    .state.routing_table.indices
    | .[] | .shards | .[] | .[]
    | select(.unassigned_info.reason=="ALLOCATION_FAILED")
    '

Note: if you get command jq not found then you will need to run this first, then once installed, run the above again: yum install jq