Elasticsearch data migration

Lately, I have been working on migrating from elasticsearch 0.19 to 1.7 version. One of the tasks involves migrating data from old to new cluster.  In this post, I write about a tool for migrating the data over to the new version.

Firstly, you cannot move the index files directly, attempting results in following error in 1.7 ES:

Error injecting constructor, org.elasticsearch.ElasticsearchIllegalStateException: Could not find a state file to recover from among

Elasticsearch 0.19 version is ancient and not compatible for a direct migration. Hence the only solution is to read it from the old elasticsearch cluster and write to new elastic.

Followed a simple guideline while coming up for the tool:

1. Primary goal: "Just move the data over to new version!"
2. Nice to haves: "Monitor its progress"

A bare bone version of it is here in github capturing the essence of this migration. Primary goal is achieved by using Nodejs elasticsearch client library where given a index name, migrates based on created field in the document. A assumption being made that a document with timestamp is present but the solution can be used for any increasing sequence. Secondary goal is achieved by a in memory map which stores and updates the index status.

Checkout the README on github to set it up locally.

Whats pending:
A nice user experience.