When you are operating Elasticsearch clusters, you will probably need some form of managing the lifetime of your indices. You could also need a way of handling migration from hot to warm nodes. Previously this was typically handled by a separate tool from Elastic called curator.
We have been running with Curator handling our indices for a long time, but we feel handling indices should be builtin functionality. We have had some scripts running in 3 steps as a cron job:
- rollover – runs every 15 min
- migration – runs every hour
- deletion – runs once per day
The configuration files for Curator could prove to a little bit of a pain.
But when Elastic released 6.7.0 – their new Index Lifecycle Management was now a builtin feature making our daily administation simpler. No longer do you need an external cron job running curator. All ILM tasks can be done from within Kibana.
Migration from curator
So this is all good. But what do you actually need to do in order to start migration away from curator and instead start using ILM.
You dont have to migrate every thing in a big bang approach, just move 1 thing at a time. Most of our big indices are using rollover approach, so that will be the focus here.
We came up with these steps in order to do the migration for a specific group of indices.
- Create ILM Policies
- Create one for rollover and one for already rolled over indices
- Apply policy for already rolled over indices
- Attach rollover policy to template
- Perform rollover
- Remove curator support for current group of indices
Create ILM Policies
It’s straightforward to create a policy, just fire up Kibana, Navigate to Management and Index LifeCycle Policies and Create Policy.
So in this example we create 2 policies
We create a set of 2 policies per group of indices. If everything you have are using the same periods for hot/warm/deletion, you only need 2 policies for everything. But in order to allow for customization per group of indices, we create policies per group.
So the dns-standard policy is used for existing indices. So it will not contain a rollover phase.
Edit the dns-standard policy again and click the save as new at the top, enable the rollover phase and save it as dns-standard-rollover.
Now please verify the settings in the ILM policies matches your configuration from your old curator jobs 😉
In order to apply a policy to a set of indices, you need to goto Kibana Developer tools.
Once you run that , your current DNS indices will be handled by ILM.
Attach rollover policy
In order to attach the rollover policy to a template click actions for the rollover policy. You will be asked to select a template name and you need to enter the write alias for the rollover index.
Next step is to do a manual rollover of your DNS alias. You will do that from Kibana Developer toools.
Now all DNS indices will be handled by ILM
Remove curator support
Goto your curator server and remove all references to the current set of indices.
Verify your work
Once you have completed the steps , you are able to see in ILM, that everything is working. You can see how many indices are handled by each policy. If you monitor this on a daily basis you will eventually see , when the dns-standard policy is no longer needed (linked-policies=0). This will happen according to your retention settings.
Please try out ILM in your test environment first to get familiar with the steps. Or learn the hard way 😉
The latest version of Curator is ILM aware. So it will not touch indices, that are marked by ILM.
One thing to be aware of , is that ILM doesn’t offer all advanced features of Curator. However it delivers, the needed functionality for most basic usecases.
We encourage you to try it out.