Controlling Stored Data Size

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Controlling Stored Data Size

Michael Leikind
Greetings!
I plain to use elasticache with logstash.
Our production servers send about 20G of logs per day, so indexing server may run out of disk space very soon.
On my current system (which I want to replace) I have a control over indexed data rolling mechanism, i.e. I can configure it to archive the old data and after a some period of time remove it completely from index DB.
Tried to search for an alternative configuration in elasticsearch, but with no luck. 
Can you please advise on what my options are ?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
Reply | Threaded
Open this post in threaded view
|

Re: Controlling Stored Data Size

Alexander Reelsen-2
Hey,

right now there is no expire on indices, but this might come sooner or later, see https://github.com/elasticsearch/elasticsearch/issues/2114

Apart from that you can use time based indices (per month), combine them for search via aliases, and maybe delete them manually via a cronjob at the beginning of each month, or even have a small monitoring script, which checks the available space and then deletes the oldest data automatically for now.


Hope this helps...

--Alex


On Wed, Aug 28, 2013 at 4:35 PM, Michael Leikind <[hidden email]> wrote:
Greetings!
I plain to use elasticache with logstash.
Our production servers send about 20G of logs per day, so indexing server may run out of disk space very soon.
On my current system (which I want to replace) I have a control over indexed data rolling mechanism, i.e. I can configure it to archive the old data and after a some period of time remove it completely from index DB.
Tried to search for an alternative configuration in elasticsearch, but with no luck. 
Can you please advise on what my options are ?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.