We are using Elasticsearch 0.90.6 on our production server, with one replica. We need to import the data via scripts and we have million of records to be saved. We are using the Bulk mode to save the documents to Elasticsearch index.
Our architecture is using two Databases, One is Mysql, and other is Elasticsearch. The data is first saved in Mysql, then it is sent to Elasticseach for documents savings. But while data import script is running, the data is being saved in Mysql, but not being pushed to Elasticsearch, On investigating the nodes via Elastic HQ, I found the following factors
The filter eviction is in red colour, saying it has crossed the limit.I need to understand, the two terms used for this filter eviction(filter_cache.eviction and query_total), How are they being calculated and How can i control them,How it is affecting the data saving and data searching too, or anything else that can cause the document skipping from saving.The search is also very very slow, although,we have given 30 GB of RAM to Elasticsearch server.
This metric/warning should have no effect in indexing. The query_total is the number of shard _searches executed so far, and filter cache evictions counts the number of times something in the filter cache is removed/replaced because you have reached the limits - this is affected by filters in your queries. Filter cache evictions are normal as part of running your queries, although the filter cache size can be increased (or decreased) as needed. But before you do that, I'd probably find out first why search is slow by analyzing your queries.