We are doing following query which is a range query and we apply sort by timestamp criteria:
EndUserTxnReportSearchCriteria criteria = new EndUserTxnReportSearchCriteria();
criteria.setJurhash(getReqAccount().getJurHash());
for(String operation: operationList)
criteria.addOperation(operation);
criteria.setSortDir("DESC");
criteria.setSortBy("Date");
Date endDate = Calendar.getInstance(getUserTimeZone()).getTime();
criteria.setEndDate(endDate);
Calendar startDateCal = Calendar.getInstance();
startDateCal.add(Calendar.YEAR, -1);
Date startDate = startDateCal.getTime();
criteria.setStartDate(startDate);
QueryBuilder jh = QueryBuilders.termQuery(ElasticSearchTransactionTypeUtil.Fields.account.toString(), criteria.getJurhash());
BoolQueryBuilder boolQb = QueryBuilders.boolQuery().must(jh);
// Add operation if present
if (criteria.getOperations()!=null && !criteria.getOperations().isEmpty()){
for(String operation: criteria.getOperations())
boolQb.should(QueryBuilders.termQuery(ElasticSearchTransactionTypeUtil.Fields.operation.toString(),operation));
boolQb.minimumNumberShouldMatch(1);
}
// Add userID to query
if (criteria.getExtUserId()!=null && !criteria.getExtUserId().equals("")){
//TODO: For demo, comment out wildcard search
//QueryBuilder uid = QueryBuilders.wildcardQuery(ElasticSearchTransactionTypeUtil.Fields.user.toString(), "*" + criteria.getExtUserId().toLowerCase() + "*");
QueryBuilder uid = QueryBuilders.wildcardQuery(ElasticSearchTransactionTypeUtil.Fields.user.toString(), "*" + criteria.getExtUserId().toLowerCase() + "*");
boolQb = boolQb.must(uid);
}
// Build query
QueryBuilder qb = QueryBuilders.filteredQuery(boolQb,
FilterBuilders.rangeFilter(ElasticSearchTransactionTypeUtil.Fields.timeStamp.toString())
.from(criteria.getStartDate())
.to(criteria.getEndDate())
.includeLower(true)
.includeUpper(false));
// Get client
TransportClient client = getClient();
if(client != null){
try{
if(log.isDebugEnabled())
log.debug("Query:" + new String(qb.buildAsBytes(XContentType.JSON)));
// Get response
SearchRequestBuilder requestBuilder= client.prepareSearch()
.setOperationThreading(SearchOperationThreading.THREAD_PER_SHARD)
.setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setQuery(qb.buildAsBytes())
.setFrom(firstResult - 1)
.setSize(pageSize)
.setIndices(esIndices)
.addSort(getSortBy(criteria.getSortBy()), getSortDir(criteria.getSortDir())) .setExplain(false);
SearchResponse response = requestBuilder.execute().actionGet(getTimeoutinmillis());
if(log.isDebugEnabled())
log.debug("SearchResponse:" + response.toString());
List<EventLog> eventLogList = new ArrayList<EventLog>();
try {
SearchHit[] hits = (response==null || response.getHits()== null)?null:response.getHits().getHits();
if(hits!=null && hits.length > 0) {
EventLog eventLog;
for(int i=0;i<hits.length;i++){
Map<String,Object> map = hits[i].sourceAsMap();
eventLog = ElasticSearchTransactionTypeUtil.convertFromESDataToEventLog(map);
if(log.isDebugEnabled())
log.debug("Adding event log " + eventLog);
eventLogList.add(eventLog);
}
return eventLogList;
}
} catch (Exception e) {
log.error("Error while parsing the results from elastic search.");
throw new RuntimeException(e);
}
return eventLogList;
}
finally{
//The client should never close because of the client is singleton.
//if(client!=null) client.close();
}
}
return null;
}
Timestamp is a very high cardinality field (Almost unique and probably we have a lots and lots of such unique terms in our data). I think sorting by timestamp is something that is causing problems with these queries.. When we search for last 2 years worth of data and do a search, the search just fails to give us back results within 20s timeout. We face this issue intermittantly and we are trying to debug why is this happening.
Currently we use Elastic search 0.18.3 .
Is there a better way of writing above query to get data for reporting? Is there some functionality like limite() in Elastic Search Search API?
Looks like we also need to move to better version of lucene to improve on memory usage (lucene 3.6+)
Thanks and regards,
Girish Khadke
On Thursday, February 14, 2013 3:20:37 AM UTC-8, Clinton Gormley wrote:
> 3. How do we upgrade a running production cluster without having a
> downtime (rolling upgrade)?
I wrote up an explanation of how we did a major upgrade without downtime
here:
https://gist.github.com/clintongormley/3888120
clint
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
[hidden email].
For more options, visit
https://groups.google.com/groups/opt_out.
Locked
