partial_fields: streaming or in-memory?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

partial_fields: streaming or in-memory?

btiernay
Hi ES'ers!

Just curious if anyone knows if partial_fields uses a streaming approach to filter _source documents. The reason I ask is that we index some relatively large documents and if ES can perform streaming filtering (as I believe Jackson supports) this would be ideal. I couldn't really figure this out by browsing the source. It looks as though this is implemented in-memory by pruning map keys. I could be wrong :)

Thanks again!

Bob

References:

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: partial_fields: streaming or in-memory?

btiernay
Doing a bit more digging it looks as though this is the case:

[1] hitExecute
calls
[1] Object value = context.lookup().source().filter(field.includes(), field.excludes());
calls
[2] loadSourceIfNeeded
calls
[2] reader.document(docId, sourceFieldVisitor);
calls
[3]

[3] <a href="http://lucene.apache.org/core/4_0_0/core/org/apache/lucene/index/IndexReader.html#document(int, org.apache.lucene.index.StoredFieldVisitor)">http://lucene.apache.org/core/4_0_0/core/org/apache/lucene/index/IndexReader.html#document(int, org.apache.lucene.index.StoredFieldVisitor)

On Monday, 29 April 2013 19:39:02 UTC-4, btiernay wrote:
Hi ES'ers!

Just curious if anyone knows if partial_fields uses a streaming approach to filter _source documents. The reason I ask is that we index some relatively large documents and if ES can perform streaming filtering (as I believe Jackson supports) this would be ideal. I couldn't really figure this out by browsing the source. It looks as though this is implemented in-memory by pruning map keys. I could be wrong :)

Thanks again!

Bob

References:

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.