Write-Read question

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Write-Read question

Jack Park-2
I observe, and expect a slight delay between writing an object to ES and being able to fetch it back.  In ordinary circumstances, this is not an issue.

But, in the context of many agents processing information resources, creating indexes for them, but also needing to refer to those indexes right away, what are the solutions available which allow to keep each index object (say, a JSONObject of the indexed document) in synch across many agents?

Many thanks in advance.
Jack

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAH6s0fxyFxBRVQUv2Xxn%2BGqsLL3y8Y7QkpntbgGVTLmn2dG7ew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Write-Read question

vineeth mohan-2
Hello Jack ,

What do you mean by "fetching it back" , is it a GET or a search.
GET is realtime and Search is near realtime.
You can use the refresh flag while indexing to make sure search is also real time , but its expensive.

Thanks
         Vineeth

On Mon, Oct 20, 2014 at 8:59 PM, Jack Park <[hidden email]> wrote:
I observe, and expect a slight delay between writing an object to ES and being able to fetch it back.  In ordinary circumstances, this is not an issue.

But, in the context of many agents processing information resources, creating indexes for them, but also needing to refer to those indexes right away, what are the solutions available which allow to keep each index object (say, a JSONObject of the indexed document) in synch across many agents?

Many thanks in advance.
Jack

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAH6s0fxyFxBRVQUv2Xxn%2BGqsLL3y8Y7QkpntbgGVTLmn2dG7ew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5mk7kMXrJLujaiAJHMZWMfidUOQFWCd%3DzkPMLcOas0p3Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Write-Read question

Jack Park-2
Hi Veneeth,

In previous experiments, where I pre-load the index with a kind of typology -- the index is a topic map --, I set the refresh flag, and still had problems. 

In doing this work, there are both GET and search queries going in.

The actual use case in question relates to a collection of JSON documents which are first indexed, and then later updated by agents reading other JSON documents which reference those indexed documents. Frequently, this happens nearly simultaneously in this sense: the first document is indexed, but it nearly immediately referenced (GET by id) by another document. When you have many agents doing this work, the penalty is that an already indexed document may not be returned to another agent if the timing is too close.

My solution to preloading the index with a typology was simply to create that typology as a collection of JSON documents which do not do any self-referencing, and load those.

I don't have that option here. As suggested above, my experience with setting the refresh flag is that it did not help in all cases. I suppose I could try again.

What I seek is something reliable. That's because I will soon begin dealing with more than 200k JSON documents, actually PubMed abstracts, and hundreds of Carrot2 clusters which reference them.

Many thanks
Jack

On Mon, Oct 20, 2014 at 8:56 AM, vineeth mohan <[hidden email]> wrote:
Hello Jack ,

What do you mean by "fetching it back" , is it a GET or a search.
GET is realtime and Search is near realtime.
You can use the refresh flag while indexing to make sure search is also real time , but its expensive.

Thanks
         Vineeth

On Mon, Oct 20, 2014 at 8:59 PM, Jack Park <[hidden email]> wrote:
I observe, and expect a slight delay between writing an object to ES and being able to fetch it back.  In ordinary circumstances, this is not an issue.

But, in the context of many agents processing information resources, creating indexes for them, but also needing to refer to those indexes right away, what are the solutions available which allow to keep each index object (say, a JSONObject of the indexed document) in synch across many agents?

Many thanks in advance.
Jack

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAH6s0fxyFxBRVQUv2Xxn%2BGqsLL3y8Y7QkpntbgGVTLmn2dG7ew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5mk7kMXrJLujaiAJHMZWMfidUOQFWCd%3DzkPMLcOas0p3Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAH6s0fxYJXo2VRmzvdSxyxfeGzhzhD7zpg3vO%3DczfgnvooMk1w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.