[ANN] Elasticsearch Simple Action Plugin

classic Classic list List threaded Threaded
30 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[ANN] Elasticsearch Simple Action Plugin

joergprante@gmail.com
Hi,

many of us want to start writing extensions for Elasticsearch.

Except submitting pull requests to the core code, one great advantage of Elasticsearch is the plugin mechanism. Here, custom code can be hooked into Elasticsearch, without having to ask for inclusion into the core code. Nevertheless, plugin code can be published on Github and easily included into a running ES instance by using the ES plugin command line tool.

Unfortunately, writing plugins is not so easy as it seems. There are many plugins, some of them are very advanced, and finding a starting point for a personal project could be quite hard.

Hence, for educational purposes, I wrote a tiny plugin, as a starting point, to demonstrate how a plugin works.

The simple plugin is indeed very simple. It makes reuse of the standard search action:

- it defines a built-in query (a "match all" query)

- it creates a custom action for it

- the action is called from Java API

- the result of the action (the search response of the "match all" query) is logged

The plugin code comes with a junit test. It is available at 


In the hope it is useful,

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH-M6%2BZroAz8Reb3e2agW0vXKSavk%3D0hD_bq%2BBHtRYLhw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Elasticsearch Simple Action Plugin

virgil
Thank you. About the plugin, I wonder if you install it on all nodes in cluster to make it work or install it in a nondata node?
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Elasticsearch Simple Action Plugin

Luiz Guilherme Pais dos Santos
In reply to this post by joergprante@gmail.com
Hi Jörg,

Thanks a lot! Some month ago I started to write one plugin and it was really difficult. This skeleton is perfect for a cold start :)


On Tue, Jun 3, 2014 at 12:15 PM, [hidden email] <[hidden email]> wrote:
Hi,

many of us want to start writing extensions for Elasticsearch.

Except submitting pull requests to the core code, one great advantage of Elasticsearch is the plugin mechanism. Here, custom code can be hooked into Elasticsearch, without having to ask for inclusion into the core code. Nevertheless, plugin code can be published on Github and easily included into a running ES instance by using the ES plugin command line tool.

Unfortunately, writing plugins is not so easy as it seems. There are many plugins, some of them are very advanced, and finding a starting point for a personal project could be quite hard.

Hence, for educational purposes, I wrote a tiny plugin, as a starting point, to demonstrate how a plugin works.

The simple plugin is indeed very simple. It makes reuse of the standard search action:

- it defines a built-in query (a "match all" query)

- it creates a custom action for it

- the action is called from Java API

- the result of the action (the search response of the "match all" query) is logged

The plugin code comes with a junit test. It is available at 


In the hope it is useful,

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH-M6%2BZroAz8Reb3e2agW0vXKSavk%3D0hD_bq%2BBHtRYLhw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.



--
Luiz Guilherme P. Santos

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAMdL%3DZEcM%3DzRmLk9%2BK7fiZ1hoT1zbcy_ScYT0F_8Kx0RzPkm%2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Elasticsearch Simple Action Plugin

joergprante@gmail.com
In reply to this post by virgil
Usually, plugins that extend internal ES functionality should be installed on all nodes. This is easy to remember and preferable from an administrative view. All the nodes in the ES cluster must have access to plugin code under all circumstances, especially when executing actions, mappers, routers, discovery helpers, analyzer code, indexing helpers... 

In this case, for the simple action demo plugin, you can install it just on the node of the cluster where you want to execute the demo "match_all" search from. The "match_all" search is then searching on all the indexes of the cluster.  

If you want to execute the demo plugin "match_all" search from other nodes, you would have to install the plugin on those other nodes, too.

Jörg



On Tue, Jun 3, 2014 at 7:33 PM, virgil <[hidden email]> wrote:
Thank you. About the plugin, I wonder if you install it on all nodes in
cluster to make it work or install it in a nondata node?



--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/ANN-Elasticsearch-Simple-Action-Plugin-tp4056971p4056981.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1401816815837-4056981.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFkQ4xL5xqrXfH%2BzwSJ0WvyLcNw6S%2B7Z0pTSCuwXHnXWw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Elasticsearch Simple Action Plugin

virgil
Thank you Jörg. I see the point. But if the plugin consumes memory (ex. hold a HashMap for customized score), installing it on all nodes may waste memory of the cluster. Is there any way to deal with this issue?
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Elasticsearch Simple Action Plugin

joergprante@gmail.com
Not sure if I understand your concern completely - as long as you're doing things right in your code, it should be possible to allocate resources only when required - this holds also for plugins.

Jörg


On Tue, Jun 3, 2014 at 11:48 PM, virgil <[hidden email]> wrote:
Thank you Jörg. I see the point. But if the plugin consumes memory (ex. hold
a HashMap for customized score), installing it on all nodes may waste memory
of the cluster. Is there any way to deal with this issue?



--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/ANN-Elasticsearch-Simple-Action-Plugin-tp4056971p4057003.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1401832124609-4057003.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFpp8ybSWqxya9s-hK6qhhBUOTno8pdgX-Ba6YT_oAHFA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Elasticsearch Simple Action Plugin

virgil
The problem is that only one copy of HashMap is needed to customize score of all documents in the cluster. But as we have to install the plugin on all nodes, the actual memory used is multiplied by the number of nodes in cluster. I try to figure out one way to save the memory. Tried on non-data node, but it seems not working.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Elasticsearch Simple Action Plugin

joergprante@gmail.com
You need resources on all nodes that hold shards, you can not do it with just one instance, because ES index is distributed. Rescoring would be very expensive if you did it on an extra central instance with an extra scatter/gather phase. It is also very expensive in scripting.

A better method is a similarity plugin like https://github.com/tlrx/elasticsearch-custom-similarity-provider

Not sure how your code looks like though, maybe you can share it with the community?

Jörg



On Wed, Jun 4, 2014 at 2:55 AM, virgil <[hidden email]> wrote:
The problem is that only one copy of HashMap is needed to customize score of
all documents in the cluster. But as we have to install the plugin on all
nodes, the actual memory used is multiplied by the number of nodes in
cluster. I try to figure out one way to save the memory. Tried on non-data
node, but it seems not working.



--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/ANN-Elasticsearch-Simple-Action-Plugin-tp4056971p4057015.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1401843345821-4057015.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH%3D228Y2PvB265Hs4NX1O_Ac4QBuWXJGcCqKaXFc3a56A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Elasticsearch Simple Action Plugin

joergprante@gmail.com
Sorry, the plugin is outdated, a better start is by looking at


Jörg


On Wed, Jun 4, 2014 at 10:07 AM, [hidden email] <[hidden email]> wrote:
You need resources on all nodes that hold shards, you can not do it with just one instance, because ES index is distributed. Rescoring would be very expensive if you did it on an extra central instance with an extra scatter/gather phase. It is also very expensive in scripting.

A better method is a similarity plugin like https://github.com/tlrx/elasticsearch-custom-similarity-provider

Not sure how your code looks like though, maybe you can share it with the community?

Jörg



On Wed, Jun 4, 2014 at 2:55 AM, virgil <[hidden email]> wrote:
The problem is that only one copy of HashMap is needed to customize score of
all documents in the cluster. But as we have to install the plugin on all
nodes, the actual memory used is multiplied by the number of nodes in
cluster. I try to figure out one way to save the memory. Tried on non-data
node, but it seems not working.



--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/ANN-Elasticsearch-Simple-Action-Plugin-tp4056971p4057015.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1401843345821-4057015.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHZTAZrAdtQAnvj_7UtO%3DaAVtN3qt337PTzDjnbCmtPaA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Elasticsearch Simple Action Plugin

Alexandra
This post has NOT been accepted by the mailing list yet.
In reply to this post by joergprante@gmail.com
Thanks for the awesome example. Another question from me : If I wanted to take the search example and make another one there ( for example if I want to do paginated search ), could I have access to the search response inside the doExecute method ?

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Elasticsearch Simple Action Plugin

Ivan Brusic
In reply to this post by joergprante@gmail.com
Jörg, thanks for the plugin to help as a starting point for plugin development.

Although I have built a few plugins during the years, they were river or analysis plugins, which are fairly easy. Writing a custom action required a lot more digging, especially since there are very few to learn from. I still would like to see a write-up regarding the different families of transport actions: BroadcastOperationRequest, MasterNodeOperationRequest, NodesOperationRequest, SingleShardOperationRequest, SingleCustomOperationRequest, etc. What is the difference? I understand it now, but it should be documented. There is little documentation about the internals and there are no code level comments.  I always meant to experiment with the different action hierarchies via simple plugins and document my findings. Perhaps one day...

Cheers,

Ivan


On Wed, Jun 4, 2014 at 1:09 AM, [hidden email] <[hidden email]> wrote:
Sorry, the plugin is outdated, a better start is by looking at


Jörg


On Wed, Jun 4, 2014 at 10:07 AM, [hidden email] <[hidden email]> wrote:
You need resources on all nodes that hold shards, you can not do it with just one instance, because ES index is distributed. Rescoring would be very expensive if you did it on an extra central instance with an extra scatter/gather phase. It is also very expensive in scripting.

A better method is a similarity plugin like https://github.com/tlrx/elasticsearch-custom-similarity-provider

Not sure how your code looks like though, maybe you can share it with the community?

Jörg



On Wed, Jun 4, 2014 at 2:55 AM, virgil <[hidden email]> wrote:
The problem is that only one copy of HashMap is needed to customize score of
all documents in the cluster. But as we have to install the plugin on all
nodes, the actual memory used is multiplied by the number of nodes in
cluster. I try to figure out one way to save the memory. Tried on non-data
node, but it seems not working.



--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/ANN-Elasticsearch-Simple-Action-Plugin-tp4056971p4057015.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1401843345821-4057015.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHZTAZrAdtQAnvj_7UtO%3DaAVtN3qt337PTzDjnbCmtPaA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCkOVMuEV67ZMCX5qoAdiob%2BfWsuWK%3D0EyAKf3VGhjYdQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Elasticsearch Simple Action Plugin

virgil
In reply to this post by joergprante@gmail.com
Yeah, but I would consider the nondata node is already doing the job. -- "These "non data" nodes are still part of the cluster, and they redirect operations exactly to the node that holds the relevant data. The other benefit is the fact that for scatter / gather based operations (such as search), these nodes will take part of the processing since they will start the scatter process, and perform the actual gather processing." I just uploaded my native script code in https://github.com/virgil0/TestPlugin. It works with the function score query. You can see that there are 3 bin file I need to load into memory. Thank you for reply.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Elasticsearch Simple Action Plugin

joergprante@gmail.com
In reply to this post by Ivan Brusic
Absolutely, agreed.

The docs are sparse in my simple plugin too. I try to find some time to add sample code for all the variants and explain the differences.

Jörg


On Wed, Jun 4, 2014 at 6:22 PM, Ivan Brusic <[hidden email]> wrote:
Jörg, thanks for the plugin to help as a starting point for plugin development.

Although I have built a few plugins during the years, they were river or analysis plugins, which are fairly easy. Writing a custom action required a lot more digging, especially since there are very few to learn from. I still would like to see a write-up regarding the different families of transport actions: BroadcastOperationRequest, MasterNodeOperationRequest, NodesOperationRequest, SingleShardOperationRequest, SingleCustomOperationRequest, etc. What is the difference? I understand it now, but it should be documented. There is little documentation about the internals and there are no code level comments.  I always meant to experiment with the different action hierarchies via simple plugins and document my findings. Perhaps one day...

Cheers,

Ivan


On Wed, Jun 4, 2014 at 1:09 AM, [hidden email] <[hidden email]> wrote:
Sorry, the plugin is outdated, a better start is by looking at


Jörg


On Wed, Jun 4, 2014 at 10:07 AM, [hidden email] <[hidden email]> wrote:
You need resources on all nodes that hold shards, you can not do it with just one instance, because ES index is distributed. Rescoring would be very expensive if you did it on an extra central instance with an extra scatter/gather phase. It is also very expensive in scripting.

A better method is a similarity plugin like https://github.com/tlrx/elasticsearch-custom-similarity-provider

Not sure how your code looks like though, maybe you can share it with the community?

Jörg



On Wed, Jun 4, 2014 at 2:55 AM, virgil <[hidden email]> wrote:
The problem is that only one copy of HashMap is needed to customize score of
all documents in the cluster. But as we have to install the plugin on all
nodes, the actual memory used is multiplied by the number of nodes in
cluster. I try to figure out one way to save the memory. Tried on non-data
node, but it seems not working.



--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/ANN-Elasticsearch-Simple-Action-Plugin-tp4056971p4057015.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1401843345821-4057015.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHZTAZrAdtQAnvj_7UtO%3DaAVtN3qt337PTzDjnbCmtPaA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCkOVMuEV67ZMCX5qoAdiob%2BfWsuWK%3D0EyAKf3VGhjYdQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGRDpkJcvWzYp1TH19j%3DWhe5XULn7eRDiPTXMPa1HR1NQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Elasticsearch Simple Action Plugin

Itamar Syn-Hershko
In reply to this post by joergprante@gmail.com
You should have released this before my talk last week, I could have mentioned it :\


--

Itamar Syn-Hershko
http://code972.com | @synhershko
Freelance Developer & Consultant


On Tue, Jun 3, 2014 at 6:15 PM, [hidden email] <[hidden email]> wrote:
Hi,

many of us want to start writing extensions for Elasticsearch.

Except submitting pull requests to the core code, one great advantage of Elasticsearch is the plugin mechanism. Here, custom code can be hooked into Elasticsearch, without having to ask for inclusion into the core code. Nevertheless, plugin code can be published on Github and easily included into a running ES instance by using the ES plugin command line tool.

Unfortunately, writing plugins is not so easy as it seems. There are many plugins, some of them are very advanced, and finding a starting point for a personal project could be quite hard.

Hence, for educational purposes, I wrote a tiny plugin, as a starting point, to demonstrate how a plugin works.

The simple plugin is indeed very simple. It makes reuse of the standard search action:

- it defines a built-in query (a "match all" query)

- it creates a custom action for it

- the action is called from Java API

- the result of the action (the search response of the "match all" query) is logged

The plugin code comes with a junit test. It is available at 


In the hope it is useful,

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH-M6%2BZroAz8Reb3e2agW0vXKSavk%3D0hD_bq%2BBHtRYLhw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zt-16RTbRh376Kxg%3Di7DmjRhav-PYk_7qs1J5wu1W5a8w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Elasticsearch Simple Action Plugin

Ivan Brusic


On Wed, Jun 4, 2014 at 2:30 PM, Itamar Syn-Hershko <[hidden email]> wrote:
You should have released this before my talk last week, I could have mentioned it :\


--

Itamar Syn-Hershko
http://code972.com | @synhershko
Freelance Developer & Consultant


On Tue, Jun 3, 2014 at 6:15 PM, [hidden email] <[hidden email]> wrote:
Hi,

many of us want to start writing extensions for Elasticsearch.

Except submitting pull requests to the core code, one great advantage of Elasticsearch is the plugin mechanism. Here, custom code can be hooked into Elasticsearch, without having to ask for inclusion into the core code. Nevertheless, plugin code can be published on Github and easily included into a running ES instance by using the ES plugin command line tool.

Unfortunately, writing plugins is not so easy as it seems. There are many plugins, some of them are very advanced, and finding a starting point for a personal project could be quite hard.

Hence, for educational purposes, I wrote a tiny plugin, as a starting point, to demonstrate how a plugin works.

The simple plugin is indeed very simple. It makes reuse of the standard search action:

- it defines a built-in query (a "match all" query)

- it creates a custom action for it

- the action is called from Java API

- the result of the action (the search response of the "match all" query) is logged

The plugin code comes with a junit test. It is available at 


In the hope it is useful,

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH-M6%2BZroAz8Reb3e2agW0vXKSavk%3D0hD_bq%2BBHtRYLhw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zt-16RTbRh376Kxg%3Di7DmjRhav-PYk_7qs1J5wu1W5a8w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQD0Pvbr0eenPiVYm032ZycyTGWKxL7MH3KNL5EBAJZCzg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Elasticsearch Simple Action Plugin

joergprante@gmail.com
In reply to this post by virgil

As said, it is true that scoring scripts (like the function score scripts o the AbstractSearchScript) need to reside on data nodes. Accessing fields is a low level operation in a script so it is not possible to install such a boost plugin that uses scripting on a data-less node. You would have to install it on all the data nodes which might become tedious (but it is doable).

Another issue is that you use scripting in a java plugin. I conclude from this, the search should work later over the HTTP API by executing a standard function score query with the boost script name (is that true?) 

Writing a plugin, in a pure java environment, you have much more degrees of freedom to supersede the script functionality and use other code paths. For example, you could reuse the resource watch service from ES (used for watching script file changes) to reload the boost info (which is in your binary files I assume). Then you could build the query internally using the Java API as a custom score query action and execute it from your favorite (data-less) node (or from two nodes, for better fault tolerance / load balancing).

Optionally, you could expose a new endpoint to the ES REST API, for example "_search_with_boost", which works like "_search", but makes use of the boost info files.

For a more generic solution, it would be convenient to convert the boost info into a JSON parameter file so this could be loaded by the standard ES settings/config routines and by other languages, also for better reuse by others in the ES community :) An example plugin name could be "boost control plugin"...

Jörg



On Wed, Jun 4, 2014 at 8:15 PM, virgil <[hidden email]> wrote:
Yeah, but I would consider the nondata node is already doing the job. --
"These "non data" nodes are still part of the cluster, and they redirect
operations exactly to the node that holds the relevant data. The other
benefit is the fact that for scatter / gather based operations (such as
search), these nodes will take part of the processing since they will start
the scatter process, and perform the actual gather processing." I just
uploaded my native script code in https://github.com/virgil0/TestPlugin. It
works with the function score query. You can see that there are 3 bin file I
need to load into memory. Thank you for reply.



--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/ANN-Elasticsearch-Simple-Action-Plugin-tp4056971p4057054.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1401905723480-4057054.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGBMEEc6oC1%3DBX7gS41se13BExO_iKJtiGC6zrhmxJqxA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Elasticsearch Simple Action Plugin

joergprante@gmail.com
In reply to this post by Itamar Syn-Hershko
Great walkthrough :) ... I only miss the mentioning of the standard language plugins for scripting (groovy, js, etc.) And rivers are not obsolete, the "pull" method from a singleton "river" node is just discouraged.

Jörg


On Wed, Jun 4, 2014 at 11:30 PM, Itamar Syn-Hershko <[hidden email]> wrote:
You should have released this before my talk last week, I could have mentioned it :\


--

Itamar Syn-Hershko
http://code972.com | @synhershko
Freelance Developer & Consultant


On Tue, Jun 3, 2014 at 6:15 PM, [hidden email] <[hidden email]> wrote:
Hi,

many of us want to start writing extensions for Elasticsearch.

Except submitting pull requests to the core code, one great advantage of Elasticsearch is the plugin mechanism. Here, custom code can be hooked into Elasticsearch, without having to ask for inclusion into the core code. Nevertheless, plugin code can be published on Github and easily included into a running ES instance by using the ES plugin command line tool.

Unfortunately, writing plugins is not so easy as it seems. There are many plugins, some of them are very advanced, and finding a starting point for a personal project could be quite hard.

Hence, for educational purposes, I wrote a tiny plugin, as a starting point, to demonstrate how a plugin works.

The simple plugin is indeed very simple. It makes reuse of the standard search action:

- it defines a built-in query (a "match all" query)

- it creates a custom action for it

- the action is called from Java API

- the result of the action (the search response of the "match all" query) is logged

The plugin code comes with a junit test. It is available at 


In the hope it is useful,

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH-M6%2BZroAz8Reb3e2agW0vXKSavk%3D0hD_bq%2BBHtRYLhw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zt-16RTbRh376Kxg%3Di7DmjRhav-PYk_7qs1J5wu1W5a8w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEyF-hh3yEfeznk6p8v1tuzreDTYgcUVohuRW%3DpXRyO2w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Elasticsearch Simple Action Plugin

virgil
In reply to this post by joergprante@gmail.com
Really good suggestion!  Yeah, the search will work by executing a standard function score query with the boost script name. Thank you!


2014-06-04 15:06 GMT-07:00 [hidden email] [via ElasticSearch Users] <[hidden email]>:

As said, it is true that scoring scripts (like the function score scripts o the AbstractSearchScript) need to reside on data nodes. Accessing fields is a low level operation in a script so it is not possible to install such a boost plugin that uses scripting on a data-less node. You would have to install it on all the data nodes which might become tedious (but it is doable).

Another issue is that you use scripting in a java plugin. I conclude from this, the search should work later over the HTTP API by executing a standard function score query with the boost script name (is that true?) 

Writing a plugin, in a pure java environment, you have much more degrees of freedom to supersede the script functionality and use other code paths. For example, you could reuse the resource watch service from ES (used for watching script file changes) to reload the boost info (which is in your binary files I assume). Then you could build the query internally using the Java API as a custom score query action and execute it from your favorite (data-less) node (or from two nodes, for better fault tolerance / load balancing).

Optionally, you could expose a new endpoint to the ES REST API, for example "_search_with_boost", which works like "_search", but makes use of the boost info files.

For a more generic solution, it would be convenient to convert the boost info into a JSON parameter file so this could be loaded by the standard ES settings/config routines and by other languages, also for better reuse by others in the ES community :) An example plugin name could be "boost control plugin"...

Jörg



On Wed, Jun 4, 2014 at 8:15 PM, virgil <[hidden email]> wrote:
Yeah, but I would consider the nondata node is already doing the job. --
"These "non data" nodes are still part of the cluster, and they redirect
operations exactly to the node that holds the relevant data. The other
benefit is the fact that for scatter / gather based operations (such as
search), these nodes will take part of the processing since they will start
the scatter process, and perform the actual gather processing." I just
uploaded my native script code in https://github.com/virgil0/TestPlugin. It
works with the function score query. You can see that there are 3 bin file I
need to load into memory. Thank you for reply.



--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/ANN-Elasticsearch-Simple-Action-Plugin-tp4056971p4057054.html

Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1401905723480-4057054.post%40n3.nabble.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGBMEEc6oC1%3DBX7gS41se13BExO_iKJtiGC6zrhmxJqxA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



If you reply to this email, your message will be added to the discussion below:
http://elasticsearch-users.115913.n3.nabble.com/ANN-Elasticsearch-Simple-Action-Plugin-tp4056971p4057071.html
To unsubscribe from [ANN] Elasticsearch Simple Action Plugin, click here.
NAML



--

Fei (Virgil) Xie

M.S. in Very Large Information System, Institue for Software Research

School of Computer Science, Carnegie Mellon University

[hidden email]

Alumni, Tsinghua University

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Elasticsearch Simple Action Plugin

joergprante@gmail.com
One more hint, you see 

org.elasticsearch.common.lucene.search.function.FieldValueFunction

This implements the ScoreFunction and fetches boost values from a configured field in the doc, for use by the Java API for FunctionScoreQuery.

If you can write a custom ScoreFunction, you could implement another ScoreFunction how to obtain the boost value, in dependency of other fields in the doc. From a map with the product price info you could pass customer name or product codes. In the custom action, at each query execution, the map would have to be passed from the executing data-less node to the nodes with the shards. Caution: the map should be not too big (a few hundred entries maybe). This could be an overhead otherwise. 

The alternative would be simpler, a script with groovy JsonSlurper code for example, executing the boost function score search, of course on all the data nodes. This may be preferable since the search is really faster when the product price boost info is set up once from the local file system of the data node (or once a day).

Jörg




On Thu, Jun 5, 2014 at 2:17 AM, virgil <[hidden email]> wrote:
Really good suggestion!  Yeah, the search will work by executing a standard function score query with the boost script name. Thank you!


2014-06-04 15:06 GMT-07:00 [hidden email] [via ElasticSearch Users] <[hidden email]>:

As said, it is true that scoring scripts (like the function score scripts o the AbstractSearchScript) need to reside on data nodes. Accessing fields is a low level operation in a script so it is not possible to install such a boost plugin that uses scripting on a data-less node. You would have to install it on all the data nodes which might become tedious (but it is doable).

Another issue is that you use scripting in a java plugin. I conclude from this, the search should work later over the HTTP API by executing a standard function score query with the boost script name (is that true?) 

Writing a plugin, in a pure java environment, you have much more degrees of freedom to supersede the script functionality and use other code paths. For example, you could reuse the resource watch service from ES (used for watching script file changes) to reload the boost info (which is in your binary files I assume). Then you could build the query internally using the Java API as a custom score query action and execute it from your favorite (data-less) node (or from two nodes, for better fault tolerance / load balancing).

Optionally, you could expose a new endpoint to the ES REST API, for example "_search_with_boost", which works like "_search", but makes use of the boost info files.

For a more generic solution, it would be convenient to convert the boost info into a JSON parameter file so this could be loaded by the standard ES settings/config routines and by other languages, also for better reuse by others in the ES community :) An example plugin name could be "boost control plugin"...

Jörg



On Wed, Jun 4, 2014 at 8:15 PM, virgil <[hidden email]> wrote:
Yeah, but I would consider the nondata node is already doing the job. --
"These "non data" nodes are still part of the cluster, and they redirect
operations exactly to the node that holds the relevant data. The other
benefit is the fact that for scatter / gather based operations (such as
search), these nodes will take part of the processing since they will start
the scatter process, and perform the actual gather processing." I just
uploaded my native script code in https://github.com/virgil0/TestPlugin. It
works with the function score query. You can see that there are 3 bin file I
need to load into memory. Thank you for reply.



--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/ANN-Elasticsearch-Simple-Action-Plugin-tp4056971p4057054.html

Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGBMEEc6oC1%3DBX7gS41se13BExO_iKJtiGC6zrhmxJqxA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



If you reply to this email, your message will be added to the discussion below:
http://elasticsearch-users.115913.n3.nabble.com/ANN-Elasticsearch-Simple-Action-Plugin-tp4056971p4057071.html
To unsubscribe from [ANN] Elasticsearch Simple Action Plugin, click here.
NAML



--

Fei (Virgil) Xie

M.S. in Very Large Information System, Institue for Software Research

School of Computer Science, Carnegie Mellon University

[hidden email]

Alumni, Tsinghua University



View this message in context: Re: [ANN] Elasticsearch Simple Action Plugin

Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPRQSRfZTu0vFYOn0gbFnAUuaWQgB16vGGZspatGrL-ACt8jtA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFPsWWBqM2EH86ZCW_uyoFespTnfGgTdVgRVvH5612GTg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Elasticsearch Simple Action Plugin

virgil
Good idea! One thing I am not quite clear is that writing a custom ScoreFunction, I will have to modify elasticsearch source code and compile it right? Or there is any other way to do it? Thank you.
12