Running ES Client inside ES Plugin

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Running ES Client inside ES Plugin

Vadim Voituk
Hello, 

I'm developing ES plugin that need to get some static data (let's call it "list") from the ES-index.
What is best way to do this?

Of course it can be implemented the same way as "external" ES-client (create client, connect to node, etc)
But i guess there are more effective way to do this.

For example, something like: get current node, send and some kind of low-level request to the index needed.

Looking forward for any comments, ideas, considerations...

-- 
Vadim


Reply | Threaded
Open this post in threaded view
|

Re: Running ES Client inside ES Plugin

Alexander Reelsen
Hi

On 20 Apr., 14:09, Vadim Voituk <[hidden email]> wrote:
> Of course it can be implemented the same way as "external" ES-client
> (create client, connect to node, etc)
> But i guess there are more effective way to do this.
No need to create a client, simply use the @Inject annotation in your
plugin to insert a client instance - see other sample plugins and
rivers how they do this.
Or am I missing something why this is not possible in your sample?


--Alexander
Reply | Threaded
Open this post in threaded view
|

Re: Running ES Client inside ES Plugin

Vadim Voituk

On Friday, April 20, 2012 2:22:53 PM UTC+2, Alexander Reelsen wrote:
> Of course it can be implemented the same way as "external" ES-client
> (create client, connect to node, etc)
> But i guess there are more effective way to do this.
No need to create a client, simply use the @Inject annotation in your
plugin to insert a client instance - see other sample plugins and
rivers how they do this.
Or am I missing something why this is not possible in your sample?


Alexander, 
thanks for your answer.
Could you please point me to an example.

The only i found – injection of the client into child class BaseRestController, but in my case i need to inject it into NativeScriptFactory.

Something like:

public class XXXScriptFactory implements NativeScriptFactory 
    @Inject  Client client;
     ....
}

But i'm getting error like:

"Initialization Failed ...
1) Tried proxying org.elasticsearch.script.ScriptService to support a circular dependency, but it is not an interface."


Reply | Threaded
Open this post in threaded view
|

Re: Running ES Client inside ES Plugin

kimchy
Administrator
Add the Client to the constructor of your implementation of NativeScriptFactory (and make sure to have @Inject on the constructor as well).

On Fri, Apr 20, 2012 at 4:33 PM, Vadim Voituk <[hidden email]> wrote:

On Friday, April 20, 2012 2:22:53 PM UTC+2, Alexander Reelsen wrote:
> Of course it can be implemented the same way as "external" ES-client
> (create client, connect to node, etc)
> But i guess there are more effective way to do this.
No need to create a client, simply use the @Inject annotation in your
plugin to insert a client instance - see other sample plugins and
rivers how they do this.
Or am I missing something why this is not possible in your sample?


Alexander, 
thanks for your answer.
Could you please point me to an example.

The only i found – injection of the client into child class BaseRestController, but in my case i need to inject it into NativeScriptFactory.

Something like:

public class XXXScriptFactory implements NativeScriptFactory 
    @Inject  Client client;
     ....
}

But i'm getting error like:

"Initialization Failed ...
1) Tried proxying org.elasticsearch.script.ScriptService to support a circular dependency, but it is not an interface."



Reply | Threaded
Open this post in threaded view
|

Re: Running ES Client inside ES Plugin

Vadim Voituk
Hello, Shay

I'm trying to do in this way:

On Saturday, April 21, 2012 5:07:01 PM UTC+2, kimchy wrote:
Add the Client to the constructor of your implementation of NativeScriptFactory (and make sure to have @Inject on the constructor as well).

.. and here  what i getting: 

1) Tried proxying org.elasticsearch.script.ScriptService to support a circular dependency, but it is not an interface.
2) IllegalStateException[This is a proxy used to support circular references involving constructors. The object we're proxying is not constructed yet. Please wait until after injection has completed to use this object.]

And here is my code:

public class PreorderSortScriptFactory implements NativeScriptFactory {
    @Inject  public PreorderSortScriptFactory(Client client) {

        System.out.println("PreorderSortScriptFactory.PreorderSortScriptFactory() + " + client );
    }


Reply | Threaded
Open this post in threaded view
|

Re: Running ES Client inside ES Plugin

avasilenko
Vadim,

Looks like you encountered circular dependencies. Have you tried injecting via setter?

Alexandr Vasilenko
Skype:menterr

2012/4/23 Vadim Voituk <[hidden email]>
Hello, Shay

I'm trying to do in this way:


On Saturday, April 21, 2012 5:07:01 PM UTC+2, kimchy wrote:
Add the Client to the constructor of your implementation of NativeScriptFactory (and make sure to have @Inject on the constructor as well).

.. and here  what i getting: 

1) Tried proxying org.elasticsearch.script.ScriptService to support a circular dependency, but it is not an interface.
2) IllegalStateException[This is a proxy used to support circular references involving constructors. The object we're proxying is not constructed yet. Please wait until after injection has completed to use this object.]

And here is my code:

public class PreorderSortScriptFactory implements NativeScriptFactory {
    @Inject  public PreorderSortScriptFactory(Client client) {

        System.out.println("PreorderSortScriptFactory.PreorderSortScriptFactory() + " + client );
    }



Reply | Threaded
Open this post in threaded view
|

Re: Running ES Client inside ES Plugin

Vadim Voituk
Hello, Alex, 

On Monday, April 23, 2012 12:49:41 PM UTC+2, Alex Vasilenko wrote:
Vadim,

Looks like you encountered circular dependencies. Have you tried injecting via setter?


Yeah, ai've tried in this way too: 

@Inject public void setClient(Client client) { ... }

But the error is the same.

 And i don't think that there are much of differences in injection lifecycle there :)

Reply | Threaded
Open this post in threaded view
|

Re: Running ES Client inside ES Plugin

kimchy
Administrator
Yea, I see why it happens... . Try and do the (not as nice solution) of injecting the factory with Injector and keep it in a field. And then, on newScript called on the factor, call: injector.getInstance(Client.class) to get the client.

On Mon, Apr 23, 2012 at 3:44 PM, Vadim Voituk <[hidden email]> wrote:
Hello, Alex, 


On Monday, April 23, 2012 12:49:41 PM UTC+2, Alex Vasilenko wrote:
Vadim,

Looks like you encountered circular dependencies. Have you tried injecting via setter?


Yeah, ai've tried in this way too: 

@Inject public void setClient(Client client) { ... }

But the error is the same.

 And i don't think that there are much of differences in injection lifecycle there :)


Reply | Threaded
Open this post in threaded view
|

Re: Running ES Client inside ES Plugin

Vadim Voituk
Hello, Shay

Trying in this way: 

@Inject public PreorderSortScriptFactory(Injector injector) {
    super();
    logger.info("" + injector);
}

but as result ES even not starts – java process silently dies.

Here is what i see from log 

[2012-04-26 10:57:14,227][INFO ][node                     ] [Williams, Simon] {0.19.2}[7659]: initializing ...

[2012-04-26 10:57:14,236][INFO ][plugins                  ] [Williams, Simon] loaded [Pre-Ordered Sorting], sites []

[2012-04-26 10:57:16,512][INFO ][node                     ] [Bounty Hunter] {0.19.2}[7659]: initializing ...
[2012-04-26 10:57:16,513][INFO ][plugins                  ] [Bounty Hunter] loaded [Pre-Ordered Sorting], sites []


--
Voituk Vadim 
[hidden email]
SkypeID: voituk


On Wed, Apr 25, 2012 at 17:54, Shay Banon <[hidden email]> wrote:
Yea, I see why it happens... . Try and do the (not as nice solution) of injecting the factory with Injector and keep it in a field. And then, on newScript called on the factor, call: injector.getInstance(Client.class) to get the client.

On Mon, Apr 23, 2012 at 3:44 PM, Vadim Voituk <[hidden email]> wrote:
Hello, Alex, 


On Monday, April 23, 2012 12:49:41 PM UTC+2, Alex Vasilenko wrote:
Vadim,

Looks like you encountered circular dependencies. Have you tried injecting via setter?


Yeah, ai've tried in this way too: 

@Inject public void setClient(Client client) { ... }

But the error is the same.

 And i don't think that there are much of differences in injection lifecycle there :)



Reply | Threaded
Open this post in threaded view
|

Re: Running ES Client inside ES Plugin

kimchy
Administrator
I;ve ran a simpel test with injecting the Injector, and it seems to work..., maybe you can share your code and I can have a look?

On Thu, Apr 26, 2012 at 1:58 PM, Vadim Voituk <[hidden email]> wrote:
Hello, Shay

Trying in this way: 

@Inject public PreorderSortScriptFactory(Injector injector) {
    super();
    logger.info("" + injector);
}

but as result ES even not starts – java process silently dies.

Here is what i see from log 

[2012-04-26 10:57:14,227][INFO ][node                     ] [Williams, Simon] {0.19.2}[7659]: initializing ...

[2012-04-26 10:57:14,236][INFO ][plugins                  ] [Williams, Simon] loaded [Pre-Ordered Sorting], sites []

[2012-04-26 10:57:16,512][INFO ][node                     ] [Bounty Hunter] {0.19.2}[7659]: initializing ...
[2012-04-26 10:57:16,513][INFO ][plugins                  ] [Bounty Hunter] loaded [Pre-Ordered Sorting], sites []


--
Voituk Vadim 
[hidden email]
SkypeID: voituk



On Wed, Apr 25, 2012 at 17:54, Shay Banon <[hidden email]> wrote:
Yea, I see why it happens... . Try and do the (not as nice solution) of injecting the factory with Injector and keep it in a field. And then, on newScript called on the factor, call: injector.getInstance(Client.class) to get the client.

On Mon, Apr 23, 2012 at 3:44 PM, Vadim Voituk <[hidden email]> wrote:
Hello, Alex, 


On Monday, April 23, 2012 12:49:41 PM UTC+2, Alex Vasilenko wrote:
Vadim,

Looks like you encountered circular dependencies. Have you tried injecting via setter?


Yeah, ai've tried in this way too: 

@Inject public void setClient(Client client) { ... }

But the error is the same.

 And i don't think that there are much of differences in injection lifecycle there :)




Reply | Threaded
Open this post in threaded view
|

Re: Running ES Client inside ES Plugin

Vadim Voituk
In reply to this post by Vadim Voituk
Hello, Shay

The code is pretty straightforward: 

public class PreorderSortScriptFactory implements NativeScriptFactory {

  private static final ESLogger logger = Loggers.getLogger("com.voituk.PreorderSort");

  private Injector injector;
  @Inject
  public PreorderSortScriptFactory(Injector injector) {
    super();
    this.injector = injector;
  }

  @Override
  public ExecutableScript newScript(Map<String, Object> params) {
    super();
    logger.info("" + injector);

  }

}

I'm using ES 0.19.2 on Debian with 4Gb specified as ES_HEAP_SIZE variable.
And the most strange thing that i can't see any exception or something like that even in debug mode, so probably it's some kind of internal exception happens that crashes JVM silently.


Anyway, i've managed the initial problem by creating separate client node in this way:

Node node = NodeBuilder.nodeBuilder().client(true).node();

But it would be nice to to solve this in a right way - via @Injection.

Thanks.


On Friday, April 27, 2012 12:22:59 PM UTC+2, kimchy wrote:
I;ve ran a simpel test with injecting the Injector, and it seems to work..., maybe you can share your code and I can have a look?


Reply | Threaded
Open this post in threaded view
|

Re: Running ES Client inside ES Plugin

kimchy
Administrator
Thats what I run and it works. If you can share a full project I can have a look why it fails.

On Mon, Apr 30, 2012 at 10:54 AM, Vadim Voituk <[hidden email]> wrote:
Hello, Shay

The code is pretty straightforward: 

public class PreorderSortScriptFactory implements NativeScriptFactory {

  private static final ESLogger logger = Loggers.getLogger("com.voituk.PreorderSort");

  private Injector injector;
  @Inject
  public PreorderSortScriptFactory(Injector injector) {
    super();
    this.injector = injector;
  }

  @Override
  public ExecutableScript newScript(Map<String, Object> params) {
    super();
    logger.info("" + injector);

  }

}

I'm using ES 0.19.2 on Debian with 4Gb specified as ES_HEAP_SIZE variable.
And the most strange thing that i can't see any exception or something like that even in debug mode, so probably it's some kind of internal exception happens that crashes JVM silently.


Anyway, i've managed the initial problem by creating separate client node in this way:

Node node = NodeBuilder.nodeBuilder().client(true).node();

But it would be nice to to solve this in a right way - via @Injection.

Thanks.


On Friday, April 27, 2012 12:22:59 PM UTC+2, kimchy wrote:
I;ve ran a simpel test with injecting the Injector, and it seems to work..., maybe you can share your code and I can have a look?



Reply | Threaded
Open this post in threaded view
|

Re: Running ES Client inside ES Plugin

Vadim Voituk
Hello, 

Will try to make empty project with this code only and see if it'll crash.

Will keep you updated.

Thanks,

Thats what I run and it works. If you can share a full project I can have a look why it fails.

Hello, Shay

The code is pretty straightforward: 

public class PreorderSortScriptFactory implements NativeScriptFactory {

  private static final ESLogger logger = Loggers.getLogger("com.voituk.PreorderSort");

  private Injector injector;
  @Inject
  public PreorderSortScriptFactory(Injector injector) {
    super();
    this.injector = injector;
  }

  @Override
  public ExecutableScript newScript(Map<String, Object> params) {
    super();
    logger.info("" + injector);

  }

}

I'm using ES 0.19.2 on Debian with 4Gb specified as ES_HEAP_SIZE variable.
And the most strange thing that i can't see any exception or something like that even in debug mode, so probably it's some kind of internal exception happens that crashes JVM silently.


Anyway, i've managed the initial problem by creating separate client node in this way:

Node node = NodeBuilder.nodeBuilder().client(true).node();

But it would be nice to to solve this in a right way - via @Injection.

Thanks.


On Friday, April 27, 2012 12:22:59 PM UTC+2, kimchy wrote:
I;ve ran a simpel test with injecting the Injector, and it seems to work..., maybe you can share your code and I can have a look?




Reply | Threaded
Open this post in threaded view
|

Re: Running ES Client inside ES Plugin

Vadim Voituk
Hello, Shay

Here is entire project code
https://github.com/voituk/elasticsearch-test-plugin

and that's what i see in log
[2012-05-07 13:57:06,944][INFO ][node                     ] [Sludge] {0.19.3}[19516]: stopping ...
[2012-05-07 13:57:08,105][INFO ][node                     ] [Sludge] {0.19.3}[19516]: stopped
[2012-05-07 13:57:08,105][INFO ][node                     ] [Sludge] {0.19.3}[19516]: closing ...
[2012-05-07 13:57:08,177][INFO ][node                     ] [Sludge] {0.19.3}[19516]: closed
[2012-05-07 13:57:13,942][INFO ][node                     ] [Cobra] {0.19.3}[23642]: initializing ...
[2012-05-07 13:57:13,952][INFO ][plugins                  ] [Cobra] loaded [Test Plugin], sites []

And nothing after that, and no even java process running.

> java -version 
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

Here is now the ES is started (but don't think that it matters in this case):
/usr/lib/jvm/java-6-sun/bin/java -Xms1g -Xmx1g -Xss256k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.config=/etc/elasticsearch/elasticsearch.yml -Des.path.home=/usr/share/elasticsearch -Des.path.logs=/var/log/elasticsearch -Des.path.data=/var/lib/elasticsearch -Des.path.work=/tmp/elasticsearch -Des.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.ElasticSearch
Reply | Threaded
Open this post in threaded view
|

Re: Running ES Client inside ES Plugin

Vadim Voituk
Hmm,

I've just removed everything not related to the problem described, and it works fine.
So definitely it's not an ElasticSearch but my project setup problem.

Sorry for disturbing.


On Monday, May 7, 2012 4:06:28 PM UTC+2, Vadim Voituk wrote:
Hello, Shay

Here is entire project code

and that's what i see in log
[2012-05-07 13:57:06,944][INFO ][node                     ] [Sludge] {0.19.3}[19516]: stopping ...
[2012-05-07 13:57:08,105][INFO ][node                     ] [Sludge] {0.19.3}[19516]: stopped
[2012-05-07 13:57:08,105][INFO ][node                     ] [Sludge] {0.19.3}[19516]: closing ...
[2012-05-07 13:57:08,177][INFO ][node                     ] [Sludge] {0.19.3}[19516]: closed
[2012-05-07 13:57:13,942][INFO ][node                     ] [Cobra] {0.19.3}[23642]: initializing ...
[2012-05-07 13:57:13,952][INFO ][plugins                  ] [Cobra] loaded [Test Plugin], sites []

And nothing after that, and no even java process running.

> java -version 
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

Here is now the ES is started (but don't think that it matters in this case):
/usr/lib/jvm/java-6-sun/bin/java -Xms1g -Xmx1g -Xss256k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.config=/etc/elasticsearch/elasticsearch.yml -Des.path.home=/usr/share/elasticsearch -Des.path.logs=/var/log/elasticsearch -Des.path.data=/var/lib/elasticsearch -Des.path.work=/tmp/elasticsearch -Des.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.ElasticSearch