jclouds provider

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

jclouds provider

Josh Devins
What happened to the jclouds provider that was in ES pre-0.8? I'm
facing deploying ES to RackSpace cloud or SoftLayer and would like to
use my cloud providers blob storage and discovery.

Is there still a plugin available for jclouds or for cloud providers
other than AWS?

Thanks,

Josh
Reply | Threaded
Open this post in threaded view
|

Re: jclouds provider

James Cook
https://github.com/elasticsearch/elasticsearch/issues/733
Reply | Threaded
Open this post in threaded view
|

Re: jclouds provider

James Cook
In reply to this post by Josh Devins
I know that doesn't answer your question directly, but Shay dropped jClouds because it was having some problems early in its life. I agree that it would be nice to take another look at it. 
Reply | Threaded
Open this post in threaded view
|

Re: jclouds provider

Josh Devins
The issue mentions that RackSpace is supported but I don't see any
documentation or reference to this support. Is it hiding somewhere? :)

@James, nice AWS post too. Helps to describe how ES should integrate
with clouds along with providing easy to follow steps for getting
started on AWS.


On Sep 9, 3:51 pm, James Cook <[hidden email]> wrote:
> I know that doesn't answer your question directly, but Shay dropped jClouds
> because it was having some problems early in its life. I agree that it would
> be nice to take another look at it.
Reply | Threaded
Open this post in threaded view
|

Re: jclouds provider

James Cook
Thanks, I'm glad it helped. 

No, Rackspace is not supported. The idea was that it was "supported" back in pre 0.8 builds because jClouds was used over the current cloud discovery process which only supports EC2 currently.

Since Shay has externalized the cloud discovery process and gateway support for AWS specifically, it shouldn't be too hard to change it back over to jClouds support. I'm sure he would appreciate a pull request. I wish I had the time to work on it, because AWS is $$$, and I also would like other alternatives.

The key would be to make sure that jClouds supports the necessary contracts for discovery (discover other nodes and provide region, group, tags), and the appropriate methods to support the cloud-based gateway (which I am unsure offhand how this works, i.e. sync/async transfer, auto chunking?, reading)
Reply | Threaded
Open this post in threaded view
|

Re: jclouds provider

Josh Devins
We are in the same boat looking at growing AWS monthly bills ;)

I will file a ticket and also if I have time, work on an
implementation and a pull request. Would be great to get Shay's input
as to why the jclouds-based integration was backed out, what sorts of
problems he had, etc. At the very least, basic RackSpace support
through their API should be possible.

See: https://github.com/elasticsearch/elasticsearch/issues/1319
Reply | Threaded
Open this post in threaded view
|

Re: jclouds provider

James Cook
This might explain a bit:
https://github.com/elasticsearch/elasticsearch/issues/197
Reply | Threaded
Open this post in threaded view
|

Re: jclouds provider

kimchy
Administrator
Heya,

  First, a quick explanation of what the aws cloud plugin provides. Basically, it provides two main features:

1. Automatic discovery based on cloud API. This is a nice extension to the zen discovery module, where the unicast list of hosts is built automatically using the AWS API. It would be nice to have it using jclouds or directly using other cloud APIs. The problem with jclouds back then was that it was doing far too much in order to just get the current running instances. Need to check if it improved.

   But, and this is important, you don't have to use it. You can build your own unicast discovery list and use it on any cloud provider or environment that does not support multicast. The lack of support for rackspace *does not* mean you can't run elasticsearch on rackspace.

2. If the cloud provider has support for blob store, then a blob store gateway can be implemented. This is again not necessarily required. You can run elasticsearch with the default local gateway which will work.


Mentioning it again: both extensions, the auto disco based on the cloud API And blob store gateway are nice, but are not required in order to run elasticsearch on cloud providers. Without them, elasticsearch is similar to other "nosql" solutions, or even mysql (and the like) in terms of how you deploy it on your favorite cloud. Of course, it would be nice to have those extensions available to simplify even more the deployment on other clouds.

On Fri, Sep 9, 2011 at 5:14 PM, James Cook <[hidden email]> wrote:
This might explain a bit:

Reply | Threaded
Open this post in threaded view
|

Re: jclouds provider

Josh Devins
Shay, thanks for the notes. Good to know the problems with jclouds in
the past. I'm definitely more interested in getting the discovery and
blob storage working as an optimization. I think it's clear that
unicast discovery+local gateway works on any cloud provider.

I hope to have some time to look into at least an initial jclouds
solution, but could take some months given that it's an
optimization ;)

Cheers,

Josh
Reply | Threaded
Open this post in threaded view
|

Re: jclouds provider

grkvlt
Josh,

I would be really interested in helping with the jclouds work, if I
can. I work with the jclouds founder, Adrian, so that will hopefully
make it easier to sort out any integration issues with jclouds and the
blob store if they occur ;) I think the 1.1.1 release of jclouds is
much improved, and as mentioned above it would greatly simplify
deployment to a new cloud provider.

As for 'doing too much to get the current running instances', I'm not
sure what was happening there. It *should* be a pretty simple task in
jclouds...

Cheers,
Andrew.

On Sep 12, 9:27 am, Josh Devins <[hidden email]> wrote:

> Shay, thanks for the notes. Good to know the problems withjcloudsin
> the past. I'm definitely more interested in getting the discovery and
> blob storage working as an optimization. I think it's clear that
> unicast discovery+local gateway works on any cloud provider.
>
> I hope to have some time to look into at least an initialjclouds
> solution, but could take some months given that it's an
> optimization ;)
>
> Cheers,
>
> Josh
Reply | Threaded
Open this post in threaded view
|

Re: jclouds provider

Josh Devins-2
Hey Andrew,

(As I mentioned in my email earlier) I haven't looked into starting
the jclouds integration yet but probably will in the next 6-weeks. Let
me know if you get started
before that! Will update the ticket as I go.

I agree that in principle/theory it should not be too complicated, but
practice will always have some gotcha's! Hopefully any issues that
arise can be dealt with my you/Adrian/jclouds. If you have time to
work on this before I get to it, that would be great. It's low on my
priority list right now since, as Shay mentioned, this is an
optimization. You can still use local storage and unicast discovery on
any cloud provider/DC/network setup.

Josh
Reply | Threaded
Open this post in threaded view
|

Re: jclouds provider

James Cook
My brief glance at the API seemed to indicate that the concept of tags in (i.e. EC2) is not supported. Groups are supported.