Error while backing up ES to HDFS via plugin

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Error while backing up ES to HDFS via plugin

Salman
Hadoop nodes are NOT co-located with ElasticSearch and are on CDH5.1

I installed the plugin [ elasticsearch/elasticsearch-repository-hdfs/2.0.2] on all Elasticsearch nodes, and restarted the nodes

When I try to setup the repo, I get the following error:

curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{"type": "hdfs","uri": "hdfs://nn-ip:8020/","conf_location":"/etc/conf/hadoop/conf/hdfs-site.xml,/etc/conf/hadoop/conf/core-site.xml","path": "elasticsearch","compress": true} }'

{"error":"RepositoryException[[my_backup] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, org.elasticsearch.ElasticsearchIllegalArgumentException: no 'path' defined for hdfs snapshot/restore\n  at org.elasticsearch.repositories.hdfs.HdfsRepository.<init>(Unknown Source)\n  while locating org.elasticsearch.repositories.hdfs.HdfsRepository\n  while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: ElasticsearchIllegalArgumentException[no 'path' defined for hdfs snapshot/restore]; ","status":500}

Please help me, with what additional configuration needs to be done to back up ES indices to Hadoop which is not colocated on same nodes

Ref:https://github.com/elasticsearch/elasticsearch-hadoop/tree/master/repository-hdfs

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0a6d17b7-0494-4624-98ce-a0a6c3860183%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Error while backing up ES to HDFS via plugin

Salman
I made a bit more progress, now when I run this I get this:

curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{"type": "hdfs","uri": "hds://nnip:8020/","conf_location":"/etc/conf/hadoop/conf/hdfs-site.xml,/etc/conf/hadoop/conf/core-site.xml","path": "elasticsearch","load_defaults": "true", "compress": true} }'

{"error":"RemoteTransportException[[node][inet[/10.3.89.102:9300]][cluster:admin/repository/put]]; nested: RepositoryException[[my_backup] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, java.lang.NoClassDefFoundError: org/apache/commons/cli/ParseException\n  at org.elasticsearch.repositories.hdfs.HdfsRepository.<init>()\n  at org.elasticsearch.repositories.hdfs.HdfsRepository\n  at Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1 error]; nested: NoClassDefFoundError[org/apache/commons/cli/ParseException]; nested: ClassNotFoundException[org.apache.commons.cli.ParseException]; ","status":500}



On Tuesday, February 10, 2015 at 2:58:12 PM UTC-8, Salman Ahmed wrote:
Hadoop nodes are NOT co-located with ElasticSearch and are on CDH5.1

I installed the plugin [ elasticsearch/elasticsearch-repository-hdfs/2.0.2] on all Elasticsearch nodes, and restarted the nodes

When I try to setup the repo, I get the following error:

curl -XPUT '<a href="http://localhost:9200/_snapshot/my_backup" target="_blank" rel="nofollow" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_snapshot%2Fmy_backup\46sa\75D\46sntz\0751\46usg\75AFQjCNHUWGQdgStMzqJ5NyJrRX_E-aHQHA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_snapshot%2Fmy_backup\46sa\75D\46sntz\0751\46usg\75AFQjCNHUWGQdgStMzqJ5NyJrRX_E-aHQHA';return true;">http://localhost:9200/_snapshot/my_backup' -d '{"type": "hdfs","uri": "hdfs://nn-ip:8020/","conf_location":"/etc/conf/hadoop/conf/hdfs-site.xml,/etc/conf/hadoop/conf/core-site.xml","path": "elasticsearch","compress": true} }'

{"error":"RepositoryException[[my_backup] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, org.elasticsearch.ElasticsearchIllegalArgumentException: no 'path' defined for hdfs snapshot/restore\n  at org.elasticsearch.repositories.hdfs.HdfsRepository.<init>(Unknown Source)\n  while locating org.elasticsearch.repositories.hdfs.HdfsRepository\n  while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: ElasticsearchIllegalArgumentException[no 'path' defined for hdfs snapshot/restore]; ","status":500}

Please help me, with what additional configuration needs to be done to back up ES indices to Hadoop which is not colocated on same nodes

Ref:<a href="https://github.com/elasticsearch/elasticsearch-hadoop/tree/master/repository-hdfs" target="_blank" rel="nofollow" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-hadoop%2Ftree%2Fmaster%2Frepository-hdfs\46sa\75D\46sntz\0751\46usg\75AFQjCNH6Xuf6uQ1_yHqy3moR0Hh2dmurNA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-hadoop%2Ftree%2Fmaster%2Frepository-hdfs\46sa\75D\46sntz\0751\46usg\75AFQjCNH6Xuf6uQ1_yHqy3moR0Hh2dmurNA';return true;">https://github.com/elasticsearch/elasticsearch-hadoop/tree/master/repository-hdfs

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4db8c55e-be0d-4532-a0c6-fd8662b2542b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Error while backing up ES to HDFS via plugin

dadoonet
What gives GET /_cat/plugins ?

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 11 févr. 2015 à 02:07, Salman Ahmed <[hidden email]> a écrit :

I made a bit more progress, now when I run this I get this:

curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{"type": "hdfs","uri": "hds://nnip:8020/","conf_location":"/etc/conf/hadoop/conf/hdfs-site.xml,/etc/conf/hadoop/conf/core-site.xml","path": "elasticsearch","load_defaults": "true", "compress": true} }'

{"error":"RemoteTransportException[[node][inet[/10.3.89.102:9300]][cluster:admin/repository/put]]; nested: RepositoryException[[my_backup] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, java.lang.NoClassDefFoundError: org/apache/commons/cli/ParseException\n  at org.elasticsearch.repositories.hdfs.HdfsRepository.<init>()\n  at org.elasticsearch.repositories.hdfs.HdfsRepository\n  at Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1 error]; nested: NoClassDefFoundError[org/apache/commons/cli/ParseException]; nested: ClassNotFoundException[org.apache.commons.cli.ParseException]; ","status":500}



On Tuesday, February 10, 2015 at 2:58:12 PM UTC-8, Salman Ahmed wrote:
Hadoop nodes are NOT co-located with ElasticSearch and are on CDH5.1

I installed the plugin [ elasticsearch/elasticsearch-repository-hdfs/2.0.2] on all Elasticsearch nodes, and restarted the nodes

When I try to setup the repo, I get the following error:

curl -XPUT '<a href="http://localhost:9200/_snapshot/my_backup" target="_blank" rel="nofollow" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_snapshot%2Fmy_backup\46sa\75D\46sntz\0751\46usg\75AFQjCNHUWGQdgStMzqJ5NyJrRX_E-aHQHA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_snapshot%2Fmy_backup\46sa\75D\46sntz\0751\46usg\75AFQjCNHUWGQdgStMzqJ5NyJrRX_E-aHQHA';return true;">http://localhost:9200/_snapshot/my_backup' -d '{"type": "hdfs","uri": "hdfs://nn-ip:8020/","conf_location":"/etc/conf/hadoop/conf/hdfs-site.xml,/etc/conf/hadoop/conf/core-site.xml","path": "elasticsearch","compress": true} }'

{"error":"RepositoryException[[my_backup] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, org.elasticsearch.ElasticsearchIllegalArgumentException: no 'path' defined for hdfs snapshot/restore\n  at org.elasticsearch.repositories.hdfs.HdfsRepository.<init>(Unknown Source)\n  while locating org.elasticsearch.repositories.hdfs.HdfsRepository\n  while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: ElasticsearchIllegalArgumentException[no 'path' defined for hdfs snapshot/restore]; ","status":500}

Please help me, with what additional configuration needs to be done to back up ES indices to Hadoop which is not colocated on same nodes

Ref:<a href="https://github.com/elasticsearch/elasticsearch-hadoop/tree/master/repository-hdfs" target="_blank" rel="nofollow" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-hadoop%2Ftree%2Fmaster%2Frepository-hdfs\46sa\75D\46sntz\0751\46usg\75AFQjCNH6Xuf6uQ1_yHqy3moR0Hh2dmurNA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-hadoop%2Ftree%2Fmaster%2Frepository-hdfs\46sa\75D\46sntz\0751\46usg\75AFQjCNH6Xuf6uQ1_yHqy3moR0Hh2dmurNA';return true;">https://github.com/elasticsearch/elasticsearch-hadoop/tree/master/repository-hdfs

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4db8c55e-be0d-4532-a0c6-fd8662b2542b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/60CACB99-C626-48DB-B440-9CE23715EDA5%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Error while backing up ES to HDFS via plugin

dadoonet
I think I understand.

You need to wrap hdfs settings under a settings field:

$ curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{
   
"type": "hdfs",
   
"settings": {
        // settings here

   
}
}'
HTH

David

Le 11 févr. 2015 à 06:39, David Pilato <[hidden email]> a écrit :

What gives GET /_cat/plugins ?

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 11 févr. 2015 à 02:07, Salman Ahmed <[hidden email]> a écrit :

I made a bit more progress, now when I run this I get this:

curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{"type": "hdfs","uri": "hds://nnip:8020/","conf_location":"/etc/conf/hadoop/conf/hdfs-site.xml,/etc/conf/hadoop/conf/core-site.xml","path": "elasticsearch","load_defaults": "true", "compress": true} }'

{"error":"RemoteTransportException[[node][inet[/10.3.89.102:9300]][cluster:admin/repository/put]]; nested: RepositoryException[[my_backup] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, java.lang.NoClassDefFoundError: org/apache/commons/cli/ParseException\n  at org.elasticsearch.repositories.hdfs.HdfsRepository.<init>()\n  at org.elasticsearch.repositories.hdfs.HdfsRepository\n  at Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1 error]; nested: NoClassDefFoundError[org/apache/commons/cli/ParseException]; nested: ClassNotFoundException[org.apache.commons.cli.ParseException]; ","status":500}



On Tuesday, February 10, 2015 at 2:58:12 PM UTC-8, Salman Ahmed wrote:
Hadoop nodes are NOT co-located with ElasticSearch and are on CDH5.1

I installed the plugin [ elasticsearch/elasticsearch-repository-hdfs/2.0.2] on all Elasticsearch nodes, and restarted the nodes

When I try to setup the repo, I get the following error:

curl -XPUT '<a href="http://localhost:9200/_snapshot/my_backup" target="_blank" rel="nofollow" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_snapshot%2Fmy_backup\46sa\75D\46sntz\0751\46usg\75AFQjCNHUWGQdgStMzqJ5NyJrRX_E-aHQHA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Flocalhost%3A9200%2F_snapshot%2Fmy_backup\46sa\75D\46sntz\0751\46usg\75AFQjCNHUWGQdgStMzqJ5NyJrRX_E-aHQHA';return true;">http://localhost:9200/_snapshot/my_backup' -d '{"type": "hdfs","uri": "hdfs://nn-ip:8020/","conf_location":"/etc/conf/hadoop/conf/hdfs-site.xml,/etc/conf/hadoop/conf/core-site.xml","path": "elasticsearch","compress": true} }'

{"error":"RepositoryException[[my_backup] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, org.elasticsearch.ElasticsearchIllegalArgumentException: no 'path' defined for hdfs snapshot/restore\n  at org.elasticsearch.repositories.hdfs.HdfsRepository.<init>(Unknown Source)\n  while locating org.elasticsearch.repositories.hdfs.HdfsRepository\n  while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: ElasticsearchIllegalArgumentException[no 'path' defined for hdfs snapshot/restore]; ","status":500}

Please help me, with what additional configuration needs to be done to back up ES indices to Hadoop which is not colocated on same nodes

Ref:<a href="https://github.com/elasticsearch/elasticsearch-hadoop/tree/master/repository-hdfs" target="_blank" rel="nofollow" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-hadoop%2Ftree%2Fmaster%2Frepository-hdfs\46sa\75D\46sntz\0751\46usg\75AFQjCNH6Xuf6uQ1_yHqy3moR0Hh2dmurNA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch-hadoop%2Ftree%2Fmaster%2Frepository-hdfs\46sa\75D\46sntz\0751\46usg\75AFQjCNH6Xuf6uQ1_yHqy3moR0Hh2dmurNA';return true;">https://github.com/elasticsearch/elasticsearch-hadoop/tree/master/repository-hdfs

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4db8c55e-be0d-4532-a0c6-fd8662b2542b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/60CACB99-C626-48DB-B440-9CE23715EDA5%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/F0F088A7-0F75-4593-A427-BEEB7D82CE08%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Error while backing up ES to HDFS via plugin

Salman
Thanks for the tip, now I am getting the following error:

 curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{"type": "hdfs","settings":{"uri": "hdfs://ip01:8020/","conf_location":"/etc/conf/hadoop/conf/hdfs-site.xml,/etc/conf/hadoop/conf/core-site.xml","path": "elasticsearch","load_defaults": "true", "compress": true} }'
{"error":"RemoteTransportException[[ip03][inet[/10.3.89.103:9300]][cluster:admin/repository/put]]; nested: RepositoryException[[my_backup] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, java.lang.NoClassDefFoundError: org/elasticsearch/common/blobstore/ImmutableBlobContainer\n  at org.elasticsearch.repositories.hdfs.HdfsRepository.<init>()\n  at org.elasticsearch.repositories.hdfs.HdfsRepository\n  at Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1 error]; nested: NoClassDefFoundError[org/elasticsearch/common/blobstore/ImmutableBlobContainer]; nested: ClassNotFoundException[org.elasticsearch.common.blobstore.ImmutableBlobContainer]; ","status":500}
Reply | Threaded
Open this post in threaded view
|

Re: Error while backing up ES to HDFS via plugin

dadoonet
What gives GET /_cat/plugins ?
What is you Elasticsearch version?

-- 
David Pilato | Technical Advocate | Elasticsearch.com



Le 13 févr. 2015 à 15:41, Salman <[hidden email]> a écrit :

Thanks for the tip, now I am getting the following error:

curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{"type":
"hdfs","settings":{"uri":
"<a href="hdfs://ip01:8020/" class="">hdfs://ip01:8020/","conf_location":"/etc/conf/hadoop/conf/hdfs-site.xml,/etc/conf/hadoop/conf/core-site.xml","path":
"elasticsearch","load_defaults": "true", "compress": true} }'
{"error":"RemoteTransportException[[ip03][inet[/10.3.89.103:9300]][cluster:admin/repository/put]];
nested: RepositoryException[[my_backup] failed to create repository];
nested: CreationException[Guice creation errors:\n\n1) Error injecting
constructor, java.lang.NoClassDefFoundError:
org/elasticsearch/common/blobstore/ImmutableBlobContainer\n  at
org.elasticsearch.repositories.hdfs.HdfsRepository.<init>()\n  at
org.elasticsearch.repositories.hdfs.HdfsRepository\n  at
Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1
error]; nested:
NoClassDefFoundError[org/elasticsearch/common/blobstore/ImmutableBlobContainer];
nested:
ClassNotFoundException[org.elasticsearch.common.blobstore.ImmutableBlobContainer];
","status":500}



--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Error-while-backing-up-ES-to-HDFS-via-plugin-tp4070310p4070515.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1423838504957-4070515.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/765E437F-1708-40C7-B202-981E00D593D3%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Error while backing up ES to HDFS via plugin

Salman
In reply to this post by Salman
I finally got it to work with the latest beta plugin. Thank you guys for your help.
Reply | Threaded
Open this post in threaded view
|

Re: Error while backing up ES to HDFS via plugin

Mungeol Heo
I met the same problem with Salman, and, also, solved by using latest
beta version. BTW, I tested on elasticsearch-1.4.2.

On Sat, Feb 14, 2015 at 1:47 AM, Salman <[hidden email]> wrote:

> I finally got it to work with the latest beta plugin. Thank you guys for your
> help.
>
>
>
> --
> View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Error-while-backing-up-ES-to-HDFS-via-plugin-tp4070310p4070526.html
> Sent from the ElasticSearch Users mailing list archive at Nabble.com.
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
> To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1423846039443-4070526.post%40n3.nabble.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CADQPeWwSBmZ%2BZyvjPywuS-1VXLxmt8DWRTRrcqtFQQY%2BSBuVQw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Error while backing up ES to HDFS via plugin

dgligorov
This post has NOT been accepted by the mailing list yet.
Same issue here with ES 1.4.4 and 2.0.2-light, with 2.1.0.Beta4-light is OK

...was doing upgrade from ES 1.2 with 2.0.0-light