Elastic on Secure YARN

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Elastic on Secure YARN

Eric Walk
Hi All,

I'm working on running elasticsearch on yarn in a secured hadoop cluster (HDP 2.2).

It is failing when instantiating the containers with the error below. The issue appears that the Kerberos based JAAS Principal is not being passed from the launching container to the child containers. This could be a configuration issue, it could also just be something that is not implemented in the elasticsearch on yarn module. Was hoping to get a quick clarification before I spend too much time troubleshooting.

The files are uploaded in HDFS:

[root@ldevawshdp0001 elasticsearch-yarn]# hadoop fs -ls /apps/elasticsearch
Found 2 items
-rw-r--r--   2 ewalk hdfs   31442427 2015-04-21 18:35 /apps/elasticsearch/elasticsearch-1.5.1.zip
-rw-r--r--   2 ewalk hdfs      53819 2015-04-21 18:35 /apps/elasticsearch/elasticsearch-yarn-2.1.0.BUILD-SNAPSHOT.jar

We are able to run other YARN jobs successfully, this is something specific to elastic's spin up process.

Any help would be appreciated.

Best,
Eric

15/04/21 19:17:32 INFO am.ApplicationMaster: Starting ApplicationMaster...
15/04/21 19:17:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/04/21 19:17:32 INFO client.RMProxy: Connecting to ResourceManager at ldevawshdp0002.cedargatepartners.pvc/10.4.0.7:8030
15/04/21 19:17:33 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
15/04/21 19:17:33 INFO am.EsCluster: Allocating Elasticsearch cluster with 2 nodes
15/04/21 19:17:38 INFO impl.AMRMClientImpl: Received new token for : ldevawshdp0002.cedargatepartners.pvc:45454
15/04/21 19:17:38 INFO impl.AMRMClientImpl: Received new token for : ldevawshdp0001.cedargatepartners.pvc:45454
15/04/21 19:17:38 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
15/04/21 19:17:38 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
15/04/21 19:17:38 INFO am.EsCluster: Cluster has completed running...
15/04/21 19:17:53 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
Exception in thread "main" java.lang.IllegalArgumentException: Cannot find Elasticsearch zip at [/apps/elasticsearch/elasticsearch-1.5.1.zip]; make sure the artifacts have been properly provisioned and the correct permissions are in place.
	at org.elasticsearch.hadoop.yarn.am.EsCluster.setupEsZipResource(EsCluster.java:224)
	at org.elasticsearch.hadoop.yarn.am.EsCluster.launchContainer(EsCluster.java:191)
	at org.elasticsearch.hadoop.yarn.am.EsCluster.start(EsCluster.java:115)
	at org.elasticsearch.hadoop.yarn.am.ApplicationMaster.run(ApplicationMaster.java:79)
	at org.elasticsearch.hadoop.yarn.am.ApplicationMaster.main(ApplicationMaster.java:104)
Caused by: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "ldevawshdp0002/10.4.0.7"; destination host is: "ldevawshdp0001.cedargatepartners.pvc":8020; 
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
	at org.apache.hadoop.ipc.Client.call(Client.java:1472)
	at org.apache.hadoop.ipc.Client.call(Client.java:1399)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
	at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
	at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
	at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
	at org.elasticsearch.hadoop.yarn.am.EsCluster.setupEsZipResource(EsCluster.java:222)
	... 4 more
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
	at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
	at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
	at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
	at org.apache.hadoop.ipc.Client.call(Client.java:1438)
	... 21 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
	at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
	at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
	at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
	... 24 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
	... 33 more

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/89b33a2b-a46b-4614-9fc4-a4ce09711680%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elastic on Secure YARN

Costin Leau
Hi,

Can you please raise an issue? From the looks of it the security token is indeed not passed along however I'm unclear
whether you are configuring one yourself or not.
You mention that launching other containers doesn't expose this issue, do you use them in a different way? Who holds the
Kerberos credentials?

Cheers,

On 4/21/15 10:47 PM, Eric Walk wrote:

> Hi All,
>
> I'm working on running elasticsearch on yarn in a secured hadoop cluster (HDP 2.2).
>
> It is failing when instantiating the containers with the error below. The issue appears that the Kerberos based JAAS
> Principal is not being passed from the launching container to the child containers. This could be a configuration issue,
> it could also just be something that is not implemented in the elasticsearch on yarn module. Was hoping to get a quick
> clarification before I spend too much time troubleshooting.
>
> The files are uploaded in HDFS:
>
> [root@ldevawshdp0001 elasticsearch-yarn]# hadoop fs -ls /apps/elasticsearch
> Found 2 items
> -rw-r--r--   2 ewalk hdfs   31442427 2015-04-21 18:35 /apps/elasticsearch/elasticsearch-1.5.1.zip
> -rw-r--r--   2 ewalk hdfs      53819 2015-04-21 18:35 /apps/elasticsearch/elasticsearch-yarn-2.1.0.BUILD-SNAPSHOT.jar
>
> We are able to run other YARN jobs successfully, this is something specific to elastic's spin up process.
>
> Any help would be appreciated.
>
> Best,
> Eric
>
> 15/04/21 19:17:32 INFO am.ApplicationMaster: Starting ApplicationMaster...
> 15/04/21 19:17:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 15/04/21 19:17:32 INFO client.RMProxy: Connecting to ResourceManager at ldevawshdp0002.cedargatepartners.pvc/10.4.0.7:8030
> 15/04/21 19:17:33 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
> 15/04/21 19:17:33 INFO am.EsCluster: Allocating Elasticsearch cluster with 2 nodes
> 15/04/21 19:17:38 INFO impl.AMRMClientImpl: Received new token for : ldevawshdp0002.cedargatepartners.pvc:45454
> 15/04/21 19:17:38 INFO impl.AMRMClientImpl: Received new token for : ldevawshdp0001.cedargatepartners.pvc:45454
> 15/04/21 19:17:38 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
> 15/04/21 19:17:38 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
> 15/04/21 19:17:38 INFO am.EsCluster: Cluster has completed running...
> 15/04/21 19:17:53 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
> Exception in thread "main" java.lang.IllegalArgumentException: Cannot find Elasticsearch zip at [/apps/elasticsearch/elasticsearch-1.5.1.zip]; make sure the artifacts have been properly provisioned and the correct permissions are in place.
> at org.elasticsearch.hadoop.yarn.am.EsCluster.setupEsZipResource(EsCluster.java:224)
> at org.elasticsearch.hadoop.yarn.am.EsCluster.launchContainer(EsCluster.java:191)
> at org.elasticsearch.hadoop.yarn.am.EsCluster.start(EsCluster.java:115)
> at org.elasticsearch.hadoop.yarn.am.ApplicationMaster.run(ApplicationMaster.java:79)
> at org.elasticsearch.hadoop.yarn.am.ApplicationMaster.main(ApplicationMaster.java:104)
> Caused by: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "ldevawshdp0002/10.4.0.7"; destination host is: "ldevawshdp0001.cedargatepartners.pvc":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
> at org.apache.hadoop.ipc.Client.call(Client.java:1472)
> at org.apache.hadoop.ipc.Client.call(Client.java:1399)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
> at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
> at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
> at org.elasticsearch.hadoop.yarn.am.EsCluster.setupEsZipResource(EsCluster.java:222)
> ... 4 more
> Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
> at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
> at org.apache.hadoop.ipc.Client.call(Client.java:1438)
> ... 21 more
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
> at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
> at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
> at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
> at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
> ... 24 more
> Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
> at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
> at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
> at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
> ... 33 more
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> [hidden email] <mailto:[hidden email]>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/89b33a2b-a46b-4614-9fc4-a4ce09711680%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/89b33a2b-a46b-4614-9fc4-a4ce09711680%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/553773A7.7080901%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elastic on Secure YARN

Eric Walk
Hi Costin,

I'll go ahead and open an issue in Git. Here's a little more thought on the matter:

I'm not a developer, I'm an Infrastructure guy, so I don't know all the details of how it is generally done.

The issue occurs only when the containers that are launched to be the elastic cluster members try to access HDFS.

There are thousands of different applications that can run in YARN containers, many of them do not have this issue. For instance the Spark Hadoop RDD is able to successfully connect using this code:
 

Essentially, when you start an application in YARN, the master has the credentials of the user that launched it by default, but the application containers out on the node managers will not have any credentials unless they are specifically delegated out to them by the code running on the master.

Best,
Eric

On Wed, Apr 22, 2015 at 6:11 AM Costin Leau <[hidden email]> wrote:
Hi,

Can you please raise an issue? From the looks of it the security token is indeed not passed along however I'm unclear
whether you are configuring one yourself or not.
You mention that launching other containers doesn't expose this issue, do you use them in a different way? Who holds the
Kerberos credentials?

Cheers,

On 4/21/15 10:47 PM, Eric Walk wrote:
> Hi All,
>
> I'm working on running elasticsearch on yarn in a secured hadoop cluster (HDP 2.2).
>
> It is failing when instantiating the containers with the error below. The issue appears that the Kerberos based JAAS
> Principal is not being passed from the launching container to the child containers. This could be a configuration issue,
> it could also just be something that is not implemented in the elasticsearch on yarn module. Was hoping to get a quick
> clarification before I spend too much time troubleshooting.
>
> The files are uploaded in HDFS:
>
> [root@ldevawshdp0001 elasticsearch-yarn]# hadoop fs -ls /apps/elasticsearch
> Found 2 items
> -rw-r--r--   2 ewalk hdfs   31442427 2015-04-21 18:35 /apps/elasticsearch/elasticsearch-1.5.1.zip
> -rw-r--r--   2 ewalk hdfs      53819 2015-04-21 18:35 /apps/elasticsearch/elasticsearch-yarn-2.1.0.BUILD-SNAPSHOT.jar
>
> We are able to run other YARN jobs successfully, this is something specific to elastic's spin up process.
>
> Any help would be appreciated.
>
> Best,
> Eric
>
> 15/04/21 19:17:32 INFO am.ApplicationMaster: Starting ApplicationMaster...
> 15/04/21 19:17:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 15/04/21 19:17:32 INFO client.RMProxy: Connecting to ResourceManager at ldevawshdp0002.cedargatepartners.pvc/10.4.0.7:8030
> 15/04/21 19:17:33 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
> 15/04/21 19:17:33 INFO am.EsCluster: Allocating Elasticsearch cluster with 2 nodes
> 15/04/21 19:17:38 INFO impl.AMRMClientImpl: Received new token for : ldevawshdp0002.cedargatepartners.pvc:45454
> 15/04/21 19:17:38 INFO impl.AMRMClientImpl: Received new token for : ldevawshdp0001.cedargatepartners.pvc:45454
> 15/04/21 19:17:38 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
> 15/04/21 19:17:38 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
> 15/04/21 19:17:38 INFO am.EsCluster: Cluster has completed running...
> 15/04/21 19:17:53 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
> Exception in thread "main" java.lang.IllegalArgumentException: Cannot find Elasticsearch zip at [/apps/elasticsearch/elasticsearch-1.5.1.zip]; make sure the artifacts have been properly provisioned and the correct permissions are in place.
>       at org.elasticsearch.hadoop.yarn.am.EsCluster.setupEsZipResource(EsCluster.java:224)
>       at org.elasticsearch.hadoop.yarn.am.EsCluster.launchContainer(EsCluster.java:191)
>       at org.elasticsearch.hadoop.yarn.am.EsCluster.start(EsCluster.java:115)
>       at org.elasticsearch.hadoop.yarn.am.ApplicationMaster.run(ApplicationMaster.java:79)
>       at org.elasticsearch.hadoop.yarn.am.ApplicationMaster.main(ApplicationMaster.java:104)
> Caused by: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "ldevawshdp0002/10.4.0.7"; destination host is: "ldevawshdp0001.cedargatepartners.pvc":8020;
>       at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
>       at org.apache.hadoop.ipc.Client.call(Client.java:1472)
>       at org.apache.hadoop.ipc.Client.call(Client.java:1399)
>       at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
>       at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)
>       at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>       at java.lang.reflect.Method.invoke(Method.java:606)
>       at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
>       at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>       at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source)
>       at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
>       at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
>       at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
>       at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>       at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
>       at org.elasticsearch.hadoop.yarn.am.EsCluster.setupEsZipResource(EsCluster.java:222)
>       ... 4 more
> Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>       at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
>       at java.security.AccessController.doPrivileged(Native Method)
>       at javax.security.auth.Subject.doAs(Subject.java:415)
>       at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>       at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
>       at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
>       at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
>       at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
>       at org.apache.hadoop.ipc.Client.call(Client.java:1438)
>       ... 21 more
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>       at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
>       at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
>       at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
>       at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
>       at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
>       at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
>       at java.security.AccessController.doPrivileged(Native Method)
>       at javax.security.auth.Subject.doAs(Subject.java:415)
>       at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>       at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
>       ... 24 more
> Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
>       at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
>       at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
>       at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
>       at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
>       at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
>       at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
>       at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
>       ... 33 more
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> [hidden email] <mailto:[hidden email]>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/89b33a2b-a46b-4614-9fc4-a4ce09711680%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/89b33a2b-a46b-4614-9fc4-a4ce09711680%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/6BGX8bKNWak/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/553773A7.7080901%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPSACbvQHAmJjmN3YxFC9c2sAYdean-BRai6HsiEFegOLEueDQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.