rails client getting socket connect timeouts regularly

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

rails client getting socket connect timeouts regularly

T Vinod Gupta
hi,
we have an ES index that is accessed by a rails server.. things work well in general. but every few minutes (once/twice an hour), the client gets a socket connect failure from the ES server. has anyone seen this and if yes, what is the solution? our ES host is running healthy with heavy write/read operations going in parallel from java clients. the only difference i see is that in the java clients, we have a pool of Client objects that we reuse and dont create a new client for each operation. 

are there any best practices here? should there be a http connection pooling between rails client and ES server? it is not trivial to do in rails as each web request goes to a separate process.

here is the stack trace every time it happens - 

/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:in `initialize'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:in `open'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:in `open_socket'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:227:in `connect'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:294:in `socket'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:159:in `request'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/adapter/excon.rb:15:in `call'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/response.rb:8:in `call'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/request/json.rb:32:in `call'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/connection.rb:210:in `run_request'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/connection.rb:93:in `get'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/elasticsearch-client-0.0.6/lib/elasticsearch.rb:54:in `get'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/elasticsearch-client-0.0.6/lib/elasticsearch.rb:79:in `mget'

thanks 
Reply | Threaded
Open this post in threaded view
|

Re: rails client getting socket connect timeouts regularly

kimchy
Administrator
First, from Java there isn't a need to pool Client objects, you can just use a single Client in your application (its perfectly thread safe).

Regarding the rails client, can you possibly monitor the number of open sockets you have on the machine, maybe you are opening and closing sockets from rails where the OS is starting to throttle it? Any chance of trying and use persistent connections from rails?

On Aug 6, 2012, at 5:56 PM, T Vinod Gupta <[hidden email]> wrote:

hi,
we have an ES index that is accessed by a rails server.. things work well in general. but every few minutes (once/twice an hour), the client gets a socket connect failure from the ES server. has anyone seen this and if yes, what is the solution? our ES host is running healthy with heavy write/read operations going in parallel from java clients. the only difference i see is that in the java clients, we have a pool of Client objects that we reuse and dont create a new client for each operation. 

are there any best practices here? should there be a http connection pooling between rails client and ES server? it is not trivial to do in rails as each web request goes to a separate process.

here is the stack trace every time it happens - 

/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:in `initialize'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:in `open'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:in `open_socket'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:227:in `connect'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:294:in `socket'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:159:in `request'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/adapter/excon.rb:15:in `call'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/response.rb:8:in `call'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/request/json.rb:32:in `call'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/connection.rb:210:in `run_request'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/connection.rb:93:in `get'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/elasticsearch-client-0.0.6/lib/elasticsearch.rb:54:in `get'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/elasticsearch-client-0.0.6/lib/elasticsearch.rb:79:in `mget'

thanks 

Reply | Threaded
Open this post in threaded view
|

Re: rails client getting socket connect timeouts regularly

T Vinod Gupta
thanks for the tip on java client object. i didnt know it was thread safe.

ill take a look at open sockets as you suggested. do you know of any rails library/gem that lets you do persistent connections to ES? do people implement their own wrapper/abstraction here?

thanks 

On Tue, Aug 7, 2012 at 2:36 PM, Shay Banon <[hidden email]> wrote:
First, from Java there isn't a need to pool Client objects, you can just use a single Client in your application (its perfectly thread safe).

Regarding the rails client, can you possibly monitor the number of open sockets you have on the machine, maybe you are opening and closing sockets from rails where the OS is starting to throttle it? Any chance of trying and use persistent connections from rails?


On Aug 6, 2012, at 5:56 PM, T Vinod Gupta <[hidden email]> wrote:

hi,
we have an ES index that is accessed by a rails server.. things work well in general. but every few minutes (once/twice an hour), the client gets a socket connect failure from the ES server. has anyone seen this and if yes, what is the solution? our ES host is running healthy with heavy write/read operations going in parallel from java clients. the only difference i see is that in the java clients, we have a pool of Client objects that we reuse and dont create a new client for each operation. 

are there any best practices here? should there be a http connection pooling between rails client and ES server? it is not trivial to do in rails as each web request goes to a separate process.

here is the stack trace every time it happens - 

/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:in `initialize'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:in `open'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:in `open_socket'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:227:in `connect'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:294:in `socket'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:159:in `request'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/adapter/excon.rb:15:in `call'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/response.rb:8:in `call'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/request/json.rb:32:in `call'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/connection.rb:210:in `run_request'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/connection.rb:93:in `get'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/elasticsearch-client-0.0.6/lib/elasticsearch.rb:54:in `get'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/elasticsearch-client-0.0.6/lib/elasticsearch.rb:79:in `mget'

thanks 


Reply | Threaded
Open this post in threaded view
|

Re: rails client getting socket connect timeouts regularly

Shairon Toledo
We've been used rubberband ruby gem for more than 4 months we didn't get any issue with timeout.
Rubberband gets nodes list from http://node:9200/_nodes before each search or index in the same request, it also has built-in retry, failover and others features, maybe this implementation fits better in your Rails env.



On Tue, Aug 7, 2012 at 6:42 PM, T Vinod Gupta <[hidden email]> wrote:
thanks for the tip on java client object. i didnt know it was thread safe.

ill take a look at open sockets as you suggested. do you know of any rails library/gem that lets you do persistent connections to ES? do people implement their own wrapper/abstraction here?

thanks 


On Tue, Aug 7, 2012 at 2:36 PM, Shay Banon <[hidden email]> wrote:
First, from Java there isn't a need to pool Client objects, you can just use a single Client in your application (its perfectly thread safe).

Regarding the rails client, can you possibly monitor the number of open sockets you have on the machine, maybe you are opening and closing sockets from rails where the OS is starting to throttle it? Any chance of trying and use persistent connections from rails?


On Aug 6, 2012, at 5:56 PM, T Vinod Gupta <[hidden email]> wrote:

hi,
we have an ES index that is accessed by a rails server.. things work well in general. but every few minutes (once/twice an hour), the client gets a socket connect failure from the ES server. has anyone seen this and if yes, what is the solution? our ES host is running healthy with heavy write/read operations going in parallel from java clients. the only difference i see is that in the java clients, we have a pool of Client objects that we reuse and dont create a new client for each operation. 

are there any best practices here? should there be a http connection pooling between rails client and ES server? it is not trivial to do in rails as each web request goes to a separate process.

here is the stack trace every time it happens - 

/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:in `initialize'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:in `open'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:in `open_socket'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:227:in `connect'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:294:in `socket'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:159:in `request'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/adapter/excon.rb:15:in `call'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/response.rb:8:in `call'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/request/json.rb:32:in `call'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/connection.rb:210:in `run_request'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/connection.rb:93:in `get'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/elasticsearch-client-0.0.6/lib/elasticsearch.rb:54:in `get'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/elasticsearch-client-0.0.6/lib/elasticsearch.rb:79:in `mget'

thanks 





--











Shairon Toledo
http://hashcode.me