jdbc river keeping connection open

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

jdbc river keeping connection open

Abid Hussain
Hi all,

we have some JUnit based integration tests in which we basically setup a clean system, create a river and do some index operations like inserting/updating documents. We are not using ElasticSearchIntegrationTest but are running our tests against a dedicated cluster.

In the setup which is performed before every test run we do the following:
  1. remove jdbc river from last test run
  2. remove index from last test run
  3. truncate all tables in database
We are facing the issue that step 3 sometimes blocks because there is an open connection from the river.

The deletion is performed by this code:
DeleteMappingResponse response = client.admin().indices()
 
.prepareDeleteMapping("_river")
 
.setType(rivername)
 
.execute()
 
.actionGet();

return response.isAcknowledged();

Is this a known issue or do we have some misunderstanding of how to deal with river deletion?

May it be the case that the table truncation is performed when the river deletion isn't fully completed? If so, is there a way to wait until river deletion completed?

Regards,

Abid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/00a6d81c-5f72-4c16-a814-dcb771d3111b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: jdbc river keeping connection open

joergprante@gmail.com

Are you sure the deletion of a (non-existent) mapping "_river" makes sense?

What connection is open? Database connection?

Jörg

Am 16.02.2015 12:29 schrieb "Abid Hussain" <[hidden email]>:
Hi all,

we have some JUnit based integration tests in which we basically setup a clean system, create a river and do some index operations like inserting/updating documents. We are not using ElasticSearchIntegrationTest but are running our tests against a dedicated cluster.

In the setup which is performed before every test run we do the following:
  1. remove jdbc river from last test run
  2. remove index from last test run
  3. truncate all tables in database
We are facing the issue that step 3 sometimes blocks because there is an open connection from the river.

The deletion is performed by this code:
DeleteMappingResponse response = client.admin().indices()
 
.prepareDeleteMapping("_river")
 
.setType(rivername)
 
.execute()
 
.actionGet();

return response.isAcknowledged();

Is this a known issue or do we have some misunderstanding of how to deal with river deletion?

May it be the case that the table truncation is performed when the river deletion isn't fully completed? If so, is there a way to wait until river deletion completed?

Regards,

Abid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/00a6d81c-5f72-4c16-a814-dcb771d3111b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHs1Yr2hGRv-Hg1Q0ufCEZY720boZwoU%2BXAeudx-nJ1gQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: jdbc river keeping connection open

Abid Hussain

Are you sure the deletion of a (non-existent) mapping "_river" makes sense?

The given code results in river deletion. If there is a better way to do so, I would be happy if you could tell me.
 

What connection is open? Database connection?

Yes, a database connection used by river remains open.

Regards,

Abid
 

Jörg

Am 16.02.2015 12:29 schrieb "Abid Hussain" <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="yO9OK8Qz5xQJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">hus...@...>:
Hi all,

we have some JUnit based integration tests in which we basically setup a clean system, create a river and do some index operations like inserting/updating documents. We are not using ElasticSearchIntegrationTest but are running our tests against a dedicated cluster.

In the setup which is performed before every test run we do the following:
  1. remove jdbc river from last test run
  2. remove index from last test run
  3. truncate all tables in database
We are facing the issue that step 3 sometimes blocks because there is an open connection from the river.

The deletion is performed by this code:
DeleteMappingResponse response = client.admin().indices()
 
.prepareDeleteMapping("_river")
 
.setType(rivername)
 
.execute()
 
.actionGet();

return response.isAcknowledged();

Is this a known issue or do we have some misunderstanding of how to deal with river deletion?

May it be the case that the table truncation is performed when the river deletion isn't fully completed? If so, is there a way to wait until river deletion completed?

Regards,

Abid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="yO9OK8Qz5xQJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/00a6d81c-5f72-4c16-a814-dcb771d3111b%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/00a6d81c-5f72-4c16-a814-dcb771d3111b%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/00a6d81c-5f72-4c16-a814-dcb771d3111b%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/00a6d81c-5f72-4c16-a814-dcb771d3111b%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7108a8e1-8695-40e2-a269-4d3b32773dee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: jdbc river keeping connection open

Abid Hussain
I noticed in the elasticsearch logs:
[2015-02-16 15:25:20,398][INFO ][river.jdbc.RiverMetrics  ] pipeline org.xbib.elasticsearch.plugin.jdbc.RiverPipeline@25042719 complete: river jdbc/ctv4ch_junit_abid_interview metrics: 1 rows, 0.23379861173437763 mean, (0.0 0.0 0.0), ingest metrics: elapsed 4 seconds, 0.0 bytes bytes, 0.0 bytes avg, 0 MB/s
[2015-02-16 15:25:20,398][ERROR][river.jdbc.BulkNodeClient] after bulk [1] error
java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
        at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
        at org.elasticsearch.action.bulk.BulkProcessor.execute(BulkProcessor.java:325)
        at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.xbib.elasticsearch.plugin.jdbc.client.BulkProcessorHelper.flush(BulkProcessorHelper.java:43)

This seems to occur when the river, which has just been created, is removed again by the next test run. So it seems the an InterruptedException results in an open connection remaining.

I did the following: before deletion of river; I wait until the previously created one did finish (via _river/jdbc/<river-name>/_state. Then I delete.

This seems to work.

Regards,

Abid

Am Montag, 16. Februar 2015 14:43:37 UTC+1 schrieb Abid Hussain:

Are you sure the deletion of a (non-existent) mapping "_river" makes sense?

The given code results in river deletion. If there is a better way to do so, I would be happy if you could tell me.
 

What connection is open? Database connection?

Yes, a database connection used by river remains open.

Regards,

Abid
 

Jörg

Am 16.02.2015 12:29 schrieb "Abid Hussain" <[hidden email]>:
Hi all,

we have some JUnit based integration tests in which we basically setup a clean system, create a river and do some index operations like inserting/updating documents. We are not using ElasticSearchIntegrationTest but are running our tests against a dedicated cluster.

In the setup which is performed before every test run we do the following:
  1. remove jdbc river from last test run
  2. remove index from last test run
  3. truncate all tables in database
We are facing the issue that step 3 sometimes blocks because there is an open connection from the river.

The deletion is performed by this code:
DeleteMappingResponse response = client.admin().indices()
 
.prepareDeleteMapping("_river")
 
.setType(rivername)
 
.execute()
 
.actionGet();

return response.isAcknowledged();

Is this a known issue or do we have some misunderstanding of how to deal with river deletion?

May it be the case that the table truncation is performed when the river deletion isn't fully completed? If so, is there a way to wait until river deletion completed?

Regards,

Abid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/00a6d81c-5f72-4c16-a814-dcb771d3111b%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/00a6d81c-5f72-4c16-a814-dcb771d3111b%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/00a6d81c-5f72-4c16-a814-dcb771d3111b%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/00a6d81c-5f72-4c16-a814-dcb771d3111b%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3885a810-218e-44a9-b31d-2fd51a37c86e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.