Quantcast

couch-ES integration

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

couch-ES integration

andrea crotti
I'm trying to write some integration tests to try out the integration between couchdb and ES.

The idea is that I should:
- create a database
- add some entries to the eb
- create a river

And it should be automatically indexed and be able to search from it.

I can't however get this working, no matter how many sleep and refresh() I add there..

The setup and teardown functions look like this:


def setup_es():
    global ES_CON
    ES_CON = ES(ES_URI)
    test_river = CouchDBRiver(host=COUCH_HOST, port=COUCH_PORT, db=TEST_DATABASE)
    ES_CON.create_river(test_river, river_name=TEST_INDEX)
    ES_CON.refresh()


def setup_db():
    global SERVER
    SERVER = Server(uri=COUCH_URI)
    if TEST_DATABASE in SERVER.all_dbs():
        SERVER.delete_db(TEST_DATABASE)

    db = SERVER.create_db(TEST_DATABASE)
    db.save_doc(USER_DOC)


def clear():
    SERVER.delete_db(TEST_DATABASE)
    ES_CON.delete_river(TEST_INDEX)
    ES_CON.indices.delete_index(TEST_INDEX)



And this test should for example pass:
    def test_user_doc_indexed(self):
        self.assertEqual(ES_CON.search(MatchAllQuery(), indices=[TEST_INDEX]).count(), 1)


Is it a timing problem?
I'll just leave a test db running maybe if I have no choice, but would prefer to do it all in the test module..

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: couch-ES integration

andrea crotti
Even trying to don't delete the database and the river doesn't seem to work unfortunately.

Now the problem is also that if I just create the river and the index I get this:

SearchPhaseExecutionException: Failed to execute phase [query], total failure; shardFailures {[_na_][test_index][0]: No active shards}{[_na_][test_index][1]: No active shards}{[_na_][test_index][2]: No active shards}{[_na_][test_index][3]: No active shards}{[_na_][test_index][4]: No active shards}

BUT if I actually force indexing some data with something like this:
    ES_CON.index(ud, TEST_INDEX, doc_type='User')

it works fine and the data is found, but the river is still not doing its job..

On Tuesday, February 26, 2013 12:12:25 PM UTC, andrea crotti wrote:
I'm trying to write some integration tests to try out the integration between couchdb and ES.

The idea is that I should:
- create a database
- add some entries to the eb
- create a river

And it should be automatically indexed and be able to search from it.

I can't however get this working, no matter how many sleep and refresh() I add there..

The setup and teardown functions look like this:


def setup_es():
    global ES_CON
    ES_CON = ES(ES_URI)
    test_river = CouchDBRiver(host=COUCH_HOST, port=COUCH_PORT, db=TEST_DATABASE)
    ES_CON.create_river(test_river, river_name=TEST_INDEX)
    ES_CON.refresh()


def setup_db():
    global SERVER
    SERVER = Server(uri=COUCH_URI)
    if TEST_DATABASE in SERVER.all_dbs():
        SERVER.delete_db(TEST_DATABASE)

    db = SERVER.create_db(TEST_DATABASE)
    db.save_doc(USER_DOC)


def clear():
    SERVER.delete_db(TEST_DATABASE)
    ES_CON.delete_river(TEST_INDEX)
    ES_CON.indices.delete_index(TEST_INDEX)



And this test should for example pass:
    def test_user_doc_indexed(self):
        self.assertEqual(ES_CON.search(MatchAllQuery(), indices=[TEST_INDEX]).count(), 1)


Is it a timing problem?
I'll just leave a test db running maybe if I have no choice, but would prefer to do it all in the test module..

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: couch-ES integration

dadoonet
Hi Andrea,


I think you should understand something.
When you create the river, the river takes some time to really start, establish the contact with CouchDB and index documents.
So, if 1ms after the river creation (even if you *refresh* the river), you are not sure that you will have the first document in it.
Or, the index is on the way of being created but all shards are not yet available.

Here are some suggestions:
1/ create the index first. waitForAYellowClusterStatus. Then add the river. Your search won't fail after that
2/ wait for a reasonable time, let's say 5 seconds.

I prefer the first option.

Does it help?

-- 
David Pilato | Technical Advocate | Elasticsearch.com



Le 26 févr. 2013 à 13:36, andrea crotti <[hidden email]> a écrit :

Even trying to don't delete the database and the river doesn't seem to work unfortunately.

Now the problem is also that if I just create the river and the index I get this:

SearchPhaseExecutionException: Failed to execute phase [query], total failure; shardFailures {[_na_][test_index][0]: No active shards}{[_na_][test_index][1]: No active shards}{[_na_][test_index][2]: No active shards}{[_na_][test_index][3]: No active shards}{[_na_][test_index][4]: No active shards}

BUT if I actually force indexing some data with something like this:
    ES_CON.index(ud, TEST_INDEX, doc_type='User')

it works fine and the data is found, but the river is still not doing its job..

On Tuesday, February 26, 2013 12:12:25 PM UTC, andrea crotti wrote:
I'm trying to write some integration tests to try out the integration between couchdb and ES.

The idea is that I should:
- create a database
- add some entries to the eb
- create a river

And it should be automatically indexed and be able to search from it.

I can't however get this working, no matter how many sleep and refresh() I add there..

The setup and teardown functions look like this:


def setup_es():
    global ES_CON
    ES_CON = ES(ES_URI)
    test_river = CouchDBRiver(host=COUCH_HOST, port=COUCH_PORT, db=TEST_DATABASE)
    ES_CON.create_river(test_river, river_name=TEST_INDEX)
    ES_CON.refresh()


def setup_db():
    global SERVER
    SERVER = Server(uri=COUCH_URI)
    if TEST_DATABASE in SERVER.all_dbs():
        SERVER.delete_db(TEST_DATABASE)

    db = SERVER.create_db(TEST_DATABASE)
    db.save_doc(USER_DOC)


def clear():
    SERVER.delete_db(TEST_DATABASE)
    ES_CON.delete_river(TEST_INDEX)
    ES_CON.indices.delete_index(TEST_INDEX)



And this test should for example pass:
    def test_user_doc_indexed(self):
        self.assertEqual(ES_CON.search(MatchAllQuery(), indices=[TEST_INDEX]).count(), 1)


Is it a timing problem?
I'll just leave a test db running maybe if I have no choice, but would prefer to do it all in the test module..

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Loading...