adding document not working by using Bulk API JAVA

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

adding document not working by using Bulk API JAVA

fabup
This post has NOT been accepted by the mailing list yet.
Hi! I'm trying to test Bulk API JAVA to reindex Crawler2 to Crawler3. Here is my code

public static void main(String[] args) throws Exception {
                String str_id = "";
                String old_index = "crawler2";
                String new_index = "crawler3";
                String table = "page";
               
                Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name", "test-cluster").build();        
                TransportClient client = new TransportClient(settings).addTransportAddress(new InetSocketTransportAddress("10.20.8.51",9300));
                QueryBuilder qb = QueryBuilders.matchAllQuery();
                SearchResponse scrollResp = client.prepareSearch(old_index)
                        .setSearchType(SearchType.SCAN)
                        .setScroll(new TimeValue(600000))
                        .setQuery(qb)
                        .setSize(100).execute().actionGet(); //100 hits per shard will be returned for each scroll
                //Scroll until no hits are returned
                System.out.println("-------Start-----");
                while (true) {
                        try{
                                BulkRequestBuilder bulkRequest = client.prepareBulk();
                            for (SearchHit hit : scrollResp.getHits().getHits()) {
                            //System.out.println("str_id : ");
                            str_id = String.valueOf(hit.getId());    
                           
                            bulkRequest.add(client.prepareIndex(new_index, table, str_id)
                            .setSource(hit.getSourceAsString()));    
                            BulkResponse bulkResponse = bulkRequest.execute().actionGet();    
                            }
                            scrollResp = client.prepareSearchScroll(scrollResp.getScrollId()).setScroll(new TimeValue(600000)).execute().actionGet();    
                            //Break condition: No hits are returned
                            if (scrollResp.getHits().getHits().length == 0) {    
                            System.out.println("len = 0");
                                break;
                            }
                        }
                        catch(Exception e){
                                System.out.println("e : "+e);
                                break;
                        }
                }
                System.out.println("-------Stop-----");
                client.close();
               
    }

This code can add 700-900 documents to index and then it will appear error

 org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes were available: [[Chameleon][Yts0j3wlSQaYQA9JUYT4vQ][HawaiSearchEngineTest][inet[/10.20.8.51:9300]]{master=true}]

As picture below, at the third line "docs: 781 (1,910)" , the number in bracket (1,910) still increase but document number (781) nothing change.



could you please help me for resolving this issue. Thank you for your help.