Elastic Search configuration

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Elastic Search configuration

Hardik Dobariya

Please let me know if below elastic server configuration is feasible or not.

Total 1 cluster with 3 nodes having 5 shards and 1 replica with multiple indices having total around 100 million data ,sizing 1tb and still increasing


  1. Node 1:Master=true,data=true (30gb ram,8 cores)
  2. Node 2:Master=true,data=true (30gb ram,8 cores)
  3. Node 3:Master=false,data=false (30gb ram,8 cores)

All nodes are pointing to same UNC path(\\pc1\\data) data location.Data is not stored locally on nodes,all data is stored on one location as mentioned above.we are using Nas to store data and seems Nas is very fast so we are storing data on UNC path and not locally.

While googling found if shards are decreased ,it will increase search performance.According to the specified configuration should i drop no of shards to 1?.Will it increase the performance or does the configuration need any changes?.
Please suggest me the configuration.
Thanks

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elastic Search configuration

Mark Walkom-2
Are you having performance issues now?

If so I'd start with replacing your disk setup, chances are it's not as fast as you think, especially if you are storing the data from both nodes on the same path.
Also what ES version are you on, what Java release and version?

On 12 May 2015 at 19:01, Hardik Dobariya <[hidden email]> wrote:

Please let me know if below elastic server configuration is feasible or not.

Total 1 cluster with 3 nodes having 5 shards and 1 replica with multiple indices having total around 100 million data ,sizing 1tb and still increasing


  1. Node 1:Master=true,data=true (30gb ram,8 cores)
  2. Node 2:Master=true,data=true (30gb ram,8 cores)
  3. Node 3:Master=false,data=false (30gb ram,8 cores)

All nodes are pointing to same UNC path(\\pc1\\data) data location.Data is not stored locally on nodes,all data is stored on one location as mentioned above.we are using Nas to store data and seems Nas is very fast so we are storing data on UNC path and not locally.

While googling found if shards are decreased ,it will increase search performance.According to the specified configuration should i drop no of shards to 1?.Will it increase the performance or does the configuration need any changes?.
Please suggest me the configuration.
Thanks

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_7DjNa%2B82_B-v95%2BW1HbSs-VsUC%3DHxa8OuRLOhJCUV-Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elastic Search configuration

Hardik Dobariya
No am not facing any performance issue.Just want to confirm if my configuration is feasible or not and what will happen if i bring down shard to 1.
My ES version is 1.5.2 and java release is 8.45

On Tuesday, May 12, 2015 at 3:38:40 PM UTC+5:30, Mark Walkom wrote:
Are you having performance issues now?

If so I'd start with replacing your disk setup, chances are it's not as fast as you think, especially if you are storing the data from both nodes on the same path.
Also what ES version are you on, what Java release and version?

On 12 May 2015 at 19:01, Hardik Dobariya <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="QLDQNvxkgUkJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">hardikdob...@...> wrote:

Please let me know if below elastic server configuration is feasible or not.

Total 1 cluster with 3 nodes having 5 shards and 1 replica with multiple indices having total around 100 million data ,sizing 1tb and still increasing


  1. Node 1:Master=true,data=true (30gb ram,8 cores)
  2. Node 2:Master=true,data=true (30gb ram,8 cores)
  3. Node 3:Master=false,data=false (30gb ram,8 cores)

All nodes are pointing to same UNC path(\\pc1\\data) data location.Data is not stored locally on nodes,all data is stored on one location as mentioned above.we are using Nas to store data and seems Nas is very fast so we are storing data on UNC path and not locally.

While googling found if shards are decreased ,it will increase search performance.According to the specified configuration should i drop no of shards to 1?.Will it increase the performance or does the configuration need any changes?.
Please suggest me the configuration.
Thanks

--
Please update your bookmarks! We have moved to <a href="https://discuss.elastic.co/" target="_blank" rel="nofollow" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fdiscuss.elastic.co%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFguF42tUvBArAl-xFnDHHIi5v0cA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fdiscuss.elastic.co%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFguF42tUvBArAl-xFnDHHIi5v0cA';return true;">https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="QLDQNvxkgUkJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ecbaf990-27d2-46f0-8ae5-2aba7a012b85%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elastic Search configuration

Mark Walkom-2
We do not recommend the use of networked file systems.
We definitely don't recommend using those and putting both sets of data in the one location.

Reducing shard count may help, it'd certainly reduce the overhead in maintaining all those shards. Just keep shard sizes under 50GB if you can.
Also having two masters can lead to potential split brain, you'd be better off making node 3 a master: true, data: false and setting minimum_masters.

On 12 May 2015 at 20:22, Hardik Dobariya <[hidden email]> wrote:
No am not facing any performance issue.Just want to confirm if my configuration is feasible or not and what will happen if i bring down shard to 1.
My ES version is 1.5.2 and java release is 8.45

On Tuesday, May 12, 2015 at 3:38:40 PM UTC+5:30, Mark Walkom wrote:
Are you having performance issues now?

If so I'd start with replacing your disk setup, chances are it's not as fast as you think, especially if you are storing the data from both nodes on the same path.
Also what ES version are you on, what Java release and version?

On 12 May 2015 at 19:01, Hardik Dobariya <[hidden email]> wrote:

Please let me know if below elastic server configuration is feasible or not.

Total 1 cluster with 3 nodes having 5 shards and 1 replica with multiple indices having total around 100 million data ,sizing 1tb and still increasing


  1. Node 1:Master=true,data=true (30gb ram,8 cores)
  2. Node 2:Master=true,data=true (30gb ram,8 cores)
  3. Node 3:Master=false,data=false (30gb ram,8 cores)

All nodes are pointing to same UNC path(\\pc1\\data) data location.Data is not stored locally on nodes,all data is stored on one location as mentioned above.we are using Nas to store data and seems Nas is very fast so we are storing data on UNC path and not locally.

While googling found if shards are decreased ,it will increase search performance.According to the specified configuration should i drop no of shards to 1?.Will it increase the performance or does the configuration need any changes?.
Please suggest me the configuration.
Thanks

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ecbaf990-27d2-46f0-8ae5-2aba7a012b85%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9qjR17AOO4hinjxbvAKYO5L5tttPncz_fPL-ZQ5mGCKQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elastic Search configuration

Hardik Dobariya
yes i can understand the use of network file system is not good.
Actually we do not have any physical machines.we are using virtual environment.This is the reason we are using NAS to store data on network.
Any suggestion on using virtual environment for elastic search?

On Tuesday, May 12, 2015 at 4:39:15 PM UTC+5:30, Mark Walkom wrote:
We do not recommend the use of networked file systems.
We definitely don't recommend using those and putting both sets of data in the one location.

Reducing shard count may help, it'd certainly reduce the overhead in maintaining all those shards. Just keep shard sizes under 50GB if you can.
Also having two masters can lead to potential split brain, you'd be better off making node 3 a master: true, data: false and setting minimum_masters.

On 12 May 2015 at 20:22, Hardik Dobariya <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="CC5SzL3Hx48J" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">hardikdob...@...> wrote:
No am not facing any performance issue.Just want to confirm if my configuration is feasible or not and what will happen if i bring down shard to 1.
My ES version is 1.5.2 and java release is 8.45

On Tuesday, May 12, 2015 at 3:38:40 PM UTC+5:30, Mark Walkom wrote:
Are you having performance issues now?

If so I'd start with replacing your disk setup, chances are it's not as fast as you think, especially if you are storing the data from both nodes on the same path.
Also what ES version are you on, what Java release and version?

On 12 May 2015 at 19:01, Hardik Dobariya <[hidden email]> wrote:

Please let me know if below elastic server configuration is feasible or not.

Total 1 cluster with 3 nodes having 5 shards and 1 replica with multiple indices having total around 100 million data ,sizing 1tb and still increasing


  1. Node 1:Master=true,data=true (30gb ram,8 cores)
  2. Node 2:Master=true,data=true (30gb ram,8 cores)
  3. Node 3:Master=false,data=false (30gb ram,8 cores)

All nodes are pointing to same UNC path(\\pc1\\data) data location.Data is not stored locally on nodes,all data is stored on one location as mentioned above.we are using Nas to store data and seems Nas is very fast so we are storing data on UNC path and not locally.

While googling found if shards are decreased ,it will increase search performance.According to the specified configuration should i drop no of shards to 1?.Will it increase the performance or does the configuration need any changes?.
Please suggest me the configuration.
Thanks

--
Please update your bookmarks! We have moved to <a href="https://discuss.elastic.co/" rel="nofollow" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fdiscuss.elastic.co%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFguF42tUvBArAl-xFnDHHIi5v0cA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fdiscuss.elastic.co%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFguF42tUvBArAl-xFnDHHIi5v0cA';return true;">https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to <a href="https://discuss.elastic.co/" target="_blank" rel="nofollow" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fdiscuss.elastic.co%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFguF42tUvBArAl-xFnDHHIi5v0cA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fdiscuss.elastic.co%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFguF42tUvBArAl-xFnDHHIi5v0cA';return true;">https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="CC5SzL3Hx48J" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/ecbaf990-27d2-46f0-8ae5-2aba7a012b85%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/ecbaf990-27d2-46f0-8ae5-2aba7a012b85%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/ecbaf990-27d2-46f0-8ae5-2aba7a012b85%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/ecbaf990-27d2-46f0-8ae5-2aba7a012b85%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/66346979-7579-41a0-9e9c-d3650bf81227%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elastic Search configuration

Hardik Dobariya
yes i can understand the use of network file system is not good.
Actually we do not have any physical machines.we are using virtual environment.This is the reason we are using NAS to store data on network.
Any suggestion on using virtual environment for elastic search?
and yes am already using 3 nodes.2 masters with data and one child 

On Tuesday, May 12, 2015 at 4:52:25 PM UTC+5:30, Hardik Dobariya wrote:
yes i can understand the use of network file system is not good.
Actually we do not have any physical machines.we are using virtual environment.This is the reason we are using NAS to store data on network.
Any suggestion on using virtual environment for elastic search?

On Tuesday, May 12, 2015 at 4:39:15 PM UTC+5:30, Mark Walkom wrote:
We do not recommend the use of networked file systems.
We definitely don't recommend using those and putting both sets of data in the one location.

Reducing shard count may help, it'd certainly reduce the overhead in maintaining all those shards. Just keep shard sizes under 50GB if you can.
Also having two masters can lead to potential split brain, you'd be better off making node 3 a master: true, data: false and setting minimum_masters.

On 12 May 2015 at 20:22, Hardik Dobariya <[hidden email]> wrote:
No am not facing any performance issue.Just want to confirm if my configuration is feasible or not and what will happen if i bring down shard to 1.
My ES version is 1.5.2 and java release is 8.45

On Tuesday, May 12, 2015 at 3:38:40 PM UTC+5:30, Mark Walkom wrote:
Are you having performance issues now?

If so I'd start with replacing your disk setup, chances are it's not as fast as you think, especially if you are storing the data from both nodes on the same path.
Also what ES version are you on, what Java release and version?

On 12 May 2015 at 19:01, Hardik Dobariya <[hidden email]> wrote:

Please let me know if below elastic server configuration is feasible or not.

Total 1 cluster with 3 nodes having 5 shards and 1 replica with multiple indices having total around 100 million data ,sizing 1tb and still increasing


  1. Node 1:Master=true,data=true (30gb ram,8 cores)
  2. Node 2:Master=true,data=true (30gb ram,8 cores)
  3. Node 3:Master=false,data=false (30gb ram,8 cores)

All nodes are pointing to same UNC path(\\pc1\\data) data location.Data is not stored locally on nodes,all data is stored on one location as mentioned above.we are using Nas to store data and seems Nas is very fast so we are storing data on UNC path and not locally.

While googling found if shards are decreased ,it will increase search performance.According to the specified configuration should i drop no of shards to 1?.Will it increase the performance or does the configuration need any changes?.
Please suggest me the configuration.
Thanks

--
Please update your bookmarks! We have moved to <a href="https://discuss.elastic.co/" rel="nofollow" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fdiscuss.elastic.co%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFguF42tUvBArAl-xFnDHHIi5v0cA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fdiscuss.elastic.co%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFguF42tUvBArAl-xFnDHHIi5v0cA';return true;">https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to <a href="https://discuss.elastic.co/" rel="nofollow" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fdiscuss.elastic.co%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFguF42tUvBArAl-xFnDHHIi5v0cA';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fdiscuss.elastic.co%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNFguF42tUvBArAl-xFnDHHIi5v0cA';return true;">https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/ecbaf990-27d2-46f0-8ae5-2aba7a012b85%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/ecbaf990-27d2-46f0-8ae5-2aba7a012b85%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/ecbaf990-27d2-46f0-8ae5-2aba7a012b85%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/ecbaf990-27d2-46f0-8ae5-2aba7a012b85%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a820c9da-0cb1-485c-a0b2-fc1001cab710%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elastic Search configuration

Magnus Bäck
On Tuesday, May 12, 2015 at 13:24 CEST,
     Hardik Dobariya <[hidden email]> wrote:

> yes i can understand the use of network file system is not good.
> Actually we do not have any physical machines.we are using virtual
> environment.This is the reason we are using NAS to store data on
> network.

Sure, but you don't have to use NAS just because you use VMs.
Local disks or volumes mounted from a SAN are still preferred
to accessing the file system over SMB or NFS.

> Any suggestion on using virtual environment for elastic search?
> and yes am already using 3 nodes.2 masters with data and one child

Yes, and as Mark says that's a bad idea. You're better off allowing
all three nodes to be masters. Unless you have a serious query load
you should keep data on all three nodes. Having a third of the
cluster's capacity dedicated to processing queries is most likely
very wasteful.

--
Magnus Bäck                | Software Engineer, Development Tools
[hidden email] | Sony Mobile Communications

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150512121645.GA806%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elastic Search configuration

Hardik Dobariya
oh yes i forgot to mentioned that we are using SAN mounted volumes.
And according to your reply i can say my configuration will work if am using SAN mounted volumes?
one last question if my configuration is correct then do i have to drop down my shard size from 5 to 1 or to something less to 3 which might increase performance?
These are the three nodes configured

  1. Node 1:Master=true,data=true (30gb ram,8 cores)
  2. Node 2:Master=true,data=true (30gb ram,8 cores)
  3. Node 3:Master=false,data=false (30gb ram,8 cores)

On Tuesday, May 12, 2015 at 5:46:54 PM UTC+5:30, Magnus Bäck wrote:
On Tuesday, May 12, 2015 at 13:24 CEST,
     Hardik Dobariya <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="kGuHPcOPkrsJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">hardikdob...@...> wrote:

> yes i can understand the use of network file system is not good.
> Actually we do not have any physical machines.we are using virtual
> environment.This is the reason we are using NAS to store data on
> network.

Sure, but you don't have to use NAS just because you use VMs.
Local disks or volumes mounted from a SAN are still preferred
to accessing the file system over SMB or NFS.

> Any suggestion on using virtual environment for elastic search?
> and yes am already using 3 nodes.2 masters with data and one child

Yes, and as Mark says that's a bad idea. You're better off allowing
all three nodes to be masters. Unless you have a serious query load
you should keep data on all three nodes. Having a third of the
cluster's capacity dedicated to processing queries is most likely
very wasteful.

--
Magnus Bäck                | Software Engineer, Development Tools
<a href="javascript:" target="_blank" gdf-obfuscated-mailto="kGuHPcOPkrsJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">magnu...@... | Sony Mobile Communications

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8bb923dd-67e2-457d-982b-f38c8d372c0e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elastic Search configuration

Magnus Bäck
On Tuesday, May 12, 2015 at 14:24 CEST,
     Hardik Dobariya <[hidden email]> wrote:

> oh yes i forgot to mentioned that we are using SAN mounted volumes.

Um, okay. But you said you accessed them via a UNC path (\\pc1\data).
Then it's not a locally mounted volume.

> And according to your reply i can say my configuration will work if am
> using SAN mounted volumes?

Yes. SAN and NAS both work but SAN is preferred.

> one last question if my configuration is correct then do i have to
> drop down my shard size from 5 to 1 or to something less to 3 which
> might increase performance?

There are rarely any "correct" configurations, and what's optimal
depends on many factors. While there are rules of thumb you'd have
to measure for yourself what's best for you.

The optimal number of shards depends on how many indexes you have,
how many nodes you have (that we know; three) how big the indexes
are etc.

Shards have a certain overhead so you don't want to have too many
(you'll waste resources to shard book keeping) but you don't want
them to be too big either and you'll want to spread them out between
your nodes.

> These are the three nodes configured
>  1. Node 1:Master=true,data=true (30gb ram,8 cores)
>  2. Node 2:Master=true,data=true (30gb ram,8 cores)
>  3. Node 3:Master=false,data=false (30gb ram,8 cores)

Yes, and again, you should turn node 3 into master=true, data=true
as well.

--
Magnus Bäck                | Software Engineer, Development Tools
[hidden email] | Sony Mobile Communications

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150512201910.GB2679%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elastic Search configuration

Hardik Dobariya

we have around 500 indices all sizing to approx 50gb respectively and total size goes around 1tb for all indices and will keep on increasing .
The reason behind setting node3 as Master=false,data=false because in configuration file i read this type of node will only work for searching ,aggregation etc. Do i still need to set node3 three as mater and data

On Wednesday, May 13, 2015 at 1:49:18 AM UTC+5:30, Magnus Bäck wrote:
On Tuesday, May 12, 2015 at 14:24 CEST,
     Hardik Dobariya <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="pBlbkLtT-VYJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">hardikdob...@...> wrote:

> oh yes i forgot to mentioned that we are using SAN mounted volumes.

Um, okay. But you said you accessed them via a UNC path (\\pc1\data).
Then it's not a locally mounted volume.

> And according to your reply i can say my configuration will work if am
> using SAN mounted volumes?

Yes. SAN and NAS both work but SAN is preferred.

> one last question if my configuration is correct then do i have to
> drop down my shard size from 5 to 1 or to something less to 3 which
> might increase performance?

There are rarely any "correct" configurations, and what's optimal
depends on many factors. While there are rules of thumb you'd have
to measure for yourself what's best for you.

The optimal number of shards depends on how many indexes you have,
how many nodes you have (that we know; three) how big the indexes
are etc.

Shards have a certain overhead so you don't want to have too many
(you'll waste resources to shard book keeping) but you don't want
them to be too big either and you'll want to spread them out between
your nodes.

> These are the three nodes configured
>  1. Node 1:Master=true,data=true (30gb ram,8 cores)
>  2. Node 2:Master=true,data=true (30gb ram,8 cores)
>  3. Node 3:Master=false,data=false (30gb ram,8 cores)

Yes, and again, you should turn node 3 into master=true, data=true
as well.

--
Magnus Bäck                | Software Engineer, Development Tools
<a href="javascript:" target="_blank" gdf-obfuscated-mailto="pBlbkLtT-VYJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">magnu...@... | Sony Mobile Communications

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bcc44119-1521-4ef5-8024-73eee4947537%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elastic Search configuration

Magnus Bäck
On Wednesday, May 13, 2015 at 08:43 CEST,
     Hardik Dobariya <[hidden email]> wrote:

> we have around 500 indices all sizing to approx 50gb respectively and
> total size goes around 1tb for all indices and will keep on increasing.
> The reason behind setting node3 as Master=false,data=false because in
> configuration file i read this type of node will only work for
> searching ,aggregation etc. Do i still need to set node3 three as mater
> and data

You'll definitely want all three nodes to be master-eligible (as Mark
explained this prevents a split brain situation). Whether it makes sense
to dedicate one of the nodes to queries depends a bit on the types of
queries you make but I'd say it's unlikely that dedicating a third of
your cluster's capacity for queries is the best use of your money.

--
Magnus Bäck                | Software Engineer, Development Tools
[hidden email] | Sony Mobile Communications

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150513071628.GA13466%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Elastic Search configuration

Hardik Dobariya
Thank you Mark and Magnus for the help provided :) 

On Wednesday, May 13, 2015 at 12:46:46 PM UTC+5:30, Magnus Bäck wrote:
On Wednesday, May 13, 2015 at 08:43 CEST,
     Hardik Dobariya <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="BxImWJGG9kgJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">hardikdob...@...> wrote:

> we have around 500 indices all sizing to approx 50gb respectively and
> total size goes around 1tb for all indices and will keep on increasing.
> The reason behind setting node3 as Master=false,data=false because in
> configuration file i read this type of node will only work for
> searching ,aggregation etc. Do i still need to set node3 three as mater
> and data

You'll definitely want all three nodes to be master-eligible (as Mark
explained this prevents a split brain situation). Whether it makes sense
to dedicate one of the nodes to queries depends a bit on the types of
queries you make but I'd say it's unlikely that dedicating a third of
your cluster's capacity for queries is the best use of your money.

--
Magnus Bäck                | Software Engineer, Development Tools
<a href="javascript:" target="_blank" gdf-obfuscated-mailto="BxImWJGG9kgJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">magnu...@... | Sony Mobile Communications

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/
---
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/17866b5d-f6df-4333-8fc8-c450f8b48954%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.