About replica shard use many field data

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

About replica shard use many field data

Junheng Gong
we have about 2.2B data in our elasticsearch, and we using facet and function score query on those data. 

it will load many data to fielddata, and replica double it.

so my question is "how to set replica to make sure it won't use field data expect primary shard down"?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/83db33a6-0248-42eb-b913-8da49a45006f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: About replica shard use many field data

Mark Walkom-2
Replica's don't double the amount of field data loaded. A query will only load what it needs from the shards it needs, but it won't load primary and replica, just one or the other.

Ideally you should 1) upgrade to aggs, facets are deprecated and not as performant, and 2) upgrade to doc_values.

On 4 May 2015 at 18:50, Junheng Gong <[hidden email]> wrote:
we have about 2.2B data in our elasticsearch, and we using facet and function score query on those data. 

it will load many data to fielddata, and replica double it.

so my question is "how to set replica to make sure it won't use field data expect primary shard down"?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/83db33a6-0248-42eb-b913-8da49a45006f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9ag56CtN%3D%3DnkMdNqfUfeOxgNEDTnKQ1LXtDZw%3DRuhzhw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: About replica shard use many field data

Junheng Gong
In reply to this post by Junheng Gong
Thanks for you replay Mark,

"active": {
         "primaries": {
            "fielddata": {
               "memory_size_in_bytes": 77076457764,
               "evictions": 0,
               "fields": {
                  "commentsCount": {
                     "memory_size_in_bytes": 2416090508
                  },
                  "_uid": {
                     "memory_size_in_bytes": 46517210860
                  },
                  "sourceTimeUnit": {
                     "memory_size_in_bytes": 4160954108
                  },
                  "likesCount": {
                     "memory_size_in_bytes": 4225460468
                  },
                  "imageCount": {
                     "memory_size_in_bytes": 435807840
                  },
                  "geo": {
                     "memory_size_in_bytes": 19320933980
                  }
               }
            }
         },
         "total": {
            "fielddata": {
               "memory_size_in_bytes": 90893341516,
               "evictions": 0,
               "fields": {
                  "commentsCount": {
                     "memory_size_in_bytes": 2661775356
                  },
                  "_uid": {
                     "memory_size_in_bytes": 55220225456
                  },
                  "sourceTimeUnit": {
                     "memory_size_in_bytes": 4846336504
                  },
                  "likesCount": {
                     "memory_size_in_bytes": 4712145912
                  },
                  "imageCount": {
                     "memory_size_in_bytes": 517060176
                  },
                  "geo": {
                     "memory_size_in_bytes": 22935798112
                  }
               }
            }
         }
      },

we got field data usage by using GET /_stats/fielddata?fields=*

in this result total is more than primaries, so i thought maybe some replica load data to field data too. (cause some query will send to replica shard)

在 2015年5月4日星期一 UTC+8下午4:50:38,Junheng Gong写道:
we have about 2.2B data in our elasticsearch, and we using facet and function score query on those data. 

it will load many data to fielddata, and replica double it.

so my question is "how to set replica to make sure it won't use field data expect primary shard down"?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8928f963-47c9-478f-acbb-5b594cd4a9ff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.