don't store certain fields by default

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

don't store certain fields by default

Allan Johns
I can't quite figure out from the docs how to do this. I know I need to use the exclude functionality shown here http://www.elasticsearch.org/guide/reference/mapping/source-field/, but the documentation frustratingly provides the code snippets, but doesn't say where to put them.

I want to not store fields beginning with an underscore. I tried adding this to my config/elasticsearch.yml:

mapping:
    MYTYPE:
        _source:
            excludes: [ "_*" ]

But this didn't work.
What do I do?

thx
A

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: don't store certain fields by default

Allan Johns
I also tried this in my default-mapping.json, no luck (see template_2):

{
    "_default_" : {
        "dynamic_templates" : [
            {
                "template_1" : {
                    "match" : "folder",
                    "match_mapping_type" : "string",
                    "mapping" : {
                        "type" : "string",
                        "index" : "analyzed",
                        "analyzer": "gas_folder_analyzer"
                    }
                }
            },
            {
                "template_2" : {
                    "match" : "_*",
                    "mapping" : {
                        "store": "no"
                    }
                }
            },
            {
                "template_3" : {
                    "match" : "*",
                    "match_mapping_type" : "string",
                    "mapping" : {
                        "type" : "string",
                        "index" : "not_analyzed"
                    }
                }
            }
        ]
    }
}



On Wed, Jun 12, 2013 at 7:01 PM, Allan Johns <[hidden email]> wrote:
I can't quite figure out from the docs how to do this. I know I need to use the exclude functionality shown here http://www.elasticsearch.org/guide/reference/mapping/source-field/, but the documentation frustratingly provides the code snippets, but doesn't say where to put them.

I want to not store fields beginning with an underscore. I tried adding this to my config/elasticsearch.yml:

mapping:
    MYTYPE:
        _source:
            excludes: [ "_*" ]

But this didn't work.
What do I do?

thx
A


--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: don't store certain fields by default

Allan Johns
Anyone?

I tried:
"store" : false
also, this didn't work either.


You can actually do this, right?

A



On Wed, Jun 12, 2013 at 7:07 PM, Allan Johns <[hidden email]> wrote:
I also tried this in my default-mapping.json, no luck (see template_2):

{
    "_default_" : {
        "dynamic_templates" : [
            {
                "template_1" : {
                    "match" : "folder",
                    "match_mapping_type" : "string",
                    "mapping" : {
                        "type" : "string",
                        "index" : "analyzed",
                        "analyzer": "gas_folder_analyzer"
                    }
                }
            },
            {
                "template_2" : {
                    "match" : "_*",
                    "mapping" : {
                        "store": "no"
                    }
                }
            },
            {
                "template_3" : {
                    "match" : "*",
                    "match_mapping_type" : "string",
                    "mapping" : {
                        "type" : "string",
                        "index" : "not_analyzed"
                    }
                }
            }
        ]
    }
}



On Wed, Jun 12, 2013 at 7:01 PM, Allan Johns <[hidden email]> wrote:
I can't quite figure out from the docs how to do this. I know I need to use the exclude functionality shown here http://www.elasticsearch.org/guide/reference/mapping/source-field/, but the documentation frustratingly provides the code snippets, but doesn't say where to put them.

I want to not store fields beginning with an underscore. I tried adding this to my config/elasticsearch.yml:

mapping:
    MYTYPE:
        _source:
            excludes: [ "_*" ]

But this didn't work.
What do I do?

thx
A



--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: don't store certain fields by default

Allan Johns
On further reading, the "store" property doesn't quite do what I want.

I have a field that I want to be able to search on, but I don't want it in my _source. Is this possible?
A



On Thu, Jun 13, 2013 at 11:35 AM, Allan Johns <[hidden email]> wrote:
Anyone?

I tried:
"store" : false
also, this didn't work either.


You can actually do this, right?

A



On Wed, Jun 12, 2013 at 7:07 PM, Allan Johns <[hidden email]> wrote:
I also tried this in my default-mapping.json, no luck (see template_2):

{
    "_default_" : {
        "dynamic_templates" : [
            {
                "template_1" : {
                    "match" : "folder",
                    "match_mapping_type" : "string",
                    "mapping" : {
                        "type" : "string",
                        "index" : "analyzed",
                        "analyzer": "gas_folder_analyzer"
                    }
                }
            },
            {
                "template_2" : {
                    "match" : "_*",
                    "mapping" : {
                        "store": "no"
                    }
                }
            },
            {
                "template_3" : {
                    "match" : "*",
                    "match_mapping_type" : "string",
                    "mapping" : {
                        "type" : "string",
                        "index" : "not_analyzed"
                    }
                }
            }
        ]
    }
}



On Wed, Jun 12, 2013 at 7:01 PM, Allan Johns <[hidden email]> wrote:
I can't quite figure out from the docs how to do this. I know I need to use the exclude functionality shown here http://www.elasticsearch.org/guide/reference/mapping/source-field/, but the documentation frustratingly provides the code snippets, but doesn't say where to put them.

I want to not store fields beginning with an underscore. I tried adding this to my config/elasticsearch.yml:

mapping:
    MYTYPE:
        _source:
            excludes: [ "_*" ]

But this didn't work.
What do I do?

thx
A




--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: don't store certain fields by default

dadoonet
In reply to this post by Allan Johns
You should know that Elasticsearch stores your source doc as is and don't modify it.
That probably explains that when you get your document back, you are still seeing source with all fields.

But, can you GIST a complete curl recreation that shows what you are doing?

Note that you could probably create default mapping when creating your index:
curl -XPOST localhost:9200/test -d '{
    "settings" : {
        "number_of_shards" : 1
    },
    "mappings" : {
        "type1" : {
            "_source" : { "enabled" : false },
            "properties" : {
                "field1" : { "type" : "string", "index" : "not_analyzed" }
            }
        }
    }
}'



--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 13 juin 2013 à 03:35, Allan Johns <[hidden email]> a écrit :

Anyone?

I tried:
"store" : false
also, this didn't work either.


You can actually do this, right?

A



On Wed, Jun 12, 2013 at 7:07 PM, Allan Johns <[hidden email]> wrote:
I also tried this in my default-mapping.json, no luck (see template_2):

{
    "_default_" : {
        "dynamic_templates" : [
            {
                "template_1" : {
                    "match" : "folder",
                    "match_mapping_type" : "string",
                    "mapping" : {
                        "type" : "string",
                        "index" : "analyzed",
                        "analyzer": "gas_folder_analyzer"
                    }
                }
            },
            {
                "template_2" : {
                    "match" : "_*",
                    "mapping" : {
                        "store": "no"
                    }
                }
            },
            {
                "template_3" : {
                    "match" : "*",
                    "match_mapping_type" : "string",
                    "mapping" : {
                        "type" : "string",
                        "index" : "not_analyzed"
                    }
                }
            }
        ]
    }
}



On Wed, Jun 12, 2013 at 7:01 PM, Allan Johns <[hidden email]> wrote:
I can't quite figure out from the docs how to do this. I know I need to use the exclude functionality shown here http://www.elasticsearch.org/guide/reference/mapping/source-field/, but the documentation frustratingly provides the code snippets, but doesn't say where to put them.

I want to not store fields beginning with an underscore. I tried adding this to my config/elasticsearch.yml:

mapping:
    MYTYPE:
        _source:
            excludes: [ "_*" ]

But this didn't work.
What do I do?

thx
A



--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: don't store certain fields by default

dadoonet
In reply to this post by Allan Johns
{
    "tweet" : {
        "_source" : {"enabled" : false}
    }
}
Why do you feel you need to disable it?

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 13 juin 2013 à 03:56, Allan Johns <[hidden email]> a écrit :

On further reading, the "store" property doesn't quite do what I want.

I have a field that I want to be able to search on, but I don't want it in my _source. Is this possible?
A



On Thu, Jun 13, 2013 at 11:35 AM, Allan Johns <[hidden email]> wrote:
Anyone?

I tried:
"store" : false
also, this didn't work either.


You can actually do this, right?

A



On Wed, Jun 12, 2013 at 7:07 PM, Allan Johns <[hidden email]> wrote:
I also tried this in my default-mapping.json, no luck (see template_2):

{
    "_default_" : {
        "dynamic_templates" : [
            {
                "template_1" : {
                    "match" : "folder",
                    "match_mapping_type" : "string",
                    "mapping" : {
                        "type" : "string",
                        "index" : "analyzed",
                        "analyzer": "gas_folder_analyzer"
                    }
                }
            },
            {
                "template_2" : {
                    "match" : "_*",
                    "mapping" : {
                        "store": "no"
                    }
                }
            },
            {
                "template_3" : {
                    "match" : "*",
                    "match_mapping_type" : "string",
                    "mapping" : {
                        "type" : "string",
                        "index" : "not_analyzed"
                    }
                }
            }
        ]
    }
}



On Wed, Jun 12, 2013 at 7:01 PM, Allan Johns <[hidden email]> wrote:
I can't quite figure out from the docs how to do this. I know I need to use the exclude functionality shown here http://www.elasticsearch.org/guide/reference/mapping/source-field/, but the documentation frustratingly provides the code snippets, but doesn't say where to put them.

I want to not store fields beginning with an underscore. I tried adding this to my config/elasticsearch.yml:

mapping:
    MYTYPE:
        _source:
            excludes: [ "_*" ]

But this didn't work.
What do I do?

thx
A




--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: don't store certain fields by default

Allan Johns
David, I cannot.

I need to disable only certain fields from _source, not all of them.

I need to be able to search for documents via an indexed field, but not actually store that field in _source. I'm a bit surprised this isn't possible to be honest.

I feel I need to disable, ie remove, a particular field from _source because I simply never use it in that context and it is just taking up extra storage space. I only want to search based on the field - the actual value of the field is not useful to me.

Thanks,
A






On Thu, Jun 13, 2013 at 3:16 PM, David Pilato <[hidden email]> wrote:
{
    "tweet" : {
        "_source" : {"enabled" : false}
    }
}
Why do you feel you need to disable it?

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 13 juin 2013 à 03:56, Allan Johns <[hidden email]> a écrit :

On further reading, the "store" property doesn't quite do what I want.

I have a field that I want to be able to search on, but I don't want it in my _source. Is this possible?
A



On Thu, Jun 13, 2013 at 11:35 AM, Allan Johns <[hidden email]> wrote:
Anyone?

I tried:
"store" : false
also, this didn't work either.


You can actually do this, right?

A



On Wed, Jun 12, 2013 at 7:07 PM, Allan Johns <[hidden email]> wrote:
I also tried this in my default-mapping.json, no luck (see template_2):

{
    "_default_" : {
        "dynamic_templates" : [
            {
                "template_1" : {
                    "match" : "folder",
                    "match_mapping_type" : "string",
                    "mapping" : {
                        "type" : "string",
                        "index" : "analyzed",
                        "analyzer": "gas_folder_analyzer"
                    }
                }
            },
            {
                "template_2" : {
                    "match" : "_*",
                    "mapping" : {
                        "store": "no"
                    }
                }
            },
            {
                "template_3" : {
                    "match" : "*",
                    "match_mapping_type" : "string",
                    "mapping" : {
                        "type" : "string",
                        "index" : "not_analyzed"
                    }
                }
            }
        ]
    }
}



On Wed, Jun 12, 2013 at 7:01 PM, Allan Johns <[hidden email]> wrote:
I can't quite figure out from the docs how to do this. I know I need to use the exclude functionality shown here http://www.elasticsearch.org/guide/reference/mapping/source-field/, but the documentation frustratingly provides the code snippets, but doesn't say where to put them.

I want to not store fields beginning with an underscore. I tried adding this to my config/elasticsearch.yml:

mapping:
    MYTYPE:
        _source:
            excludes: [ "_*" ]

But this didn't work.
What do I do?

thx
A




--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: don't store certain fields by default

dadoonet
For that specific need, you have to disable _source field and field by field define if you want to store it or not.
Then, when searching, ask for fields you want back.

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 13 juin 2013 à 07:44, Allan Johns <[hidden email]> a écrit :

David, I cannot.

I need to disable only certain fields from _source, not all of them.

I need to be able to search for documents via an indexed field, but not actually store that field in _source. I'm a bit surprised this isn't possible to be honest.

I feel I need to disable, ie remove, a particular field from _source because I simply never use it in that context and it is just taking up extra storage space. I only want to search based on the field - the actual value of the field is not useful to me.

Thanks,
A






On Thu, Jun 13, 2013 at 3:16 PM, David Pilato <[hidden email]> wrote:
{
    "tweet" : {
        "_source" : {"enabled" : false}
    }
}
Why do you feel you need to disable it?

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 13 juin 2013 à 03:56, Allan Johns <[hidden email]> a écrit :

On further reading, the "store" property doesn't quite do what I want.

I have a field that I want to be able to search on, but I don't want it in my _source. Is this possible?
A



On Thu, Jun 13, 2013 at 11:35 AM, Allan Johns <[hidden email]> wrote:
Anyone?

I tried:
"store" : false
also, this didn't work either.


You can actually do this, right?

A



On Wed, Jun 12, 2013 at 7:07 PM, Allan Johns <[hidden email]> wrote:
I also tried this in my default-mapping.json, no luck (see template_2):

{
    "_default_" : {
        "dynamic_templates" : [
            {
                "template_1" : {
                    "match" : "folder",
                    "match_mapping_type" : "string",
                    "mapping" : {
                        "type" : "string",
                        "index" : "analyzed",
                        "analyzer": "gas_folder_analyzer"
                    }
                }
            },
            {
                "template_2" : {
                    "match" : "_*",
                    "mapping" : {
                        "store": "no"
                    }
                }
            },
            {
                "template_3" : {
                    "match" : "*",
                    "match_mapping_type" : "string",
                    "mapping" : {
                        "type" : "string",
                        "index" : "not_analyzed"
                    }
                }
            }
        ]
    }
}



On Wed, Jun 12, 2013 at 7:01 PM, Allan Johns <[hidden email]> wrote:
I can't quite figure out from the docs how to do this. I know I need to use the exclude functionality shown here http://www.elasticsearch.org/guide/reference/mapping/source-field/, but the documentation frustratingly provides the code snippets, but doesn't say where to put them.

I want to not store fields beginning with an underscore. I tried adding this to my config/elasticsearch.yml:

mapping:
    MYTYPE:
        _source:
            excludes: [ "_*" ]

But this didn't work.
What do I do?

thx
A




--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: don't store certain fields by default

dadoonet
But if it's only to save disk space, I'm wondering if you really want to do it (complexity wise).
Are your fields very huge?

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 13 juin 2013 à 07:52, David Pilato <[hidden email]> a écrit :

For that specific need, you have to disable _source field and field by field define if you want to store it or not.
Then, when searching, ask for fields you want back.

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 13 juin 2013 à 07:44, Allan Johns <[hidden email]> a écrit :

David, I cannot.

I need to disable only certain fields from _source, not all of them.

I need to be able to search for documents via an indexed field, but not actually store that field in _source. I'm a bit surprised this isn't possible to be honest.

I feel I need to disable, ie remove, a particular field from _source because I simply never use it in that context and it is just taking up extra storage space. I only want to search based on the field - the actual value of the field is not useful to me.

Thanks,
A






On Thu, Jun 13, 2013 at 3:16 PM, David Pilato <[hidden email]> wrote:
{
    "tweet" : {
        "_source" : {"enabled" : false}
    }
}
Why do you feel you need to disable it?

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 13 juin 2013 à 03:56, Allan Johns <[hidden email]> a écrit :

On further reading, the "store" property doesn't quite do what I want.

I have a field that I want to be able to search on, but I don't want it in my _source. Is this possible?
A



On Thu, Jun 13, 2013 at 11:35 AM, Allan Johns <[hidden email]> wrote:
Anyone?

I tried:
"store" : false
also, this didn't work either.


You can actually do this, right?

A



On Wed, Jun 12, 2013 at 7:07 PM, Allan Johns <[hidden email]> wrote:
I also tried this in my default-mapping.json, no luck (see template_2):

{
    "_default_" : {
        "dynamic_templates" : [
            {
                "template_1" : {
                    "match" : "folder",
                    "match_mapping_type" : "string",
                    "mapping" : {
                        "type" : "string",
                        "index" : "analyzed",
                        "analyzer": "gas_folder_analyzer"
                    }
                }
            },
            {
                "template_2" : {
                    "match" : "_*",
                    "mapping" : {
                        "store": "no"
                    }
                }
            },
            {
                "template_3" : {
                    "match" : "*",
                    "match_mapping_type" : "string",
                    "mapping" : {
                        "type" : "string",
                        "index" : "not_analyzed"
                    }
                }
            }
        ]
    }
}



On Wed, Jun 12, 2013 at 7:01 PM, Allan Johns <[hidden email]> wrote:
I can't quite figure out from the docs how to do this. I know I need to use the exclude functionality shown here http://www.elasticsearch.org/guide/reference/mapping/source-field/, but the documentation frustratingly provides the code snippets, but doesn't say where to put them.

I want to not store fields beginning with an underscore. I tried adding this to my config/elasticsearch.yml:

mapping:
    MYTYPE:
        _source:
            excludes: [ "_*" ]

But this didn't work.
What do I do?

thx
A




--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: don't store certain fields by default

Allan Johns
No they aren't huge, but the number of documents in the database will become huge (100s of millions) so any disk space savings I can get become significant.

I don't think that disabling _source and storing all fields into the indexes is the right approach, I have read specifically that this adversely affects performance, especially when you have a lot of fields - each field needs to be piecemeal extracted from the index, and returning _source all in one go is faster.

It seems that this should be something fairly trivial to implement... all I want to be able to do is discard certain fields before the document _source is written to the database, but after the indexes have been generated.

I'd like to see some new mapping field that can be set, like so:

{
                "type" : "string",
                "index" : "not_analyzed",
                "store" : true,
                "in_source" : false
}

thx
A




On Thu, Jun 13, 2013 at 3:54 PM, David Pilato <[hidden email]> wrote:
But if it's only to save disk space, I'm wondering if you really want to do it (complexity wise).
Are your fields very huge?

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 13 juin 2013 à 07:52, David Pilato <[hidden email]> a écrit :

For that specific need, you have to disable _source field and field by field define if you want to store it or not.
Then, when searching, ask for fields you want back.

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 13 juin 2013 à 07:44, Allan Johns <[hidden email]> a écrit :

David, I cannot.

I need to disable only certain fields from _source, not all of them.

I need to be able to search for documents via an indexed field, but not actually store that field in _source. I'm a bit surprised this isn't possible to be honest.

I feel I need to disable, ie remove, a particular field from _source because I simply never use it in that context and it is just taking up extra storage space. I only want to search based on the field - the actual value of the field is not useful to me.

Thanks,
A






On Thu, Jun 13, 2013 at 3:16 PM, David Pilato <[hidden email]> wrote:
{
    "tweet" : {
        "_source" : {"enabled" : false}
    }
}
Why do you feel you need to disable it?

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 13 juin 2013 à 03:56, Allan Johns <[hidden email]> a écrit :

On further reading, the "store" property doesn't quite do what I want.

I have a field that I want to be able to search on, but I don't want it in my _source. Is this possible?
A



On Thu, Jun 13, 2013 at 11:35 AM, Allan Johns <[hidden email]> wrote:
Anyone?

I tried:
"store" : false
also, this didn't work either.


You can actually do this, right?

A



On Wed, Jun 12, 2013 at 7:07 PM, Allan Johns <[hidden email]> wrote:
I also tried this in my default-mapping.json, no luck (see template_2):

{
    "_default_" : {
        "dynamic_templates" : [
            {
                "template_1" : {
                    "match" : "folder",
                    "match_mapping_type" : "string",
                    "mapping" : {
                        "type" : "string",
                        "index" : "analyzed",
                        "analyzer": "gas_folder_analyzer"
                    }
                }
            },
            {
                "template_2" : {
                    "match" : "_*",
                    "mapping" : {
                        "store": "no"
                    }
                }
            },
            {
                "template_3" : {
                    "match" : "*",
                    "match_mapping_type" : "string",
                    "mapping" : {
                        "type" : "string",
                        "index" : "not_analyzed"
                    }
                }
            }
        ]
    }
}



On Wed, Jun 12, 2013 at 7:01 PM, Allan Johns <[hidden email]> wrote:
I can't quite figure out from the docs how to do this. I know I need to use the exclude functionality shown here http://www.elasticsearch.org/guide/reference/mapping/source-field/, but the documentation frustratingly provides the code snippets, but doesn't say where to put them.

I want to not store fields beginning with an underscore. I tried adding this to my config/elasticsearch.yml:

mapping:
    MYTYPE:
        _source:
            excludes: [ "_*" ]

But this didn't work.
What do I do?

thx
A




--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: don't store certain fields by default

q42jaap
Hi Allen,

Have you looked at:


{
   
"my_type" : {
       
"_source" : {
           
"includes" : ["path1.*", "path2.*"],
           
"excludes" : ["pat3.*"]
       
}
   
}
}

You say you've specified this in a yml file, in my experience this is not practical, I'd supply the mapping on index creation, I don't know for sure, but I think the _source mapping info stays the same AFTER the index has been created. Changing the yml file will not give you an error if the mapping doesn't merge. If you create a lot of indexes, you can make a template with the mapping. The template is part of the cluster state so it is persisted forever in the cluster. You don't need to keep the yml file in sync for every node in your cluster.

Have you already tried to look what the running mapping is for your index?

Jaap

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: don't store certain fields by default

Allan Johns
Hi Jaap,

That looks to be exactly what I am after, I somehow missed this when going through the docs.

Thanks!
A



On Tue, Jun 18, 2013 at 4:56 AM, Jaap Taal <[hidden email]> wrote:
Hi Allen,

Have you looked at:


{
   
"my_type" : {
       
"_source" : {
           
"includes" : ["path1.*", "path2.*"],
           
"excludes" : ["pat3.*"]
       
}
   
}
}

You say you've specified this in a yml file, in my experience this is not practical, I'd supply the mapping on index creation, I don't know for sure, but I think the _source mapping info stays the same AFTER the index has been created. Changing the yml file will not give you an error if the mapping doesn't merge. If you create a lot of indexes, you can make a template with the mapping. The template is part of the cluster state so it is persisted forever in the cluster. You don't need to keep the yml file in sync for every node in your cluster.

Have you already tried to look what the running mapping is for your index?

Jaap

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.