Flexible index mappings

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Flexible index mappings

Ivan Brusic
In my current setup, I am configuring new index mappings via a file on
each node. File-based mappings allow me to change items at the
application level and not via code, and are easier to manage in
source-control.

However, with using index aliases, the only way I can apply to same
mapping to each new index is by putting the file in the _default
directory. Currently the system is only using one index, so the
solution is acceptable, but other indices will appear in the future.

Having a more flexible approach is needed, but I am unsure about any
practical suggestions. Having wildcards in the directory name is ugly
and not supported by Windows. Perhaps a default.json in the
config/mappings directory with a slightly different JSON structure to
allow index names (with wildcards)?

e.g.

{
    "myindex-*" : {
        "mytype" : {
            "_source" : {"enabled" : true},
            "_all" : {"enabled" : true},
            "properties" : {
                ...
This file would have the lowest precedence in determining the correct mapping.

Or perhaps am I missing another solution altogether?

Cheers,

Ivan
Reply | Threaded
Open this post in threaded view
|

Re: Flexible index mappings

Karussell
> Or perhaps am I missing another solution altogether?

In ElasticSearch you define per type

http://www.elasticsearch.org/guide/reference/mapping/object-type.html

Or did I misunderstand something?

Regards,
Peter.

Reply | Threaded
Open this post in threaded view
|

Re: Flexible index mappings

Ivan Brusic
There are only two options for placing the mapping file: inside the
_default directory which applies to all indices, or inside a directory
with the same name as an index. Types defined inside a mapping inside
that _default will apply to all indices, even if the type is not used.
No performance issues, but the index metadata will have this mapping,
which can be confusing.

On Thu, Feb 9, 2012 at 1:56 AM, Karussell <[hidden email]> wrote:

>> Or perhaps am I missing another solution altogether?
>
> In ElasticSearch you define per type
>
> http://www.elasticsearch.org/guide/reference/mapping/object-type.html
>
> Or did I misunderstand something?
>
> Regards,
> Peter.
>
Reply | Threaded
Open this post in threaded view
|

Re: Flexible index mappings

kimchy
Administrator
How about using index templates? You can't place index templates in the configuration location of node(s), but you can place them when the system starts, and then they will apply to indices matching the relevant index name.

On Thursday, February 9, 2012 at 6:03 PM, Ivan Brusic wrote:

There are only two options for placing the mapping file: inside the
_default directory which applies to all indices, or inside a directory
with the same name as an index. Types defined inside a mapping inside
that _default will apply to all indices, even if the type is not used.
No performance issues, but the index metadata will have this mapping,
which can be confusing.

On Thu, Feb 9, 2012 at 1:56 AM, Karussell <[hidden email]> wrote:
Or perhaps am I missing another solution altogether?

In ElasticSearch you define per type


Or did I misunderstand something?

Regards,
Peter.

Reply | Threaded
Open this post in threaded view
|

Re: Flexible index mappings

Ivan Brusic
I did consider index templates since they allow the use of wildcards
in the index name, but did not explore further since they are not file
based.

Not sure what you mean by "you can place them when the system starts".
Do you mean they can be added via the API at startup?

Ultimately, the file based approach might not work since they require a reboot.

--
Ivan

On Sun, Feb 12, 2012 at 3:20 AM, Shay Banon <[hidden email]> wrote:

> How about using index templates? You can't place index templates in the
> configuration location of node(s), but you can place them when the system
> starts, and then they will apply to indices matching the relevant index
> name.
>
> On Thursday, February 9, 2012 at 6:03 PM, Ivan Brusic wrote:
>
> There are only two options for placing the mapping file: inside the
> _default directory which applies to all indices, or inside a directory
> with the same name as an index. Types defined inside a mapping inside
> that _default will apply to all indices, even if the type is not used.
> No performance issues, but the index metadata will have this mapping,
> which can be confusing.
>
> On Thu, Feb 9, 2012 at 1:56 AM, Karussell <[hidden email]>
> wrote:
>
> Or perhaps am I missing another solution altogether?
>
>
> In ElasticSearch you define per type
>
> http://www.elasticsearch.org/guide/reference/mapping/object-type.html
>
> Or did I misunderstand something?
>
> Regards,
> Peter.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Flexible index mappings

kimchy
Administrator
I added this so you can place templates under config: https://github.com/elasticsearch/elasticsearch/issues/1703.

On Wednesday, February 15, 2012 at 12:56 AM, Ivan Brusic wrote:

I did consider index templates since they allow the use of wildcards
in the index name, but did not explore further since they are not file
based.

Not sure what you mean by "you can place them when the system starts".
Do you mean they can be added via the API at startup?

Ultimately, the file based approach might not work since they require a reboot.

--
Ivan

On Sun, Feb 12, 2012 at 3:20 AM, Shay Banon <[hidden email]> wrote:
How about using index templates? You can't place index templates in the
configuration location of node(s), but you can place them when the system
starts, and then they will apply to indices matching the relevant index
name.

On Thursday, February 9, 2012 at 6:03 PM, Ivan Brusic wrote:

There are only two options for placing the mapping file: inside the
_default directory which applies to all indices, or inside a directory
with the same name as an index. Types defined inside a mapping inside
that _default will apply to all indices, even if the type is not used.
No performance issues, but the index metadata will have this mapping,
which can be confusing.

On Thu, Feb 9, 2012 at 1:56 AM, Karussell <[hidden email]>
wrote:

Or perhaps am I missing another solution altogether?


In ElasticSearch you define per type


Or did I misunderstand something?

Regards,
Peter.

Reply | Threaded
Open this post in threaded view
|

Re: Flexible index mappings

Ivan Brusic
Finally had a chance to test out the change with 0.19.0 (maintenance
coding is taking up all my time). Works like a charm, but any
modifications to a template are not reflected in the index, even after
a restart. Is this an issue with dynamic templates (which I have never
used) in general?

--
Ivan

On Wed, Feb 15, 2012 at 2:36 AM, Shay Banon <[hidden email]> wrote:

> I added this so you can place templates under
> config: https://github.com/elasticsearch/elasticsearch/issues/1703.
>
> On Wednesday, February 15, 2012 at 12:56 AM, Ivan Brusic wrote:
>
> I did consider index templates since they allow the use of wildcards
> in the index name, but did not explore further since they are not file
> based.
>
> Not sure what you mean by "you can place them when the system starts".
> Do you mean they can be added via the API at startup?
>
> Ultimately, the file based approach might not work since they require a
> reboot.
>
> --
> Ivan
>
> On Sun, Feb 12, 2012 at 3:20 AM, Shay Banon <[hidden email]> wrote:
>
> How about using index templates? You can't place index templates in the
> configuration location of node(s), but you can place them when the system
> starts, and then they will apply to indices matching the relevant index
> name.
>
> On Thursday, February 9, 2012 at 6:03 PM, Ivan Brusic wrote:
>
> There are only two options for placing the mapping file: inside the
> _default directory which applies to all indices, or inside a directory
> with the same name as an index. Types defined inside a mapping inside
> that _default will apply to all indices, even if the type is not used.
> No performance issues, but the index metadata will have this mapping,
> which can be confusing.
>
> On Thu, Feb 9, 2012 at 1:56 AM, Karussell <[hidden email]>
> wrote:
>
> Or perhaps am I missing another solution altogether?
>
>
> In ElasticSearch you define per type
>
> http://www.elasticsearch.org/guide/reference/mapping/object-type.html
>
> Or did I misunderstand something?
>
> Regards,
> Peter.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Flexible index mappings

Radu Gheorghe
Hi Ivan,

I know it's a long shot, but are you sure you actually modified the
template, and not entered a new one without deleting the old template?
That's what I did yesterday, so I thought I should share :D

To get all your templates, you can do something like this:

# curl -XGET 'http://localhost:9200/_cluster/state?pretty=true' > /tmp/
cluster_state_dump

and then look in that file for "templates".

Best regards,
Radu

On Mar 14, 11:21 pm, Ivan Brusic <[hidden email]> wrote:

> Finally had a chance to test out the change with 0.19.0 (maintenance
> coding is taking up all my time). Works like a charm, but any
> modifications to a template are not reflected in the index, even after
> a restart. Is this an issue with dynamic templates (which I have never
> used) in general?
>
> --
> Ivan
>
>
>
>
>
>
>
> On Wed, Feb 15, 2012 at 2:36 AM, Shay Banon <[hidden email]> wrote:
> > I added this so you can place templates under
> > config: https://github.com/elasticsearch/elasticsearch/issues/1703.
>
> > On Wednesday, February 15, 2012 at 12:56 AM, Ivan Brusic wrote:
>
> > I did consider index templates since they allow the use of wildcards
> > in the index name, but did not explore further since they are not file
> > based.
>
> > Not sure what you mean by "you can place them when the system starts".
> > Do you mean they can be added via the API at startup?
>
> > Ultimately, the file based approach might not work since they require a
> > reboot.
>
> > --
> > Ivan
>
> > On Sun, Feb 12, 2012 at 3:20 AM, Shay Banon <[hidden email]> wrote:
>
> > How about using index templates? You can't place index templates in the
> > configuration location of node(s), but you can place them when the system
> > starts, and then they will apply to indices matching the relevant index
> > name.
>
> > On Thursday, February 9, 2012 at 6:03 PM, Ivan Brusic wrote:
>
> > There are only two options for placing the mapping file: inside the
> > _default directory which applies to all indices, or inside a directory
> > with the same name as an index. Types defined inside a mapping inside
> > that _default will apply to all indices, even if the type is not used.
> > No performance issues, but the index metadata will have this mapping,
> > which can be confusing.
>
> > On Thu, Feb 9, 2012 at 1:56 AM, Karussell <[hidden email]>
> > wrote:
>
> > Or perhaps am I missing another solution altogether?
>
> > In ElasticSearch you define per type
>
> >http://www.elasticsearch.org/guide/reference/mapping/object-type.html
>
> > Or did I misunderstand something?
>
> > Regards,
> > Peter.
Reply | Threaded
Open this post in threaded view
|

Re: Flexible index mappings

kimchy
Administrator
In reply to this post by Ivan Brusic
Modifying the template will only apply to new indices created, not to existing indices...

On Wed, Mar 14, 2012 at 11:21 PM, Ivan Brusic <[hidden email]> wrote:
Finally had a chance to test out the change with 0.19.0 (maintenance
coding is taking up all my time). Works like a charm, but any
modifications to a template are not reflected in the index, even after
a restart. Is this an issue with dynamic templates (which I have never
used) in general?

--
Ivan

On Wed, Feb 15, 2012 at 2:36 AM, Shay Banon <[hidden email]> wrote:
> I added this so you can place templates under
> config: https://github.com/elasticsearch/elasticsearch/issues/1703.
>
> On Wednesday, February 15, 2012 at 12:56 AM, Ivan Brusic wrote:
>
> I did consider index templates since they allow the use of wildcards
> in the index name, but did not explore further since they are not file
> based.
>
> Not sure what you mean by "you can place them when the system starts".
> Do you mean they can be added via the API at startup?
>
> Ultimately, the file based approach might not work since they require a
> reboot.
>
> --
> Ivan
>
> On Sun, Feb 12, 2012 at 3:20 AM, Shay Banon <[hidden email]> wrote:
>
> How about using index templates? You can't place index templates in the
> configuration location of node(s), but you can place them when the system
> starts, and then they will apply to indices matching the relevant index
> name.
>
> On Thursday, February 9, 2012 at 6:03 PM, Ivan Brusic wrote:
>
> There are only two options for placing the mapping file: inside the
> _default directory which applies to all indices, or inside a directory
> with the same name as an index. Types defined inside a mapping inside
> that _default will apply to all indices, even if the type is not used.
> No performance issues, but the index metadata will have this mapping,
> which can be confusing.
>
> On Thu, Feb 9, 2012 at 1:56 AM, Karussell <[hidden email]>
> wrote:
>
> Or perhaps am I missing another solution altogether?
>
>
> In ElasticSearch you define per type
>
> http://www.elasticsearch.org/guide/reference/mapping/object-type.html
>
> Or did I misunderstand something?
>
> Regards,
> Peter.
>
>