Example config file

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Example config file

Clinton Gormley
Hiya

Does anybody have a real life example of an elasticsearch config file.

I'm somewhat confused about what goes into the config file, and what
gets specified when (eg) creating an index.

thanks

Clint

--
Web Announcements Limited is a company registered in England and Wales,
with company number 05608868, with registered address at 10 Arvon Road,
London, N5 1PR.

Reply | Threaded
Open this post in threaded view
|

Re: Example config file

kimchy
Administrator
HI,

  If you can give an example of what you try and configure, then I can probably give a better answer. I tried and explain it here: http://www.elasticsearch.com/docs/elasticsearch/setup/configuration/, what specifically do you struggle with so I can explain it better in the docs....

-shay.banon

On Tue, Mar 9, 2010 at 5:40 PM, Clinton Gormley <[hidden email]> wrote:
Hiya

Does anybody have a real life example of an elasticsearch config file.

I'm somewhat confused about what goes into the config file, and what
gets specified when (eg) creating an index.

thanks

Clint

--
Web Announcements Limited is a company registered in England and Wales,
with company number 05608868, with registered address at 10 Arvon Road,
London, N5 1PR.


Reply | Threaded
Open this post in threaded view
|

Re: Example config file

Clinton Gormley
Hiya
>
>
>   If you can give an example of what you try and configure, then I can
> probably give a better answer. I tried and explain it here:
> http://www.elasticsearch.com/docs/elasticsearch/setup/configuration/,
> what specifically do you struggle with so I can explain it better in
> the docs....

Yeah sorry, crap question :)

OK, for instance, do the following settings make sense. I realise that
some are the same as the defaults (and thus redundant) but I wanted to
be sure that I was putting them in the right place.

    cluster:
        name:               MyCluster

    network:
        bindHost:           192.168.5.100
        publishHost:        192.168.5.100

    node:
        data:               true

    http:
        enabled:            true


    gateway:
        type:               fs
        fs:            
            location:       /opt/elasticsearch/data/cluster

    index:
        gateway:
            type:           fs
            fs:            
                location:   /opt/elasticsearch/data/index

        store:
            type:           memory
            cacheSize:      100m

Of course, when I create an index, I can set the gateway.fs.location to
a different directory, which would make sense, no?

For things like mappings, would/could you put them in the config file,
would you just put_mapping via the API?

ta

clint

--
Web Announcements Limited is a company registered in England and Wales,
with company number 05608868, with registered address at 10 Arvon Road,
London, N5 1PR.

Reply | Threaded
Open this post in threaded view
|

Re: Example config file

kimchy
Administrator
Answered below:

On Tue, Mar 9, 2010 at 7:00 PM, Clinton Gormley <[hidden email]> wrote:
Hiya
>
>
>   If you can give an example of what you try and configure, then I can
> probably give a better answer. I tried and explain it here:
> http://www.elasticsearch.com/docs/elasticsearch/setup/configuration/,
> what specifically do you struggle with so I can explain it better in
> the docs....

Yeah sorry, crap question :)

OK, for instance, do the following settings make sense. I realise that
some are the same as the defaults (and thus redundant) but I wanted to
be sure that I was putting them in the right place.

   cluster:
       name:               MyCluster

   network:
       bindHost:           192.168.5.100
       publishHost:        192.168.5.100

   node:
       data:               true

   http:
       enabled:            true


   gateway:
       type:               fs
       fs:
           location:       /opt/elasticsearch/data/cluster

   index:
       gateway:
           type:           fs
           fs:
               location:   /opt/elasticsearch/data/index

       store:
           type:           memory
           cacheSize:      100m

The store one is wrong, and I would play with the bufferSize as well:

store:
   type: memory
   memory:
        cacheSize: 100m
        bufferSize: 10k
 

Of course, when I create an index, I can set the gateway.fs.location to
a different directory, which would make sense, no?

Depends. When you explicitly set the one for the index, then you need to make sure that you don't have two indices pointing to the same location. If you don't specify one, then the global gateway location will be used, with the index name appended as a sub directory. So I would say just leave the global one to be fs, and let the index level ones "inherit it" (the index will automatically use fs gateway if the global gateway is fs).
 

For things like mappings, would/could you put them in the config file,
would you just put_mapping via the API?

Currently, you can only place do it through the API. There is no option for the cluster to start up and read mappings from a specific location. Of course, there is also the question of which mappings in such a case apply to which index...
 

ta

clint

--
Web Announcements Limited is a company registered in England and Wales,
with company number 05608868, with registered address at 10 Arvon Road,
London, N5 1PR.


Reply | Threaded
Open this post in threaded view
|

Re: Example config file

Clinton Gormley

>
> Depends. When you explicitly set the one for the index, then you need
> to make sure that you don't have two indices pointing to the same
> location. If you don't specify one, then the global gateway location
> will be used, with the index name appended as a sub directory. So I
> would say just leave the global one to be fs, and let the index level
> ones "inherit it" (the index will automatically use fs gateway if the
> global gateway is fs).

OK - makes sense.

So if I wanted to change the mapping for an existing index/type, I could
do something like this:

 - create index 'temp_index'
 - put mapping 'new_type'
 - put other mappings from 'index' (except for 'old_type')
 - foreach document in 'index'
    - save to temp_index
 - stop nodes
 - mv index old_index
 - mv temp_index index
 - restart nodes

ta

clint
>
--
Web Announcements Limited is a company registered in England and Wales,
with company number 05608868, with registered address at 10 Arvon Road,
London, N5 1PR.

Reply | Threaded
Open this post in threaded view
|

Re: Example config file

kimchy
Administrator
> - mv index old_index
> - mv temp_index index

I assume you mean mv the gateway location? Yes, this should work, though I don't like the fact that you need to stop the nodes and start them again!. I guess once elasticsearch will have the ability to "rename", or at the very least "alias/ln -s" indices then you won't need to restart nodes as you can create work with the aliases names and then "re-alias".

Also, iterating over the documents and reindexing should be simpler with elasticsearch. One step towards this is the ability to continue to scroll search requests (which I am working on now). The second is to have an API for that.

-shay.banon


On Tue, Mar 9, 2010 at 7:37 PM, Clinton Gormley <[hidden email]> wrote:

>
> Depends. When you explicitly set the one for the index, then you need
> to make sure that you don't have two indices pointing to the same
> location. If you don't specify one, then the global gateway location
> will be used, with the index name appended as a sub directory. So I
> would say just leave the global one to be fs, and let the index level
> ones "inherit it" (the index will automatically use fs gateway if the
> global gateway is fs).

OK - makes sense.

So if I wanted to change the mapping for an existing index/type, I could
do something like this:

 - create index 'temp_index'
 - put mapping 'new_type'
 - put other mappings from 'index' (except for 'old_type')
 - foreach document in 'index'
   - save to temp_index
 - stop nodes
 - mv index old_index
 - mv temp_index index
 - restart nodes

ta

clint
>
--
Web Announcements Limited is a company registered in England and Wales,
with company number 05608868, with registered address at 10 Arvon Road,
London, N5 1PR.


Reply | Threaded
Open this post in threaded view
|

Re: Example config file

kimchy
Administrator
One last point, I am also slowly making the put_mapping API more smarter with merging new definitions with old ones. Changing the type from integer to string will probably not be supported (though you can use the new multi_field mapping to handle that), but changing things like term vectors and such should be supported. Of course, adding new mappings is supported already.

-shay.banon

On Tue, Mar 9, 2010 at 7:52 PM, Shay Banon <[hidden email]> wrote:
> - mv index old_index
> - mv temp_index index

I assume you mean mv the gateway location? Yes, this should work, though I don't like the fact that you need to stop the nodes and start them again!. I guess once elasticsearch will have the ability to "rename", or at the very least "alias/ln -s" indices then you won't need to restart nodes as you can create work with the aliases names and then "re-alias".

Also, iterating over the documents and reindexing should be simpler with elasticsearch. One step towards this is the ability to continue to scroll search requests (which I am working on now). The second is to have an API for that.

-shay.banon


On Tue, Mar 9, 2010 at 7:37 PM, Clinton Gormley <[hidden email]> wrote:

>
> Depends. When you explicitly set the one for the index, then you need
> to make sure that you don't have two indices pointing to the same
> location. If you don't specify one, then the global gateway location
> will be used, with the index name appended as a sub directory. So I
> would say just leave the global one to be fs, and let the index level
> ones "inherit it" (the index will automatically use fs gateway if the
> global gateway is fs).

OK - makes sense.

So if I wanted to change the mapping for an existing index/type, I could
do something like this:

 - create index 'temp_index'
 - put mapping 'new_type'
 - put other mappings from 'index' (except for 'old_type')
 - foreach document in 'index'
   - save to temp_index
 - stop nodes
 - mv index old_index
 - mv temp_index index
 - restart nodes

ta

clint
>
--
Web Announcements Limited is a company registered in England and Wales,
with company number 05608868, with registered address at 10 Arvon Road,
London, N5 1PR.



Reply | Threaded
Open this post in threaded view
|

Re: Example config file

Clinton Gormley
On Tue, 2010-03-09 at 19:54 +0200, Shay Banon wrote:
> One last point, I am also slowly making the put_mapping API more
> smarter with merging new definitions with old ones. Changing the type
> from integer to string will probably not be supported (though you can
> use the new multi_field mapping to handle that), but changing things
> like term vectors and such should be supported. Of course, adding new
> mappings is supported already.

w00t!

>
--
Web Announcements Limited is a company registered in England and Wales,
with company number 05608868, with registered address at 10 Arvon Road,
London, N5 1PR.