adding asciifolding in default analyzer?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

adding asciifolding in default analyzer?

Colin Surprenant
Hi,

I am (still) running 0.8 on a production box and I would like to add
the asciifolding filter. I haven't defined mappings for my index
(using the default dynamic mapping).

Can I simply add the asciifolding filter to the "default" analyzer like this:

index :
        analysis :
                analyzer :
                        default :
                                tokenizer : standard
                                filter : [standard, lowercase, stop,
asciifolding]

I tried it on my dev box but it does not seems to pickup the
asciifolding filter.

Do I absolutely need to define a custom analyzer and assign it using a
custom mapping?

Thanks,
Colin
Reply | Threaded
Open this post in threaded view
|

Re: adding asciifolding in default analyzer?

Clinton Gormley
Hiya

> I am (still) running 0.8 on a production box and I would like to add
> the asciifolding filter. I haven't defined mappings for my index
> (using the default dynamic mapping).
>
> Can I simply add the asciifolding filter to the "default" analyzer like this:
>
> index :
>         analysis :
>                 analyzer :
>                         default :
>                                 tokenizer : standard
>                                 filter : [standard, lowercase, stop,asciifolding]

Where are you adding this? To config/elasticsearch.yml?

It looks right to me.

Alternatively, you can also add it (minus the first 'index:') when you
create the index.

>
> I tried it on my dev box but it does not seems to pickup the
> asciifolding filter.

Did you create the index before or after adding those settings? It'll
only affect indices created afterwards.

clint



Reply | Threaded
Open this post in threaded view
|

Re: adding asciifolding in default analyzer?

Colin Surprenant
> Where are you adding this? To config/elasticsearch.yml?

Yup.

> Did you create the index before or after adding those settings? It'll
> only affect indices created afterwards.

Actually I tried both. I tried after changing the config and
restarting the node (recovers the index from the gateway), did not
work, even for data added after the change.
I also tried after deleting the index using the REST API and I also
tried after wiping the indices and gateway directories and restarting
the node.

Thanks,
Colin
Reply | Threaded
Open this post in threaded view
|

Re: adding asciifolding in default analyzer?

kimchy
Administrator
The configuration looks good (btw, when you create an index, you still need the index "prefix" in the settings provided). I tried it locally and I see that it gets added. Can you gist a sample that shows it does not work for you?

On Mon, Aug 30, 2010 at 11:16 PM, Colin Surprenant <[hidden email]> wrote:
> Where are you adding this? To config/elasticsearch.yml?

Yup.

> Did you create the index before or after adding those settings? It'll
> only affect indices created afterwards.

Actually I tried both. I tried after changing the config and
restarting the node (recovers the index from the gateway), did not
work, even for data added after the change.
I also tried after deleting the index using the REST API and I also
tried after wiping the indices and gateway directories and restarting
the node.

Thanks,
Colin

Reply | Threaded
Open this post in threaded view
|

Re: adding asciifolding in default analyzer?

Colin Surprenant
> The configuration looks good (btw, when you create an index, you still need
> the index "prefix" in the settings provided). I tried it locally and I see
> that it gets added. Can you gist a sample that shows it does not work for
> you?

Well, in this case, I am not using the API Create Index. I am just
starting to directly add documents which triggers the index creation
on the first item. This is where I am hoping that it will pickup the
default analyser redefinition in the elasticsearch.yml config.

Is this the problem? Do I absolutely need to use Create Index and pass
it the analyzer config?

Thanks,
Colin
Reply | Threaded
Open this post in threaded view
|

Re: adding asciifolding in default analyzer?

kimchy
Administrator
No, you don't need to, the new index gets automatically created for you. Can you gist your conf, index creation, and how you check that it was not applied (using curl)? I will test it.

On Tue, Aug 31, 2010 at 6:57 AM, Colin Surprenant <[hidden email]> wrote:
> The configuration looks good (btw, when you create an index, you still need
> the index "prefix" in the settings provided). I tried it locally and I see
> that it gets added. Can you gist a sample that shows it does not work for
> you?

Well, in this case, I am not using the API Create Index. I am just
starting to directly add documents which triggers the index creation
on the first item. This is where I am hoping that it will pickup the
default analyser redefinition in the elasticsearch.yml config.

Is this the problem? Do I absolutely need to use Create Index and pass
it the analyzer config?

Thanks,
Colin

Reply | Threaded
Open this post in threaded view
|

Re: adding asciifolding in default analyzer?

Colin Surprenant
> No, you don't need to, the new index gets automatically created for you. Can
> you gist your conf, index creation, and how you check that it was not
> applied (using curl)? I will test it.

Thanks Shay but no need, I just found the problem. After giving it a
try on 0.10 I realized my problem was not with the config but with the
actual encoding in my documents. doh.

So, now, my documents are correctly utf8 encoded and the asciifolding
filter does work as specified on the default analyzer, as decribed in
my previous message.

Thanks for your help.
Colin