asciifolding analyzer

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

asciifolding analyzer

Carlos Eduardo da Fonseca
Hi, I've done a lot of research on groups and ES guides on how to use this analyzer but somehow it is not working.

Here's my elasticsearch.yml:

index:
  analysis:
    analyzer:
      standard:
        tokenizer: standard
        type: standard
        filter: [standard, lowercase, stop, asciifolding]


I already stopped ES, removed the data/ dir and indexed everything after changing the file. Still not working.

My ES version is 1.17.0

What may be wrong?

Thanks,

--
Carlos Eduardo da Fonseca
Reply | Threaded
Open this post in threaded view
|

Re: asciifolding analyzer

kimchy
Administrator
You change the standard analyzer, this means that in the mapping, if you set for a field explicitly to use the standard analyzer (set analyzer="standard") then it will use it.

If you want to change the *default* analyzer, then replace the name standard with default.

On Thu, Jul 28, 2011 at 1:22 AM, Carlos Eduardo da Fonseca <[hidden email]> wrote:
Hi, I've done a lot of research on groups and ES guides on how to use this analyzer but somehow it is not working.

Here's my elasticsearch.yml:

index:
  analysis:
    analyzer:
      standard:
        tokenizer: standard
        type: standard
        filter: [standard, lowercase, stop, asciifolding]


I already stopped ES, removed the data/ dir and indexed everything after changing the file. Still not working.

My ES version is 1.17.0

What may be wrong?

Thanks,

--
Carlos Eduardo da Fonseca

Reply | Threaded
Open this post in threaded view
|

Re: asciifolding analyzer

Carlos Eduardo da Fonseca
Hi Shay, thanks for the answer. 

I changed it to "default", restarted ES (cleaning data/) and reindexed, but it doesn't worked... 

elasticsearch.yml:

index:
  analysis:
    analyzer:
      default:
        tokenizer: standard
        type: standard
        filter: [standard, lowercase, stop, asciifolding]



On Thu, Jul 28, 2011 at 9:53 AM, Shay Banon <[hidden email]> wrote:
You change the standard analyzer, this means that in the mapping, if you set for a field explicitly to use the standard analyzer (set analyzer="standard") then it will use it.

If you want to change the *default* analyzer, then replace the name standard with default.

On Thu, Jul 28, 2011 at 1:22 AM, Carlos Eduardo da Fonseca <[hidden email]> wrote:
Hi, I've done a lot of research on groups and ES guides on how to use this analyzer but somehow it is not working.

Here's my elasticsearch.yml:

index:
  analysis:
    analyzer:
      standard:
        tokenizer: standard
        type: standard
        filter: [standard, lowercase, stop, asciifolding]


I already stopped ES, removed the data/ dir and indexed everything after changing the file. Still not working.

My ES version is 1.17.0

What may be wrong?

Thanks,

--
Carlos Eduardo da Fonseca




--
Carlos Eduardo da Fonseca
Reply | Threaded
Open this post in threaded view
|

Re: asciifolding analyzer

kimchy
Administrator
Gist an example then of why it does not work. One way to make sure it has taked affect on indices created is to issue get settings and see if its there.

On Thu, Jul 28, 2011 at 4:09 PM, Carlos Eduardo da Fonseca <[hidden email]> wrote:
Hi Shay, thanks for the answer. 

I changed it to "default", restarted ES (cleaning data/) and reindexed, but it doesn't worked... 

elasticsearch.yml:

index:
  analysis:
    analyzer:
      default:
        tokenizer: standard
        type: standard
        filter: [standard, lowercase, stop, asciifolding]



On Thu, Jul 28, 2011 at 9:53 AM, Shay Banon <[hidden email]> wrote:
You change the standard analyzer, this means that in the mapping, if you set for a field explicitly to use the standard analyzer (set analyzer="standard") then it will use it.

If you want to change the *default* analyzer, then replace the name standard with default.

On Thu, Jul 28, 2011 at 1:22 AM, Carlos Eduardo da Fonseca <[hidden email]> wrote:
Hi, I've done a lot of research on groups and ES guides on how to use this analyzer but somehow it is not working.

Here's my elasticsearch.yml:

index:
  analysis:
    analyzer:
      standard:
        tokenizer: standard
        type: standard
        filter: [standard, lowercase, stop, asciifolding]


I already stopped ES, removed the data/ dir and indexed everything after changing the file. Still not working.

My ES version is 1.17.0

What may be wrong?

Thanks,

--
Carlos Eduardo da Fonseca




--
Carlos Eduardo da Fonseca

Reply | Threaded
Open this post in threaded view
|

Re: asciifolding analyzer

Carlos Eduardo da Fonseca
Ok, here's a gist with index settings and everything I did after creating the index with settings via PUT: https://gist.github.com/1a92fe00ddbd71806892


Reply | Threaded
Open this post in threaded view
|

Re: asciifolding analyzer

Vladimir Shkurin
I'm using this settings and asciifolding works:

https://gist.github.com/1109136

/_search?q=title:botao* - OK
/_search?q=botao* - OK

You can remove the "my_delimiter", "porter_stem" and use these
settings, just write in your mappings for field-type "string" for
"index_analyzer" and "search_analyzer" your custom analyzer.
Reply | Threaded
Open this post in threaded view
|

Re: asciifolding analyzer

Carlos Eduardo da Fonseca
That worked! thanks a lot!!!

On Thu, Jul 28, 2011 at 4:41 PM, Vladimir Shkurin <[hidden email]> wrote:
I'm using this settings and asciifolding works:

https://gist.github.com/1109136

/_search?q=title:botao* - OK
/_search?q=botao* - OK

You can remove the "my_delimiter", "porter_stem" and use these
settings, just write in your mappings for field-type "string" for
"index_analyzer" and "search_analyzer" your custom analyzer.



--
Carlos Eduardo da Fonseca
Reply | Threaded
Open this post in threaded view
|

Re: asciifolding analyzer

kimchy
Administrator
Strange that it worked :). By default, wildcard and prefix using the query_string query (which is what q= is translated to) don't get analyzed. http://www.elasticsearch.org/guide/reference/query-dsl/query-string-query.html. You can pass analyze_wildcard set to true, which will cause wildcards to be analyzed as well (though its tricky).

Having said that, the analyze_wildcard parameter is not exposed when using ?q=..., though it can be added easily, something like: ?q=analyze_wildcard=true&q=... .

Most times, a better option is to use the text family of queries: http://www.elasticsearch.org/guide/reference/query-dsl/text-query.html.

On Thu, Jul 28, 2011 at 11:57 PM, Carlos Eduardo da Fonseca <[hidden email]> wrote:
That worked! thanks a lot!!!


On Thu, Jul 28, 2011 at 4:41 PM, Vladimir Shkurin <[hidden email]> wrote:
I'm using this settings and asciifolding works:

https://gist.github.com/1109136

/_search?q=title:botao* - OK
/_search?q=botao* - OK

You can remove the "my_delimiter", "porter_stem" and use these
settings, just write in your mappings for field-type "string" for
"index_analyzer" and "search_analyzer" your custom analyzer.



--
Carlos Eduardo da Fonseca

Reply | Threaded
Open this post in threaded view
|

Re: asciifolding analyzer

Carlos Eduardo da Fonseca
Shay, I searched using text and field queries, forgot to mention. I think it wasn't working before because of missing mappings...

On Fri, Jul 29, 2011 at 3:37 AM, Shay Banon <[hidden email]> wrote:
Strange that it worked :). By default, wildcard and prefix using the query_string query (which is what q= is translated to) don't get analyzed. http://www.elasticsearch.org/guide/reference/query-dsl/query-string-query.html. You can pass analyze_wildcard set to true, which will cause wildcards to be analyzed as well (though its tricky).

Having said that, the analyze_wildcard parameter is not exposed when using ?q=..., though it can be added easily, something like: ?q=analyze_wildcard=true&q=... .

Most times, a better option is to use the text family of queries: http://www.elasticsearch.org/guide/reference/query-dsl/text-query.html.


On Thu, Jul 28, 2011 at 11:57 PM, Carlos Eduardo da Fonseca <[hidden email]> wrote:
That worked! thanks a lot!!!


On Thu, Jul 28, 2011 at 4:41 PM, Vladimir Shkurin <[hidden email]> wrote:
I'm using this settings and asciifolding works:

https://gist.github.com/1109136

/_search?q=title:botao* - OK
/_search?q=botao* - OK

You can remove the "my_delimiter", "porter_stem" and use these
settings, just write in your mappings for field-type "string" for
"index_analyzer" and "search_analyzer" your custom analyzer.



--
Carlos Eduardo da Fonseca




--
Carlos Eduardo da Fonseca