TF/IDF wihout TF

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

TF/IDF wihout TF

Andrew Gaydenko
How to turn TF off (that is a single term in a field is the same as multiple ones) in scoring (keeping IDF and field length)?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d527960c-1407-4f61-b058-2baab38b4957%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: TF/IDF wihout TF

Doug Turnbull
You can turn off tf, but you also have to turn off positions at the same time. (There is no positions without term frequency). This implies you won't be able to do a phrase query. This leaves IDF and norms as the main scoring component

See the "index_options" here for disabling term freq
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html

-Doug

On Sat, Dec 27, 2014 at 10:10 AM, Andrew Gaydenko <[hidden email]> wrote:
How to turn TF off (that is a single term in a field is the same as multiple ones) in scoring (keeping IDF and field length)?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d527960c-1407-4f61-b058-2baab38b4957%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Doug Turnbull
Search & Big Data Architect

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALG6HL9ei6T1sAE30YXs%3D8QtBd%3DwuymfyOP0XUnaqW446HoQSQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: TF/IDF wihout TF

Andrew Gaydenko
Thanks! Having norms on and docs for index_options will we keep scoring on the field length? - that is "abc" in "abc" has more score than in "abd def". In other words, do norms include field-length-based scoring factor?

On Saturday, December 27, 2014 6:35:39 PM UTC+3, Doug Turnbull wrote:
You can turn off tf, but you also have to turn off positions at the same time. (There is no positions without term frequency). This implies you won't be able to do a phrase query. This leaves IDF and norms as the main scoring component

See the "index_options" here for disabling term freq
<a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmapping-core-types.html\46sa\75D\46sntz\0751\46usg\75AFQjCNG0lHgJdWep8xypKWDPnk6_4UeHww';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fwww.elasticsearch.org%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Fmapping-core-types.html\46sa\75D\46sntz\0751\46usg\75AFQjCNG0lHgJdWep8xypKWDPnk6_4UeHww';return true;">http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html

-Doug

On Sat, Dec 27, 2014 at 10:10 AM, Andrew Gaydenko <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="j3Mtrx8P6A0J" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">andrew....@...> wrote:
How to turn TF off (that is a single term in a field is the same as multiple ones) in scoring (keeping IDF and field length)?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="j3Mtrx8P6A0J" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">elasticsearc...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/elasticsearch/d527960c-1407-4f61-b058-2baab38b4957%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" onmousedown="this.href='https://groups.google.com/d/msgid/elasticsearch/d527960c-1407-4f61-b058-2baab38b4957%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;" onclick="this.href='https://groups.google.com/d/msgid/elasticsearch/d527960c-1407-4f61-b058-2baab38b4957%40googlegroups.com?utm_medium\75email\46utm_source\75footer';return true;">https://groups.google.com/d/msgid/elasticsearch/d527960c-1407-4f61-b058-2baab38b4957%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/optout.



--
Doug Turnbull
Search & Big Data Architect
<a href="http://o19s.com" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Fo19s.com\46sa\75D\46sntz\0751\46usg\75AFQjCNEDoThL2vrmhscBJPc34AzGJUhXMA';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Fo19s.com\46sa\75D\46sntz\0751\46usg\75AFQjCNEDoThL2vrmhscBJPc34AzGJUhXMA';return true;">OpenSource Connections

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a6792674-1fcd-4c96-b409-f1f8c6d9b860%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

TF/IDF wihout TF

drjz
In reply to this post by Andrew Gaydenko
You can "disable" TF by multiplying it with itself. This requires adding a configuration in the base similarity module I think. I have not tried this yet, so please let us know if this will work!

/JZ

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a15814ff-f02e-472b-9d70-dcd9a5d860b5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.