JsonFieldMapper used for default field within a multi_field element

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

JsonFieldMapper used for default field within a multi_field element

dominict
Say I have the following mapping :

{
    "stringtest" : {
       " properties" : {
            "name" : {
                "type" : "multi_field",
                "fields" : {
                    "name" : { type : "string" },
                    "name_untouched" : {type: "string"}
                }
            }
        }
    }
}

The JsonStringFieldMapper.indexedValue(String value) only gets called
for name_untouched, and not name.  Should the JsonStringFieldMapper
used for both the default "name" and "name_untouched" fields?

Mapping setup via:

 
server.client().admin().indices().putMapping(putMappingRequest("test").mappingSource(mapping)).actionGet();


Document is indexed via:

 
server.client().index(indexRequest("test").type("stringtest").id("1")
                .source(jsonBuilder().startObject().field("name",
"123").endObject())).actionGet();

 
server.client().admin().indices().refresh(refreshRequest()).actionGet();

then a count is done:

  CountResponse countResponse =
client.count(countRequest("test").querySource(termQuery("name",
"123"))).actionGet();
  CountResponse countResponse2 =
client.count(countRequest("test").querySource(termQuery("name.name_name",
"123"))).actionGet();
  CountResponse countResponse3 =
client.count(countRequest("test").querySource(termQuery("name.name_untouched",
"123"))).actionGet();
  CountResponse countResponse4 =
client.count(countRequest("test").querySource(termQuery("name.name",
"123"))).actionGet();
  CountResponse countResponse5 =
client.count(countRequest("test").querySource(termQuery("name.untouched",
"123"))).actionGet();

of the above 5 count operations only count 1 + 3 return the correct
count number of 1.

Should "name.untouched" also work?

It's probably a complete mis-reading of
http://www.elasticsearch.com/docs/elasticsearch/mapping/multi_field/
on my part.

cheers,
/dom
Reply | Threaded
Open this post in threaded view
|

Re: JsonFieldMapper used for default field within a multi_field element

kimchy
Administrator
You are being modest :). There is a bug in the docs (just fixed it). The idea is that any mapping unit a multi_field that is not using the same name as the multi_field can be accessed by navigating through the multi_field name.

So, the docs are wrong, have a look at the corrected ones (the change is to rename name_untouched to untouched). In your case, since the extra field(s) is called name_untouched, then you navigate to it using name.name_untouched. If you change it to untouched, then name.untouched should work.

cheers,
shay.banon

On Tue, Apr 6, 2010 at 10:11 PM, dominict <[hidden email]> wrote:
Say I have the following mapping :

{
   "stringtest" : {
      " properties" : {
           "name" : {
               "type" : "multi_field",
               "fields" : {
                   "name" : { type : "string" },
                   "name_untouched" : {type: "string"}
               }
           }
       }
   }
}

The JsonStringFieldMapper.indexedValue(String value) only gets called
for name_untouched, and not name.  Should the JsonStringFieldMapper
used for both the default "name" and "name_untouched" fields?

Mapping setup via:


server.client().admin().indices().putMapping(putMappingRequest("test").mappingSource(mapping)).actionGet();


Document is indexed via:


server.client().index(indexRequest("test").type("stringtest").id("1")
               .source(jsonBuilder().startObject().field("name",
"123").endObject())).actionGet();


server.client().admin().indices().refresh(refreshRequest()).actionGet();

then a count is done:

 CountResponse countResponse =
client.count(countRequest("test").querySource(termQuery("name",
"123"))).actionGet();
 CountResponse countResponse2 =
client.count(countRequest("test").querySource(termQuery("name.name_name",
"123"))).actionGet();
 CountResponse countResponse3 =
client.count(countRequest("test").querySource(termQuery("name.name_untouched",
"123"))).actionGet();
 CountResponse countResponse4 =
client.count(countRequest("test").querySource(termQuery("name.name",
"123"))).actionGet();
 CountResponse countResponse5 =
client.count(countRequest("test").querySource(termQuery("name.untouched",
"123"))).actionGet();

of the above 5 count operations only count 1 + 3 return the correct
count number of 1.

Should "name.untouched" also work?

It's probably a complete mis-reading of
http://www.elasticsearch.com/docs/elasticsearch/mapping/multi_field/
on my part.

cheers,
/dom

Reply | Threaded
Open this post in threaded view
|

Re: JsonFieldMapper used for default field within a multi_field element

dominict
Cheers for the update shay,

I'm more curious about the JsonFieldMapper that is obtained and called
when searching for the default field. i.e. for the mapping:

{
    "stringtest" : {
        "properties" : {
            "name" : {
                "type" : "multi_field",
                "fields" : {
                    "name" : { type : "string", index :
"analyzed" },
                    "orig" : { type : "string", index :
"analyzed" },
                    "untouched" : {type: "string", index :
"not_analyzed"}
                }
            }
        }
    }
}

  CountResponse countResponse1 =
server.client().count(countRequest("test").querySource(fieldQuery("name",
"123"))).actionGet();
  CountResponse countResponse2 =
server.client().count(countRequest("test").querySource(fieldQuery("name.untouched",
"123"))).actionGet();
  CountResponse countResponse3 =
server.client().count(countRequest("test").querySource(fieldQuery("name.orig",
"123"))).actionGet();

upon searching, the method JsonFieldMapper.indexedValue(String) is
called (to format the string for searching I guess).  What I'm seeing
is the following:

  name is assigned JsonLongFieldMapper
  name.orig is assigned JsonStringFieldMapper
  name.untouched is assigned JsonStringFieldMapper

Is this meant to happen, or should it be assigned
JsonStringFieldMapper.  I'm just looking at plugin development in
0.6.0 (which is working well), and this is where my question is
stemming from.


cheers
/dom


On Apr 6, 11:33 pm, Shay Banon <[hidden email]> wrote:

> You are being modest :). There is a bug in the docs (just fixed it). The
> idea is that any mapping unit a multi_field that is not using the same name
> as the multi_field can be accessed by navigating through the multi_field
> name.
>
> So, the docs are wrong, have a look at the corrected ones (the change is to
> rename name_untouched to untouched). In your case, since the extra field(s)
> is called name_untouched, then you navigate to it using name.name_untouched.
> If you change it to untouched, then name.untouched should work.
>
> cheers,
> shay.banon
>
>
>
> On Tue, Apr 6, 2010 at 10:11 PM, dominict <[hidden email]> wrote:
> > Say I have the following mapping :
>
> > {
> >    "stringtest" : {
> >       " properties" : {
> >            "name" : {
> >                "type" : "multi_field",
> >                "fields" : {
> >                    "name" : { type : "string" },
> >                    "name_untouched" : {type: "string"}
> >                }
> >            }
> >        }
> >    }
> > }
>
> > The JsonStringFieldMapper.indexedValue(String value) only gets called
> > for name_untouched, and not name.  Should the JsonStringFieldMapper
> > used for both the default "name" and "name_untouched" fields?
>
> > Mapping setup via:
>
> > server.client().admin().indices().putMapping(putMappingRequest("test").mapp ingSource(mapping)).actionGet();
>
> > Document is indexed via:
>
> > server.client().index(indexRequest("test").type("stringtest").id("1")
> >                .source(jsonBuilder().startObject().field("name",
> > "123").endObject())).actionGet();
>
> > server.client().admin().indices().refresh(refreshRequest()).actionGet();
>
> > then a count is done:
>
> >  CountResponse countResponse =
> > client.count(countRequest("test").querySource(termQuery("name",
> > "123"))).actionGet();
> >  CountResponse countResponse2 =
> > client.count(countRequest("test").querySource(termQuery("name.name_name",
> > "123"))).actionGet();
> >  CountResponse countResponse3 =
>
> > client.count(countRequest("test").querySource(termQuery("name.name_untouche d",
> > "123"))).actionGet();
> >  CountResponse countResponse4 =
> > client.count(countRequest("test").querySource(termQuery("name.name",
> > "123"))).actionGet();
> >  CountResponse countResponse5 =
> > client.count(countRequest("test").querySource(termQuery("name.untouched",
> > "123"))).actionGet();
>
> > of the above 5 count operations only count 1 + 3 return the correct
> > count number of 1.
>
> > Should "name.untouched" also work?
>
> > It's probably a complete mis-reading of
> >http://www.elasticsearch.com/docs/elasticsearch/mapping/multi_field/
> > on my part.
>
> > cheers,
> > /dom
Reply | Threaded
Open this post in threaded view
|

Re: JsonFieldMapper used for default field within a multi_field element

kimchy
Administrator
No, it should not be a JsonLong one..., can you raise an issue and a testcase? Would like to nail this for 0.6 if possible. 

Just myself being the usual curious me, what plugin are you working on?

cheers,
shay.banon

On Wed, Apr 7, 2010 at 2:27 AM, dominict <[hidden email]> wrote:
Cheers for the update shay,

I'm more curious about the JsonFieldMapper that is obtained and called
when searching for the default field. i.e. for the mapping:

{
   "stringtest" : {
       "properties" : {
           "name" : {
               "type" : "multi_field",
               "fields" : {
                   "name" : { type : "string", index :
"analyzed" },
                   "orig" : { type : "string", index :
"analyzed" },
                   "untouched" : {type: "string", index :
"not_analyzed"}
               }
           }
       }
   }
}

 CountResponse countResponse1 =
server.client().count(countRequest("test").querySource(fieldQuery("name",
"123"))).actionGet();
 CountResponse countResponse2 =
server.client().count(countRequest("test").querySource(fieldQuery("name.untouched",
"123"))).actionGet();
 CountResponse countResponse3 =
server.client().count(countRequest("test").querySource(fieldQuery("name.orig",
"123"))).actionGet();

upon searching, the method JsonFieldMapper.indexedValue(String) is
called (to format the string for searching I guess).  What I'm seeing
is the following:

 name is assigned JsonLongFieldMapper
 name.orig is assigned JsonStringFieldMapper
 name.untouched is assigned JsonStringFieldMapper

Is this meant to happen, or should it be assigned
JsonStringFieldMapper.  I'm just looking at plugin development in
0.6.0 (which is working well), and this is where my question is
stemming from.


cheers
/dom


On Apr 6, 11:33 pm, Shay Banon <[hidden email]> wrote:
> You are being modest :). There is a bug in the docs (just fixed it). The
> idea is that any mapping unit a multi_field that is not using the same name
> as the multi_field can be accessed by navigating through the multi_field
> name.
>
> So, the docs are wrong, have a look at the corrected ones (the change is to
> rename name_untouched to untouched). In your case, since the extra field(s)
> is called name_untouched, then you navigate to it using name.name_untouched.
> If you change it to untouched, then name.untouched should work.
>
> cheers,
> shay.banon
>
>
>
> On Tue, Apr 6, 2010 at 10:11 PM, dominict <[hidden email]> wrote:
> > Say I have the following mapping :
>
> > {
> >    "stringtest" : {
> >       " properties" : {
> >            "name" : {
> >                "type" : "multi_field",
> >                "fields" : {
> >                    "name" : { type : "string" },
> >                    "name_untouched" : {type: "string"}
> >                }
> >            }
> >        }
> >    }
> > }
>
> > The JsonStringFieldMapper.indexedValue(String value) only gets called
> > for name_untouched, and not name.  Should the JsonStringFieldMapper
> > used for both the default "name" and "name_untouched" fields?
>
> > Mapping setup via:
>
> > server.client().admin().indices().putMapping(putMappingRequest("test").mapp ingSource(mapping)).actionGet();
>
> > Document is indexed via:
>
> > server.client().index(indexRequest("test").type("stringtest").id("1")
> >                .source(jsonBuilder().startObject().field("name",
> > "123").endObject())).actionGet();
>
> > server.client().admin().indices().refresh(refreshRequest()).actionGet();
>
> > then a count is done:
>
> >  CountResponse countResponse =
> > client.count(countRequest("test").querySource(termQuery("name",
> > "123"))).actionGet();
> >  CountResponse countResponse2 =
> > client.count(countRequest("test").querySource(termQuery("name.name_name",
> > "123"))).actionGet();
> >  CountResponse countResponse3 =
>
> > client.count(countRequest("test").querySource(termQuery("name.name_untouche d",
> > "123"))).actionGet();
> >  CountResponse countResponse4 =
> > client.count(countRequest("test").querySource(termQuery("name.name",
> > "123"))).actionGet();
> >  CountResponse countResponse5 =
> > client.count(countRequest("test").querySource(termQuery("name.untouched",
> > "123"))).actionGet();
>
> > of the above 5 count operations only count 1 + 3 return the correct
> > count number of 1.
>
> > Should "name.untouched" also work?
>
> > It's probably a complete mis-reading of
> >http://www.elasticsearch.com/docs/elasticsearch/mapping/multi_field/
> > on my part.
>
> > cheers,
> > /dom

Reply | Threaded
Open this post in threaded view
|

Re: JsonFieldMapper used for default field within a multi_field element

dominict
I'll knock up a test case, and submit an issue.

The plugin I'm looking at is nothing special afraid, just using it as
a nice learning aid.
What I was looking at was the ability to store milliseconds in the
index, but search (and range search) using formatted dates.
I noticed some fields in documents that are being stored in a key/
value store at work, contain timestamps in milliseconds (not sure
why),
but thought it would be handy to be able to find all those docs with
timestamps from today (2010-04-06 to 2010-04-07) for example.
Not a great plugin, I know :-)

cheers
/dom


On Apr 7, 12:29 am, Shay Banon <[hidden email]> wrote:

> No, it should not be a JsonLong one..., can you raise an issue and a
> testcase? Would like to nail this for 0.6 if possible.
>
> Just myself being the usual curious me, what plugin are you working on?
>
> cheers,
> shay.banon
>
>
>
> On Wed, Apr 7, 2010 at 2:27 AM, dominict <[hidden email]> wrote:
> > Cheers for the update shay,
>
> > I'm more curious about the JsonFieldMapper that is obtained and called
> > when searching for the default field. i.e. for the mapping:
>
> > {
> >    "stringtest" : {
> >        "properties" : {
> >            "name" : {
> >                "type" : "multi_field",
> >                "fields" : {
> >                     "name" : { type : "string", index :
> > "analyzed" },
> >                    "orig" : { type : "string", index :
> > "analyzed" },
> >                    "untouched" : {type: "string", index :
> > "not_analyzed"}
> >                }
> >            }
> >        }
> >    }
> > }
>
> >  CountResponse countResponse1 =
> > server.client().count(countRequest("test").querySource(fieldQuery("name",
> > "123"))).actionGet();
> >  CountResponse countResponse2 =
>
> > server.client().count(countRequest("test").querySource(fieldQuery("name.unt ouched",
> > "123"))).actionGet();
> >  CountResponse countResponse3 =
>
> > server.client().count(countRequest("test").querySource(fieldQuery("name.ori g",
> > "123"))).actionGet();
>
> > upon searching, the method JsonFieldMapper.indexedValue(String) is
> > called (to format the string for searching I guess).  What I'm seeing
> > is the following:
>
> >  name is assigned JsonLongFieldMapper
> >  name.orig is assigned JsonStringFieldMapper
> >  name.untouched is assigned JsonStringFieldMapper
>
> > Is this meant to happen, or should it be assigned
> > JsonStringFieldMapper.  I'm just looking at plugin development in
> > 0.6.0 (which is working well), and this is where my question is
> > stemming from.
>
> > cheers
> > /dom
>
> > On Apr 6, 11:33 pm, Shay Banon <[hidden email]> wrote:
> > > You are being modest :). There is a bug in the docs (just fixed it). The
> > > idea is that any mapping unit a multi_field that is not using the same
> > name
> > > as the multi_field can be accessed by navigating through the multi_field
> > > name.
>
> > > So, the docs are wrong, have a look at the corrected ones (the change is
> > to
> > > rename name_untouched to untouched). In your case, since the extra
> > field(s)
> > > is called name_untouched, then you navigate to it using
> > name.name_untouched.
> > > If you change it to untouched, then name.untouched should work.
>
> > > cheers,
> > > shay.banon
>
> > > On Tue, Apr 6, 2010 at 10:11 PM, dominict <[hidden email]>
> > wrote:
> > > > Say I have the following mapping :
>
> > > > {
> > > >    "stringtest" : {
> > > >       " properties" : {
> > > >            "name" : {
> > > >                "type" : "multi_field",
> > > >                "fields" : {
> > > >                    "name" : { type : "string" },
> > > >                    "name_untouched" : {type: "string"}
> > > >                }
> > > >            }
> > > >        }
> > > >    }
> > > > }
>
> > > > The JsonStringFieldMapper.indexedValue(String value) only gets called
> > > > for name_untouched, and not name.  Should the JsonStringFieldMapper
> > > > used for both the default "name" and "name_untouched" fields?
>
> > > > Mapping setup via:
>
> > server.client().admin().indices().putMapping(putMappingRequest("test").mapp
> > ingSource(mapping)).actionGet();
>
> > > > Document is indexed via:
>
> > > > server.client().index(indexRequest("test").type("stringtest").id("1")
> > > >                .source(jsonBuilder().startObject().field("name",
> > > > "123").endObject())).actionGet();
>
> > server.client().admin().indices().refresh(refreshRequest()).actionGet();
>
> > > > then a count is done:
>
> > > >  CountResponse countResponse =
> > > > client.count(countRequest("test").querySource(termQuery("name",
> > > > "123"))).actionGet();
> > > >  CountResponse countResponse2 =
>
> > client.count(countRequest("test").querySource(termQuery("name.name_name",
> > > > "123"))).actionGet();
> > > >  CountResponse countResponse3 =
>
> > client.count(countRequest("test").querySource(termQuery("name.name_untouche
> > d",
> > > > "123"))).actionGet();
> > > >  CountResponse countResponse4 =
> > > > client.count(countRequest("test").querySource(termQuery("name.name",
> > > > "123"))).actionGet();
> > > >  CountResponse countResponse5 =
>
> > client.count(countRequest("test").querySource(termQuery("name.untouched",
> > > > "123"))).actionGet();
>
> > > > of the above 5 count operations only count 1 + 3 return the correct
> > > > count number of 1.
>
> > > > Should "name.untouched" also work?
>
> > > > It's probably a complete mis-reading of
> > > >http://www.elasticsearch.com/docs/elasticsearch/mapping/multi_field/
> > > > on my part.
>
> > > > cheers,
> > > > /dom
Reply | Threaded
Open this post in threaded view
|

Re: JsonFieldMapper used for default field within a multi_field element

dominict
Hi there,

It looks as though the only issue at the moment is between the chair
and keyboard (i.e. me).
And the only place I'm seeing issue, is with the plugin I'm working
on.  So please consider it a non issue.
I'll comment back if I am ever able to reproduce, in a context other
that my own plugin.  Apologies for taking up your time.
Cheers for the update to the docs on the multi_field.

/dom

On Apr 7, 12:57 am, dominict <[hidden email]> wrote:

> I'll knock up a test case, and submit an issue.
>
> The plugin I'm looking at is nothing special afraid, just using it as
> a nice learning aid.
> What I was looking at was the ability to store milliseconds in the
> index, but search (and range search) using formatted dates.
> I noticed some fields in documents that are being stored in a key/
> value store at work, contain timestamps in milliseconds (not sure
> why),
> but thought it would be handy to be able to find all those docs with
> timestamps from today (2010-04-06 to 2010-04-07) for example.
> Not a great plugin, I know :-)
>
> cheers
> /dom
>
> On Apr 7, 12:29 am, Shay Banon <[hidden email]> wrote:
>
>
>
> > No, it should not be a JsonLong one..., can you raise an issue and a
> > testcase? Would like to nail this for 0.6 if possible.
>
> > Just myself being the usual curious me, what plugin are you working on?
>
> > cheers,
> > shay.banon
>
> > On Wed, Apr 7, 2010 at 2:27 AM, dominict <[hidden email]> wrote:
> > > Cheers for the update shay,
>
> > > I'm more curious about the JsonFieldMapper that is obtained and called
> > > when searching for the default field. i.e. for the mapping:
>
> > > {
> > >    "stringtest" : {
> > >        "properties" : {
> > >            "name" : {
> > >                "type" : "multi_field",
> > >                "fields" : {
> > >                     "name" : { type : "string", index :
> > > "analyzed" },
> > >                    "orig" : { type : "string", index :
> > > "analyzed" },
> > >                    "untouched" : {type: "string", index :
> > > "not_analyzed"}
> > >                }
> > >            }
> > >        }
> > >    }
> > > }
>
> > >  CountResponse countResponse1 =
> > > server.client().count(countRequest("test").querySource(fieldQuery("name",
> > > "123"))).actionGet();
> > >  CountResponse countResponse2 =
>
> > > server.client().count(countRequest("test").querySource(fieldQuery("name.unt ouched",
> > > "123"))).actionGet();
> > >  CountResponse countResponse3 =
>
> > > server.client().count(countRequest("test").querySource(fieldQuery("name.ori g",
> > > "123"))).actionGet();
>
> > > upon searching, the method JsonFieldMapper.indexedValue(String) is
> > > called (to format the string for searching I guess).  What I'm seeing
> > > is the following:
>
> > >  name is assigned JsonLongFieldMapper
> > >  name.orig is assigned JsonStringFieldMapper
> > >  name.untouched is assigned JsonStringFieldMapper
>
> > > Is this meant to happen, or should it be assigned
> > > JsonStringFieldMapper.  I'm just looking at plugin development in
> > > 0.6.0 (which is working well), and this is where my question is
> > > stemming from.
>
> > > cheers
> > > /dom
>
> > > On Apr 6, 11:33 pm, Shay Banon <[hidden email]> wrote:
> > > > You are being modest :). There is a bug in the docs (just fixed it). The
> > > > idea is that any mapping unit a multi_field that is not using the same
> > > name
> > > > as the multi_field can be accessed by navigating through the multi_field
> > > > name.
>
> > > > So, the docs are wrong, have a look at the corrected ones (the change is
> > > to
> > > > rename name_untouched to untouched). In your case, since the extra
> > > field(s)
> > > > is called name_untouched, then you navigate to it using
> > > name.name_untouched.
> > > > If you change it to untouched, then name.untouched should work.
>
> > > > cheers,
> > > > shay.banon
>
> > > > On Tue, Apr 6, 2010 at 10:11 PM, dominict <[hidden email]>
> > > wrote:
> > > > > Say I have the following mapping :
>
> > > > > {
> > > > >    "stringtest" : {
> > > > >       " properties" : {
> > > > >            "name" : {
> > > > >                "type" : "multi_field",
> > > > >                "fields" : {
> > > > >                    "name" : { type : "string" },
> > > > >                    "name_untouched" : {type: "string"}
> > > > >                }
> > > > >            }
> > > > >        }
> > > > >    }
> > > > > }
>
> > > > > The JsonStringFieldMapper.indexedValue(String value) only gets called
> > > > > for name_untouched, and not name.  Should the JsonStringFieldMapper
> > > > > used for both the default "name" and "name_untouched" fields?
>
> > > > > Mapping setup via:
>
> > > server.client().admin().indices().putMapping(putMappingRequest("test").mapp
> > > ingSource(mapping)).actionGet();
>
> > > > > Document is indexed via:
>
> > > > > server.client().index(indexRequest("test").type("stringtest").id("1")
> > > > >                .source(jsonBuilder().startObject().field("name",
> > > > > "123").endObject())).actionGet();
>
> > > server.client().admin().indices().refresh(refreshRequest()).actionGet();
>
> > > > > then a count is done:
>
> > > > >  CountResponse countResponse =
> > > > > client.count(countRequest("test").querySource(termQuery("name",
> > > > > "123"))).actionGet();
> > > > >  CountResponse countResponse2 =
>
> > > client.count(countRequest("test").querySource(termQuery("name.name_name",
> > > > > "123"))).actionGet();
> > > > >  CountResponse countResponse3 =
>
> > > client.count(countRequest("test").querySource(termQuery("name.name_untouche
> > > d",
> > > > > "123"))).actionGet();
> > > > >  CountResponse countResponse4 =
> > > > > client.count(countRequest("test").querySource(termQuery("name.name",
> > > > > "123"))).actionGet();
> > > > >  CountResponse countResponse5 =
>
> > > client.count(countRequest("test").querySource(termQuery("name.untouched",
> > > > > "123"))).actionGet();
>
> > > > > of the above 5 count operations only count 1 + 3 return the correct
> > > > > count number of 1.
>
> > > > > Should "name.untouched" also work?
>
> > > > > It's probably a complete mis-reading of
> > > > >http://www.elasticsearch.com/docs/elasticsearch/mapping/multi_field/
> > > > > on my part.
>
> > > > > cheers,
> > > > > /dom
Reply | Threaded
Open this post in threaded view
|

Re: JsonFieldMapper used for default field within a multi_field element

kimchy
Administrator
In reply to this post by dominict
What you are working on is actually a very valid feature, and I was thinking about just that the other day. It should be simple to allow for the date type of handle long as well as strings (that represent the date). What I was not so sure about is what it should return if stores (simplest I was thinking about it formatted dates), and what should be stored in the _all field?

cheers,
shay.banon

On Wed, Apr 7, 2010 at 2:57 AM, dominict <[hidden email]> wrote:
I'll knock up a test case, and submit an issue.

The plugin I'm looking at is nothing special afraid, just using it as
a nice learning aid.
What I was looking at was the ability to store milliseconds in the
index, but search (and range search) using formatted dates.
I noticed some fields in documents that are being stored in a key/
value store at work, contain timestamps in milliseconds (not sure
why),
but thought it would be handy to be able to find all those docs with
timestamps from today (2010-04-06 to 2010-04-07) for example.
Not a great plugin, I know :-)

cheers
/dom


On Apr 7, 12:29 am, Shay Banon <[hidden email]> wrote:
> No, it should not be a JsonLong one..., can you raise an issue and a
> testcase? Would like to nail this for 0.6 if possible.
>
> Just myself being the usual curious me, what plugin are you working on?
>
> cheers,
> shay.banon
>
>
>
> On Wed, Apr 7, 2010 at 2:27 AM, dominict <[hidden email]> wrote:
> > Cheers for the update shay,
>
> > I'm more curious about the JsonFieldMapper that is obtained and called
> > when searching for the default field. i.e. for the mapping:
>
> > {
> >    "stringtest" : {
> >        "properties" : {
> >            "name" : {
> >                "type" : "multi_field",
> >                "fields" : {
> >                     "name" : { type : "string", index :
> > "analyzed" },
> >                    "orig" : { type : "string", index :
> > "analyzed" },
> >                    "untouched" : {type: "string", index :
> > "not_analyzed"}
> >                }
> >            }
> >        }
> >    }
> > }
>
> >  CountResponse countResponse1 =
> > server.client().count(countRequest("test").querySource(fieldQuery("name",
> > "123"))).actionGet();
> >  CountResponse countResponse2 =
>
> > server.client().count(countRequest("test").querySource(fieldQuery("name.unt ouched",
> > "123"))).actionGet();
> >  CountResponse countResponse3 =
>
> > server.client().count(countRequest("test").querySource(fieldQuery("name.ori g",
> > "123"))).actionGet();
>
> > upon searching, the method JsonFieldMapper.indexedValue(String) is
> > called (to format the string for searching I guess).  What I'm seeing
> > is the following:
>
> >  name is assigned JsonLongFieldMapper
> >  name.orig is assigned JsonStringFieldMapper
> >  name.untouched is assigned JsonStringFieldMapper
>
> > Is this meant to happen, or should it be assigned
> > JsonStringFieldMapper.  I'm just looking at plugin development in
> > 0.6.0 (which is working well), and this is where my question is
> > stemming from.
>
> > cheers
> > /dom
>
> > On Apr 6, 11:33 pm, Shay Banon <[hidden email]> wrote:
> > > You are being modest :). There is a bug in the docs (just fixed it). The
> > > idea is that any mapping unit a multi_field that is not using the same
> > name
> > > as the multi_field can be accessed by navigating through the multi_field
> > > name.
>
> > > So, the docs are wrong, have a look at the corrected ones (the change is
> > to
> > > rename name_untouched to untouched). In your case, since the extra
> > field(s)
> > > is called name_untouched, then you navigate to it using
> > name.name_untouched.
> > > If you change it to untouched, then name.untouched should work.
>
> > > cheers,
> > > shay.banon
>
> > > On Tue, Apr 6, 2010 at 10:11 PM, dominict <[hidden email]>
> > wrote:
> > > > Say I have the following mapping :
>
> > > > {
> > > >    "stringtest" : {
> > > >       " properties" : {
> > > >            "name" : {
> > > >                "type" : "multi_field",
> > > >                "fields" : {
> > > >                    "name" : { type : "string" },
> > > >                    "name_untouched" : {type: "string"}
> > > >                }
> > > >            }
> > > >        }
> > > >    }
> > > > }
>
> > > > The JsonStringFieldMapper.indexedValue(String value) only gets called
> > > > for name_untouched, and not name.  Should the JsonStringFieldMapper
> > > > used for both the default "name" and "name_untouched" fields?
>
> > > > Mapping setup via:
>
> > server.client().admin().indices().putMapping(putMappingRequest("test").mapp
> > ingSource(mapping)).actionGet();
>
> > > > Document is indexed via:
>
> > > > server.client().index(indexRequest("test").type("stringtest").id("1")
> > > >                .source(jsonBuilder().startObject().field("name",
> > > > "123").endObject())).actionGet();
>
> > server.client().admin().indices().refresh(refreshRequest()).actionGet();
>
> > > > then a count is done:
>
> > > >  CountResponse countResponse =
> > > > client.count(countRequest("test").querySource(termQuery("name",
> > > > "123"))).actionGet();
> > > >  CountResponse countResponse2 =
>
> > client.count(countRequest("test").querySource(termQuery("name.name_name",
> > > > "123"))).actionGet();
> > > >  CountResponse countResponse3 =
>
> > client.count(countRequest("test").querySource(termQuery("name.name_untouche
> > d",
> > > > "123"))).actionGet();
> > > >  CountResponse countResponse4 =
> > > > client.count(countRequest("test").querySource(termQuery("name.name",
> > > > "123"))).actionGet();
> > > >  CountResponse countResponse5 =
>
> > client.count(countRequest("test").querySource(termQuery("name.untouched",
> > > > "123"))).actionGet();
>
> > > > of the above 5 count operations only count 1 + 3 return the correct
> > > > count number of 1.
>
> > > > Should "name.untouched" also work?
>
> > > > It's probably a complete mis-reading of
> > > >http://www.elasticsearch.com/docs/elasticsearch/mapping/multi_field/
> > > > on my part.
>
> > > > cheers,
> > > > /dom

Reply | Threaded
Open this post in threaded view
|

Re: JsonFieldMapper used for default field within a multi_field element

dominict
I managed to create a test case this morning with Date and String
types.  Not sure what was confusing me last night.
I've created the issue, with code sample:
http://github.com/elasticsearch/elasticsearch/issues/issue/123

I hadn't really thought much into what would get stored in the _all
field.  What I was mulling over in my head was,
if someone searched for 20100407, and the field stored the millis for
say now 1270640073794; they wouldn't match
as 20100407 == 1270594800794.

I was going to have a 2 formats associated with the type, so that the
timestamp, was chomped to say the start of the day

{type: "millis", store_format : "basic_date", search_format :
"YYYYMMDD:HHmmss" }


Then is someone searched for 20100407 or 20100407:124304 then both
search terms would be reduced to the store_format.

Hadn't quite worked out in my head how to actually search for things
for say: YYYYMMDD:HH, but that is where I was going with the
multi_fields investigation.

cheers
/dom

On Apr 7, 9:26 am, Shay Banon <[hidden email]> wrote:

> What you are working on is actually a very valid feature, and I was thinking
> about just that the other day. It should be simple to allow for the date
> type of handle long as well as strings (that represent the date). What I was
> not so sure about is what it should return if stores (simplest I was
> thinking about it formatted dates), and what should be stored in the _all
> field?
>
> cheers,
> shay.banon
>
>
>
> On Wed, Apr 7, 2010 at 2:57 AM, dominict <[hidden email]> wrote:
> > I'll knock up a test case, and submit an issue.
>
> > The plugin I'm looking at is nothing special afraid, just using it as
> > a nice learning aid.
> > What I was looking at was the ability to store milliseconds in the
> > index, but search (and range search) using formatted dates.
> > I noticed some fields in documents that are being stored in a key/
> > value store at work, contain timestamps in milliseconds (not sure
> > why),
> > but thought it would be handy to be able to find all those docs with
> > timestamps from today (2010-04-06 to 2010-04-07) for example.
> > Not a great plugin, I know :-)
>
> > cheers
> > /dom
>
> > On Apr 7, 12:29 am, Shay Banon <[hidden email]> wrote:
> > > No, it should not be a JsonLong one..., can you raise an issue and a
> > > testcase? Would like to nail this for 0.6 if possible.
>
> > > Just myself being the usual curious me, what plugin are you working on?
>
> > > cheers,
> > > shay.banon
>
> > > On Wed, Apr 7, 2010 at 2:27 AM, dominict <[hidden email]>
> > wrote:
> > > > Cheers for the update shay,
>
> > > > I'm more curious about the JsonFieldMapper that is obtained and called
> > > > when searching for the default field. i.e. for the mapping:
>
> > > > {
> > > >    "stringtest" : {
> > > >        "properties" : {
> > > >            "name" : {
> > > >                "type" : "multi_field",
> > > >                "fields" : {
> > > >                     "name" : { type : "string", index :
> > > > "analyzed" },
> > > >                    "orig" : { type : "string", index :
> > > > "analyzed" },
> > > >                    "untouched" : {type: "string", index :
> > > > "not_analyzed"}
> > > >                }
> > > >            }
> > > >        }
> > > >    }
> > > > }
>
> > > >  CountResponse countResponse1 =
>
> > server.client().count(countRequest("test").querySource(fieldQuery("name",
> > > > "123"))).actionGet();
> > > >  CountResponse countResponse2 =
>
> > server.client().count(countRequest("test").querySource(fieldQuery("name.unt
> > ouched",
> > > > "123"))).actionGet();
> > > >  CountResponse countResponse3 =
>
> > server.client().count(countRequest("test").querySource(fieldQuery("name.ori
> > g",
> > > > "123"))).actionGet();
>
> > > > upon searching, the method JsonFieldMapper.indexedValue(String) is
> > > > called (to format the string for searching I guess).  What I'm seeing
> > > > is the following:
>
> > > >  name is assigned JsonLongFieldMapper
> > > >  name.orig is assigned JsonStringFieldMapper
> > > >  name.untouched is assigned JsonStringFieldMapper
>
> > > > Is this meant to happen, or should it be assigned
> > > > JsonStringFieldMapper.  I'm just looking at plugin development in
> > > > 0.6.0 (which is working well), and this is where my question is
> > > > stemming from.
>
> > > > cheers
> > > > /dom
>
> > > > On Apr 6, 11:33 pm, Shay Banon <[hidden email]> wrote:
> > > > > You are being modest :). There is a bug in the docs (just fixed it).
> > The
> > > > > idea is that any mapping unit a multi_field that is not using the
> > same
> > > > name
> > > > > as the multi_field can be accessed by navigating through the
> > multi_field
> > > > > name.
>
> > > > > So, the docs are wrong, have a look at the corrected ones (the change
> > is
> > > > to
> > > > > rename name_untouched to untouched). In your case, since the extra
> > > > field(s)
> > > > > is called name_untouched, then you navigate to it using
> > > > name.name_untouched.
> > > > > If you change it to untouched, then name.untouched should work.
>
> > > > > cheers,
> > > > > shay.banon
>
> > > > > On Tue, Apr 6, 2010 at 10:11 PM, dominict <[hidden email]
>
> > > > wrote:
> > > > > > Say I have the following mapping :
>
> > > > > > {
> > > > > >    "stringtest" : {
> > > > > >       " properties" : {
> > > > > >            "name" : {
> > > > > >                "type" : "multi_field",
> > > > > >                "fields" : {
> > > > > >                    "name" : { type : "string" },
> > > > > >                    "name_untouched" : {type: "string"}
> > > > > >                }
> > > > > >            }
> > > > > >        }
> > > > > >    }
> > > > > > }
>
> > > > > > The JsonStringFieldMapper.indexedValue(String value) only gets
> > called
> > > > > > for name_untouched, and not name.  Should the JsonStringFieldMapper
> > > > > > used for both the default "name" and "name_untouched" fields?
>
> > > > > > Mapping setup via:
>
> > server.client().admin().indices().putMapping(putMappingRequest("test").mapp
> > > > ingSource(mapping)).actionGet();
>
> > > > > > Document is indexed via:
>
> > server.client().index(indexRequest("test").type("stringtest").id("1")
> > > > > >                .source(jsonBuilder().startObject().field("name",
> > > > > > "123").endObject())).actionGet();
>
> > server.client().admin().indices().refresh(refreshRequest()).actionGet();
>
> > > > > > then a count is done:
>
> > > > > >  CountResponse countResponse =
> > > > > > client.count(countRequest("test").querySource(termQuery("name",
> > > > > > "123"))).actionGet();
> > > > > >  CountResponse countResponse2 =
>
> > client.count(countRequest("test").querySource(termQuery("name.name_name",
> > > > > > "123"))).actionGet();
> > > > > >  CountResponse countResponse3 =
>
> > client.count(countRequest("test").querySource(termQuery("name.name_untouche
> > > > d",
> > > > > > "123"))).actionGet();
> > > > > >  CountResponse countResponse4 =
> > > > > > client.count(countRequest("test").querySource(termQuery("name.name
> > ",
> > > > > > "123"))).actionGet();
> > > > > >  CountResponse countResponse5 =
>
> > client.count(countRequest("test").querySource(termQuery("name.untouched",
> > > > > > "123"))).actionGet();
>
> > > > > > of the above 5 count operations only count 1 + 3 return the correct
> > > > > > count number of 1.
>
> > > > > > Should "name.untouched" also work?
>
> > > > > > It's probably a complete mis-reading of
>
> >http://www.elasticsearch.com/docs/elasticsearch/mapping/multi_field/
> > > > > > on my part.
>
> > > > > > cheers,
> > > > > > /dom