Joda date format error

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Joda date format error

Mahendra M
Hi,

While applying mappings for a date field, I tried the following in the
date_formats field.
"MM/yyyy", "yyyy/MM/dd HH:mm:ss"

When I submitted content for indexing, this format was generating an error.

java.lang.IllegalArgumentException: Invalid format: "05/2010" is
malformed at "/2010"
        at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:634)

In the second case, it was "2010/05/03 10:43:00" is malformed at "/05/03"

I noticed that this error is coming up for formats where a slash (/)
is used like yyyy/MM/dd or MM/yyyy

From the Joda documentation and some examples, I figured that formats
like this are valid.

Could this possibly be a bug somewhere or am I missing something ?

Regards,
Mahendra

http://twitter.com/mahendra
Reply | Threaded
Open this post in threaded view
|

Re: Joda date format error

kimchy
Administrator
Hi,

  Can you post the mapping you use?

cheers,
shay.banon

On May 3, 4:40 pm, Mahendra M <[hidden email]> wrote:

> Hi,
>
> While applying mappings for a date field, I tried the following in the
> date_formats field.
> "MM/yyyy", "yyyy/MM/dd HH:mm:ss"
>
> When I submitted content for indexing, this format was generating an error.
>
> java.lang.IllegalArgumentException: Invalid format: "05/2010" is
> malformed at "/2010"
>         at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:6 34)
>
> In the second case, it was "2010/05/03 10:43:00" is malformed at "/05/03"
>
> I noticed that this error is coming up for formats where a slash (/)
> is used like yyyy/MM/dd or MM/yyyy
>
> From the Joda documentation and some examples, I figured that formats
> like this are valid.
>
> Could this possibly be a bug somewhere or am I missing something ?
>
> Regards,
> Mahendra
>
> http://twitter.com/mahendra
Reply | Threaded
Open this post in threaded view
|

Re: Joda date format error

Mahendra M
On Mon, May 3, 2010 at 7:39 PM, Shay Banon <[hidden email]> wrote:
> Hi,
>
>  Can you post the mapping you use?

Hi,

The JSON is as follows - this is a test one. The field 'genre' is
marked as date.

{
  "a" : {
    "type" : "object",
    "dynamic" : false,
    "enabled" : true,
    "path" : "full",
    "date_formats" : [ "yyyy/MM/dd" ],
    "_boost" : {
      "name" : "_boost"
    },
    "_id" : {
      "store" : "no"
    },
    "_all" : {
      "enabled" : true,
      "store" : "no",
      "term_vector" : "no"
    },
    "_source" : {
      "name" : "_source",
      "enabled" : true
    },
    "properties" : {
      "genre" : {
        "type" : "date",
        "date_formats" : [ "yyyy/MM/dd" ],
        "index_name" : "genre",
        "index" : "analyzed",
        "store" : "no",
        "term_vector" : "no",
        "boost" : 1.0,
        "omit_norms" : false,
        "include_in_all": false,
        "omit_term_freq_and_positions" : false
      },
      "sub_genre" : {
        "type" : "string",
        "index_name" : "sub_genre",
        "index" : "analyzed",
        "store" : "no",
        "term_vector" : "no",
        "boost" : 1.0,
        "omit_norms" : false,
        "include_in_all": false,
        "omit_term_freq_and_positions" : false
      }
    }
  }
}

The following content fails here:
{
    "genre": "2010/05/03",
    "sub_genre": "test"
}

PS: I am trying this on elasticsearch 0.6

Regards,
Mahendra

http://twitter.com/mahendra
Reply | Threaded
Open this post in threaded view
|

Re: Joda date format error

Mahendra M
Hi,

How I made the JSON was like this.

* Index a document without any mappings.
* Check elasticsearch logs for the mappings that were generated dynamically.
* Modify the mappings according to my needs.

I hope this is the correct approach..

Regards,
Mahendra

On Mon, May 3, 2010 at 9:57 PM, Mahendra M <[hidden email]> wrote:

> Hi,
>
> The JSON is as follows - this is a test one. The field 'genre' is
> marked as date.
>
> {
>  "a" : {
>    "type" : "object",
>    "dynamic" : false,
>    "enabled" : true,
>    "path" : "full",
>    "date_formats" : [ "yyyy/MM/dd" ],
>    "_boost" : {
>      "name" : "_boost"
>    },
>    "_id" : {
>      "store" : "no"
>    },
>    "_all" : {
>      "enabled" : true,
>      "store" : "no",
>      "term_vector" : "no"
>    },
>    "_source" : {
>      "name" : "_source",
>      "enabled" : true
>    },
>    "properties" : {
>      "genre" : {
>        "type" : "date",
>        "date_formats" : [ "yyyy/MM/dd" ],
>        "index_name" : "genre",
>        "index" : "analyzed",
>        "store" : "no",
>        "term_vector" : "no",
>        "boost" : 1.0,
>        "omit_norms" : false,
>        "include_in_all": false,
>        "omit_term_freq_and_positions" : false
>      },
>      "sub_genre" : {
>        "type" : "string",
>        "index_name" : "sub_genre",
>        "index" : "analyzed",
>        "store" : "no",
>        "term_vector" : "no",
>        "boost" : 1.0,
>        "omit_norms" : false,
>        "include_in_all": false,
>        "omit_term_freq_and_positions" : false
>      }
>    }
>  }
> }
>
> The following content fails here:
> {
>    "genre": "2010/05/03",
>    "sub_genre": "test"
> }
>
> PS: I am trying this on elasticsearch 0.6
>
> Regards,
> Mahendra
>
> http://twitter.com/mahendra
>
Reply | Threaded
Open this post in threaded view
|

Re: Joda date format error

kimchy
Administrator
Hi, 

  The mapping for specifying a format on a date field is called 'format' not 'data_formats'. The 'date_formats' only applies to the object mapping, and is relevant when a new field is introduced that has no mappings in order to try and derive if its a date type or not.

  Regarding the automatic detection of date fields based on the object mapping, the format you specified in the `date_formats` section still won't make date types to be detected properly. This is since I also check for the existence of '-' or ':' in the string. This is because the date parser is too lenient and manages to parse things that should be string with just numbers in them ;). I will add the '/' to it as well while I think of a better way to work around this...

cheers,
shay.banon

On Tue, May 4, 2010 at 11:19 AM, Mahendra M <[hidden email]> wrote:
Hi,

How I made the JSON was like this.

* Index a document without any mappings.
* Check elasticsearch logs for the mappings that were generated dynamically.
* Modify the mappings according to my needs.

I hope this is the correct approach..

Regards,
Mahendra

On Mon, May 3, 2010 at 9:57 PM, Mahendra M <[hidden email]> wrote:
> Hi,
>
> The JSON is as follows - this is a test one. The field 'genre' is
> marked as date.
>
> {
>  "a" : {
>    "type" : "object",
>    "dynamic" : false,
>    "enabled" : true,
>    "path" : "full",
>    "date_formats" : [ "yyyy/MM/dd" ],
>    "_boost" : {
>      "name" : "_boost"
>    },
>    "_id" : {
>      "store" : "no"
>    },
>    "_all" : {
>      "enabled" : true,
>      "store" : "no",
>      "term_vector" : "no"
>    },
>    "_source" : {
>      "name" : "_source",
>      "enabled" : true
>    },
>    "properties" : {
>      "genre" : {
>        "type" : "date",
>        "date_formats" : [ "yyyy/MM/dd" ],
>        "index_name" : "genre",
>        "index" : "analyzed",
>        "store" : "no",
>        "term_vector" : "no",
>        "boost" : 1.0,
>        "omit_norms" : false,
>        "include_in_all": false,
>        "omit_term_freq_and_positions" : false
>      },
>      "sub_genre" : {
>        "type" : "string",
>        "index_name" : "sub_genre",
>        "index" : "analyzed",
>        "store" : "no",
>        "term_vector" : "no",
>        "boost" : 1.0,
>        "omit_norms" : false,
>        "include_in_all": false,
>        "omit_term_freq_and_positions" : false
>      }
>    }
>  }
> }
>
> The following content fails here:
> {
>    "genre": "2010/05/03",
>    "sub_genre": "test"
> }
>
> PS: I am trying this on elasticsearch 0.6
>
> Regards,
> Mahendra
>
> http://twitter.com/mahendra
>

Reply | Threaded
Open this post in threaded view
|

Re: Joda date format error

Mahendra M
On Tue, May 4, 2010 at 5:56 PM, Shay Banon <[hidden email]> wrote:
> I will add the '/' to it as well while I think of a
> better way to work around this...

That would be cool. I am actually writing a couchdb module to
integrate it with elasticsearch (will apply to my company for
opensourcing it). All of couchdb examples and documents use the date
fields with a '/' (saying parsing it is easy in JS)

Format is : yyyy/MM/dd HH:mm:ss +SSSS

Having support for this will make couchdb integration a lot smoother.

Regards,
Mahendra

http://twitter.com/mahendra
Reply | Threaded
Open this post in threaded view
|

Re: Joda date format error

kimchy
Administrator
Already in master (the support for auto detection of date fields with '/'), but, still the default date_formats for objects is the ISO one, we can make the integration even smoother if I add this format to the list of default date_formats if none are specified... .

On Wed, May 5, 2010 at 9:16 AM, Mahendra M <[hidden email]> wrote:
On Tue, May 4, 2010 at 5:56 PM, Shay Banon <[hidden email]> wrote:
> I will add the '/' to it as well while I think of a
> better way to work around this...

That would be cool. I am actually writing a couchdb module to
integrate it with elasticsearch (will apply to my company for
opensourcing it). All of couchdb examples and documents use the date
fields with a '/' (saying parsing it is easy in JS)

Format is : yyyy/MM/dd HH:mm:ss +SSSS

Having support for this will make couchdb integration a lot smoother.

Regards,
Mahendra

http://twitter.com/mahendra

Reply | Threaded
Open this post in threaded view
|

Re: Joda date format error

kimchy
Administrator
By the way, where in couchdb does is this format specified, I can't find it. It seems like it all depends on how you input the string that represents the date...

On Wed, May 5, 2010 at 11:27 AM, Shay Banon <[hidden email]> wrote:
Already in master (the support for auto detection of date fields with '/'), but, still the default date_formats for objects is the ISO one, we can make the integration even smoother if I add this format to the list of default date_formats if none are specified... .


On Wed, May 5, 2010 at 9:16 AM, Mahendra M <[hidden email]> wrote:
On Tue, May 4, 2010 at 5:56 PM, Shay Banon <[hidden email]> wrote:
> I will add the '/' to it as well while I think of a
> better way to work around this...

That would be cool. I am actually writing a couchdb module to
integrate it with elasticsearch (will apply to my company for
opensourcing it). All of couchdb examples and documents use the date
fields with a '/' (saying parsing it is easy in JS)

Format is : yyyy/MM/dd HH:mm:ss +SSSS

Having support for this will make couchdb integration a lot smoother.

Regards,
Mahendra

http://twitter.com/mahendra


Reply | Threaded
Open this post in threaded view
|

Re: Joda date format error

Mahendra M
On Wed, May 5, 2010 at 2:11 PM, Shay Banon <[hidden email]> wrote:
> By the way, where in couchdb does is this format specified, I can't find it.
> It seems like it all depends on how you input the string that represents the
> date...

It is not mandatory. The examples are mostly in "/" format. In
couchdb, the sorting happens based on string comparison. So, it does
not matter if it is /,- or :

You can see some examples here:
http://books.couchdb.org/relax/design-documents/views

--
Mahendra

http://twitter.com/mahendra
Reply | Threaded
Open this post in threaded view
|

Re: Joda date format error

kimchy
Administrator
Committed: http://github.com/elasticsearch/elasticsearch/issues/issue/159.

On Wed, May 5, 2010 at 1:19 PM, Mahendra M <[hidden email]> wrote:
On Wed, May 5, 2010 at 2:11 PM, Shay Banon <[hidden email]> wrote:
> By the way, where in couchdb does is this format specified, I can't find it.
> It seems like it all depends on how you input the string that represents the
> date...

It is not mandatory. The examples are mostly in "/" format. In
couchdb, the sorting happens based on string comparison. So, it does
not matter if it is /,- or :

You can see some examples here:
http://books.couchdb.org/relax/design-documents/views

--