leave content in mySQL and use ElasticSearch only for Index

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

leave content in mySQL and use ElasticSearch only for Index

aseknoppik
Hello,

I want to use Elasticsearch or only indexing and searching E-Mails. We want to store the meta-info within Elasticsearch, keeping the content/body of every Mail in an mySQL database. So Elasticsearch shall have a reference to the mail body.

Is that possible and how?

Regards
Michael

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/928f30a8-31fd-456c-a527-e2e51ebd69f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: leave content in mySQL and use ElasticSearch only for Index

joergprante@gmail.com
Have a look at the JDBC plugin. With that plugin, you can push metadata from MySQL to Elasticsearch.


Jörg


On Wed, Aug 6, 2014 at 1:21 PM, <[hidden email]> wrote:
Hello,

I want to use Elasticsearch or only indexing and searching E-Mails. We want to store the meta-info within Elasticsearch, keeping the content/body of every Mail in an mySQL database. So Elasticsearch shall have a reference to the mail body.

Is that possible and how?

Regards
Michael

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/928f30a8-31fd-456c-a527-e2e51ebd69f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGP6w%2BxS%2BQLoBR9r2q6tz_iz1rez1zAE%2B3j56WgFnSTKQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: leave content in mySQL and use ElasticSearch only for Index

aseknoppik
In reply to this post by aseknoppik
Using this plugin would lead to a migration from mysql data into Elasticsearch.

So let me reformulate my question:

My infrastructure is like this:

client---->Elasticsearch
   |
   |
    -------->mySQL

So I have a client which generates an index and some metadata for a mail(header and body). The mail is stored in mySQL. And the client-side generated index and metadata is stored in Elasticsearch.

The reason is because I have > 1 TB of mail content every day. This content shall still be written to mySQL. Elasticsearch shall keep only the index. Is that possible? And how?

Regards
Michael


Am Mittwoch, 6. August 2014 13:21:09 UTC+2 schrieb [hidden email]:
Hello,

I want to use Elasticsearch or only indexing and searching E-Mails. We want to store the meta-info within Elasticsearch, keeping the content/body of every Mail in an mySQL database. So Elasticsearch shall have a reference to the mail body.

Is that possible and how?

Regards
Michael

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bb04102a-9aca-469d-8145-d5e644481d39%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: leave content in mySQL and use ElasticSearch only for Index

Andrej
What I don't understand is why you generate an index and want to store it in elasticsearch. You could use the plugin as Jörg suggested, transfer you data to elasticsearch, set index:true for the fields you want and set store:false in the mapping. This way you get an index build by elasticsearch, can search on it, get the id as result and the data is not stored (except metadata, if you set it to be stored). See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#mapping-core-types.

Cheers,
Andrej

Am Mittwoch, 6. August 2014 15:34:11 UTC+2 schrieb [hidden email]:
Using this plugin would lead to a migration from mysql data into Elasticsearch.

So let me reformulate my question:

My infrastructure is like this:

client---->Elasticsearch
   |
   |
    -------->mySQL

So I have a client which generates an index and some metadata for a mail(header and body). The mail is stored in mySQL. And the client-side generated index and metadata is stored in Elasticsearch.

The reason is because I have > 1 TB of mail content every day. This content shall still be written to mySQL. Elasticsearch shall keep only the index. Is that possible? And how?

Regards
Michael


Am Mittwoch, 6. August 2014 13:21:09 UTC+2 schrieb [hidden email]:
Hello,

I want to use Elasticsearch or only indexing and searching E-Mails. We want to store the meta-info within Elasticsearch, keeping the content/body of every Mail in an mySQL database. So Elasticsearch shall have a reference to the mail body.

Is that possible and how?

Regards
Michael

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d1ffc059-abad-4b11-8179-35ed3c077cbf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: leave content in mySQL and use ElasticSearch only for Index

joergprante@gmail.com
JDBC plugin is not for migration.

It can be configured to select the data from the RDBMS you want. You can fetch the metadata fields and index them into Elasticsearch with a simple SQL select statement.

Jörg 


On Wed, Aug 6, 2014 at 3:48 PM, Andrej Rosenheinrich <[hidden email]> wrote:
What I don't understand is why you generate an index and want to store it in elasticsearch. You could use the plugin as Jörg suggested, transfer you data to elasticsearch, set index:true for the fields you want and set store:false in the mapping. This way you get an index build by elasticsearch, can search on it, get the id as result and the data is not stored (except metadata, if you set it to be stored). See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#mapping-core-types.

Cheers,
Andrej

Am Mittwoch, 6. August 2014 15:34:11 UTC+2 schrieb [hidden email]:
Using this plugin would lead to a migration from mysql data into Elasticsearch.

So let me reformulate my question:

My infrastructure is like this:

client---->Elasticsearch
   |
   |
    -------->mySQL

So I have a client which generates an index and some metadata for a mail(header and body). The mail is stored in mySQL. And the client-side generated index and metadata is stored in Elasticsearch.

The reason is because I have > 1 TB of mail content every day. This content shall still be written to mySQL. Elasticsearch shall keep only the index. Is that possible? And how?

Regards
Michael


Am Mittwoch, 6. August 2014 13:21:09 UTC+2 schrieb [hidden email]:
Hello,

I want to use Elasticsearch or only indexing and searching E-Mails. We want to store the meta-info within Elasticsearch, keeping the content/body of every Mail in an mySQL database. So Elasticsearch shall have a reference to the mail body.

Is that possible and how?

Regards
Michael

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d1ffc059-abad-4b11-8179-35ed3c077cbf%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFRoWCW1ea-ZXr%2B9VfLzFDSvtenP8xHX35Tyi6Q6zDwSw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.