Introduction to Elasticsearch – How does a search engine work? – How to get started with Elasticsearch?

View Github Repository Open presentation in a new window

eoaksnes

See all presentation from eoaksnes

Introduction to Elasticsearch – How does a search engine work? – How to get started with Elasticsearch?

0 0

introduction-to-elasticsearch

Presentation

On Github eoaksnes / introduction-to-elasticsearch

Introduction to Elasticsearch

Eirik Ola Aksnes

  • Velkommen, i dag skal jeg holde en veldig kort introduksjon til Elasticsearch.

Agenda

  • How does a search engine work
  • Elasticsearch
    • What is
    • How to get started
    • Own experience
    • Use cases
  • Agendaen idag er som følgende...

How does a search engine work?

  • Hvordan fungerer en søkemotor?
Your document collection is big! Scan through all the documents every time you search for something?
  • Dette ville tatt evigheter
Pre-process the documents and create an index!
  • For å gjøre dine søk raskt og effektivt vil en søkemotor forhåndsbehandle dokumentene og lage en index

Create an inverted index

  • Man lager seg da noe som heter en "invertert index"
  • På venstre siden har vi tre dokumenter...
  • Siden dette er en BigOne konferanse, så vil mye av innholdet i dag være pizza relatert...
  • Det som skjer er at man lager seg en invertert index av disse dokumentene (dokumentene blir indeksert, som det heter)
  • En invertert index (som vi ser på høyre side her nå) inneholder alle ordene som finnes i dokumentene, og for hvert ord så lister man opp hvilke dokumenter som inneholder ordet...
  • Så ordet "pizza" finnes i dokument 0 og 2

Find unique terms

  • Så hvordan finner man unike ord?
    • Hvis man f.eks tar for seg dokumentet "Turles loves pizza", så vil det gå igjennom forskjellige steg...
      • Man splitter opp dokumentet i ord
      • Man gjør alle bokstaver små
      • Man finner grunnstammer for ord, f.eks "Loves" blir "love"
    • Dette er ett forenklet eksempel...

Search against the inverted index

Sort by relevance

How well each document matches the query

By default, Elasticsearch sorts matching results by their relevance score, that is, by how well each document matches the query.
  • Jeg skal nå bruke sirka 1 minutt på å si hva Elasticsearch er.
    • Lucene
      • Cumbersome to use directly
      • Provides few features for scaling past a single machine
    • Real time
      • Det går fort å indeksere dokumenter
      • Data er tilgjenglig for søk nesten med en gang etter indeksering

How to get started with Elasticsearch?

  • Så hvordan kan man komme igang med Elasticsearch...

It is that easy

  • Download Elasticsearch from www.elastic.co
  • Elasticsearch only requires Java to run
wget https://download.elasticsearch.org/elasticsearch/release/...
tar -zxvf elasticsearch-2.2.0.tar.gz
cd elasticsearch-2.2.0/bin
./elasticsearch.sh

Zero configurations

  • Elasticsearch just works
    • No configuration is needed
    • It has sensible defaults settings
It is easy to get started with Elasticsearch!

Is Elasticsearch alive?

You can access it at http://localhost:9200 on your web browser, which returns this:

{
   "status":200,
   "name":"Cypher",
   "cluster_name":"elasticsearch",
   "version":{
      "number":"1.5.2",
      "build_hash":"62ff9868b4c8a0c45860bebb259e21980778ab1c",
      "build_timestamp":"2015-04-27T09:21:06Z",
      "build_snapshot":false,
      "lucene_version":"4.10.4"
   },
   "tagline":"You Know, for Search"
}

REST API

  • Elasticsearch hides the complexities of Lucene behind a REST API
    • POST (create)
    • GET (read)
    • PUT (update)
    • DELETE (delete)

DEMO - CURL works just fine

  • An index is like a database
  • An type is like a SQL table

What is stored in Elasticsearch?

JSON documents!

{
   "title": "Introduction to Elasticsearch",
   "date": "2016-04-07",
   "author": "Eirik Ola Aksnes"
}

Let's do an example - A BigOne pizza website

  • We are building a website to find BigOne pizzas
  • We have a collection of BigOne pizzas
  • We want simple text based searching

How to store the pizzas?

The act of storing data in Elasticsearch is called indexing.

$curl -X POST localhost:9200/big-one/pizza/1 --data
'{
    "name": "California Sunset Chicken"
}'

$curl -X POST localhost:9200/big-one/pizza/2 --data
'{
    "name": "American Bacon"
}'

$curl -X POST localhost:9200/big-one/pizza/3 --data
'{
    "name": "Classic American"
}'
It is much like the INSERT keyword in SQL except that, if the document already exists, the new document would replace the old. The second part indicates on which index (an index could be compared to an SQL database, though I don’t like this comparison) your query will be performed, and what is the type (a type could be compared to an SQL table, though I don’t like this comparison either) of the document. From now, I will write indices and types in orange

Get

$curl -X GET localhost:9200/big-one/pizza/1

Result:

{
   "_index":"big-one",
   "_type":"pizza",
   "_id":"1",
   "_version":1,
   "found":true,
   "_source":{
      "name":"California Sunset Chicken"
   }
}

Update

$curl -X PUT localhost:9200/big-one/pizza/1 --data
'{
   "name":"California Sunset Chicken Awesome"
}'

Result:

{
   "_index":"big-one",
   "_type":"pizza",
   "_id":"1",
   "_version":2,
   "created":false
}

Delete

$curl -X DELETE localhost:9200/big-one/pizza/1

So far

  • All we have is NoSQL document store which is
    • Fast
    • Scalable
    • Easy to use
  • Now to the really cool part, full-text search...

Full-text search

Find all the pizzas that contains the word "American"

$curl -X GET localhost:9200/big-one/pizza/_search?q=American

Full-text search - Result

{
   "took":4,
   "timed_out":false,
   "_shards":{
      "total":5,
      "successful":5,
      "failed":0
   },
   "hits":{
      "total":2,
      "max_score":0.19178301,
      "hits":[
         {
            "_index":"big-one",
            "_type":"pizza",
            "_id":"2",
            "_score":0.19178301,
            "_source":{
               "name":"American Bacon"
            }
         },
         {
            "_index":"big-one",
            "_type":"pizza",
            "_id":"3",
            "_score":0.19178301,
            "_source":{
               "name":"Classic American"
            }
         }
      ]
   }
}

Sorted by relevance!

Alternate Approach

  • Search using Query DSL

Full-text search

Find the pizzas with a name that contains the word "American"

$curl -XGET localhost:9200/big-one/pizza/_search -d
'{
   "query":{
      "match":{
         "name":"American"
      }
   }
}'

Full-text search - result

{
   "took": 8,
   "timed_out": false,
   "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
   },
   "hits": {
      "total": 2,
      "max_score": 0.19178301,
      "hits": [
         {
            "_index": "big-one",
            "_type": "pizza",
            "_id": "2",
            "_score": 0.19178301,
            "_source": {
               "name": "American Bacon"
            }
         },
         {
            "_index": "big-one",
            "_type": "pizza",
            "_id": "3",
            "_score": 0.19178301,
            "_source": {
               "name": "Classic American"
            }
         }
      ]
   }
}

Own experience with Elasticsearch

  • Grunnen til at jeg ønsket å lære meg om søkemotorer var...
  • At jeg lenge har samlet på mat oppskrifter...
  • Og etter hvert har det blitt veldig mange mat oppskrifter, nesten 400 sider...
  • Som gjør at det nesten er umulig å finne det man søke etter...
  • F.eks jeg ønsker å finne alle hovedretter som inneholder kylling...

Alt Mulig Mat

  • Text based searching
  • Structured searching (get all "Dessert" recipes)

How to use Ealsticsearch?

Commonly used in addition to another database...

Use Cases

What can Elasticsearch be used for?

For Big Data

Github uses Elasticsearch to search 20TB data, including 1.3 billion files and 130 billion code lines

Relationship databases:
  • This works well with smaller data sets, but is not very scalable
  • When the volume goes up, performance down (write operations)

Text search

With filtering, aggregations, highlightning, pagination...

Pure Analytics

Count things and summarize your data, lots of data, often on timestamped data!

Centralized Logging

Logs > Logstash > Elasticsearch > Kibana

Geolocation

The end!

  • It is easy to start building advanced search functionality
    • No configuration is needed
    • Just add data and start searching

Questions?

References

Introduction to Elasticsearch Eirik Ola Aksnes Velkommen, i dag skal jeg holde en veldig kort introduksjon til Elasticsearch.