• RICKY: Non-profit that delivers technology for social justice
• RICKY: Work to increase technical competance and solve social and community issues with tech
• We work on the technical side to DELIVER & HOST a number of web applications to aid other organisations

Who is @infoxchange?

• SAM: Diverse environment mix of languages and platforms
• We run at approximately 4000 - 8000 requests per minute across our apps

Our Products

• RICKY: Brief runover of our offerings
• eRef: specialist referrals, eg go to GP, referral to hospital radiology
• eWait: aged care queues
• Service coord: care plans/how to coordinate care between specialists
• QIPPS: project pre-planning, what do you want to achieve? target?, slip slop slap
• Patient management: Make sure that patient information is readily, easily accessible
• Drupal: Give small community orgs web presence

• SAM: We've been in digital search since 1991
• Think about what finding services was like in 1990...

• RICKY: 1990: Archie, the worlds first search engine aka 'query form' was launched
• Index FTP filenames
• McGill University, Montreal

1990 - Archie

1991 - ServiceSeeker

1995 - Google

• RICKY: 1996: Service Seeker 2 was released (and it's still running now)
• Only minor cosmetic changes (mostly looks the same)
• Powers lifeline call centre

1996 - ServiceSeeker 2

2014 - ServiceSeeker 3 (API)

• I'm going to talk about the OPs journey
• Our Environment
• Performance
• Monitoring
• Lessons Learnt
• and then I'll hand you over to Ricky - talk through the dev side of things

Environmental Constraints

• Small Ops team, 4 people working across all our products
• We're a non-profit so we have to be careful with our money
• Relatively low-end hardware
• No search / java experience
• When we started out on the journey we had LOTS of technical debt that needed to be paid down
• and an elderly network

• But we've learnt to leverage our stengths...
• We run modern platforms
• Our Dev and Ops teams work together
• We don't really have much internal bureaucracy getting in the way
• We're an open source shop
• We just rolled a sexy new network
• and most of all, our people

• 3 node clusters
• Debian shop
• 24 Core, 30GB clusters
• XenServer for virtalisation works very well for us

• Some general Elasticsearch deployment tweaks
• Make sure your heap size is set to half your RAM
• As we're not running fancy physical hardware
• We dont have SSDs etc - we try to get blood from a stone
• We did some tuning to allow for FASTER SHARD RECOVERY after upgrading / failing a node
• Don't let your hosts SWAP
• RICKY: ulimit: better to die than be killed
• Don't let people delete all your nice things
• Keep-alives were a pain

ES_HEAP_SIZE = 5g

routing.allocation.node_initial_primaries_recoveries = 4,
routing.allocation.node_concurrent_recoveries    = 15
recovery.max_bytes_per_sec                       = 100mb,
recovery.concurrent_stream                       = 5

mlockall                    = true,  

disable_delete_all_indices  = true, 

• We had BIG problems with Javas GC, lag up ES and queries would time out ...
• "Stop-the-world garbage collection and blocking for disk IO can cause runtime latencies on the order of seconds to minutes."
• I'm certainly no expert with java but after some reasearch our solution is to garbage collect more often, which has a far lesser impact on blocking performance

s/CMSInitiatingOccupancyFraction=[0-9]* /CMSInitiatingOccupancyFraction=35/g

Monitoring

• At Infoxchange we LOVE monitoring, I think it's something we do quite well
• I have a few slides on this I'll go quickly but I think it's really important
• ElasticHQ has been helpful for spotting obvious performance bottlenecks

• If you're already monitoring that everything is alive & well...
• And you're returning that Information to Nagios...
• Why not use & graph the returned performance metrics from the checks for trending

Monitoring

Monitoring

Monitoring

• Let's talk briefly about logs
• We pass all our app logs to Logstash
• Logstash does some magic then stores them in elasticsearch
• Our devs can see those logs using Kibana (web interface to ES)
• It's not as good as splunk, but it's pretty decent

• Just like Elasticsearch itself we deploy & control logstash using puppet
• Both Elasticsearch & Logstash have excellent (official) puppet modules
• Logstash deployment looks a bit like this ...

  ixalogstash::input { 'syslog-docker':
    input_type     => 'syslog',
    type_tag       => 'docker',
    input_port     => 5550,
  }
  ixalogstash::input { 'syslog-nginx':
    input_type     => 'syslog',
    type_tag       => 'nginx',
    input_port     => 5551,
  }

  ixalogstash::filter { 'syslog-nginx':
    type_tag       => 'nginx',
    filter_match   => '%{COMBINEDAPACHELOG} %{QS:vhost}',
  }

  ixalogstash::output { "${elastic_host}":
    output_type    => 'elastichttp',
  }

The Dev Journey

• Gunicorn/nginx, no Apache
• TastyPie is a RESTful framework
• ElasticUtils is like an ORM for ES

ISS3

• GIS
• Private
• Query strings
• Keywords
• Cascaded data

• Cover this quickly
• Django routes to TastyPie
• TastyPie handles RESTful
• obj_get_list builds query
• Query string parser returns query chunks
• Queries ES and we get JSON doc source
• Dehyrate adds dynamic fields, remove _id etc
• TastyPie adds meta, returns 200/etc

• Must match in area perfectly
• Must fall back to matching by distance
• Country vs city is tricky, because you have to normalize distance vs search
• In the country, 100km is acceptable
• In the city, 10km is acceptable

• inverse document frequency
• boosts terms that appear less often
• GIS is not boosted
• Funcscore query combines queries on a given function
• Decay functions like gauss, etc
• Set scoremode to whatever suits you (add or max; we use add)
• Set boostmode to multiply (default)
• Can match on things like price, etc

GIS and IDF

Like oil and water

• key words to pick out
• or not....
• multiple suburbs, same name
• multi word states
• landmarks
• multiple different keywords to parse
• sometimes no keywords

Intelligent Query Strings

• doctor in richmond
• doctor near richmond
• doctor richmond
• doctor near richmond nsw
• doctor near richmond new south wales
• doctor near richmond town hall
• doctor around richmond town hall that speaks chinese
• female doctor around richmond qld that speaks chinese

• Any family in Banks before I insult it?
• blood banks
• blood in banks
• top search is needles exchange, because many local councils static link to this search
• needles exchange -> needles, TAS
• the rocks/the rock

Keywords

Keywords

• New South Wales
• Allied Health Services
• Primary school
• Alternative therapy

  "type": "synonym",
  "synonyms": [
    "new south wales => new___south___wales,nsw",
    "primari school => primari school,primary___school",
    "altern therapi => altern therapi,alternative___therapy"
  ]

• comp: ONLY a prefix suggester
• comp: More like a completer for a list of tags
• term: Only does "fuzzy" spell correction
• term: Type "doct", more likely to suggest a 4 letter word with 3 letters changed that "doctor"
• comp/term: Only for single words; "new s" wont come up as new south wales
• phra: Multiple words/whole phrase (in place replacement)
• phra: Fuzzy corrections
• phra: Completion
• phra: Replaces anywhere in the string; more like a "did you mean"

Fun With "Markov Chains"

• celery is like a message queue, for tasks
• to index, we get all the doc ids that need refresh
• we split into chunks of 500
• we send a job for each chunk of 500
• each chunk may spawn other jobs (org -> site -> service)

Thank you