Building a scalable recommendation engine

using play framework, couchbase and elasticsearch

Note: This article is more on technology stack selection than how to use different algorithms to generate recommendations.

Intro

Recommendation engines are everywhere trying to suggest relevant things to users and relevant recommendation increases the business value. Recommendation engine is usually built on top of insights data. Insights are generated as follows.

• Data is ingested, cleaned, enriched and standardized in required format.
• Various algorithms are run on top this data to produce output.
• Insights data is created based on the output produced by algorithms.

In short, insights are generated by the algorithms from the standardized data.

Goals

• Recommendation engine, responding within a second for large number of concurrent users.
• Horizontally scalable system. Add more nodes to increase capacity.

In short, speed and easy scalability.

Tech Stack

1. Nginx serves as end point for APIs. It serves two roles,
   1. Load balancing : Distributing the load to actual play framework apps.
   2. SSL Termination : It is always a recommended practice to use SSL if any user data is involved and set it up in the reverse proxy.
2. Play framework app works as Rest API server providing recommendations. It is Sessionless and Servlet free. It uses iteratee and netty to easily scale. Also, it has better productivity with Hit refresh workflow. This application acted more like a wrapper around the core module.
3. Spring framework is primarily used for dependency injection. Alternatively you can use google guice as well. Core application logic is written here. Usually it would be better to have the core logic in a separate module and add that as dependency in the Rest API application. It would be much easier to provide different kind of integration later, if required.
4. Couchbase is used as a cache database server, data retrieval is relatively much faster than traditional RDBMS and even alternate document based storage systems. It has GUI to easily add/remove nodes. Now, it supports key and value eviction and so, it is no more RAM limited. Prefer to bulk get the documents.
5. Elasticsearch is used as a search server. It has numerous Search APIs. In our use case, most of the recommendations had geo location associated with it. Elastic search is helpful in the location based searches. Also, used in auto completes where user location and input text is taken up. Elasticsearch also horizontally scalable search engine. We can easily add more nodes if required.

Learnings

• Most of the heavy duty work is done in batch mode before insights are generated.
• Often insights are limited in size due to speed constraints on real time recommendation. It is trade off between speed and relevance.
• Choice of Play Framework, Couchbase and Elasticsearch worked well together.
• JSONs Everywhere. Both cache server and search server data is stored as Json. As well as recommendations are in JSON.
• Except Nginx, all other components used here are horizontally scalable. As Nginx run on evented model and can support more concurrent users.
• More data, add more nodes to Couchbase and Elasticsearch clusters.
• More concurrent users, add more Play framework nodes.