Low latency item-to-item recommendation system

Low latency item-to-item recommendation solution based on BiqQuery and ScaNN library for efficient vector similarity search at scale

This template builds a low latency item-to-item recommendation solution. The foundation of the solution are BigQuery and ScaNN - an open source library for efficient vector similarity search at scale.

Build a training infrastructure using this template


This template creates a new repository containing example notebooks and training pipelines. Also, the template creates Google Cloud Platform AI notebook with pre-loaded (cloned) repository, as well as a Continuous Integration running the notebooks sequentially on any change in the repository.

Example Dataset

This template uses the public bigquery-samples.playlists BigQuery dataset to demonstrate the solutions. The playlist data are used to learn embeddings for songs based on their co-occurrences in different playlists. The learnt embeddings can be used to match and recommend relevant songs to a given song or playlist.


The system utilizes BigQuery ML Matrix Factorization model to train the embeddings, and the open-source ScaNN framework to build and approximate nearest neighbour index.

  1. Compute pointwise mutual information (PMI) between items based on their co-occurrences.
  2. Train item embeddings using BigQuery ML Matrix Factorization, with item PMI as implicit feedback.
  3. Export and post-process the embeddings from BigQuery ML model to Cloud Storage as CSV files using Cloud Dataflow.
  4. Implement an embedding lookup model using Keras and deploy it to AI Platform Prediction.
  5. Serve the embedding as an approximate nearest neighbor index using ScaNN on AI Platform Prediction for real-time similar items matching.


Running the solution

  1. 01trainbqmlmfpmi.ipynb - This notebook covers computing pairwise item co-occurrences to train the the BigQuery ML Matrix Factorization model, and generate embeddings for the items.
  2. 02exportbqmlmfembeddings.ipynb - This notebook covers exporting the trained embeddings from the Matrix Factorization BigQuery ML Model to Cloud Storage, as CSV files, using Apache Beam and Cloud Dataflow.
  3. 03createembeddinglookupmodel.ipynb - This notebook covers wrapping the item embeddings in a Keras model and exporting it as a SavedModel, to act as an item-embedding lookup.
  4. 04buildembeddings_scann.ipynb - This notebook covers building an approximate nearest neighbor index for the embeddings using ScaNN and AI Platform Training. The built ScaNN index then is stored in Cloud Storage.


This template is created based on the this github repository.