This is a normal E-commerce use case that had been relying on earlier standalone Solr and later Master-Slave for ages.

Everything was running smoothly with no performance glitches until one day we had this requirement to support “Unlimited Products” in our shops.

We were confident that things would be smooth with full reindex 2 times a day but looking at the index pipeline duration of 4+ hrs, we realized it needed some cloud magic.

This is where we started and transformed a normal batch index pipeline to a stream index pipeline leveraging Kafka and Redis and achieved a NRT indexing.

Along with that, we did not shy to give our infra a bonus upgrade from Solr 6.6.2 to Solr 8.7 and an add-on migration from the existing Master-Slave to Cloud.

We want to talk about our journey and important lessons learned during this process.