Image & text similarity problems have plenty of textbook ML solutions. However in the wild, these solutions often fail. In this talk I'll present a use-case about inferring product similarity from multiple sources of data (images, text, etc) and discuss how we developed a practical & scalable approach using our understanding of domain knowledge.