Anything can be represented by a vector. Text can be represented by vectors describing the text's meaning. Images can be represented by the objects it contains. Users of a system can be represented by their interests and preferences. Even time-based entities such as video, sound, or user interactions can be represented by vectors. Finding the most similar vectors has all kinds of useful applications. 

There are many libraries to choose from for similarity search. However, in real-world applications, there are additional complications that need to be addressed. For instance, similarity search needs to scale up while ensuring that data indexed in the system is searchable immediately without any time-consuming index building in the background. Most importantly, however, additional search filters are often combined with the similarity search. This can severely limit the end result's quality, as post-filtering can prevent otherwise relevant results from surfacing. 

In this talk, we'll explore some real cases where combining approximate nearest neighbors (ANN) search with filtering causes problems. The solution is to integrate the ANN search with filtering, however, most libraries for nearest-neighbor search work in isolation and do not support this. To our knowledge, the only open-source platform that does is Vespa.ai, and we'll delve into how Vespa.ai solves this problem.