At FedGeoDay in April, I attended a workshop conducted by my friend Steve Pousty on the use of vector databases. Steve is a great teacher and I learned a lot that day. I’ve been on a bit of a journey since. I’ve gotten some project work with AI and I find vector databases to be the most intriguing part of an AI tech stack.
Vector databases are a component of many generative AI systems and support tasks such as semantic search and contextual retrieval via techniques such as retrieval augmented generation (RAG). They are highly optimized for similarity search as opposed to inverted index searches like Google which are optimized to find exact matches. Vector databases, with their ability to search on semantic similarity, are useful for tasks such as finding documents that “talk around” a topic without explicitly mentioning it.
Vector databases get their name from the fact that they convert data into mathematical vectors which are embedded into vector space. Vectors can be defined by an large number of dimensions. These are mathematical dimensions, similar in concept to the three spatial dimensions that humans can perceive, but vastly different in practicality. The same is true for the “space” they are embedded into. Much like Tobler’s First Law in our 3D spatial world, similarity is measured by proximity in vector space. The closer two vectors are to each other, the more similar to each other they are. That’s a little bit of an oversimplification, but I’ll leave it there in the interest of progressing with the topic of this post.
One aspect of vector databases I find interesting is the ability to analyze embeddings created directly from binary data, such as images. This opens up the ability to say “Show me images that look like this one.” I’ll work through a very basic example that I put together, but first, I’ll cover some basics.
The model matters. Steve told me to start with pre-trained models and fine-tune as necessary. He was right. As recently as five years ago, the big push was for training data. Now, thousands of pre-trained models are available on HuggingFace – the Github for AI models. For this example, I used a pre-trained Resnet-50 model, mainly because it works well with the machine on which I am writing this post. It is a 50-layer convolutional neural network (CNN) which, in this case, has already been trained.
The model also matters for querying. Vector database inputs are encoded to vector for querying, so you have to use the same model. This is similar to the idea that you have to query a SQL database using the same language as the data. You can’t use string literals in Spanish to query data in English and expect to get useful results. The same is true for vector databases.
Moving on, I set up a basic workflow:
- Acquire the images
- Embed the images
- Store the embeddings
- Query the embeddings using an input image
I’ll walk through each step next, but what I am attempting to accomplish with this example is find the top three most similar images to my input image from my vector database. In this case, I will be using pgVector, a vector database extension to PostgreSQL. I will be embedding images of beach, desert, glacier, prairie, and rainforest.
Acquire the Images
In this case, I decided to scrape some images from Wikimedia Commons using the categories above as query strings. I wrote a Python script that issues the query and pulls the first ten images for each category. This screenshot shows some of the images that were pulled.
The code for this step is at the end of the post.
Generating and Storing Embedding
The next step is to create embeddings from the downloaded images and store those embeddings in pgVector.
Embedding images involves converting them into a format that a computer can easily compare and analyze. This process starts with a pre-trained model like ResNet-50, which has already learned to recognize various features in images, such as shapes, colors, and textures. When an image is fed into the model, it breaks down the image into a vector representing these features. It is similar to creating a unique fingerprint for the image. This vector captures the essential characteristics of the image in a way that allows a computer to compare it with other images. By storing these vectors in a database, we can quickly find and retrieve images that are similar based on their embedded features, even if they don’t look exactly the same.
The first step here is to create a table to hold the embeddings. This assumes you have installed the pgvector extension. Here is the DDL for the table that I used
Next, I wrote a python script to create embeddings for each image I downloaded previously. As mentioned above, this code uses the Resnet-50 model, which can be found on Huggingface. The workflow is simple, create the embeddings in Python and then execute an INSERT statement using psycopg2.
The code for this step is at the end of this post.
Querying Embeddings Using an Image
So far, the workflows has been fairly straightforward: Get the data, encode the data, store the data.
Now I’m going to query the data. I’ll take another image, encode it using Resnet-50, and run a SELECT statement through psycopg2 to find the three most similar images from those I downloaded in the first step. I’ll be looking for desert photos by querying with this image:
As I said previously, the model matters. I have to use the same model to encode the input image. The script does that and then passes the resulting embedding through to PgVector. In this case, the query uses the ‘<->’ operator to calculate the Euclidean (straight-line) distance between the input embedding and the stored embeddings. Using the ‘<=>’ would cause the query to use cosine similarity.
With the sample data, the three most similar images were:
Those results were pretty good and you can see how each image has aspects of similarity. Similarity is based on features such as edges, textures, patterns and more which are broken down into layers. The Resnet-50 model supports up to 50 layers, and it can perform fine-grained similarity search.
The code for this step is at the end of this post.
Wrapping Up
Vector databases are an incredibly interesting technology to me and I am continuing my journey with them. This post shows some of my first steps, but I’m excited about learning and doing more. Often, the top-level technology that gets the hype, such as AI, is supported by some compelling technology that gets less attention.
I’m particularly intrigued by PgVector because of its ability to put vectors alongside scalar data and enable hybrid search scenarios in a way that is familiar. I’m looking forward to doing more.
Code For This Post
Acquiring Images:
Generating and Storing Embeddings:
Querying Images:
