Google doesn't show search result counts anymore. By contrast, Bing still shows them. For example, when you search Bing for "palantir", it says that there are "about 125,000 results".
But does this number actually mean anything?
Not really. At least not with modern search engines (when I say "search engine" here, I mean "text search engine", just to keep things a bit simpler).
Basically, this is because modern search engines try to capture the meaning of your search query, not just the literal words. And the way "meaning" is captured depends on various algorithm design decisions and properties that I'll describe below—and the consequences of these design decisions and properties are not always deterministic. And if something does not behave deterministically, you can't really use it to count things.
"But management wants a number"
Yes, but here is the harsh reality: With modern search engines, you can't really get clear-cut numbers for queries such as:
- How many competitors does Palantir have?
- How many science publications are there on treatment of long COVID?
- How many patents are there on tablet computer designs?
For (1), results depend on how you scope "competitor".
For (2), you might also—or perhaps particularly—want papers that don't mention "long COVID" but present a treatment that might work, based on what you know about physiological mechanisms of long COVID. But how exactly do you draw the line there?
And for (3)... sure, there are patent classes, for example. But remember the lawsuit between Apple and Samsung about smartphone and tablet designs in the 2010s? If there were a clear-cut definition of what a patent actually applies to, we would not have such lawsuits.
Keyword search vs. vector search
Modern search engines typically use "vector search" (some people also call this "topic search" or "semantic search"). By contrast, traditional search engines used "keyword search".
Keyword search and vector search differ in how they determine results. Keyword search is clear-cut, simply checking if a text contains the exact word. But vector search uses semantic similarity to find related content. This makes vector search more complex. Results depend on various factors in the algorithm's design. Unlike keyword search, vector search can find relevant information even when exact words don't match. But it is also less deterministic.
Let's look at all of this in more detail.
Why you can count results with keyword search...
When you do keyword search, you only get documents that contain your exact keyword. For example, if you search for...
molecule...you only get texts that contain this exact term, molecule. Let's say we ignore case, so we also get Molecule.
But you won't get molecules, molecular, etc.. If you also want these, you have to use a wildcard, in this case an asterisk:
molecul*...gives you all texts that contain molecule, molecules, molecular, and so on (again, ignoring case).
Both searches, molecule and molecul*, do strict pattern matching, based on literal string matches. There is no room for interpretation.
And because there is no room for interpretation, you can count search results: A text either contains your keyword or it doesn't. In or out. Very simple.
...but not really with vector search
Vector search is what most modern search engines use (this is an oversimplification because many search engines use some kind of hybrid, but let's keep things simple, and it does not affect the point I want to make here).
The goal of vector search is to give you results that might not use the exact wording but that mean the same thing. For example:
- When you search for bank, you might also be interested in texts that mention financial institution.
- When you search for apple, you're probably interested either in the company or the fruit, but not both.
Keyword search can't do either of these.
How vector search works
Enter vector search. It is called vector search because it converts words, or texts, into numerical vectors. Then, in order to determine how similar two words are to each other (or texts, or texts to words), it calculates the distance between the vectors that represent the words or the texts. The more similar the vectors, the more similar the words or texts.
How does this help you get words or texts with the same meaning but different wording?
Well, per se, it doesn't. For example, let's say your vocabulary has three words, bank, financial institution, and apple (for comparison, English has around 1,000,000 words, according to a study by Harvard University and Google). You could represent your three words as vectors in a three-dimensional vector space like this:
bank: [1, 0, 0]
financial institution: [0, 1, 0]
apple: [0, 0, 1]This would be equivalent to a keyword search because the vectors are not similar. In fact, you could view keyword search as just a special case of vector search. And this special case, where all words are represented by vectors that are orthogonal to each other, is called sparse vector search.
But what if you reduced the dimensionality of your vector space from three to two, and got something like this?
bank: [0.9, 0]
financial institution: [0.8, 0]
apple: [0, 1]Now you can see that bank and financial institution are quite similar but apple is not.
This is the idea behind vector search: You use a machine learning method, called "embedding", that maps the words into a vector space—and, crucially, the dimensionality of this vector space is smaller than the size of your vocabulary (unless sparse vector search is what you want).
How much smaller?
Currently, many embedding models produce 768- or 1536-dimensional vectors for most languages. In other words, for English, you reduce a 1,000,000-dimensional vector space to a 768- or 1536-dimensional one (this article is a good overview of current embedding models).
Put differently, you pack your words into a rather small suitcase. And—and this is crucial again—you do this in a way so that you pack similar words next to each other (socks next to socks, t-shirts next to t-shirts, etc.).
How do embedding algorithms "know" what words are similar?
By the contexts of the words. For example, bank and financial institution typically appear in very similar contexts but apple does not. And embeddings capture this. They represent words by their contexts.
Why vector search makes counting search results practically useless
Vector search, unlike keyword search, allows room for interpretation in how many results you should get for your query.
You can understand intuitively why this actually has to be this way. For example, let's revisit our query from above:
How many competitors does Palantir have?
You could rephrase this to...
Find companies that do similar things to Palantir.
Clearly, keyword search won't get you there. It would simply return texts that contain Palantir. And sure, you could count these results, but the results would be useless.
What you want instead is a list of Palantir-like companies. But should this list include Siemens? Or Helsing? Or IBM? Or SAP? Or Salesforce? Or even Lockheed Martin?
You could probably argue each way for each of these companies, depending on the context. What's similar is, at least to a considerable extent, in the eye of the beholder. And this means that there can't be the one and only definition that holds in all contexts, and that simply lets you count Palantir competitors in a completely objective, or rather context-free, way.
And the same principle applies to our other queries from above (the papers and the patents).
But even if you had an unanimously agreed-upon definition of similarity for each of these queries, there are search engine design decisions and properties of vector search algorithms that make search result counting basically useless.
Search engine design decisions that affect your search result numbers (but usually not in a deterministic way)
Before we discuss design decisions, let's quickly recap what you need for vector search:
- An embeddings model: the machine learning method that converts words or texts into numerical vectors.
- A place to store your word or text embeddings (a "vector database").
- A method of converting a query into a numerical vector (should be the same method as in (1), otherwise you compare apples to oranges).
- A method for calculating vector similarity. Many search engines use vector cosine similarity for this.
- Some kind of similarity cutoff (how similar do two vectors have to be so that you still consider them representing the same thing).
- A method for doing the actual "nearest neighbor search" in your vector database.
When you put all this together, you need to make several design decisions that directly affect the number of results you get for a given query. For example:
- What embeddings model should I use, and with what dimensionality? The consequences of this decision are notoriously hard to predict, this often feels more like an art than a science.
- Should I do embeddings for text chunks or entire documents? If I use chunks, how do I delimit them? How long should they be?
- How do I calculate vector similarity? Cosine or euclidean distance? Or something different?
- What threshold do I use for my similarity cutoff?
- What is my floating point precision for all of the above? Depending on your precision, you will probably get rounding errors at some point, which means you get different results for the same query.
- What method should I use for results retrieval?
This last point is interesting. Many current vector databases use an algorithm called Hierarchical Navigable Small Worlds (HNSW) for this. HNSW enables fast vector search, at the (tolerable) expense of some accuracy. And because it is an approximate method, it necessarily introduces yet another set of factors that affect how many search results you get for your query. You can read more about HNSW here.
Why does Bing still show result counts?
I'm not at Microsoft, so I don't know. It is certainly not because they don't use vector search. They do. But perhaps they think users want the numbers, even if they actually don't make sense anymore. This is not a knock on the users—it is not the users' job to be aware of the technical details behind search.
So... how do I count competitors of Palantir then?
This might not be a satisfactory answer… but it depends. It depends on your definition of how similar to Palantir a company has to be in order to be considered a competitor.
But there might be something positive to this: It forces you to really think through your definition of "competitor", in this example. And this might ultimately be much more valuable than getting a simple but shallow answer to a question that is in fact quite complex.