Decided to go to the basics of RAG

Building a proper project with RAG this time - the BioASQ challenge. Using the API from PubMed to retrieve titles and abstracts from published articles to create a corpus.

During this project I decided to go into a bit more detail about the Ranking Metrics used for searching in an RAG system. Normally I’ve stuck to MRR, but lately I have been seeing a lot of job descriptions explicitly mentioning something called recall@k, which turns out to be one such metric.

This week, I did spend some time, took the project slow, and built it ground up so I can subconsciously pay attention to some details. And well enough, while taking the decision for the ranking metric, while learning about recall@k, straight up my intuition says it is useless.
recall@k is the ratio between the number of relevant documents found among the top ‘k’ retrieved documents to the total number of relevant documents in the dataset. Now imagine using that to see whether your search algorithm is doing well. If that statement alone does not scream “I am a doodoohead metric” to you, you must build on this intuition.
The value of the numerator is capped at k, but the denominator has no limit.
Similar problem with precision@k, which is the ratio of the relevant documents at first k retrievals, to k.
k can be anything. The number of documents can be much less than k. This brings misleading numbers.
Both of these have no place being so prominent in the job descriptions of so many openings these days.

Then there’s BM25, the search formula. It turns out to be an enhanced version of the TFIDF formula. But as it turns out, it was just built out of experimentation with not much theoretical grounding. So there is not much point in dwelling too deep. It is a variation of TFIDF that penalizes document length, and it is the 25th experiment formula (so the name bm25).

I discovered a new search algorithm called Dense, which I am yet to learn about. Apparently this one converts words into tokens and finds similarity based on meanings. This should be very relevant to a medical problem.

MRR: (Mean Reciprocal Rank) remains the same. Reciprocal rank (position) of the first relevant document in the list -> calculate for N searches and average. Can change as we change the search term. No problem for us because we may have a good dataset.

MAP:
This time I learnt about MAP (Mean average precision). For a search result list, the MAP calculates the precision at each index where the document is relevant. Let’s say relevant documents are found at indices 1 and 4 only.
Then AP is just the average of [precision@1 and precision@4]
Then perform this for a number of queries and get AP values, and then average them to get MAP.