RAG retrieval metrics

Resuming the RAG project today. Foundation on a bunch of things. Learnt a bunch of other things.

Rate@k: seems like a nice metric. It does N number of searches, and for each search, just cares about whether a relevant document is present among the top k results. It is simply
(Number of searches where top k contains a relevant document)/N
K should be kept at a relatively smaller value if this was ever to be used. It captures how well a search algorithm can do among different types of searches.
Shows whether the search is even retrieving relevant documents.

NDCG: (Normalised Discounted Cumulative Gain) is the most popular search metric. Rewards documents appearing higher in the list (vs recall@k and precision@k), and also for relevance. But this one is complex.
Documents are first graded for their relevance manually (3 is more relevant, 2 is less, … 0 is irrelevant)

DCG: For every search result, DCG is calculated as:

(relevance of first document) + Mean of [ (relevance of i) / log₂(i) ] for all other

The reason why relevance of first document is separated out is because log₂(1) is 0 and that breaks the formula. It works to reward first occurrence anyway.

IDCG (Ideal Discounted Cumulative Gain): This formula treats the “ideal” situation as documents being listed in an order from highest relevance to lowest. So it first sorts all search results as such and then same formula as DCG above.

NDCG = DCG / IDCG

While this is a popular metric, I see a problem here because of IDCG simply sorting the results. Let’s say for a particular query there is a single document with relevance 4 which is not captured, but multiple other documents of relevance 2 are captured. NDCG does not punish the search algorithm for missing a more relevant document.

Example:
Search returns documents with relevance: [2, 0, 1, 1, 2, 0, 0]. But the corpus has a document with relevance 4
DCG is calculated for [2, 0, 1, 1, 2, 0, 0]
IDCG is calculated for [2, 2, 1, 1, 0, 0, 0]
And the ratio of the two is taken. The omission of the document with relevance 4.
In my opinion, this omission should in some way be fit into the denominator

In any case, this algorithm does not fit into this project because document relevance rating is not part of the dataset.