Checking the List a Reader Actually Sees

Botie may rank a useful guide correctly relative to most of the catalogue and still place it below everything the reader sees. Average rating error does not directly answer whether the visible list helped.

We will evaluate a fixed candidate set at a chosen cutoff, then distinguish finding relevant guides from putting the most useful ones near the top.

Retrieved: 3 guides
Relevant among those retrieved: 2
Relevant among all candidates: 4
Precision at 3 = 2 / 3
Recall at 3 = 2 / 4 = 1 / 2

Keep the displayed list separate from the full catalogue. Change which shown guides are relevant and compare the useful count with the number of recommendations delivered.

0.6667

When Botie made the recommendation, which guides could he have shown? We should evaluate against that candidate set, excluding unavailable items and deciding how to handle guides already read. Ranking against ten sampled alternatives can give a different score from ranking against the whole catalogue.

The cutoff should match the number of recommendations the reader sees. Keep the interactions used for fitting separate from the held-out relevance labels we use to judge that list.

Comparisons need the same candidates and exclusion rules.

Precision at k is the fraction of the returned k items that are relevant. Recall at k is the fraction of all relevant candidates that appear there. The denominators answer different questions.

When no relevant candidates are labelled for a reader, recall is undefined unless an aggregation convention is declared. Do not silently create a perfect score.

Top-three relevance = [1,0,1]; total relevant candidates = 4
Precision@3 = 2/3
Recall@3 = 2/4 = 0.5

These metrics ignore ordering within the selected top-k set.

Discounted cumulative gain divides a gain by a logarithmic rank discount. NDCG normalises by the best possible ordering of the same relevance labels at the same cutoff. Define whether gains are binary, raw graded relevance, or exponential graded gains.

Our example uses binary gains. If ideal gain is zero, report the declared zero or exclusion convention instead of dividing by zero.

Binary DCG = sum(relevance_at_rank / log2(rank + 1))
Ranks start at 1
NDCG = observed DCG / ideal DCG

A perfect ordering gives one when positive ideal gain exists.

import numpy as np
relevance = np.array([0, 1, 1, 0])
visible = relevance[:2]
total_relevant = 3
print(f'{visible.mean():.3f}')
print(f'{visible.sum() / total_relevant:.3f}')

0.500
0.333

Average per-reader metrics so highly active readers do not dominate unintentionally. Report performance for sparse histories and cold starts, along with catalogue coverage and repeated recommendations.

Offline labels reflect past exposure and may miss items a reader would have liked. A live experiment can assess actual behaviour, but it needs an explicit objective and careful interpretation rather than treating clicks as the whole goal.

The workshop calculates precision from a supplied visible list. Python also checks position-sensitive gain.

A two-stage recommender first retrieves a manageable candidate set, then ranks it. If the useful guide never reaches that set, the ranker cannot place it first. Evaluate candidate recall separately from the quality of the final ordering.

Use the same candidate policy when comparing rankers. A model scored against a handful of easy random negatives is solving a different problem from one ranking the full eligible catalogue.

The first count asks how many relevant guides retrieval included. The second asks how many appear in the visible list after ranking. Keep the full relevance set fixed while calculating both.

An unobserved interaction is not necessarily a confirmed irrelevant item. The example supplies relevance explicitly so the metric calculation has an unambiguous target.

import numpy as np
relevant = {2,5}
candidates = [0,1,2,3]
visible = [1,2]
candidate_recall = len(relevant.intersection(candidates))/len(relevant)
precision = len(relevant.intersection(visible))/len(visible)
print(candidate_recall, precision)

0.5 0.5

Some evaluation requests have no labelled relevant guide. Recall then has a zero denominator. Decide whether those requests are excluded from that metric or assigned a documented convention, and report how many there are.

Inspect new readers, new guides and low-exposure topics separately. An average dominated by frequent readers can conceal poor results for everyone else. Offline relevance reflects the collection and exposure process; an online change still needs a careful comparison of user outcomes.