Relevance Assessment

Draft To determine the performance of our EntitySuggester it is crucial to determine the relevance of the suggestions and based on that make further investigations on recall/precision. Relevance is a very subjective topic, which can lead to different answers depending on who to ask for evaluation.

relevance judgements

• ask many people to reduce subjectivity
• sample size?
• difference user relevance and topical relevance relevance feedback query transformation -binary relevance: relevant or not-relevant? Bayes Decision or graded relevance

Relevance Feedback

• For a given entity the EntitySuggestor shows suggestions
• the user evaluates the given output into relevant or not-relevant
• use of probailistic model or vector space model? step 2: if probabilistic: calculate new probabilities
• output changed query
• user can evaluate changed results

Step2 hard to implement.

Blind Relevance Feedback? (Pseudo Relevance Feedback)

The probabilistic model

• assumption relevance is binary
• classify suggestion S as relevant(R) or non-relevant (NR)
• put suggestion in class which has the highest probability
• S is relevant if: P(R|S)> P(NR|S) Bayes Decision Rule (conditional probability)
• how to calculate P(S|R)?
• Bayes Rule: P(R|S) = P(S|R) P(R) / P(S)

naive: just evaluationg binary relevance of suggestions for a given entity