All tasks

Ranking Discount Evaluator

PythonHardClorvin Labs original task

A repeated document should not earn relevance credit twice in the same ranking. Finish the normalized gain calculation so duplicates and the ideal ranking are handled consistently.

Build the implementation in Python under /app, with /app/run.sh as its entry point. Read one JSON request from standard input and write one JSON result to standard output. Input has positive integer k, relevance mapping document IDs to nonnegative integer grades, and ranked IDs. Take the first k positions without deduplication; only the first occurrence of a document earns its grade, later occurrences earn zero but still occupy a rank. Gain is 2**grade-1 and discount log2(rank+1) with rank starting at one. Ideal ranking is the largest k grades of distinct relevance keys. Return ndcg rounded to six decimals or zero for zero ideal gain.

Reject inputs that violate an explicit validation rule with exit status 2, no standard output, and {"error":"invalid_input"} on standard error. For valid input, exit with status 0. Keep the implementation reusable for new inputs; it must work offline and must not depend on the example data being the only data it receives.

Repair or replace the unfinished /app/run.sh and add source files under /app. Malformed JSON must use the same invalid-input response. Do not read /tests or /oracle from the application.

Inspect the checks. Explore the results.

9 recorded reference checks

Review the checks and explore the evaluation view for ranking discount evaluator.

Example dataExplore the controls with illustrative values. These are not recorded GPT-5.5 or Opus 4.8 results.
Read the example values
Illustrative reward values, not model test results
AttemptGPT-5.5 (demo)Opus 4.8 (demo)
100.25
20.250.5
30.50.5
40.751
50.50.75
610.75
70.751
811

Tell us what needs doing.

Share the scope, the expected volume and your timeline.
We’ll work out where our team can help.