A decision-theoretic characterization of perfect calibration is that an agent seeking to minimize a proper loss in expectation cannot improve their outcome by post-processing a perfectly calibrated predictor. Hu and Wu (FOCS’24) use this to define an approximate calibration measure called calibration decision loss (CDL), which measures the maximal improvement achievable by any post-processing over any proper loss. Unfortunately, CDL turns out to be intractable to even weakly approximate in the offline setting, given black-box access to the predictions and labels. We suggest circumventing this… Read More
Efficient Calibration for Decision Making
A decision-theoretic characterization of perfect calibration is that an agent seeking to minimize a proper loss in expectation cannot improve their outcome by post-processing a perfectly calibrated predictor. Hu and Wu (FOCS’24) use this to define an approximate calibration measure called calibration decision loss (CDL), which measures the maximal improvement achievable by any post-processing over any proper loss. Unfortunately, CDL turns out to be intractable to even weakly approximate in the offline setting, given black-box access to the predictions and labels. We suggest circumventing this…
Related Posts
The philosophical puzzle of rational artificial intelligence
As AI technology advances, a new interdisciplinary course seeks to equip students with foundational critical…
10 Python One-Liners for Calling LLMs from Your Code
You don’t always need a heavy wrapper, a big client class, or dozens of lines…
Introducing the Gemini 2.5 Computer Use model
Today we are releasing the Gemini 2.5 Computer Use model via the API, which outperforms…