Models and systems assessed by the tasks, data, baselines, error modes and real-world constraints behind benchmark claims.
The framework improved benchmark research tasks, but automated synthesis still requires source checking and domain experts.
NIST researchers argue that static constraints cannot guarantee desired behavior as an adaptive system and its environment continue to change.
GenCast was faster and more skillful on a large benchmark. Operational forecasting still requires reliability, calibration, infrastructure, and human judgment.