Hugging Face policy chief calls for clearer AI evaluation standards

Hugging Face chief policy officer Irene Solaiman said AI testing needs more consistent standards, including clearer requirements for outside evaluators working within AI companies. Speaking during an industry conference discussion, she questioned how evaluation results can be compared as testing methods continue to develop.

Solaiman said confidence in a benchmark depends partly on whether others can replicate its findings and have sufficient access to the model being tested. She pointed to open model weights as an important part of that access, while warning that evaluations can quickly lose their usefulness.

AI labs have promoted embedded third-party evaluators as an alternative to government regulation of model safety. But Solaiman said the qualifications needed to assess different tasks require greater standardization. Questions about evaluators’ expertise have already drawn scrutiny: Anthropic faced criticism after choosing Accenture as its first evaluator.

Testing is also broadening to cover agents and the tools used to access models, known as harnesses. Solaiman said a stronger harness can allow a weaker model to outperform a frontier model. Evaluating agents is difficult and costly, she said, with fewer tests available for harnesses than for models and work continuing on assessments of complete agent systems.