Meta has denied allegations that its Llama 4 fashions have been skilled on benchmark check units. In a submit on X, Ahmad Al-Dahle, Meta’s VP of GenAI, stated, “We’ve additionally heard claims that we skilled on check units — that’s merely not true, and we’d by no means do this.” He added that the corporate launched the fashions as quickly as they have been prepared and that “it’ll take a number of days for all the general public implementations to get dialed in.” Meta attributed the blended efficiency studies to implementation stability relatively than flaws within the coaching course of.
Meta lately launched two new Llama 4 fashions, Scout and Maverik.
Maverick rapidly reached the second spot on LMArena, the AI benchmark platform the place customers vote on one of the best responses in head-to-head mannequin comparisons. In its press launch, Meta pointed to Maverick’s ELO rating of 1417, rating it above OpenAI’s GPT-4o and just under Gemini 2.5 Professional.
Nevertheless, the model of Maverick evaluated on LMArena isn’t equivalent to what Meta has made publicly accessible. In its weblog submit, Meta stated that it used an “experimental chat model” tailor-made to enhance “conversationality.”
Chatbot Area, run by lmarena.ai (previously lmsys.org), acknowledged group considerations and shared over 2,000 head-to-head battle outcomes for evaluate. “To make sure full transparency, we’re releasing 2,000+ head-to-head battle outcomes for public evaluate. This consists of person prompts, mannequin responses, and person preferences,” the corporate stated.
Additionally they stated Meta’s interpretation of Area’s insurance policies didn’t align with expectations, prompting a leaderboard coverage replace to make sure honest and reproducible future evaluations.
“As well as, we’re additionally including the HF model of Llama-4-Maverick to Area, with leaderboard outcomes printed shortly. Meta’s interpretation of our coverage didn’t match what we anticipate from mannequin suppliers. Meta ought to have made it clearer that “Llama-4-Maverick-03-26-Experimental” was a customized mannequin to optimise for human choice,” the corporate stated.
The drama round Llama 4 benchmarks began when a now-viral Reddit submit citing a Chinese language report , allegedly from a Meta worker concerned in Llama 4’s improvement, claiming inside stress to mix benchmark check units throughout post-training.
“Firm management steered mixing check units from numerous benchmarks through the post-training course of, aiming to fulfill the targets throughout numerous metrics,” the submit learn. Within the report, the worker wrote that that they had submitted their resignation and requested to be excluded from the technical report.
AIM reached out to Meta sources and confirmed that the worker has not left the corporate, and the Chinese language submit is faux.
Nevertheless, a number of AI researchers have famous a distinction between the benchmarks reported by Meta and those they noticed. “Llama 4 on LMSys is a very totally different type than Llama 4 elsewhere, even for those who use the really useful system immediate. Tried numerous prompts myself,” stated a person on X.
“4D chess transfer: use Llama 4 experimental to hack LMSys, expose the slop choice, and at last discredit all the rating system,” quipped Susan Zhang, senior workers analysis engineer at Google DeepMind.
Questions have been additionally raised concerning the weekend launch of Llama 4, as tech giants normally make bulletins on weekdays. It is usually stated that Meta was beneath stress to launch Llama 4 earlier than DeepSeek launches its subsequent reasoning mannequin, R2. In the meantime, Meta has introduced that it’s going to launch its reasoning mannequin quickly.
Earlier than the discharge of Llama 4, The Data had reported that Meta had pushed again the discharge date no less than twice, because the mannequin didn’t carry out as effectively on technical benchmarks as hoped—notably in reasoning and math duties. Meta has additionally had considerations that Llama 4 is much less succesful than OpenAI’s fashions at conducting humanlike voice conversations.
The submit Meta Denies Any Wrongdoing in Llama 4 Benchmarks appeared first on Analytics India Journal.
