SolidityBench by IQ has been launched as the first leaderboard that evaluates LLMs in Solidity code generation. Available on Hugging faceit introduces two innovative benchmarks, NaiveJudge and HumanEval for Solidity, designed to assess and rank the proficiency of AI models in generating smart contract code.
Developed by IQs BrainDAO As part of the upcoming IQ Code suite, SolidityBench serves to refine and compare their proprietary EVMind LLMs against generalist and community-created models. IQ Code aims to provide AI models tailored to generate and control smart contract code, meeting the growing need for secure and efficient blockchain applications.
As IQ said CryptoSlateNaiveJudge offers a new approach by tasking LLMs with implementing smart contracts based on detailed specifications derived from audited OpenZeppelin contracts. These contracts provide a gold standard for correctness and efficiency. The generated code is evaluated against a reference implementation using criteria such as functional completeness, compliance with Solidity best practices and security standards, and optimization efficiency.
The review process uses state-of-the-art LLMs, including several versions of OpenAI’s GPT-4 and Claude 3.5 Sonnet as unbiased code reviewers. They review the code based on strict criteria, including implementing all major functionalities, handling edge cases, error management, correct syntax usage, and overall code structure and maintainability.
Optimization considerations such as gas efficiency and storage management are also evaluated. Scores range from 0 to 100 and provide a comprehensive assessment of functionality, security, and efficiency, reflecting the complexity of smart contract professional development.
Which AI models are best for developing solid smart contracts?
Benchmark results showed that OpenAI’s GPT-4o model achieved the highest overall score of 80.05, with a NaiveJudge score of 72.18 and HumanEval for Solidity pass rates of 80% on pass@1 and 92% on pass@3 .
Interestingly, newer reasoning models like OpenAI’s o1-preview and o1-mini were beaten into first place, with scores of 77.61 and 75.08 respectively. Models from Anthropic and XAI, including Claude 3.5 Sonnet and Grok-2, showed competitive performance with overall scores hovering around 74. Nvidia’s Llama-3.1-Nemotron-70B scored the lowest in the top 10 with 52.54.
Per IQ, HumanEval for Solidity adapts OpenAI’s original HumanEval benchmark from Python to Solidity, and includes 25 tasks of varying difficulty. Each task includes corresponding tests compatible with Hardhat, a popular Ethereum development environment, which allows accurate compilation and testing of the generated code. The evaluation metrics, pass@1 and pass@3, measure the model’s success on first attempts and over multiple attempts, providing insight into both accuracy and problem-solving ability.
Objectives of using AI models in smart contract development
By introducing these benchmarks, SolidityBench aims to promote the AI-enabled development of smart contracts. It encourages the creation of more advanced and reliable AI models and provides developers and researchers with valuable insights into the current capabilities and limitations of AI in Solidity development.
The benchmarking toolkit aims to advance IQ Code’s EVMind LLMs and also sets new standards for the development of AI-enabled smart contracts in the blockchain ecosystem. The initiative hopes to address a critical need in the industry, where demand for secure and efficient smart contracts continues to grow.
Developers, researchers, and AI enthusiasts are invited to explore and contribute to SolidityBench, which aims to drive the continued refinement of AI models, advance best practices, and advance decentralized applications.
Visit the SolidityBench ranking on Hugging Face for more information and to start benchmarking Solidity generation models.