A Verification-Guided Multi-Agent Framework for Reliable RTL Generation and Optimization
DOI:
https://doi.org/10.62051/r6y8jj34Keywords:
Register-transfer level; Large language models; Multi-agent systems; Automated program repair; Fault localization; Functional verification; Equivalence checking; Logic synthesis.Abstract
Large language models (LLMs) already emit syntactically plausible Verilog, yet a large share of generated designs fails functional verification, and the designs that do pass are not necessarily good implementations. We present VGMA, a verification-guided multi-agent framework that closes the loop between an RTL generator and open-source EDA tooling. VGMA synthesises a differential testbench directly from the reference port interface, compresses every failing simulation into a counterexample signature of nine semantic tags, maps that signature to a prior over repair operators, ranks suspicious lines with a backward cone-of-influence slice refined by the differing bit mask and by elaboration diagnostics, and admits a patch only after a bounded sequential equivalence proof. Every number reported here is produced by Icarus Verilog and Yosys. On 261 injected defects across 152 VerilogEval v2 problems, the full signal set places the faulty line first for 47.9% of defects (75.1% within the top five) against 31.4% (64.0%) without elaboration diagnostics, and repairs 73.3% of defects within 25 verification queries against 36.7% for an undirected search over an identical edit space, while using 40% fewer queries. On 220 real GPT-3.5-turbo and GPT-4 generations shipped with RTLLM, of which 125 fail, guided repair makes 40.0% of the 35 non-elaborating designs elaborate against 5.7% for undirected search, but end-to-end functional repair reaches only 5.6% against 1.6%: mutation corpora substantially overstate what line-local repair achieves on real LLM defects. Finally, 25.3% of functionally passing LLM designs synthesise larger than the human reference (mean 1.27x cells, worst case 18.8x), whereas a portfolio of synthesis scripts recovers only 2.0% on average, which places the optimisation opportunity at the RTL level rather than in the synthesis flow.
Downloads
References
[1] Liu, M., Pinckney, N., Khailany, B., & Ren, H. (2023). VerilogEval: Evaluating large language models for Verilog code generation. In Proceedings of the IEEE/ACM International Conference on Computer Aided Design (ICCAD) (pp. 1 8).
[2] Pinckney, N., Batten, C., Liu, M., Ren, H., & Khailany, B. (2024). Revisiting VerilogEval: A year of improvements in large language models for hardware code generation. arXiv preprint arXiv:2408.11053.
[3] Lu, Y., Liu, S., Zhang, Q., & Xie, Z. (2024). RTLLM: An open source benchmark for design RTL generation with large language model. In Proceedings of the Asia and South Pacific Design Automation Conference (ASP DAC) (pp. 722 727).
[4] Thakur, S., Ahmad, B., Fan, Z., Pearce, H., Tan, B., Karri, R., Dolan Gavitt, B., & Garg, S. (2023). Benchmarking large language models for automated Verilog RTL code generation. In Proceedings of the Design, Automation and Test in Europe (DATE) (pp. 1 6).
[5] Thakur, S., Ahmad, B., Pearce, H., Tan, B., Dolan Gavitt, B., Karri, R., & Garg, S. (2024). VeriGen: A large language model for Verilog code generation. ACM Transactions on Design Automation of Electronic Systems, 29(3), 1 31.
[6] Liu, S., Fang, W., Lu, Y., Wang, J., Zhang, Q., Zhang, H., & Xie, Z. (2025). RTLCoder: Fully open source and efficient LLM assisted RTL code generation technique. IEEE Transactions on Computer Aided Design of Integrated Circuits and Systems, 44(4), 1448 1461.
[7] Zhao, Y., Huang, D., Li, C., Jin, P., Nan, Z., Ma, T., Qi, L., Pan, Y., Zhang, Z., Zhang, R., Zhang, X., Du, Z., Guo, Q., Hu, X., & Chen, Y. (2024). CodeV: Empowering LLMs with HDL generation through multi level summarization. arXiv preprint arXiv:2407.10424.
[8] Thakur, S., Blocklove, J., Pearce, H., Tan, B., Garg, S., & Karri, R. (2023). AutoChip: Automating HDL generation using LLM feedback. arXiv preprint arXiv:2311.04887.
[9] Ho, C. T., Ren, H., & Khailany, B. (2025). VerilogCoder: Autonomous Verilog coding agents with graph based planning and abstract syntax tree based waveform tracing tool. In Proceedings of the AAAI Conference on Artificial Intelligence.
[10] Zhao, Y., Zhang, H., Huang, H., Yu, Z., & Zhao, J. (2025). MAGE: A multi agent engine for automated RTL code generation. In Proceedings of the ACM/IEEE Design Automation Conference (DAC).
[11] Wei, Y., Huang, Z., Li, H., Xing, W. W., Lin, T. J., & He, L. (2025). VFlow: Discovering optimal agentic workflows for Verilog generation. arXiv preprint arXiv:2504.03723.
[12] Tsai, Y. D., Liu, M., & Ren, H. (2024). RTLFixer: Automatically fixing RTL syntax errors with large language models. In Proceedings of the ACM/IEEE Design Automation Conference (DAC).
[13] Smith, E. K., Barr, E. T., Le Goues, C., & Brun, Y. (2015). Is the cure worse than the disease? Overfitting in automated program repair. In Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE) (pp. 532 543).
[14] Qi, Z., Long, F., Achour, S., & Rinard, M. (2015). An analysis of patch plausibility and correctness for generate and validate patch generation systems. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA) (pp. 24 36).
[15] Zhao, W., Chen, T., Yang, J. S., & Qiu, L. (2026). AutoML Pipeline: A RAG enhanced code generation framework with pre validation for cloud native machine learning workflows. IEEE Access, 14, 41932 41945.
[16] Williams, S., & Baxter, M. (2002). Icarus Verilog: Open source Verilog more than a year later. Linux Journal, 99.
[17] Wolf, C., & Glaser, J. (2013). Yosys: A free Verilog synthesis suite. In Proceedings of the 21st Austrian Workshop on Microelectronics (Austrochip) (pp. 47 52).
[18] Cui, F., Yin, C., Zhou, K., Xiao, Y., Sun, G., Xu, Q., Guo, Q., Liang, Y., Zhang, X., Song, D., & Others. (2024). OriGen: Enhancing RTL code generation with code to code augmentation and self reflection. In Proceedings of the IEEE/ACM International Conference on Computer Aided Design (ICCAD) (pp. 1 9).
[19] Yang, Y., Teng, F., Liu, P., Qi, M., Lv, C., Li, J., Zhang, X., & He, Z. (2025). HaVen: Hallucination mitigated LLM for Verilog code generation aligned with HDL engineers. In Proceedings of the Design, Automation and Test in Europe (DATE) (pp. 1 7).
[20] Wang, Y., Sun, G., Ye, W., Qu, G., & Li, A. (2025). VeriReason: Reinforcement learning with testbench feedback for reasoning enhanced Verilog generation. arXiv preprint arXiv:2505.11849.
[21] Pinckney, N., Deng, C., Ho, C. T., Tsai, Y. D., Liu, M., Zhou, W., Khailany, B., & Ren, H. (2025). Comprehensive Verilog design problems: A next generation benchmark dataset for evaluating large language models and agents on RTL design and verification. arXiv preprint arXiv:2506.14074.
[22] Ahmad, H., Huang, Y., & Weimer, W. (2022). CirFix: Automatically repairing defects in hardware design code. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) (pp. 990 1003).
[23] Le Goues, C., Nguyen, T., Forrest, S., & Weimer, W. (2012). GenProg: A generic method for automatic software repair. IEEE Transactions on Software Engineering, 38(1), 54 72.
[24] Xu, B., Ma, Y., Wang, C., Xia, Y., & Others. (2024). Location is key: Leveraging large language model for functional bug localization in Verilog. arXiv preprint arXiv:2409.15186.
[25] Vasudevan, S., Sheridan, D., Patel, S., Tcheng, D., Tuohy, B., & Johnson, D. (2010). GoldMine: Automatic assertion generation using data mining and static analysis. In Proceedings of the Design, Automation and Test in Europe (DATE) (pp. 626 629).
[26] Jones, J. A., & Harrold, M. J. (2005). Empirical evaluation of the Tarantula automatic fault localization technique. In Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering (ASE) (pp. 273 282).
[27] Abreu, R., Zoeteweij, P., & van Gemund, A. J. C. (2006). An evaluation of similarity coefficients for software fault localization. In Proceedings of the 12th Pacific Rim International Symposium on Dependable Computing (PRDC) (pp. 39 46).
[28] Abreu, R., Zoeteweij, P., & van Gemund, A. J. C. (2007). On the accuracy of spectrum based fault localization. In Proceedings of Testing: Academic and Industrial Conference Practice and Research Techniques (TAICPART MUTATION) (pp. 89 98).
[29] Wu, J., Zhang, Z., Yang, D., Meng, X., He, J., Mao, X., & Lei, Y. (2022). Fault localization for hardware design code with time aware program spectrum. In Proceedings of the IEEE International Conference on Computer Design (ICCD) (pp. 537 544).
[30] Mo, T., Zhang, C., Zou, J., Guo, Z., & Rhee, M. (2026). Self evolving AI agents with dual memory for automated software testing and bug localization. IEEE Access, 14, 111086 111102.
[31] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023). ReAct: Synergizing reasoning and acting in language models. In Proceedings of the International Conference on Learning Representations (ICLR).
[32] Wu, Q., Bansal, G., Zhang, J., Wu, Y., Zhang, S., Zhu, E., Li, B., Jiang, L., Zhang, X., & Wang, C. (2023). AutoGen: Enabling next gen LLM applications via multi agent conversation framework. arXiv preprint arXiv:2308.08155.
[33] Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024). SWE agent: Agent computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems (Vol. 37, pp. 50528 50652).
[34] Rhee, M., Zou, J., Mo, T., Teng, D., & Yang, J. S. (2026). Enhancing web search agents with self play contrastive fine tuning. IEEE Access. https://doi.org/10.1109/ACCESS.2026.3717594
[35] Zhang, H., Ge, Y., Zhao, X., & Wang, J. (2025). Hierarchical deep reinforcement learning for multi objective integrated circuit physical layout optimization with congestion aware reward shaping. IEEE Access, 13, 162533 162551.
[36] Brayton, R., & Mishchenko, A. (2010). ABC: An academic industrial strength verification tool. In Proceedings of the International Conference on Computer Aided Verification (CAV) (pp. 24 40).
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Transactions on Computer Science and Intelligent Systems Research
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
