Analysis of the Code Quality of Code Automatic Generation Tool Github Copilot

Github Copilot is a generative AI-based code auto-generation tool launched by Github and OpenAI in 2022.One of its core functions is to generate corresponding implementation code based on natural language annotations describing functions.This expansion of AI in the field of programming has attracted...

Full description

Bibliographic Details
Published in:Jisuanji kexue
Main Author: WANG Dongyu, MO Ran, ZHAN Wenjing, JIANG Yingjie
Format: Article
Language:Chinese
Published: Editorial office of Computer Science 2025-07-01
Subjects:
Online Access:https://www.jsjkx.com/fileup/1002-137X/PDF/1002-137X-2025-52-7-37.pdf
Description
Summary:Github Copilot is a generative AI-based code auto-generation tool launched by Github and OpenAI in 2022.One of its core functions is to generate corresponding implementation code based on natural language annotations describing functions.This expansion of AI in the field of programming has attracted heated discussion and attention in recent years.At this stage,people's focus is mainly on the comparison between AI programming and human programming,such as the comparison of programming efficiency and code performance between AI programmers and human programmers.However,there is currently limited research on the characteristics of Copilot-generated code itself,particularly regarding code quality issues,such as defects in the AI-generated code,whether these defects might lead to program errors,and the understandability of the code.Code quality directly determines the life and durability of a software project.Analyzing and summarizing its code quality characteristics helps to better use and improve such AI code tools.This paper utilizes tools to extract all open-source problems from LeetCode(2,033 in total) as data samples to test Copilot,generating code suggestions in three programming languages(Java,JavaScript,and Python),submitting them,and recording the execution results of the generated code.By statically analyzing the code suggestions with SonarQube and integrating their execution results,this paper evaluates Copilot's code quality in terms of reliability,maintainability,and complexity.The results reveal that:1)Copilot-generated code is relatively reliable.For Java,JavaScript,and Python,7,5,and 9 types of bugs are identified respectively.The proportion of code suggestions involving bugs do not exceed 3% across all three languages,but over 50% of bug-related code suggestions fail test cases.2)Copilot's code suggestions exhibit poor maintainability.For Java,JavaScript,and Python,47,23,and 20 types of code smells are detected respectively.Over 40% of code suggestions in all three languages contain code smells,and more than 50% of smell-related suggestions failetest cases.3)Copilot-generated code is easy to understand.The complexity of most code suggestions do not exceed predefined thresholds,with less than 6% of suggestions flag for excessive complexity.Finally,based on the experimental findings,practical recommendations for improving Copilot are proposed,and potential future research directions for such tools are discussed.
ISSN:1002-137X