Analysis of the Code Quality of Code Automatic Generation Tool Github Copilot

Github Copilot is a generative AI-based code auto-generation tool launched by Github and OpenAI in 2022.One of its core functions is to generate corresponding implementation code based on natural language annotations describing functions.This expansion of AI in the field of programming has attracted...

Full description

Bibliographic Details
Published in:Jisuanji kexue
Main Author: WANG Dongyu, MO Ran, ZHAN Wenjing, JIANG Yingjie
Format: Article
Language:Chinese
Published: Editorial office of Computer Science 2025-07-01
Subjects:
Online Access:https://www.jsjkx.com/fileup/1002-137X/PDF/1002-137X-2025-52-7-37.pdf
_version_ 1849362043992276992
author WANG Dongyu, MO Ran, ZHAN Wenjing, JIANG Yingjie
author_facet WANG Dongyu, MO Ran, ZHAN Wenjing, JIANG Yingjie
author_sort WANG Dongyu, MO Ran, ZHAN Wenjing, JIANG Yingjie
collection DOAJ
container_title Jisuanji kexue
description Github Copilot is a generative AI-based code auto-generation tool launched by Github and OpenAI in 2022.One of its core functions is to generate corresponding implementation code based on natural language annotations describing functions.This expansion of AI in the field of programming has attracted heated discussion and attention in recent years.At this stage,people's focus is mainly on the comparison between AI programming and human programming,such as the comparison of programming efficiency and code performance between AI programmers and human programmers.However,there is currently limited research on the characteristics of Copilot-generated code itself,particularly regarding code quality issues,such as defects in the AI-generated code,whether these defects might lead to program errors,and the understandability of the code.Code quality directly determines the life and durability of a software project.Analyzing and summarizing its code quality characteristics helps to better use and improve such AI code tools.This paper utilizes tools to extract all open-source problems from LeetCode(2,033 in total) as data samples to test Copilot,generating code suggestions in three programming languages(Java,JavaScript,and Python),submitting them,and recording the execution results of the generated code.By statically analyzing the code suggestions with SonarQube and integrating their execution results,this paper evaluates Copilot's code quality in terms of reliability,maintainability,and complexity.The results reveal that:1)Copilot-generated code is relatively reliable.For Java,JavaScript,and Python,7,5,and 9 types of bugs are identified respectively.The proportion of code suggestions involving bugs do not exceed 3% across all three languages,but over 50% of bug-related code suggestions fail test cases.2)Copilot's code suggestions exhibit poor maintainability.For Java,JavaScript,and Python,47,23,and 20 types of code smells are detected respectively.Over 40% of code suggestions in all three languages contain code smells,and more than 50% of smell-related suggestions failetest cases.3)Copilot-generated code is easy to understand.The complexity of most code suggestions do not exceed predefined thresholds,with less than 6% of suggestions flag for excessive complexity.Finally,based on the experimental findings,practical recommendations for improving Copilot are proposed,and potential future research directions for such tools are discussed.
format Article
id doaj-art-a63b58f650044f79b419fc4f511709b1
institution Directory of Open Access Journals
issn 1002-137X
language zho
publishDate 2025-07-01
publisher Editorial office of Computer Science
record_format Article
spelling doaj-art-a63b58f650044f79b419fc4f511709b12025-08-27T02:47:16ZzhoEditorial office of Computer ScienceJisuanji kexue1002-137X2025-07-01527374910.11896/jsjkx.240600076Analysis of the Code Quality of Code Automatic Generation Tool Github CopilotWANG Dongyu, MO Ran, ZHAN Wenjing, JIANG Yingjie0School of Computer Science, Central China Normal University, Wuhan 430079, ChinaGithub Copilot is a generative AI-based code auto-generation tool launched by Github and OpenAI in 2022.One of its core functions is to generate corresponding implementation code based on natural language annotations describing functions.This expansion of AI in the field of programming has attracted heated discussion and attention in recent years.At this stage,people's focus is mainly on the comparison between AI programming and human programming,such as the comparison of programming efficiency and code performance between AI programmers and human programmers.However,there is currently limited research on the characteristics of Copilot-generated code itself,particularly regarding code quality issues,such as defects in the AI-generated code,whether these defects might lead to program errors,and the understandability of the code.Code quality directly determines the life and durability of a software project.Analyzing and summarizing its code quality characteristics helps to better use and improve such AI code tools.This paper utilizes tools to extract all open-source problems from LeetCode(2,033 in total) as data samples to test Copilot,generating code suggestions in three programming languages(Java,JavaScript,and Python),submitting them,and recording the execution results of the generated code.By statically analyzing the code suggestions with SonarQube and integrating their execution results,this paper evaluates Copilot's code quality in terms of reliability,maintainability,and complexity.The results reveal that:1)Copilot-generated code is relatively reliable.For Java,JavaScript,and Python,7,5,and 9 types of bugs are identified respectively.The proportion of code suggestions involving bugs do not exceed 3% across all three languages,but over 50% of bug-related code suggestions fail test cases.2)Copilot's code suggestions exhibit poor maintainability.For Java,JavaScript,and Python,47,23,and 20 types of code smells are detected respectively.Over 40% of code suggestions in all three languages contain code smells,and more than 50% of smell-related suggestions failetest cases.3)Copilot-generated code is easy to understand.The complexity of most code suggestions do not exceed predefined thresholds,with less than 6% of suggestions flag for excessive complexity.Finally,based on the experimental findings,practical recommendations for improving Copilot are proposed,and potential future research directions for such tools are discussed.https://www.jsjkx.com/fileup/1002-137X/PDF/1002-137X-2025-52-7-37.pdfautomatic code generation|code quality|code reliability|code maintainability|code complexity
spellingShingle WANG Dongyu, MO Ran, ZHAN Wenjing, JIANG Yingjie
Analysis of the Code Quality of Code Automatic Generation Tool Github Copilot
automatic code generation|code quality|code reliability|code maintainability|code complexity
title Analysis of the Code Quality of Code Automatic Generation Tool Github Copilot
title_full Analysis of the Code Quality of Code Automatic Generation Tool Github Copilot
title_fullStr Analysis of the Code Quality of Code Automatic Generation Tool Github Copilot
title_full_unstemmed Analysis of the Code Quality of Code Automatic Generation Tool Github Copilot
title_short Analysis of the Code Quality of Code Automatic Generation Tool Github Copilot
title_sort analysis of the code quality of code automatic generation tool github copilot
topic automatic code generation|code quality|code reliability|code maintainability|code complexity
url https://www.jsjkx.com/fileup/1002-137X/PDF/1002-137X-2025-52-7-37.pdf
work_keys_str_mv AT wangdongyumoranzhanwenjingjiangyingjie analysisofthecodequalityofcodeautomaticgenerationtoolgithubcopilot