Can AI be encoded? Research Mapping Barriers to Automatic Software Engineering | MIT News

Imagine a future where AI quietly spreads the boom of software development: refactoring tangled code, migrating legacy systems and pursuing racial conditions so that human engineers can dedicate themselves to architecture, design and real novelty issues, but still beyond the influence of machines. Recent advancements seem to have driven the temptation of the future, but a new paper (CSAIL) by researchers at the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) and some collaborative institutions believe that this potential future reality requires a difficult view of today’s challenges.

Titled “Challenges and Ways to AI for Software Engineering,” the work draws many software engineering tasks beyond code generation, identifies current bottlenecks, and highlights research directions to overcome them, aiming to focus humans on advanced design, while routine work is automated.

“Everyone is talking about how we no longer need programmers and now we can use all of this automation,” said Armando Solar-Lezama, MIT professor of electrical engineering and computer science, CSAIL, and senior author of the study. “On the one hand, there’s a huge step forward in the field. The tools we have are stronger than anything we’ve seen before. But there’s still a long way to go to really get the full promise of automation we’d expect.”

Solar-Lezama believes that popular narratives often narrow down software engineering to “undergraduate programming part: someone handed you a small feature specification that you could implement, or solve Leetcode-style programming interviews.” The real practice is much broader. It includes refactoring of everyday designs, as well as a thorough migration that transfers millions of rows from Cobol to Java and reshapes the entire enterprise. It requires uninterrupted testing and analysis (fuzzy, attribute-based testing, and other methods) to catch concurrent errors or patch zero-day defects. It involves maintaining grinding: a ten-year-old code that summarizes the change history of new teammates, and viewing the pull request for “style, performance, and security”.

Industry-scale code optimization – considering the relentless multi-layer patching behind retuning the GPU core or Chrome’s V8 engine – is still difficult to evaluate. Today’s standard metrics are designed for short, independent questions, and although multiple choice tests still dominate natural language research, they are never the norm for AI code. The de facto standard Swe-Bench in the field just requires a model to patch the GitHub problem: useful, but still similar to the “undergraduate programming exercise” paradigm. It touches only a few hundred lines of code, risks leaking data from public repositories, and ignores other real-world contexts – AI-assisted refactoring, human-AI-critical rewrites of programming or involving millions of lines. Until the benchmarks expand to capture those higher risk situations, measuring progress and thus accelerating situations will be an open challenge.

If measurement is one obstacle, then human-computer communication is another obstacle. Alex Gu, a graduate student at MIT in electrical engineering and computer science, sees today’s interactions as “thin communication.” When he asks the system to generate code, he often receives a large, unstructured file or even a set of unit tests, but these tests are often superficial. The gap extends to the ability of AI to effectively use a wider range of software engineering tools, from debaters to static analyzers, which are the ability of humans to rely on for precise control and deeper understanding. “I don’t have much control over the writing of the model,” he said. “If there is no channel for AI, it can reveal your confidence – ‘This part is right… This part, maybe double checked’ – the hallucinatory logic that developers might compile blindly trustworthy, but crash but crash in production. Another key aspect is letting AI know when to delay the user to clarify.”

Proportional proportionalization of these difficulties. Current AI models struggle hard in large code bases, often covering millions of lines. The basic models are learned from public Github, GU says, but “every company’s code base is different and unique,” Gu says, making proprietary coding conventions and specification requirements fundamentally undistributed. The result is code that looks reasonable but is called a non-existent feature, violates internal style rules or fails to continually integrate the pipeline. This often leads to AI-generated code “illusion”, meaning it creates something that looks reasonable but doesn’t match the specific internal conventions, assistant functions, or building patterns of a given company.

Models are also often retrieved incorrectly because it retrieves code with a similar name (synchronym) rather than functionality and logic, which is code that the model may need to know how to write functions. “Standard retrieval techniques are easily fooled by making the same but seemingly different code pieces,” Solar -Lezama said.

The authors mention that since there is no silver bullet to these issues, they’re calling instead for community‑scale efforts: richer, having data that captures the process of developers writing code (for example, which code developers keep versus throw away, how code gets retro time, etc.), shared evaluation suites that measure progress on reactor quality, bug‑fix longevity, and migration correctness; and transparent tools that can expose uncertainty and invite humans to turn instead of passive acceptance. The GU sees the agenda as a “cale of action” for larger open source collaborations, and no lab can be called separately. Solar -Lezama imagines incremental advances – “The findings take a bite from these challenges separately” – feeding it back into commercial tools and gradually shifting AI from auto-filled peers to real engineering partners.

“Why doesn’t this matter? Software has supported the details of finance, transportation, healthcare and everyday life, as well as the human effort required to build and maintain safely is becoming a bottleneck. An artificial intelligence that can take the grunt and can not introduce hidden failures without introducing hidden failures – can unleash developers’ creativity, strategy, strategy and ethics,” said. “But the future depends on realizing that code completion is the easy part; the difficult part is everything else. Our goal is not to replace programmers. This is the goal of amplifying their goals. When AI can solve the tedious tediousness and horror, human engineers can eventually spend their time on the only human thing. ”

“With so many new works in AI, the community often chases the latest trends, so it’s hard to take a step back and reflect on which issues are most important to solve,” said Baptiste Rozière, an AI scientist at Mistral AI. “I like reading this article because it clearly outlines the critical tasks and challenges of AI in software engineering. It also outlines the promising directions for future research in the field.”

GU and Solar-Lezama are with Koushik Sen, University of California Berkeley professors, and Phd students Naman Jain and Manish Shetty, Cornell Assistant Professor Kevin Ellis and PhD students Wen-Ding Li, Stanford University University Diyi Yang Yang and Phd Student Yijia Shao Yijia Shao and John Sypect ZIING ZIING ZIYING ZIYING ZIYING ZIYEN. Their work is supported by the National Science Foundation (NSF), Sky Laboratory industrial sponsor and affiliate, Intel Corporation through the Office of NSF grants and Naval Research.

Researchers are presenting their work at the International Conference on Machine Learning (ICML).