Snowflake proposes Excot: a novel AI framework that iteratively optimizes open source LLMS by combining COT reasoning with non-policy and policy DPOs, relying solely on execution accuracy as feedback

Text-to-SQL translation is the task of converting natural language queries into structured SQL statements and is crucial to fostering user-friendly database interactions. However, this task involves significant complexity, especially pattern linking, handling constituent SQL syntax, and resolving ambiguity in user queries. Although large language models (LLMS) show powerful capabilities in various fields, structured inference techniques (e.g., by thinking chains (COT)) are still limited in text-to-SQL contexts. The prior attempt to adopt zero-emission COT or direct preference optimization (DPO) in the absence of structural reasoning yielded marginal improvements, suggesting a more stringent methodology is needed.

Snowflake launched Excot, a structured framework designed to optimize open source LLMS through a combination of COT inference and iterative priority optimization optimization, especially leveraging exclusive guidance through execution accuracy feedback and up-to-policy DPO. Excot relies on internally generated inference steps and execution results rely on external reward models and human annotations. The method runs in two main stages: Initially, it generates candidate COT data validated by non-policy DPOs, forming the basis for oversight fine-tuning. Subsequently, the model iteratively generates and perfects COT data, and gradually improves accuracy through policy DPO through feedback derived from execution correctness.

Excot uses detailed COT inference, especially split and splicing strategies, where complex queries are broken down into simpler subconquests. In a coherent final query, each subproblem is analyzed and resolved independently. This structured decomposition allows the model to more effectively manage complexities and nested structures common in SQL operations. Execution-based verification is the core mechanism of correctness evaluation, in which the query is verified by comparing its execution output with the ground truth results. Errors and correct queries are system-paired, providing a clear signal for preference-based learning. Iterative improvements in the above policy DPO phase gradually improve the model’s inference accuracy.

Excot’s experimental evaluation showed a significant improvement in execution accuracy. Specifically, using the Llama-3.1 70B model, Excot increased the execution accuracy of bird development to 57.37% to 68.51%, and improved the performance of spider tests from 78.81% to 86.59%. Comparable performance enhancements were recorded using the QWEN-2.5-CODER 32B model. These results position Excot as a leading approach to single-mode evaluation of these benchmarks, surpassing established methods such as xiyansql and proprietary models including OpenAI variants. It is worth noting that these improvements always maintain a high query validity rate (more than 98%), confirming enhanced semantic correctness as well as syntactic accuracy.

In summary, Excot represents a methodical advancement in open source LLMS structured inference optimization applied to text-to-SQL tasks. By integrating structured COT inference with preference optimization, booted by execution-based feedback only, Excot effectively addresses the limitations identified in previous methods. Its iterative improvement capabilities ensure continuous improvements that do not rely on external reward structures or manual annotations. Further research may explore the extension of this framework to more complex pattern environments and other structured inference tasks, thereby expanding the applicability and reliability of LLM in structured query generation environments.

Check paper, github pages and details. All credits for this study are to the researchers on the project. Also, please stay tuned for us twitter And don’t forget to join us 85k+ ml reddit.

🔥 (Register now) Open Source AI’s Minicon Virtual Conference: Free Registration + Attendance Certificate + 3-hour Short Event (April 12, 9am to 12pm)

Asif Razzaq is CEO of Marktechpost Media Inc. As a visionary entrepreneur and engineer, ASIF is committed to harnessing the potential of artificial intelligence to achieve social benefits. His recent effort is to launch Marktechpost, an artificial intelligence media platform that has an in-depth coverage of machine learning and deep learning news that can sound both technically, both through technical voices and be understood by a wide audience. The platform has over 2 million views per month, demonstrating its popularity among its audience.