suyiyun/llm-text2sql-taxonomy — explained in plain English
Analysis updated 2026-05-18
Read the paper and code to understand how text-to-SQL systems are classified by autonomy level.
Reproduce the Spider benchmark case study comparing chain-of-thought and non-chain-of-thought fine-tuning.
Use the case_study_spider_cot pipeline as a template for a similar fine-tuning experiment.
Reference the leaderboard aggregation approach when surveying existing text-to-SQL systems.
| suyiyun/llm-text2sql-taxonomy | 0pen-sourcer/hearth | 13127905/deep-learning-based-air-gesture-text-recognition- | |
|---|---|---|---|
| Stars | 15 | 15 | 15 |
| Language | Python | Python | Python |
| Setup difficulty | hard | moderate | moderate |
| Complexity | 4/5 | 3/5 | 3/5 |
| Audience | researcher | vibe coder | developer |
Figures from each repo's GitHub metadata at analysis time.
Reproducing the case study requires GPU fine-tuning of Qwen3-8B/LLaMA-3.1-8B and DeepSeek V3 access for CoT generation.
This repository is the official code release for a research paper called "Agentic-SQL Revisited: Autonomy-Based Taxonomy and Empirical Benchmark Analysis for LLM Text-to-SQL." It supports academic work on text-to-SQL, meaning systems that turn a plain English question into a database query written in SQL. The paper's core idea is to treat evaluation of these systems as a leaderboard problem. Instead of running every system itself, it collects results that other researchers have already reported and organizes them along a single scale of how much the system acts on its own, running from constrained systems, through those that use examples in context, to iterative systems, then agentic systems, and finally systems where reasoning is built directly into the model. The repository also includes a focused case study built on the Spider benchmark, a widely used dataset for testing text-to-SQL systems, to test this scale with real numbers. The code is organized into four folders. The leaderboard folder is meant to hold a Python tool for gathering and organizing the reported results, marked in the README as a placeholder still to be filled in. The case_study_spider_cot folder contains the full pipeline for the empirical case study: preparing the Spider data, generating step by step reasoning, called chain of thought, with DeepSeek V3, fine tuning smaller models, Qwen3-8B and LLaMA-3.1-8B, using a technique called LoRA, comparing versions with and without the reasoning step, and comparing all of that against DeepSeek V3 and GLM-4 used with three examples. A docs folder holds supporting material such as dataset statistics and a paper outline, and a papers folder has the PDF of this paper plus a companion paper introducing the taxonomy this work builds on. For instructions on running the case study itself, from extracting database schemas to generating reasoning data, building fine tuning data, training, and evaluating results, the README points to a separate README inside the case_study_spider_cot folder. This is a research artifact aimed at people working on natural language to database query systems, not a general purpose tool.
Official code for a research paper that ranks Text-to-SQL AI systems by how autonomous they are and includes a case study fine-tuning models with chain-of-thought reasoning.
Mainly Python. The stack also includes Python, LoRA, Qwen3-8B.
No license information is stated in the README, so terms of use are unclear.
Setup difficulty is rated hard, with roughly 1day+ to a first successful run.
Mainly researcher.
This repo across BitVibe Labs
double-check against the repo, no cap.