NL2SQL360: A Multi-angle Evaluation Framework

Boyan Li1, Yuyu Luo1,2, Chengliang Chai3, Guoliang Li4, Nan Tang1,2,
1HKUST(GZ), 2HKUST, 3Beijing Institute of Technology, 4Tsinghua University
Email to: bli303@connect.hkust-gz.edu.cn , yuyuluo@hkust-gz.edu.cn
Paper Code Package Data Poster Leaderboard
NL2SQL Overview **NL2SQL360** is a multi-angle evaluation framework. It can assess different NL2SQL methods against established benchmarks or tailor their evaluations based on specific criteria (e.g., varying data domains or SQL characteristics).

Abstract

Translating users' natural language questions into SQL queries (i.e., NL2SQL) significantly lowers the barriers to accessing relational databases. The emergence of Large Language Models has introduced a novel paradigm in NL2SQL tasks, enhancing capabilities dramatically. However, this raises a critical question: *Are we fully prepared to deploy NL2SQL models in production?* To address the posed questions, we present a multi-angle NL2SQL evaluation framework, NL2SQL360, to facilitate the design and test of new NL2SQL methods for researchers. Through NL2SQL360, we conduct a detailed comparison of leading NL2SQL methods across a range of application scenarios, such as different data domains and SQL characteristics, offering valuable insights for selecting the most appropriate NL2SQL methods for specific needs. Moreover, we explore the NL2SQL design space, leveraging NL2SQL360 to automate the identification of an optimal NL2SQL solution tailored to user-specific needs. Specifically, NL2SQL360 identifies an effective NL2SQL method, SuperSQL, distinguished under the Spdier dataset using the execution accuracy metric. Remarkably, SuperSQL achieves competitive performance with execution accuracy of 87% and 62.66% on the Spider and BIRD test sets, respectively.

NL2SQL360 Framework

infrastructure Overview of **NL2SQL360**, which is featured by: - Easy-to-use Evaluation: Command Line Usage / Python Code Usage. - Integrated Metrics: Execution Accuracy / Exact-Match Accuracy / Valid Efficiency Score / Question Variance Testing. - Multi-angle Performance: Fine-grained performance (JOIN, Sub-query, etc.) / Scenario-based (Business Intelligence, etc.)

NL2SQL360-AAS: NL2SQL Automated Architecture Search Algorithm

executable environment - A modular NL2SQL design space. - An algorithm to automatically search NL2SQL solutions for specific scenarios. - Our case study of SuperSQL shows competitive results in Spider and BIRD datasets.

Leaderboard

We experiment with state-of-the-art LLM-based and PLM-based NL2SQL methods in Spider Dataset. We show the overall results into different subsets, including JOIN, Subquery, QVT, and Competition Domain. **We are actively updating the benchmark with new methods. Pull requests welcomed!** đź‘Ź
Rank Model Details Score

1

Aug 29, 2024 		SFT CodeS-15B

LLM-based

Finetuning-based

 84.9

2

Aug 29, 2024 		RESDSQL-3B + NatSQL

PLM-based

Finetuning-based

 84.1

3

Aug 29, 2024 		DAILSQL + Self-Consistency

LLM-based

Prompting-based

 83.6

4

Aug 29, 2024 		DAILSQL

LLM-based

Prompting-based

 83.1

5

Aug 29, 2024 		DINSQL

LLM-based

Prompting-based

 82.8

6

Aug 29, 2024 		C3SQL

LLM-based

Prompting-based

 82.0

BibTeX

@misc{li2024dawn,
        title={The Dawn of Natural Language to SQL: Are We Fully Ready?}, 
        author={Boyan Li and Yuyu Luo and Chengliang Chai and Guoliang Li and Nan Tang},
        year={2024},
        eprint={2406.01265},
        archivePrefix={arXiv},
        primaryClass={id='cs.DB' full_name='Databases' is_active=True alt_name=None in_archive='cs' is_general=False description='Covers database management, datamining, and data processing. Roughly includes material in ACM Subject Classes E.2, E.5, H.0, H.2, and J.1.'}
  }