feat: implement DuckDB filesystem integration for Vortex file handling

[iceboundrock]

• This PR addresses DuckDB extension to use DuckDB FileSystem #5767
• This PR utilizes AI (codex from OpenAI) to generate code.

[codspeed-hq]

Merging this PR will not alter performance

✅ 1135 untouched benchmarks
⏩ 1268 skipped benchmarks¹

---

_{Comparing Lychee-Technology:develop (b2db9de) with develop (6f10740)}

Footnotes

1. 1268 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[0ax1]

Thanks for taking the time to look into this. There's a couple of issues around thread safety / locking, error handling and memory leaks.

If you used AI to assist in writing the code for your PR please mention this in your PR description (see: https://github.com/vortex-data/vortex/blob/develop/CONTRIBUTING.md).

[0ax1]

Shouldn't we also hook into the duckdb_fs_glob for local paths? Or asked differently, should we make any assumptions at all here about globbing details of duckdb_fs_glob but rather unconditionally delegate to duckdb? Whether or not gcs is supported is now a Duckdb implementation detail.

[0ax1]

My suggestion is to just do this let file_urls = duckdb_fs_glob(..).

[iceboundrock]

ok, I will move the local file handling logic into duckdb_fs_glob

[0ax1]

Almost there, thanks for taking the time to address the comments!

[iceboundrock]

Almost there, thanks for taking the time to address the comments!

Hi @0ax1 , any other comments?

[0ax1]

Almost there, thanks for taking the time to address the comments!

Hi @0ax1 , any other comments?

Currently swamped with other work but will get back to it when I find a free moment.

[gatesn]

@iceboundrock - quick question on this. Do you know if it's possible to grab the config back out from DuckDB's filesystems?

I don't want to block this PR, don't worry(!), but we do have plans to implement a high performance I/O layer at some point and it would probably make more sense to try and configure our own I/O using duckdb configs, rather than attempt to optimize calling back into DuckDB's I/O layer.

[iceboundrock]

@iceboundrock - quick question on this. Do you know if it's possible to grab the config back out from DuckDB's filesystems?

I don't want to block this PR, don't worry(!), but we do have plans to implement a high performance I/O layer at some point and it would probably make more sense to try and configure our own I/O using duckdb configs, rather than attempt to optimize calling back into DuckDB's I/O layer.

Could you please elaborate on what config refers to? If it’s S3 credentials, HTTP Proxy configuration, etc., I don’t believe we can read them from DuckDB. They are stored in Secrets Manager and the read API redacts sensitive information.

We probably can do the same as the httpfs extension does. Code link:

• 1
• 2