Databend Roadmap for 2025

[BohuTANG]

Databend Roadmap 2024-2025 · Multimodal Data+AI Warehouse

Messaging Anchor

• Hero positioning (from README): Blazing analytics, fast search, geo insights, vector AI — supercharged in a new-era Snowflake-compatible warehouse.
• Objective: every 2024-2025 deliverable should reinforce Databend as a Snowflake-compatible, multimodal warehouse that unifies BI, search, geo, and vector AI.

2024: Cloud-Native & Snowflake-Like Experience

• Cloud-native parity: Deliver a fully-managed experience comparable to Snowflake Cloud.
• 50% cost reduction: Keep the cost/performance advantage through storage and compute optimizations.
• Try Databend Cloud: https://www.databend.com/

---

2025: On-Prem Replacement with Multimodal Depth

Focus areas inherited from the README pillars:

• Analytics-native: ANSI SQL, window functions, incremental aggregates, streaming ingest.
• Search-native: JSON/full-text indexes and structured filters for hybrid search.
• Geo-native: Spatial data types plus indexing (H3/GeoHash) to power location intelligence.
• Vector-native: Vector types, embeddings, and HNSW indexes for semantic retrieval/RAG.
• Unified engine & data: One optimizer/runtime across structured, semi-structured, vector, and unstructured data on object storage.

Main Tasks for 2025

Task	Status	Comments
Dynamic Cluster Management	Done	Make query nodes more dynamic for better resource allocation.
Resource Group Management	Done	Enhance on-prem query control through better resource group management.
Disaster Recovery	Done	Implement multi-region failover solutions including point-in-time recovery and backup automation.
Stability Improvements	In Progress	Increase on-prem stability and fault tolerance for higher reliability.
Search Improvements	In Progress	v1.2.830-nightly shipped inverted indexes for VARIANT inner fields via #18861; next steps: expose MATCH/LIMIT latency tracking + VARIANT FULLTEXT scoring.
Spatial Indexing for Geometry	Planned	Geo defaults (v1.2.832-nightly, #18873) and HTTP geometry output (v1.2.841-nightly, #18963) are in; next deliverable is actual spatial/H3 index structures.
Vector Index & Embeddings Enhancements	Planned	Vector index SIMD quantization (v1.2.846-nightly, #18957) and native storage fixes (v1.2.836-nightly, #18932) landed; next up is multi-probe HNSW + native embeddings catalog.
Analytics: Table Statistics Admin API	Done	v1.2.841-nightly added #18967, giving cost-based optimizations visibility into per-table/per-column stats.
Analytics: Fuse Parquet Dictionary Toggle	Done	v1.2.840-nightly introduced enable_parquet_dictionary so BI workloads can tune compression vs. speed.
Search: External Lake Virtual Columns	Done	v1.2.844-nightly merged #18981, enabling virtual columns on external tables to project search/index keys without copy.
Geo: Defaults & Client Output	Done	v1.2.832-nightly turned on geo + virtual-column toggles by default (#18873); v1.2.841-nightly added HTTP geometry_output_format (#18963).
Vector: Index Performance Hardening	Done	v1.2.846-nightly + #18957 accelerate quantization; v1.2.836-nightly + #18932 fix panic paths when mixing vector indexes with native storage.

[GaoYusong]

Congratulations on what Databend has achieved in such a short time. Looking forward to 2024!

[Xuanwo]

What exactly is the Support Python Worksheet? Does it enable running Python in Databend?

[ZhiHanZ]

Any plan for SQL transaction and stored procedures?

[ZhiHanZ]

besides, I think we could also support query queueing, warehouse automatica scaling based on pending queue and separate another coordinator component for dispatching physical plan to warehouse compute node.

[BohuTANG]

besides, I think we could also support query queueing, warehouse automatica scaling based on pending queue and separate another coordinator component for dispatching physical plan to warehouse compute node.

This is a part of Enhancements to Concurrency and Scheduler.

[BohuTANG]

What exactly is the Support Python Worksheet? Does it enable running Python in Databend?

The goal is to make the Hugging Face Model + Python + GPU( or CPU) + Data in Databend is possible.

[djouallah]

all I want is to be able to read a delta table from a local path :)

[JasonLi-cn]

How to understand Inference ？Which abilities does it refer to?

[BohuTANG]

How to understand Inference ？Which abilities does it refer to?

Move the models(huggingface models) to the database, the database can load and run them.

[keltia]

Thanks for making this available to everyone. I'm currently interested to anything that can deal with Geospatial stuff so adding Geometry support is very nice.

[BohuTANG]

Thanks for making this available to everyone. I'm currently interested to anything that can deal with Geospatial stuff so adding Geometry support is very nice.

Working on it: #14470

[inviscid]

Thanks for making this available to everyone. I'm currently interested to anything that can deal with Geospatial stuff so adding Geometry support is very nice.

@keltia Databend uses H3 for geospatial operations. Is that what you are referring to? https://docs.databend.com/sql/sql-functions/geo-functions/

[inviscid]

Would it be possible to restructure the code a little to help with creating a fully OSS compliant version of Databend?

I know the license info specifically calls out the ee directories but we have found some other files that are also covered by the Elastic license that seem to be part of the core functionality.

• /src/meta/binaries/meta/ee-main.rs
• /src/binaries/query/ee-main.rs

Many companies will not adopt a technology that doesn't use an approved OSS license, which Apache 2.0 is, but Elastic is not (open-source vs. source-available).

In an ideal world there would be a databend-oss repo and a databend-ee repo with the latter adding all of the Enterprise features and licensing on top of the OSS version.

I know the goal is to get enterprise customers to buy a license but I think you may be losing out on more community support and growth because devs restricted by corporate governance policies will never be able to even give Databend a test drive. Plus, like many grassroots database projects within a company, things start small and cheap then turn into mission critical systems requiring licensing. If we skip the small and cheap phase, the licensing never comes in those cases.

[johnpyp]

Agree, the mixing and matching of licenses both in docs conceptually and in the literal code makes it really hard to be able to use this database.