DB Query Engine (#2265)

* changes

* read and write working

* add lightstream example

* query wip

* EOD wip

* add db types, modular bucket, auth

* remove duplicates

* remove noop store

* remove unit ls-fast test command

* remove units fast from the client

* remove WAL setup

* remove unnecessary

* adjust auth

* wip - mssql fix

* enable NULL for lock time

* some changes to documentation

* Test Suite for query engine, adjustments to stores

Discovered a flaw in sqlstore's handling of wildcard perms during testing, adjusted
Adjustd the s3 store to enable testing with this definition instead of separate mock

Added tests for different aspects of the query engine, especially rbac and unit management.

* fix docker warnings and build fail

* add missing syncs for rbac

* adjust docs, silence SQL logs, check for existence before sync

* adjust docs - PATH to DB_PATH, add prefix

* refactor with repository in mind

* make test guide more minimal

* make sure tfe can't create units

Author: breardon2011

.gitignore

@@ -17,6 +17,7 @@ __azurite*
 *.out
 go.work.sum
 
+
 # Build directories
 dist/
 build/
@@ -30,5 +31,9 @@ bin/
 *.swo
 *~
 
-# data 
-taco/data/
+
+#data 
+data/
+
+
+taco/data/

docs/ce/state-management/query-backend.mdx

@@ -0,0 +1,184 @@
+---
+title: "Query Backend"
+---
+
+OpenTaco supports using a query backend to speed up the retrieval of objects from S3. By default SQLite will initialize, but other SQL databases can be configured if desired. If the backend is not SQLite you need to setup the database first before attempting to run statesman and populate the correct environment variables.
+
+## Configuration
+
+Set the backend type using:
+
+```bash
+TACO_QUERY_BACKEND=sqlite  # Options: sqlite, postgres, mssql, mysql
+```
+
+## SQLite (Default)
+
+SQLite is the default query backend and requires no external database server and no configuration. We expose settings for convenience but you should not need to configure SQLite in most circumstances. 
+
+### Environment Variables
+
+```bash
+# Backend selection
+TACO_QUERY_BACKEND=sqlite
+
+# SQLite-specific configuration
+TACO_SQLITE_DB_PATH=./data/taco.db
+TACO_SQLITE_CACHE=shared
+TACO_SQLITE_BUSY_TIMEOUT=5s
+TACO_SQLITE_MAX_OPEN_CONNS=1
+TACO_SQLITE_MAX_IDLE_CONNS=1
+TACO_SQLITE_PRAGMA_JOURNAL_MODE=WAL
+TACO_SQLITE_PRAGMA_FOREIGN_KEYS=ON
+TACO_SQLITE_PRAGMA_BUSY_TIMEOUT=5000
+```
+
+### Defaults
+- **Path**: `./data/taco.db`
+- **Cache**: `shared`
+- **Busy Timeout**: `5s`
+- **Max Open Connections**: `1`
+- **Max Idle Connections**: `1`
+- **Journal Mode**: `WAL`
+- **Foreign Keys**: `ON`
+
+## PostgreSQL
+
+Use PostgreSQL for better concurrency and performance in production environments.
+
+### Environment Variables
+
+```bash
+# Backend selection
+TACO_QUERY_BACKEND=postgres
+
+# PostgreSQL-specific configuration
+TACO_POSTGRES_HOST=localhost
+TACO_POSTGRES_PORT=5432
+TACO_POSTGRES_USER=postgres
+TACO_POSTGRES_PASSWORD=your_password
+TACO_POSTGRES_DBNAME=taco
+TACO_POSTGRES_SSLMODE=disable
+```
+
+### Defaults
+- **Host**: `localhost`
+- **Port**: `5432`
+- **User**: `postgres`
+- **Database Name**: `taco`
+- **SSL Mode**: `disable`
+
+### Example Connection
+
+```bash
+export TACO_QUERY_BACKEND=postgres
+export TACO_POSTGRES_HOST=my-postgres-server.example.com
+export TACO_POSTGRES_PORT=5432
+export TACO_POSTGRES_USER=taco_user
+export TACO_POSTGRES_PASSWORD=secure_password
+export TACO_POSTGRES_DBNAME=taco_prod
+export TACO_POSTGRES_SSLMODE=require
+```
+
+## Microsoft SQL Server (MSSQL)
+
+Use MSSQL for enterprise environments with existing SQL Server infrastructure.
+
+### Environment Variables
+
+```bash
+# Backend selection
+TACO_QUERY_BACKEND=mssql
+
+# MSSQL-specific configuration
+TACO_MSSQL_HOST=localhost
+TACO_MSSQL_PORT=1433
+TACO_MSSQL_USER=sa
+TACO_MSSQL_PASSWORD=your_password
+TACO_MSSQL_DBNAME=taco
+```
+
+### Defaults
+- **Host**: `localhost`
+- **Port**: `1433`
+- **Database Name**: `taco`
+
+### Example Connection
+
+```bash
+export TACO_QUERY_BACKEND=mssql
+export TACO_MSSQL_HOST=sqlserver.example.com
+export TACO_MSSQL_PORT=1433
+export TACO_MSSQL_USER=taco_admin
+export TACO_MSSQL_PASSWORD=secure_password
+export TACO_MSSQL_DBNAME=taco_db
+```
+
+## MySQL
+
+Use MySQL for compatibility with existing MySQL infrastructure. 
+
+As an example I used `CREATE DATABASE taco CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;` when testing the MySQL setup. 
+
+### Environment Variables
+
+```bash
+# Backend selection
+TACO_QUERY_BACKEND=mysql
+
+# MySQL-specific configuration
+TACO_MYSQL_HOST=localhost
+TACO_MYSQL_PORT=3306
+TACO_MYSQL_USER=root
+TACO_MYSQL_PASSWORD=your_password
+TACO_MYSQL_DBNAME=taco
+TACO_MYSQL_CHARSET=utf8mb4
+```
+
+### Defaults
+- **Host**: `localhost`
+- **Port**: `3306`
+- **User**: `root`
+- **Database Name**: `taco`
+- **Charset**: `utf8mb4`
+
+### Example Connection
+
+```bash
+export TACO_QUERY_BACKEND=mysql
+export TACO_MYSQL_HOST=mysql.example.com
+export TACO_MYSQL_PORT=3306
+export TACO_MYSQL_USER=taco_user
+export TACO_MYSQL_PASSWORD=secure_password
+export TACO_MYSQL_DBNAME=taco_production
+export TACO_MYSQL_CHARSET=utf8mb4
+```
+
+## Quick Start Examples
+
+### Development (SQLite)
+
+```bash
+# No configuration needed - SQLite is the default
+./taco
+```
+
+### Production (PostgreSQL)
+
+```bash
+export TACO_QUERY_BACKEND=postgres
+export TACO_POSTGRES_HOST=prod-db.example.com
+export TACO_POSTGRES_USER=taco_prod
+export TACO_POSTGRES_PASSWORD=$PROD_DB_PASSWORD
+export TACO_POSTGRES_DBNAME=taco
+export TACO_POSTGRES_SSLMODE=require
+
+./taco
+```
+
+## Notes
+
+- **SQLite** is best for local development and testing
+- **PostgreSQL** is recommended for production deployments
+- **MSSQL** and **MySQL** are available for enterprise compatibility
+- Database schemas are automatically initialized on first run 

docs/mint.json

@@ -59,6 +59,7 @@
         "ce/state-management/digger-integration",
         "ce/state-management/development",
         "ce/state-management/analytics",
+        "ce/state-management/query-backend",
         "ce/state-management/versioning",
         "ce/state-management/gcp-quickstart",
         "ce/state-management/aws-fargate-ad-quickstart"

go.mod

@@ -1,3 +1,28 @@
 module github.com/diggerhq/digger
 
 go 1.24.0
+
+require (
+	filippo.io/edwards25519 v1.1.0 // indirect
+	github.com/go-sql-driver/mysql v1.8.1 // indirect
+	github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9 // indirect
+	github.com/golang-sql/sqlexp v0.1.0 // indirect
+	github.com/google/uuid v1.6.0 // indirect
+	github.com/jackc/pgpassfile v1.0.0 // indirect
+	github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
+	github.com/jackc/pgx/v5 v5.6.0 // indirect
+	github.com/jackc/puddle/v2 v2.2.2 // indirect
+	github.com/jinzhu/inflection v1.0.0 // indirect
+	github.com/jinzhu/now v1.1.5 // indirect
+	github.com/kelseyhightower/envconfig v1.4.0 // indirect
+	github.com/mattn/go-sqlite3 v1.14.22 // indirect
+	github.com/microsoft/go-mssqldb v1.8.2 // indirect
+	golang.org/x/crypto v0.31.0 // indirect
+	golang.org/x/sync v0.10.0 // indirect
+	golang.org/x/text v0.21.0 // indirect
+	gorm.io/driver/mysql v1.6.0 // indirect
+	gorm.io/driver/postgres v1.6.0 // indirect
+	gorm.io/driver/sqlite v1.6.0 // indirect
+	gorm.io/driver/sqlserver v1.6.1 // indirect
+	gorm.io/gorm v1.31.0 // indirect
+)