Container support and S3 support for static storage#421
Draft
nickryand wants to merge 7 commits intooxidecomputer:mainfrom
Draft
Container support and S3 support for static storage#421nickryand wants to merge 7 commits intooxidecomputer:mainfrom
nickryand wants to merge 7 commits intooxidecomputer:mainfrom
Conversation
* Added Containerfile and compose.yml allowing for rfd-api and all other necessary binaries to be built into a container image. The compose.yml can be used to stand-up a local stack which includes the database and meilisearch. * Added log_filter support to both the rfd-api and rfd-processor configuration files. This works the same as overriding RUST_LOG but allows for configuration via file instead of environment variable. * StaticStorage is now a trait. There are two implementations for the trait. Each implementation have corresponding configuration sections. Multiple static storage locations can be configured and processor will upload static storage to all configured static storage backends. ** GCS This is the original GCS implementation. ** S3 This is a new implementation using the S3 API. * PDF storage is now optional. Disable this feature by not configuring it in the configuration toml.
* Add SecretString type for path-based secret configuration
Introduces a new rfd-secret crate with a SecretString type that allows
secrets to be specified either inline or read from a file path. This
enables storing secrets outside of configuration files (e.g., in
/run/secrets for container deployments).
Secrets can now be configured as:
- Inline: key = "my-secret"
- From file: key = { path = "/run/secrets/my-key" }
Updated secrets in rfd-api:
- keys[].private and keys[].public (JWT signing keys)
- authn.oauth.*.client_id and client_secret
- search.key
- services.github.auth.private_key and token
- magic_link.email_service.resend.key
Updated secrets in rfd-processor:
- auth.github.private_key and token
- search_storage[].key
Claude Assisted
* Add DatabaseConfig struct for structured database configuration
Replace single database_url string with a [database] config section
containing host, port, user, password, and database fields. The password
field uses SecretString to support both inline values and path-based
secrets for containerized deployments.
Claude Assisted
* Refactor rfd-kube-init to use Clap for CLI argument parsing
- Add clap dependency with derive and env features
- Create CLI structure with subcommands for extensibility
- Convert environment variable configuration to Clap derive structs
- Use Clap's native integer parsing for expiration_seconds
- Use value_delimiter for comma-separated namespace arguments
- Use argument groups to require at least one namespace type
- Refactor TokenType to tuple enum containing Option<Key>
- Track failed namespaces in vector for detailed error reporting
- Return errors from init() instead of using exit codes
Claude Assisted
* added rfd-kube-init to Containerfile
* Extract rfd-kube-init as standalone crate with improved tracing
Move rfd-kube-init out of the workspace to avoid TLS/crypto feature
conflicts between jsonwebtoken (JWT signing) and rustls (TLS). The
crate now uses explicit dependency versions with compatible crypto
backends (aws-lc-rs).
- Upgrade kube crate from 0.98 to 3.0.1
- Add structured tracing with #[instrument] attributes
- Enable RUST_LOG env filter support
- Add test for crypto provider configuration
- Update Containerfile to build rfd-kube-init separately
Claude Assisted
* testing db-init with direct to database calls (#3)
* Revert "testing db-init with direct to database calls (#3)" (#5)
This reverts commit 9d6f728.
* Init d (#6)
* testing db-init with direct to database calls
* Apply formatting to config.rs
Claude Assisted
* Add /init endpoint and oauth-init CLI command
- Add /init endpoint to rfd-api for bootstrapping OAuth client
- Support multiple redirect_uris in /init request/response
- Add oauth-init subcommand to rfd-kube-init CLI
- Distribute OAuth credentials to target Kubernetes namespaces
- Idempotent: returns success on 409 Conflict
* cleaned up status check and removed redundant logging for oauth_init
* Refactor init endpoint to use Caller instead of direct v_storage access
- Replace direct OAuthClientStore/OAuthClientSecretStore/OAuthClientRedirectUriStore
calls with ctx.v_ctx().oauth methods
- Add static INIT_CALLER wrapped in OnceLock for singleton initialization
- Construct Caller with hardcoded INIT_CALLER_ID for logging visibility
- Remove v_storage field and accessor from RfdContext
- Update main.rs to create storage inline (matching pre-v_storage pattern)
Claude Assisted
* Add retry support and accept 201 status for init endpoints
- oauth_init: Use reqwest-middleware with exponential backoff retry
(1s-30s, configurable max retries via --max-retries/OAUTH_MAX_RETRIES)
- oauth_init: Accept 201 (Created) in addition to 200 (OK) for /init
- oauth_init: Refactor response handling into handle_init_response()
- meilisearch: Add manual retry loop with exponential backoff for
get_api_keys() (configurable via --max-retries/MEILI_MAX_RETRIES)
- Add reqwest-middleware and reqwest-retry dependencies
Implement a custom HttpClient for meilisearch-sdk that wraps reqwest-middleware with exponential backoff retry support. This replaces the manual retry loop with middleware-based retries, consistent with oauth_init.
Claude Assisted
Change rfd-kube-init to generate scoped write API keys instead of tenant tokens for RW namespaces. The new keys are limited to specific indexes via the MEILI_RW_INDEXES configuration. This provides better security by restricting write access to only the necessary indexes. Add automatic index creation in rfd-processor at startup, ensuring the search index exists before attempting to index documents. This removes the need for manual index creation. Claude Assisted
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added Containerfile and compose.yml allowing for rfd-api and all other necessary binaries to be built into a container image. The compose.yml can be used to stand-up a local stack which includes the database and meilisearch.
Added log_filter support to both the rfd-api and rfd-processor configuration files. This works the same as overriding RUST_LOG but allows for configuration via file instead of environment variable.
StaticStorage is now a trait. There are two implementations for the trait. Each implementation have corresponding configuration sections. Multiple static storage locations can be configured and processor will upload static storage to all configured static storage backends.
GCS Static Storage: This is the original GCS implementation.
S3 Static Storage: This is a new implementation using the S3 API.
PDF storage is now optional. Disable this feature by not configuring it in the configuration toml.