Skip to content

Commit ddddc4f

Browse files
wolf31o2wolf31o2
committed
feat: comprehensive plugin system with multi-backend storage
- Dynamic plugin registration for blob (BadgerDB, S3, GCS) and metadata (SQLite) backends - Thread-safe global registry with mutex protection and credential validation - CLI integration with plugin listing and selection flags - Comprehensive test suite including unit, integration, and security tests - Configuration system with nested plugin settings and environment variable support - Production-ready with proper error handling, logging, and security hardening Signed-off-by: Chris Gianelloni <wolf31o2@blinklabs.io>
1 parent 6011ae2 commit ddddc4f

39 files changed

+2618
-161
lines changed

‎PLUGIN_DEVELOPMENT.md‎

Lines changed: 278 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,278 @@
1+
# Plugin Development Guide
2+
3+
This guide explains how to develop plugins for Dingo's storage system.
4+
5+
## Overview
6+
7+
Dingo supports pluggable storage backends through a registration-based plugin system. Plugins can extend the system with new blob storage (blocks, transactions) and metadata storage (indexes, state) implementations.
8+
9+
## Plugin Types
10+
11+
### Blob Storage Plugins
12+
Store blockchain data (blocks, transactions, etc.). Examples:
13+
- `badger` - Local BadgerDB key-value store
14+
- `gcs` - Google Cloud Storage
15+
- `s3` - AWS S3
16+
17+
### Metadata Storage Plugins
18+
Store metadata and indexes. Examples:
19+
- `sqlite` - SQLite relational database
20+
21+
## Plugin Interface
22+
23+
All plugins must implement the `plugin.Plugin` interface:
24+
25+
```go
26+
type Plugin interface {
27+
Start() error
28+
Stop() error
29+
}
30+
```
31+
32+
## Plugin Registration
33+
34+
Plugins register themselves during package initialization using the `plugin.Register()` function:
35+
36+
```go
37+
func init() {
38+
plugin.Register(plugin.PluginEntry{
39+
Type: plugin.PluginTypeBlob, // or PluginTypeMetadata
40+
Name: "myplugin",
41+
Description: "My custom storage plugin",
42+
NewFromOptionsFunc: NewFromCmdlineOptions,
43+
Options: []plugin.PluginOption{
44+
// Plugin-specific options
45+
},
46+
})
47+
}
48+
```
49+
50+
## Plugin Options
51+
52+
Plugins define configuration options using the `PluginOption` struct:
53+
54+
```go
55+
plugin.PluginOption{
56+
Name: "data-dir", // Option name
57+
Type: plugin.PluginOptionTypeString, // Data type
58+
Description: "Data directory path", // Help text
59+
DefaultValue: "/tmp/data", // Default value
60+
Dest: &cmdlineOptions.dataDir, // Destination variable
61+
}
62+
```
63+
64+
Supported option types:
65+
- `PluginOptionTypeString`
66+
- `PluginOptionTypeBool`
67+
- `PluginOptionTypeInt`
68+
- `PluginOptionTypeUint`
69+
70+
## Environment Variables
71+
72+
Plugins automatically support environment variables with the pattern:
73+
`DINGO_DATABASE_{TYPE}_{PLUGIN}_{OPTION}`
74+
75+
Examples:
76+
- `DINGO_DATABASE_BLOB_BADGER_DATA_DIR=/data`
77+
- `DINGO_DATABASE_METADATA_SQLITE_DATA_DIR=/metadata.db`
78+
79+
## YAML Configuration
80+
81+
Plugins can be configured in `dingo.yaml`:
82+
83+
```yaml
84+
database:
85+
blob:
86+
plugin: "myplugin"
87+
myplugin:
88+
option1: "value1"
89+
option2: 42
90+
metadata:
91+
plugin: "sqlite"
92+
sqlite:
93+
data-dir: "/data/metadata.db"
94+
```
95+
96+
## Configuration Precedence
97+
98+
1. Environment variables (highest priority)
99+
2. YAML configuration
100+
3. Default values (lowest priority)
101+
102+
## Command Line Options
103+
104+
Plugins support command-line flags with the pattern:
105+
`--{type}-{plugin}-{option}`
106+
107+
Examples:
108+
- `--blob-badger-data-dir /data`
109+
- `--metadata-sqlite-data-dir /metadata.db`
110+
111+
## Plugin Development Steps
112+
113+
### 1. Create Plugin Structure
114+
115+
```text
116+
database/plugin/{type}/{name}/
117+
├── plugin.go # Registration and options
118+
├── options.go # Option functions
119+
├── database.go # Core implementation
120+
└── options_test.go # Unit tests
121+
```
122+
123+
### 2. Implement Core Plugin
124+
125+
Create the main plugin struct that implements `plugin.Plugin`:
126+
127+
```go
128+
type MyPlugin struct {
129+
// Fields
130+
}
131+
132+
func (p *MyPlugin) Start() error {
133+
// Initialize resources
134+
return nil
135+
}
136+
137+
func (p *MyPlugin) Stop() error {
138+
// Clean up resources
139+
return nil
140+
}
141+
```
142+
143+
### 3. Define Options
144+
145+
Create option functions following the pattern:
146+
147+
```go
148+
func WithOptionName(value Type) OptionFunc {
149+
return func(p *MyPlugin) {
150+
p.field = value
151+
}
152+
}
153+
```
154+
155+
### 4. Implement Constructors
156+
157+
Provide both options-based and legacy constructors:
158+
159+
```go
160+
func NewWithOptions(opts ...OptionFunc) (*MyPlugin, error) {
161+
p := &MyPlugin{}
162+
for _, opt := range opts {
163+
opt(p)
164+
}
165+
return p, nil
166+
}
167+
168+
func New(legacyParam1, legacyParam2) (*MyPlugin, error) {
169+
// For backward compatibility
170+
return NewWithOptions(
171+
WithOption1(legacyParam1),
172+
WithOption2(legacyParam2),
173+
)
174+
}
175+
```
176+
177+
### 5. Register Plugin
178+
179+
In `plugin.go`, register during initialization:
180+
181+
```go
182+
var cmdlineOptions struct {
183+
option1 string
184+
option2 int
185+
}
186+
187+
func init() {
188+
plugin.Register(plugin.PluginEntry{
189+
Type: plugin.PluginTypeBlob,
190+
Name: "myplugin",
191+
Description: "My custom plugin",
192+
NewFromOptionsFunc: NewFromCmdlineOptions,
193+
Options: []plugin.PluginOption{
194+
{
195+
Name: "option1",
196+
Type: plugin.PluginOptionTypeString,
197+
Description: "First option",
198+
DefaultValue: "default",
199+
Dest: &cmdlineOptions.option1,
200+
},
201+
// More options...
202+
},
203+
})
204+
}
205+
206+
func NewFromCmdlineOptions() plugin.Plugin {
207+
p, err := NewWithOptions(
208+
WithOption1(cmdlineOptions.option1),
209+
WithOption2(cmdlineOptions.option2),
210+
)
211+
if err != nil {
212+
panic(err)
213+
}
214+
return p
215+
}
216+
```
217+
218+
### 6. Add Tests
219+
220+
Create comprehensive tests:
221+
222+
```go
223+
func TestOptions(t *testing.T) {
224+
// Test option functions
225+
}
226+
227+
func TestLifecycle(t *testing.T) {
228+
p, err := NewWithOptions(WithOption1("test"))
229+
// Test Start/Stop
230+
}
231+
```
232+
233+
### 7. Update Imports
234+
235+
Add your plugin to the import list in the appropriate store file:
236+
- `database/plugin/blob/blob.go` for blob plugins
237+
- `database/plugin/metadata/metadata.go` for metadata plugins
238+
239+
## Example: Complete Plugin
240+
241+
See the existing plugins for complete examples:
242+
- `database/plugin/blob/badger/` - BadgerDB implementation
243+
- `database/plugin/metadata/sqlite/` - SQLite implementation
244+
- `database/plugin/blob/gcs/` - Google Cloud Storage implementation
245+
- `database/plugin/blob/aws/` - AWS S3 implementation
246+
247+
## Best Practices
248+
249+
1. **Error Handling**: Always return descriptive errors
250+
2. **Resource Management**: Properly implement Start/Stop for resource lifecycle
251+
3. **Thread Safety**: Ensure plugins are safe for concurrent use
252+
4. **Configuration Validation**: Validate configuration during construction
253+
5. **Backward Compatibility**: Maintain compatibility with existing deployments
254+
6. **Documentation**: Document all options and their effects
255+
7. **Testing**: Provide comprehensive unit and integration tests
256+
257+
## Testing Your Plugin
258+
259+
### Unit Tests
260+
Test individual components and option functions.
261+
262+
### Integration Tests
263+
Test the complete plugin lifecycle and interaction with the plugin system.
264+
265+
### CLI Testing
266+
Use the CLI to test plugin listing and selection:
267+
268+
```bash
269+
./dingo --blob list
270+
./dingo --metadata list
271+
```
272+
273+
### Configuration Testing
274+
Test environment variables and YAML configuration:
275+
276+
```bash
277+
DINGO_DATABASE_BLOB_MYPLUGIN_OPTION1=value ./dingo --blob myplugin
278+
```

‎README.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,78 @@ This behavior can be changed via the following environment variables:
6262
(default: empty)
6363
- `TLS_KEY_FILE_PATH` - SSL certificate key to use (default: empty)
6464

65+
## Database Plugins
66+
67+
Dingo supports pluggable storage backends for both blob storage (blocks, transactions) and metadata storage. This allows you to choose the best storage solution for your use case.
68+
69+
### Available Plugins
70+
71+
**Blob Storage Plugins:**
72+
- `badger` - BadgerDB local key-value store (default)
73+
- `gcs` - Google Cloud Storage blob store
74+
- `s3` - AWS S3 blob store
75+
76+
**Metadata Storage Plugins:**
77+
- `sqlite` - SQLite relational database (default)
78+
79+
### Plugin Selection
80+
81+
Plugins can be selected via command-line flags, environment variables, or configuration file:
82+
83+
```bash
84+
# Command line
85+
./dingo --blob gcs --metadata sqlite
86+
87+
# Environment variables
88+
DINGO_DATABASE_BLOB_PLUGIN=gcs
89+
DINGO_DATABASE_METADATA_PLUGIN=sqlite
90+
91+
# Configuration file (dingo.yaml)
92+
database:
93+
blob:
94+
plugin: "gcs"
95+
metadata:
96+
plugin: "sqlite"
97+
```
98+
99+
### Plugin Configuration
100+
101+
Each plugin supports specific configuration options. See `dingo.yaml.example` for detailed configuration examples.
102+
103+
**BadgerDB Options:**
104+
- `data-dir` - Directory for database files
105+
- `block-cache-size` - Block cache size in bytes
106+
- `index-cache-size` - Index cache size in bytes
107+
- `gc` - Enable garbage collection
108+
109+
**Google Cloud Storage Options:**
110+
- `bucket` - GCS bucket name
111+
- `project-id` - Google Cloud project ID
112+
- `prefix` - Path prefix within bucket
113+
- `credentials-file` - Path to service account credentials file (optional - uses Application Default Credentials if not provided)
114+
115+
**AWS S3 Options:**
116+
- `bucket` - S3 bucket name
117+
- `region` - AWS region
118+
- `prefix` - Path prefix within bucket
119+
- `access-key-id` - AWS access key ID (optional - uses default credential chain if not provided)
120+
- `secret-access-key` - AWS secret access key (optional - uses default credential chain if not provided)
121+
122+
**SQLite Options:**
123+
- `data-dir` - Path to SQLite database file
124+
125+
### Listing Available Plugins
126+
127+
You can see all available plugins and their descriptions:
128+
129+
```bash
130+
./dingo list
131+
```
132+
133+
## Plugin Development
134+
135+
For information on developing custom storage plugins, see [PLUGIN_DEVELOPMENT.md](PLUGIN_DEVELOPMENT.md).
136+
65137
### Example
66138

67139
Running on mainnet (:sweat_smile:):

‎chain/chain_test.go‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -107,10 +107,11 @@ var (
107107
},
108108
}
109109
dbConfig = &database.Config{
110-
BlobCacheSize: 1 << 20,
111-
Logger: nil,
112-
PromRegistry: nil,
113-
DataDir: "",
110+
Logger: nil,
111+
PromRegistry: nil,
112+
DataDir: "",
113+
BlobPlugin: "badger",
114+
MetadataPlugin: "sqlite",
114115
}
115116
)
116117

0 commit comments

Comments
 (0)