Skip to content

Commit 23af5a5

Browse files
wolf31o2wolf31o2
authored
feat: comprehensive plugin system with multi-backend storage (#1050)
- Dynamic plugin registration for blob (BadgerDB, S3, GCS) and metadata (SQLite) backends - Thread-safe global registry with mutex protection and credential validation - CLI integration with plugin listing and selection flags - Comprehensive test suite including unit, integration, and security tests - Configuration system with nested plugin settings and environment variable support - Production-ready with proper error handling, logging, and security hardening Signed-off-by: Chris Gianelloni <wolf31o2@blinklabs.io>
1 parent 8f60ab3 commit 23af5a5

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

43 files changed

+2995
-319
lines changed

‎PLUGIN_DEVELOPMENT.md‎

Lines changed: 279 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,279 @@
1+
# Plugin Development Guide
2+
3+
This guide explains how to develop plugins for Dingo's storage system.
4+
5+
## Overview
6+
7+
Dingo supports pluggable storage backends through a registration-based plugin system. Plugins can extend the system with new blob storage (blocks, transactions) and metadata storage (indexes, state) implementations.
8+
9+
## Plugin Types
10+
11+
### Blob Storage Plugins
12+
Store blockchain data (blocks, transactions, etc.). Examples:
13+
- `badger` - Local BadgerDB key-value store
14+
- `gcs` - Google Cloud Storage
15+
- `s3` - AWS S3
16+
17+
### Metadata Storage Plugins
18+
Store metadata and indexes. Examples:
19+
- `sqlite` - SQLite relational database
20+
21+
## Plugin Interface
22+
23+
All plugins must implement the `plugin.Plugin` interface:
24+
25+
```go
26+
type Plugin interface {
27+
Start() error
28+
Stop() error
29+
}
30+
```
31+
32+
## Plugin Registration
33+
34+
Plugins register themselves during package initialization using the `plugin.Register()` function:
35+
36+
```go
37+
func init() {
38+
plugin.Register(plugin.PluginEntry{
39+
Type: plugin.PluginTypeBlob, // or PluginTypeMetadata
40+
Name: "myplugin",
41+
Description: "My custom storage plugin",
42+
NewFromOptionsFunc: NewFromCmdlineOptions,
43+
Options: []plugin.PluginOption{
44+
// Plugin-specific options
45+
},
46+
})
47+
}
48+
```
49+
50+
## Plugin Options
51+
52+
Plugins define configuration options using the `PluginOption` struct:
53+
54+
```go
55+
plugin.PluginOption{
56+
Name: "data-dir", // Option name
57+
Type: plugin.PluginOptionTypeString, // Data type
58+
Description: "Data directory path", // Help text
59+
DefaultValue: "/tmp/data", // Default value
60+
Dest: &cmdlineOptions.dataDir, // Destination variable
61+
}
62+
```
63+
64+
Supported option types:
65+
- `PluginOptionTypeString`
66+
- `PluginOptionTypeBool`
67+
- `PluginOptionTypeInt`
68+
- `PluginOptionTypeUint`
69+
70+
## Environment Variables
71+
72+
Plugins automatically support environment variables with the pattern:
73+
`DINGO_DATABASE_{TYPE}_{PLUGIN}_{OPTION}`
74+
75+
Examples:
76+
- `DINGO_DATABASE_BLOB_BADGER_DATA_DIR=/data`
77+
- `DINGO_DATABASE_METADATA_SQLITE_DATA_DIR=/metadata.db`
78+
79+
## YAML Configuration
80+
81+
Plugins can be configured in `dingo.yaml`:
82+
83+
```yaml
84+
database:
85+
blob:
86+
plugin: "myplugin"
87+
myplugin:
88+
option1: "value1"
89+
option2: 42
90+
metadata:
91+
plugin: "sqlite"
92+
sqlite:
93+
data-dir: "/data/metadata.db"
94+
```
95+
96+
## Configuration Precedence
97+
98+
1. Command-line flags (highest priority)
99+
2. Environment variables
100+
3. YAML configuration
101+
4. Default values (lowest priority)
102+
103+
## Command Line Options
104+
105+
Plugins support command-line flags with the pattern:
106+
`--{type}-{plugin}-{option}`
107+
108+
Examples:
109+
- `--blob-badger-data-dir /data`
110+
- `--metadata-sqlite-data-dir /metadata.db`
111+
112+
## Plugin Development Steps
113+
114+
### 1. Create Plugin Structure
115+
116+
```text
117+
database/plugin/{type}/{name}/
118+
├── plugin.go # Registration and options
119+
├── options.go # Option functions
120+
├── database.go # Core implementation
121+
└── options_test.go # Unit tests
122+
```
123+
124+
### 2. Implement Core Plugin
125+
126+
Create the main plugin struct that implements `plugin.Plugin`:
127+
128+
```go
129+
type MyPlugin struct {
130+
// Fields
131+
}
132+
133+
func (p *MyPlugin) Start() error {
134+
// Initialize resources
135+
return nil
136+
}
137+
138+
func (p *MyPlugin) Stop() error {
139+
// Clean up resources
140+
return nil
141+
}
142+
```
143+
144+
### 3. Define Options
145+
146+
Create option functions following the pattern:
147+
148+
```go
149+
func WithOptionName(value Type) OptionFunc {
150+
return func(p *MyPlugin) {
151+
p.field = value
152+
}
153+
}
154+
```
155+
156+
### 4. Implement Constructors
157+
158+
Provide both options-based and legacy constructors:
159+
160+
```go
161+
func NewWithOptions(opts ...OptionFunc) (*MyPlugin, error) {
162+
p := &MyPlugin{}
163+
for _, opt := range opts {
164+
opt(p)
165+
}
166+
return p, nil
167+
}
168+
169+
func New(legacyParam1, legacyParam2) (*MyPlugin, error) {
170+
// For backward compatibility
171+
return NewWithOptions(
172+
WithOption1(legacyParam1),
173+
WithOption2(legacyParam2),
174+
)
175+
}
176+
```
177+
178+
### 5. Register Plugin
179+
180+
In `plugin.go`, register during initialization:
181+
182+
```go
183+
var cmdlineOptions struct {
184+
option1 string
185+
option2 int
186+
}
187+
188+
func init() {
189+
plugin.Register(plugin.PluginEntry{
190+
Type: plugin.PluginTypeBlob,
191+
Name: "myplugin",
192+
Description: "My custom plugin",
193+
NewFromOptionsFunc: NewFromCmdlineOptions,
194+
Options: []plugin.PluginOption{
195+
{
196+
Name: "option1",
197+
Type: plugin.PluginOptionTypeString,
198+
Description: "First option",
199+
DefaultValue: "default",
200+
Dest: &cmdlineOptions.option1,
201+
},
202+
// More options...
203+
},
204+
})
205+
}
206+
207+
func NewFromCmdlineOptions() plugin.Plugin {
208+
p, err := NewWithOptions(
209+
WithOption1(cmdlineOptions.option1),
210+
WithOption2(cmdlineOptions.option2),
211+
)
212+
if err != nil {
213+
panic(err)
214+
}
215+
return p
216+
}
217+
```
218+
219+
### 6. Add Tests
220+
221+
Create comprehensive tests:
222+
223+
```go
224+
func TestOptions(t *testing.T) {
225+
// Test option functions
226+
}
227+
228+
func TestLifecycle(t *testing.T) {
229+
p, err := NewWithOptions(WithOption1("test"))
230+
// Test Start/Stop
231+
}
232+
```
233+
234+
### 7. Update Imports
235+
236+
Add your plugin to the import list in the appropriate store file:
237+
- `database/plugin/blob/blob.go` for blob plugins
238+
- `database/plugin/metadata/metadata.go` for metadata plugins
239+
240+
## Example: Complete Plugin
241+
242+
See the existing plugins for complete examples:
243+
- `database/plugin/blob/badger/` - BadgerDB implementation
244+
- `database/plugin/metadata/sqlite/` - SQLite implementation
245+
- `database/plugin/blob/gcs/` - Google Cloud Storage implementation
246+
- `database/plugin/blob/aws/` - AWS S3 implementation
247+
248+
## Best Practices
249+
250+
1. **Error Handling**: Always return descriptive errors
251+
2. **Resource Management**: Properly implement Start/Stop for resource lifecycle
252+
3. **Thread Safety**: Ensure plugins are safe for concurrent use
253+
4. **Configuration Validation**: Validate configuration during construction
254+
5. **Backward Compatibility**: Maintain compatibility with existing deployments
255+
6. **Documentation**: Document all options and their effects
256+
7. **Testing**: Provide comprehensive unit and integration tests
257+
258+
## Testing Your Plugin
259+
260+
### Unit Tests
261+
Test individual components and option functions.
262+
263+
### Integration Tests
264+
Test the complete plugin lifecycle and interaction with the plugin system.
265+
266+
### CLI Testing
267+
Use the CLI to test plugin listing and selection:
268+
269+
```bash
270+
./dingo --blob list
271+
./dingo --metadata list
272+
```
273+
274+
### Configuration Testing
275+
Test environment variables and YAML configuration:
276+
277+
```bash
278+
DINGO_DATABASE_BLOB_MYPLUGIN_OPTION1=value ./dingo --blob myplugin
279+
```

‎README.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,78 @@ This behavior can be changed via the following environment variables:
6262
(default: empty)
6363
- `TLS_KEY_FILE_PATH` - SSL certificate key to use (default: empty)
6464

65+
## Database Plugins
66+
67+
Dingo supports pluggable storage backends for both blob storage (blocks, transactions) and metadata storage. This allows you to choose the best storage solution for your use case.
68+
69+
### Available Plugins
70+
71+
**Blob Storage Plugins:**
72+
- `badger` - BadgerDB local key-value store (default)
73+
- `gcs` - Google Cloud Storage blob store
74+
- `s3` - AWS S3 blob store
75+
76+
**Metadata Storage Plugins:**
77+
- `sqlite` - SQLite relational database (default)
78+
79+
### Plugin Selection
80+
81+
Plugins can be selected via command-line flags, environment variables, or configuration file:
82+
83+
```bash
84+
# Command line
85+
./dingo --blob gcs --metadata sqlite
86+
87+
# Environment variables
88+
DINGO_DATABASE_BLOB_PLUGIN=gcs
89+
DINGO_DATABASE_METADATA_PLUGIN=sqlite
90+
91+
# Configuration file (dingo.yaml)
92+
database:
93+
blob:
94+
plugin: "gcs"
95+
metadata:
96+
plugin: "sqlite"
97+
```
98+
99+
### Plugin Configuration
100+
101+
Each plugin supports specific configuration options. See `dingo.yaml.example` for detailed configuration examples.
102+
103+
**BadgerDB Options:**
104+
- `data-dir` - Directory for database files
105+
- `block-cache-size` - Block cache size in bytes
106+
- `index-cache-size` - Index cache size in bytes
107+
- `gc` - Enable garbage collection
108+
109+
**Google Cloud Storage Options:**
110+
- `bucket` - GCS bucket name
111+
- `project-id` - Google Cloud project ID
112+
- `prefix` - Path prefix within bucket
113+
- `credentials-file` - Path to service account credentials file (optional - uses Application Default Credentials if not provided)
114+
115+
**AWS S3 Options:**
116+
- `bucket` - S3 bucket name
117+
- `region` - AWS region
118+
- `prefix` - Path prefix within bucket
119+
- `access-key-id` - AWS access key ID (optional - uses default credential chain if not provided)
120+
- `secret-access-key` - AWS secret access key (optional - uses default credential chain if not provided)
121+
122+
**SQLite Options:**
123+
- `data-dir` - Path to SQLite database file
124+
125+
### Listing Available Plugins
126+
127+
You can see all available plugins and their descriptions:
128+
129+
```bash
130+
./dingo list
131+
```
132+
133+
## Plugin Development
134+
135+
For information on developing custom storage plugins, see [PLUGIN_DEVELOPMENT.md](PLUGIN_DEVELOPMENT.md).
136+
65137
### Example
66138

67139
Running on mainnet (:sweat_smile:):

‎chain/chain_test.go‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -107,10 +107,11 @@ var (
107107
},
108108
}
109109
dbConfig = &database.Config{
110-
BlobCacheSize: 1 << 20,
111-
Logger: nil,
112-
PromRegistry: nil,
113-
DataDir: "",
110+
Logger: nil,
111+
PromRegistry: nil,
112+
DataDir: "",
113+
BlobPlugin: "badger",
114+
MetadataPlugin: "sqlite",
114115
}
115116
)
116117

0 commit comments

Comments
 (0)