|
| 1 | +# KCL Repository Overview for AI Assistants |
| 2 | + |
| 3 | +This document provides a comprehensive overview of the KCL repository to help AI assistants understand the codebase for future work. |
| 4 | + |
| 5 | +## What is KCL? |
| 6 | + |
| 7 | +**KCL (KCL Constraint-based Record & Functional Language)** is an open-source configuration and policy language designed for cloud-native scenarios. It's a CNCF Sandbox project that enhances the writing of complex configurations through advanced programming language technology. |
| 8 | + |
| 9 | +### Main Purpose |
| 10 | +- Generate low-level static configuration data (JSON, YAML) for cloud-native applications |
| 11 | +- Reduce boilerplate in configuration through schema modeling |
| 12 | +- Define and validate configuration data with rule constraints |
| 13 | +- Manage large-scale configurations with GitOps and automation |
| 14 | +- Mutate/validate Kubernetes resources through various tool plugins |
| 15 | +- Platform engineering language for modern application delivery (used with KusionStack) |
| 16 | + |
| 17 | +### Key Use Cases |
| 18 | +- Kubernetes configuration management and abstraction |
| 19 | +- Terraform resource model abstraction |
| 20 | +- Configuration validation and constraint checking |
| 21 | +- Large-scale infrastructure as code |
| 22 | +- Platform engineering and GitOps workflows |
| 23 | + |
| 24 | +### Production Users |
| 25 | +Ant Group, Youzan, and Huawei are notable production users managing large-scale Kubernetes deployments. |
| 26 | + |
| 27 | +## Repository Structure |
| 28 | + |
| 29 | +### Top-Level Organization |
| 30 | + |
| 31 | +``` |
| 32 | +/kclvm/ Core KCL VM and compiler implementation (main codebase) |
| 33 | +/compiler_base/ Base compiler libraries and utilities (WIP, rustc-derived) |
| 34 | +/cli/ Command-line interface binary wrapper |
| 35 | +/test/ Integration and grammar tests |
| 36 | + /grammar/ Extensive grammar test cases |
| 37 | + /integration/ Integration test suites |
| 38 | +/samples/ Example KCL programs (hello.k, kubernetes.k, fib.k, etc.) |
| 39 | +/docs/ Developer guides and documentation |
| 40 | + /dev_guide/ Development guide (architecture, quick start, etc.) |
| 41 | + /design/ Design documents |
| 42 | +/scripts/ Build and release automation scripts |
| 43 | +/.github/workflows/ CI/CD pipelines for multiple platforms |
| 44 | +``` |
| 45 | + |
| 46 | +### Key Files |
| 47 | +- `Makefile` - Top-level build orchestration |
| 48 | +- `run.sh` - Build and release automation script |
| 49 | +- `VERSION` - Current version: **0.11.2** |
| 50 | +- `LICENSE` - Apache License 2.0 |
| 51 | +- `README.md` - Project documentation |
| 52 | + |
| 53 | +## Technology Stack |
| 54 | + |
| 55 | +### Primary Language: Rust |
| 56 | +- 362+ Rust source files |
| 57 | +- ~32,673 lines of Rust code in core modules |
| 58 | +- **Requires Rust 1.84+** for building |
| 59 | +- Rust 2021 edition |
| 60 | + |
| 61 | +### Secondary Languages |
| 62 | +- **KCL** - The language itself (.k files for examples and tests) |
| 63 | +- **Python** - Integration tests and test infrastructure |
| 64 | +- **Shell/Bash** - Build scripts and automation |
| 65 | +- **C/C++** - Runtime interop and FFI interfaces |
| 66 | + |
| 67 | +### Key Dependencies |
| 68 | +- **LLVM 12** - Compiler backend (optional, for high-performance compilation) |
| 69 | +- **Protobuf** - API definitions and RPC communication |
| 70 | +- **WASM** - WebAssembly compilation target support |
| 71 | +- **tokio** - Async runtime (for LSP and server) |
| 72 | +- **salsa** - Incremental computation (for LSP) |
| 73 | + |
| 74 | +## Architecture |
| 75 | + |
| 76 | +### Compilation Pipeline |
| 77 | + |
| 78 | +``` |
| 79 | +Source Code (.k files) |
| 80 | + ↓ |
| 81 | +[Lexer] → Tokens |
| 82 | + ↓ |
| 83 | +[Parser] → AST |
| 84 | + ↓ |
| 85 | +[Resolver/Sema] → Semantic Analysis & Type Checking |
| 86 | + ↓ |
| 87 | +[Compiler] → IR (LLVM IR or AST-based) |
| 88 | + ↓ |
| 89 | +[Evaluator/Runner] → Execution |
| 90 | + ↓ |
| 91 | +Output (YAML/JSON) |
| 92 | +``` |
| 93 | + |
| 94 | +### Key Components (/kclvm crates) |
| 95 | + |
| 96 | +**Frontend (Parsing & Analysis):** |
| 97 | +- `kclvm-lexer` - Lexical analysis and tokenization |
| 98 | +- `kclvm-parser` - Parse KCL source into AST |
| 99 | +- `kclvm-ast` - Abstract Syntax Tree definitions and walker |
| 100 | +- `kclvm-ast-pretty` - AST formatting and pretty-printing |
| 101 | +- `kclvm-span` - Source code span/position tracking |
| 102 | +- `kclvm-error` - Error handling and diagnostics |
| 103 | + |
| 104 | +**Semantic Analysis:** |
| 105 | +- `kclvm-sema` - Semantic analysis, type checking, and validation |
| 106 | +- `kclvm-loader` - Module loading and dependency management |
| 107 | +- `kclvm-query` - Code query and information retrieval |
| 108 | + |
| 109 | +**Compilation & Execution:** |
| 110 | +- `kclvm-compiler` - Main compilation logic with optional LLVM backend |
| 111 | +- `kclvm-evaluator` - Expression evaluation engine |
| 112 | +- `kclvm-runner` - Program execution environment |
| 113 | +- `kclvm-driver` - Compilation driver and orchestration |
| 114 | + |
| 115 | +**Runtime:** |
| 116 | +- `kclvm-runtime` - Runtime support libraries with extensive standard library |
| 117 | + - Value representation and type system |
| 118 | + - Standard library modules: json, yaml, base64, regex, crypto, datetime, math, net, etc. |
| 119 | + - Template rendering (handlebars) |
| 120 | + - File I/O and manifests |
| 121 | + |
| 122 | +**Tooling:** |
| 123 | +- `kclvm-tools` - Development tools |
| 124 | + - Format, Lint, Fix, Vet |
| 125 | + - Testing infrastructure |
| 126 | + - **LSP** (Language Server) - Full IDE support with autocomplete, goto-definition, diagnostics |
| 127 | +- `kclvm-api` - Public API layer for multi-language SDKs |
| 128 | +- `kclvm-cmd` - CLI command implementation |
| 129 | + |
| 130 | +**Utilities:** |
| 131 | +- `kclvm-config` - Configuration parsing |
| 132 | +- `kclvm-version` - Version management |
| 133 | +- `kclvm-utils` - Common utilities |
| 134 | +- `kclvm-primitives` - Primitive type definitions |
| 135 | +- `kclvm-macros` - Procedural macros |
| 136 | + |
| 137 | +### Language Server Architecture |
| 138 | +- Salsa-based incremental compilation for performance |
| 139 | +- VFS (Virtual File System) for handling unsaved changes |
| 140 | +- Thread pool for concurrent request handling |
| 141 | +- Event-driven architecture (Tasks + LSP Messages) |
| 142 | +- Compile unit discovery for projects without explicit config |
| 143 | +- Located at: `/kclvm/tools/src/LSP` |
| 144 | + |
| 145 | +## Build System |
| 146 | + |
| 147 | +### Build Tools |
| 148 | +- **Cargo** - Primary Rust build system (workspace-based with 20+ crates) |
| 149 | +- **Make** - Top-level orchestration |
| 150 | +- **Docker** - Containerized build environment (recommended: `kcllang/kcl-builder`) |
| 151 | + |
| 152 | +### Common Build Commands |
| 153 | + |
| 154 | +```bash |
| 155 | +make build # Standard build |
| 156 | +make release # Release build |
| 157 | +make check # Type checking |
| 158 | +make build-lsp # Language server |
| 159 | +make build-wasm # WASM target |
| 160 | +``` |
| 161 | + |
| 162 | +### Build Features |
| 163 | +- Workspace with 20+ member crates |
| 164 | +- Optional `llvm` feature flag for high-performance backend |
| 165 | +- Support for multiple targets: native, WASM (wasm32-wasip1), WASM-unknown |
| 166 | +- Cross-platform: Linux (AMD64, ARM64), macOS (AMD64, ARM64), Windows (MinGW) |
| 167 | +- Release profile optimized for size (opt-level = "z", LTO enabled) |
| 168 | + |
| 169 | +### Major Dependencies |
| 170 | +- **inkwell** - LLVM bindings (optional) |
| 171 | +- **serde/serde_json** - Serialization |
| 172 | +- **serde_yaml_ng** - YAML support (note: migrated from serde_yaml) |
| 173 | +- **prost/protobuf** - Protocol buffers |
| 174 | +- **tokio** - Async runtime |
| 175 | +- **lsp-server/lsp-types** - Language Server Protocol |
| 176 | +- **salsa** - Incremental computation |
| 177 | +- **rustc_lexer** - Rust's lexer for tokenization |
| 178 | +- **petgraph** - Graph data structures |
| 179 | +- **regex/fancy-regex** - Regular expressions |
| 180 | + |
| 181 | +## Testing |
| 182 | + |
| 183 | +### Testing Strategy |
| 184 | + |
| 185 | +**1. Unit Tests:** |
| 186 | +- Cargo-based unit tests across all crates |
| 187 | +- Command: `make test` or `cargo test --workspace` |
| 188 | +- Code coverage via `cargo llvm-cov` |
| 189 | + |
| 190 | +**2. Grammar Tests:** |
| 191 | +- Extensive grammar test suite in `/test/grammar` |
| 192 | +- Python-based pytest framework |
| 193 | +- Parallel execution: `pytest -v -n 5` |
| 194 | +- Tests both AST evaluator and fast evaluator |
| 195 | +- Command: `make test-grammar` |
| 196 | + |
| 197 | +**3. Integration Tests:** |
| 198 | +- `/kclvm/tests/integration` - Rust integration tests |
| 199 | +- `/test/integration` - Python integration tests |
| 200 | +- Konfig tests for real-world scenarios |
| 201 | + |
| 202 | +**4. Fuzz Testing:** |
| 203 | +- Parser fuzzing in `/kclvm/tests/fuzz` |
| 204 | +- Command: `make fuzz-parser` |
| 205 | + |
| 206 | +**5. Runtime Tests:** |
| 207 | +- Python-based runtime library tests |
| 208 | +- Command: `make test-runtime` |
| 209 | + |
| 210 | +### CI/CD |
| 211 | +Comprehensive GitHub Actions workflows (11 pipelines) for: |
| 212 | +- Linux AMD64 and ARM64 |
| 213 | +- macOS AMD64 and ARM64 |
| 214 | +- Windows (MSVC and MinGW) |
| 215 | +- Alpine Linux (musl) |
| 216 | +- CentOS 7 |
| 217 | +- WASM/WASI |
| 218 | +- Compiler base tests |
| 219 | + |
| 220 | +## Documentation |
| 221 | + |
| 222 | +### Primary Documentation Sources |
| 223 | + |
| 224 | +1. **Official Website:** https://kcl-lang.io/ |
| 225 | + - User guides, language tour, API reference |
| 226 | + - Installation instructions |
| 227 | + - Tutorial and examples |
| 228 | + |
| 229 | +2. **Repository Documentation:** |
| 230 | + - `/README.md` - Project overview and quick start |
| 231 | + - `/README-zh.md` - Chinese documentation |
| 232 | + - `/docs/dev_guide/` - Developer guide |
| 233 | + - `1.about_this_guide.md` - Guide overview |
| 234 | + - `2.quick_start.md` - Building and setup |
| 235 | + - `3.coding_conventions.md` - Code style |
| 236 | + - `4.architecture.md` - Compiler architecture |
| 237 | + - `5.source_code.md` - Source code structure |
| 238 | + - `6.languager_server.md` - LSP implementation |
| 239 | + |
| 240 | +3. **Governance:** |
| 241 | + - `/GOVERNANCE.md` - Project governance (CNCF sandbox) |
| 242 | + - `/CODE_OF_CONDUCT.md` - Community guidelines |
| 243 | + - `/ADOPTERS.md` - Production users |
| 244 | + - `/MAINTAINERS` - Maintainer list |
| 245 | + |
| 246 | +## Language Design Principles |
| 247 | + |
| 248 | +1. **Spec-driven**: Independent syntax and semantics specification |
| 249 | +2. **Functional**: Low side-effects, no system-level operations (no threads/IO) |
| 250 | +3. **Constraint-based**: Schema + Rule + Lambda for configuration validation |
| 251 | +4. **High Performance**: Rust + LLVM compilation, WASM support |
| 252 | +5. **API-first**: Multi-language SDKs (Rust, Go, Python, .NET, Java, Node.js) |
| 253 | +6. **Cloud-native**: Native support for OpenAPI, K8s CRD, KRM spec |
| 254 | +7. **Type Safety**: Static type system with constraints and validation rules |
| 255 | + |
| 256 | +## Development Workflow |
| 257 | + |
| 258 | +### Setting Up Development Environment |
| 259 | + |
| 260 | +```bash |
| 261 | +# Recommended: Use Docker builder image |
| 262 | +docker pull kcllang/kcl-builder |
| 263 | + |
| 264 | +# Or install dependencies locally |
| 265 | +# - Rust 1.84+ |
| 266 | +# - LLVM 12 (optional, for high-performance backend) |
| 267 | +# - Python 3.x (for tests) |
| 268 | +# - Protobuf compiler |
| 269 | + |
| 270 | +# Build the project |
| 271 | +make build |
| 272 | + |
| 273 | +# Run tests |
| 274 | +make test |
| 275 | +make test-grammar |
| 276 | +``` |
| 277 | + |
| 278 | +### IDE Support |
| 279 | +- Full Language Server Protocol (LSP) support |
| 280 | +- VSCode extension available |
| 281 | +- Rust-analyzer recommended for Rust development |
| 282 | +- Dev container configuration in `.devcontainer/` |
| 283 | + |
| 284 | +## Notable Features |
| 285 | + |
| 286 | +1. **Production-Ready:** Used by major companies for large-scale Kubernetes management |
| 287 | +2. **Multi-Platform:** Exceptional cross-platform support including ARM and WASM |
| 288 | +3. **Rich Ecosystem:** |
| 289 | + - Kubectl, Kustomize, Helm, KPT, Crossplane plugins |
| 290 | + - Package registry at artifacthub.io |
| 291 | + - Integration with GitOps workflows |
| 292 | +4. **Developer Experience:** |
| 293 | + - Full LSP implementation with IDE support |
| 294 | + - Comprehensive tooling (format, lint, test, vet) |
| 295 | + - Extensive test coverage and fuzzing |
| 296 | +5. **Performance Focus:** |
| 297 | + - Optional LLVM backend for native code compilation |
| 298 | + - WASM compilation target |
| 299 | + - Size-optimized release builds |
| 300 | + |
| 301 | +## Common Development Tasks |
| 302 | + |
| 303 | +### Working with the Compiler |
| 304 | +- Main compiler logic: `/kclvm/compiler/src/` |
| 305 | +- LLVM codegen: `/kclvm/compiler/src/codegen/llvm/` |
| 306 | +- Semantic analysis: `/kclvm/sema/src/` |
| 307 | + |
| 308 | +### Working with the Runtime |
| 309 | +- Runtime implementation: `/kclvm/runtime/src/` |
| 310 | +- Standard library: `/kclvm/runtime/src/_kclvm_main.ll` and various modules |
| 311 | +- Value system: `/kclvm/runtime/src/value/` |
| 312 | + |
| 313 | +### Working with the Language Server |
| 314 | +- LSP implementation: `/kclvm/tools/src/LSP/` |
| 315 | +- Entry point: `/kclvm/tools/src/LSP/src/main.rs` |
| 316 | +- Key components: state management, completion, goto-definition, diagnostics |
| 317 | + |
| 318 | +### Working with Tests |
| 319 | +- Grammar tests: `/test/grammar/` (Python-based) |
| 320 | +- Integration tests: `/kclvm/tests/integration/` (Rust) |
| 321 | +- Add new test cases by following existing patterns |
| 322 | + |
| 323 | +## Git Workflow |
| 324 | + |
| 325 | +- **Main branch:** `main` (use this for PRs) |
| 326 | +- **License:** Apache License 2.0 |
| 327 | +- **Current version:** 0.11.2 |
| 328 | +- **Recent focus areas:** YAML serialization fixes, LSP formatter improvements, documentation |
| 329 | + |
| 330 | +## Important Notes for AI Assistants |
| 331 | + |
| 332 | +1. **Always check existing code patterns** - This is a mature codebase with established conventions |
| 333 | +2. **Test coverage is critical** - Add tests for any new functionality |
| 334 | +3. **Performance matters** - This is used in production for large-scale configurations |
| 335 | +4. **Cross-platform support** - Consider multiple platforms when making changes |
| 336 | +5. **Documentation** - Keep docs in sync with code changes |
| 337 | +6. **The codebase uses workspaces** - Changes may affect multiple crates |
| 338 | +7. **LLVM backend is optional** - Code should work with or without it |
| 339 | +8. **Recent migration from serde_yaml to serde_yaml_ng** - Use the new library |
| 340 | + |
| 341 | +## Quick Reference |
| 342 | + |
| 343 | +| Task | Command | |
| 344 | +|------|---------| |
| 345 | +| Build | `make build` | |
| 346 | +| Test | `make test` | |
| 347 | +| Format | `cargo fmt` | |
| 348 | +| Lint | `cargo clippy` | |
| 349 | +| Grammar tests | `make test-grammar` | |
| 350 | +| Build LSP | `make build-lsp` | |
| 351 | +| Build WASM | `make build-wasm` | |
| 352 | +| Release build | `make release` | |
| 353 | + |
| 354 | +## Support |
| 355 | + |
| 356 | +- **Official website:** https://kcl-lang.io/ |
| 357 | +- **GitHub:** Current repository |
| 358 | +- **CNCF:** Sandbox project with formal governance |
0 commit comments