feat: add enhanced documentation system with call trees and architecture diagrams

vinod0m · vinod0m · commit d13c7f933372 · 2025-09-28T00:44:26.000+02:00
🚀 Major enhancement to documentation generation:

📊 New Features:
- Function call tree generation and visualization
- System architecture diagrams with component relationships
- AST-based code analysis for accurate dependency mapping
- Interactive documentation with modern UI components
- Code complexity analysis and reporting

🛠️ Technical Improvements:
- Advanced Sphinx extensions (graphviz, inheritance diagrams)
- Enhanced Python documentation generator script
- Automated diagram generation with Graphviz
- Local development build script for easy testing
- CI/CD integration with artifact generation

📁 New Components:
- scripts/generate-docs.py: Comprehensive documentation generator
- scripts/build-docs-local.sh: Local build automation
- Enhanced requirements-docs.txt with visualization tools
- Updated Sphinx configuration with advanced extensions
- Comprehensive documentation README with usage examples

🎯 Benefits:
- Visual understanding of codebase architecture
- Function dependency analysis for better code maintenance
- Interactive documentation with copy buttons and responsive design
- Automated generation reduces documentation maintenance overhead
- Improved developer onboarding with architectural insights
diff --git a/.github/workflows/python-docs.yml b/.github/workflows/python-docs.yml
@@ -24,11 +24,17 @@ jobs:
           python-version: '3.11'
           install-dev-reqs: 'false'
           install-docs-reqs: 'true'
-      - name: Generate Sphinx docs
+      - name: Install system dependencies
         run: |
+          sudo apt-get update
+          sudo apt-get install -y graphviz
+      - name: Generate enhanced documentation
+        run: |
+          # Run our enhanced documentation generator
+          python scripts/generate-docs.py --src src --output doc/codeDocs
           # Generate API documentation, excluding conf.py
           sphinx-apidoc -o doc/codeDocs/ src/ --force --no-toc --module-first
-          # Build the documentation
+          # Build the enhanced documentation
           sphinx-build -W -b html doc/codeDocs/ doc/codeDocs/_build/html
       - name: Upload documentation artifact
         uses: actions/upload-artifact@v4
diff --git a/doc/CodeDocs/README.md b/doc/CodeDocs/README.md
@@ -1,42 +1,141 @@
-# Code Documentation (codeDocs)
+# Enhanced Code Documentation (codeDocs)
 
-This directory contains the auto-generated API documentation for the unstructuredDataHandler project using Sphinx.
+This directory contains the **enhanced** auto-generated API documentation for the unstructuredDataHandler project using Sphinx, featuring advanced code analysis, call trees, and architectural diagrams.
 
-## Contents
+## 🚀 Enhanced Features
 
-- `conf.py` - Sphinx configuration file
-- `index.rst` - Main documentation index
-- `_static/` - Static files for documentation (CSS, images, etc.)
+### 📊 Code Analysis & Visualization
+- **System Architecture Diagrams**: High-level component relationships
+- **Function Call Trees**: Visual representation of function dependencies  
+- **Class Hierarchy Diagrams**: Inheritance and composition relationships
+- **Module Dependency Graphs**: Inter-module import analysis
+- **Complexity Analysis**: Code complexity metrics and reports
+
+### 🎨 Advanced Documentation Features  
+- **Interactive Elements**: Tabs, toggles, and collapsible sections
+- **Copy Code Buttons**: Easy code copying with syntax highlighting
+- **Responsive Design**: Modern, mobile-friendly documentation
+- **Enhanced Navigation**: Deep linking and improved search
+
+## 📁 Enhanced Directory Structure
+
+- `conf.py` - Enhanced Sphinx configuration with advanced extensions
+- `index.rst` - Main documentation index with architecture overview
+- `overview.rst` - **NEW**: System overview with architectural diagrams  
+- `_static/` - Static files including:
+  - `diagrams/` - **NEW**: Generated call trees and architecture diagrams
+  - `custom.css` - **NEW**: Enhanced styling
 - `_templates/` - Custom documentation templates
-- `_build/` - Generated HTML documentation (auto-generated, not committed)
-- `*.rst` - Auto-generated API documentation files (not committed)
+- `_build/` - Generated HTML documentation (auto-generated)
+- `*.rst` - Enhanced API documentation with call tree analysis
 
-## Building Documentation
+## 🛠️ Building Enhanced Documentation
 
-To build the documentation locally:
+### Quick Local Build
+```bash
+# Use the enhanced build script (recommended)
+./scripts/build-docs-local.sh
+```
 
+### Manual Build Process
 ```bash
-# Install documentation dependencies
+# Install enhanced documentation dependencies
 pip install -r requirements-docs.txt
 
-# Generate API documentation
-sphinx-apidoc -o doc/codeDocs/ src/
+# Generate enhanced analysis and diagrams
+python scripts/generate-docs.py --src src --output doc/codeDocs
 
-# Build HTML documentation
+# Generate API documentation  
+sphinx-apidoc -o doc/codeDocs/ src/ --force --no-toc --module-first
+
+# Build enhanced HTML documentation
 sphinx-build -b html doc/codeDocs/ doc/codeDocs/_build/html
 ```
 
-The built documentation will be available in `doc/codeDocs/_build/html/index.html`.
+### System Requirements
+- **Graphviz**: Required for diagram generation
+  ```bash
+  # Ubuntu/Debian
+  sudo apt-get install graphviz
+  
+  # macOS  
+  brew install graphviz
+  ```
+
+## 📊 Generated Outputs
+
+### Architecture Diagrams
+- `architecture.png` - Overall system component architecture
+- `*_calls.png` - Module-specific function call trees  
+- Interactive diagrams embedded in documentation
+
+### Enhanced Documentation
+- **Overview Section**: System architecture and component relationships
+- **Call Tree Analysis**: Function dependency visualization per module
+- **Complexity Reports**: Code quality metrics and analysis
+- **Interactive API Reference**: Enhanced with call flow information
+
+## 🔧 Configuration & Customization
+
+### Enhanced Sphinx Extensions
+- `sphinx.ext.graphviz` - Diagram generation support
+- `sphinx.ext.inheritance_diagram` - Class hierarchy diagrams
+- `sphinx_design` - Modern UI components and layouts  
+- `sphinx_copybutton` - Copy code functionality
+- `sphinx_autodoc_typehints` - Enhanced type hint documentation
+
+### Custom Analysis Features
+The documentation generator (`scripts/generate-docs.py`) provides:
+- **AST-based code analysis** for accurate call tree generation
+- **Module dependency mapping** for architecture visualization
+- **Complexity analysis integration** with pylint
+- **Extensible analysis framework** for custom insights
+
+## 🔄 Automated CI/CD Builds
+
+Enhanced documentation is automatically built via GitHub Actions:
+- **Workflow**: `.github/workflows/python-docs.yml`  
+- **Triggers**: Any Python file changes
+- **Artifacts**: Complete documentation package with diagrams
+- **Dependencies**: Automatic installation of Graphviz and enhanced packages
+
+## 🎯 Integration with Main Documentation
+
+This enhanced codeDocs integrates with the broader documentation structure:
+
+- **Architecture Documentation** (`doc/architecture/`) ← **Enhanced with generated diagrams**
+- **API Reference** ← **Enhanced with call trees and complexity analysis**
+- **Business Documentation** (`doc/business/`) 
+- **Development Guides** (`doc/`) ← **Enhanced with architectural insights**
+
+## 🚀 Getting Started
+
+1. **Install dependencies**: `pip install -r requirements-docs.txt`
+2. **Run enhanced build**: `./scripts/build-docs-local.sh`
+3. **View documentation**: Open `doc/codeDocs/_build/html/index.html`
+4. **Explore features**: Navigate to "Overview" for architecture diagrams
+
+## 🔍 Advanced Features
 
-## Automated Builds
+### Call Tree Analysis
+Each module includes:
+- Visual call tree diagrams
+- Function dependency lists  
+- Cross-reference linking
+- Complexity indicators
 
-The documentation is automatically built and updated via GitHub Actions on every push that modifies Python files. See `.github/workflows/python-docs.yml` for the CI configuration.
+### Architecture Insights
+- Component relationship diagrams
+- Data flow visualization
+- Module coupling analysis
+- System overview dashboard
 
-## Relationship to Main Documentation
+### Interactive Elements
+- Expandable code sections
+- Copy-to-clipboard functionality
+- Responsive diagram viewing
+- Enhanced navigation
 
-This codeDocs directory focuses specifically on API documentation generated from source code docstrings. For other types of documentation:
+---
 
-- Architecture documentation → `doc/architecture/`
-- Business documentation → `doc/business/`
-- General guides and specs → `doc/`
-- Images and references → `doc/images/`, `doc/reference/`
+**📈 This enhanced documentation system provides comprehensive insights into codebase structure, making it significantly easier for developers to understand, maintain, and extend the unstructuredDataHandler project.**
diff --git a/doc/CodeDocs/conf.py b/doc/CodeDocs/conf.py
@@ -19,9 +19,24 @@
     'sphinx.ext.viewcode',
     'sphinx.ext.napoleon',
     'sphinx.ext.intersphinx',
+    'sphinx.ext.graphviz',
+    'sphinx.ext.inheritance_diagram', 
     'sphinx_autodoc_typehints',
 ]
 
+# Enhanced documentation features
+try:
+    import sphinx_design
+    extensions.append('sphinx_design')
+except ImportError:
+    pass
+
+try:
+    import sphinx_copybutton
+    extensions.append('sphinx_copybutton')
+except ImportError:
+    pass
+
 templates_path = ['_templates']
 exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
 
diff --git a/doc/CodeDocs/index.rst b/doc/CodeDocs/index.rst
@@ -22,10 +22,18 @@ Getting Started
 
 .. toctree::
    :maxdepth: 2
-   :caption: API Reference:
+   :caption: Documentation:
 
+   overview
    modules
 
+.. toctree::
+   :maxdepth: 1
+   :caption: Code Analysis:
+
+   Call Trees & Architecture <overview>
+   API Reference <modules>
+
 Indices and tables
 ==================
 
diff --git a/requirements-docs.txt b/requirements-docs.txt
@@ -1,3 +1,24 @@
+# Core Sphinx documentation tools
 sphinx==7.3.7
 sphinx-autodoc-typehints==2.2.0
 sphinx-rtd-theme==2.0.0
+
+# Advanced documentation extensions
+sphinx-design==0.5.0
+sphinx-copybutton==0.5.2
+myst-parser==2.0.0
+
+# Call tree and code analysis tools
+pycallgraph2==1.1.3
+ast-monitor==0.4.3
+code2flow==2.5.1
+
+# Architecture diagram generation
+graphviz==0.20.1
+plantuml==0.3.0
+pylint==3.0.3
+
+# Enhanced documentation features
+sphinx-tabs==3.4.5
+sphinx-togglebutton==0.3.2
+furo==2023.9.10
diff --git a/scripts/build-docs-local.sh b/scripts/build-docs-local.sh
@@ -0,0 +1,37 @@
+#!/bin/bash
+# Local documentation generation script
+# This script generates enhanced documentation locally
+
+set -e
+
+echo "🚀 Starting local documentation generation..."
+
+# Check if we're in the right directory
+if [ ! -d "src" ]; then
+    echo "❌ Error: Please run this script from the repository root"
+    exit 1
+fi
+
+# Create output directory
+mkdir -p doc/codeDocs/_static/diagrams
+
+echo "📦 Installing documentation requirements..."
+pip install -r requirements-docs.txt
+
+echo "🔍 Generating enhanced documentation..."
+python scripts/generate-docs.py --src src --output doc/codeDocs
+
+echo "📚 Generating Sphinx API documentation..."
+sphinx-apidoc -o doc/codeDocs/ src/ --force --no-toc --module-first
+
+echo "🏗️  Building HTML documentation..."
+sphinx-build -b html doc/codeDocs/ doc/codeDocs/_build/html
+
+echo "✅ Documentation generation complete!"
+echo "📁 Open doc/codeDocs/_build/html/index.html to view the documentation"
+
+# Open documentation in browser (macOS)
+if [[ "$OSTYPE" == "darwin"* ]]; then
+    echo "🌐 Opening documentation in browser..."
+    open doc/codeDocs/_build/html/index.html
+fi
diff --git a/scripts/generate-docs.py b/scripts/generate-docs.py
