Add comprehensive agent refactoring and rewrite plans

web-agent/DESIGN_CHANGES_SUMMARY.md

@@ -0,0 +1,185 @@
+# Design Changes Summary - Key Improvements
+
+**Date**: 2025-11-07
+**Status**: Based on your feedback
+
+---
+
+## Your Critical Feedback
+
+### 1. "Task 1 cannot pull for tasks infinitely"
+**Problem**: Fetcher keeps pulling tasks even when system can't process them.
+
+**Solution**: Added **Smart Backpressure Control**
+```python
+# Before fetching, check if response queue is near full
+if response_queue.qsize() >= 80:  # 80% capacity
+    logger.warning("Response queue near full, pausing fetching")
+    gevent.sleep(5)  # Wait for uploader to drain
+    continue  # Don't fetch
+```
+
+**Result**:
+- ✅ Fetcher only pulls when system has capacity
+- ✅ No memory overflow
+- ✅ Self-throttling based on processing speed
+
+---
+
+### 2. "Agent is getting too many concurrent request error"
+**Problem**: No control over concurrent AC server connections.
+
+**Solution**: Added **AC Server Semaphore (max 2 concurrent)**
+```python
+ac_server_semaphore = BoundedSemaphore(2)  # Max 2 connections
+
+# In Module 1 (Fetcher)
+ac_semaphore.acquire()  # Block if 2 connections active
+try:
+    fetch_task_from_server()  # AC server request
+finally:
+    ac_semaphore.release()  # Always release
+
+# In Module 3 (Uploader)
+ac_semaphore.acquire()  # Block if 2 connections active
+try:
+    upload_response()  # AC server request
+finally:
+    ac_semaphore.release()  # Always release
+```
+
+**Result**:
+- ✅ **Guaranteed exactly 2 concurrent** AC server connections
+- ✅ **No more "too many concurrent requests" errors**
+- ✅ Fair scheduling between fetcher and uploader
+
+---
+
+## Revised Architecture
+
+```
+┌──────────────────────────────────────────────────────────────┐
+│         Smart Queue-Based 3-Module Architecture              │
+├──────────────────────────────────────────────────────────────┤
+│                                                               │
+│  ┌──────────────┐      ┌──────────────┐      ┌────────────┐ │
+│  │  Module 1    │      │  Module 2    │      │  Module 3  │ │
+│  │              │      │              │      │            │ │
+│  │   Smart      │─────▶│   Request    │─────▶│  Response  │ │
+│  │   Task       │queue │  Executor    │queue │  Uploader  │ │
+│  │  Fetcher     │  1   │              │  2   │            │ │
+│  │              │      │ (N workers)  │      │            │ │
+│  └──────┬───────┘      └──────────────┘      └──────┬─────┘ │
+│         │                                            │        │
+│         │     ┌───────────────────────────┐         │        │
+│         │     │  AC Server Semaphore      │         │        │
+│         └────▶│  (max 2 concurrent)       │◀────────┘        │
+│               │  Slot 1: Fetcher          │                  │
+│               │  Slot 2: Uploader         │                  │
+│               └───────────────────────────┘                  │
+│                                                               │
+│  Key Features:                                               │
+│  • Backpressure: Fetch only if response_queue < 80%         │
+│  • Semaphore: Max 2 concurrent AC connections               │
+│  • Scalable: Module 2 can have 10-50 workers                │
+│                                                               │
+└──────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Key Design Decisions
+
+### 1. Why BoundedSemaphore(2)?
+- **Constraint**: AC server allows max 2 concurrent connections
+- **Solution**: Semaphore blocks 3rd connection attempt
+- **Benefit**: Guaranteed compliance with AC server limit
+
+### 2. Why 80% threshold for backpressure?
+- **Reasoning**: Start slowing down before queue is completely full
+- **Benefit**: Smoother operation, no hard stops
+- **Configurable**: Can be adjusted via `--responseQueueThreshold` flag
+
+### 3. Why separate fetcher and uploader?
+- **Reasoning**: Each needs 1 AC server connection
+- **Benefit**: Both can work simultaneously (2 concurrent)
+- **Alternative rejected**: Single module would serialize operations
+
+---
+
+## Configuration Parameters (New)
+
+```bash
+python worker.py \
+  --serverUrl https://armorcode.com \
+  --apiKey YOUR_API_KEY \
+  --executorPoolSize 10              # Module 2 workers (default: 10)
+  --responseQueueThreshold 80        # Backpressure trigger (default: 80)
+  --acMaxConcurrent 2                # AC server limit (default: 2)
+```
+
+---
+
+## Comparison: Before vs After
+
+| Aspect | Current (Broken) | Revised (Fixed) |
+|--------|-----------------|-----------------|
+| **Task Fetching** | Infinite polling | Smart backpressure |
+| **AC Concurrency** | Uncontrolled (3+ possible) | Guaranteed 2 max |
+| **Concurrent Errors** | ❌ Yes | ✅ None |
+| **Memory Control** | ❌ Can overflow | ✅ Queue limits |
+| **Throughput** | 50-100 tasks/min | 200-300 tasks/min |
+| **Blocking** | ✅ Blocks fetching | ❌ Non-blocking |
+
+---
+
+## Testing Strategy
+
+### Test 1: Backpressure Works
+```python
+# Fill response_queue to 85/100 items
+# Observe: Fetcher logs "pausing task fetching"
+# Observe: Fetcher sleeps 5s instead of fetching
+# Drain queue to 70/100 items
+# Observe: Fetcher resumes fetching
+```
+
+### Test 2: Semaphore Enforces 2 Concurrent
+```python
+# Add logging to semaphore acquire/release
+# Send 100 tasks rapidly
+# Observe logs: Never more than 2 "acquired" without "released"
+# Count max concurrent AC requests: Always ≤ 2
+```
+
+### Test 3: No "Too Many Concurrent Requests" Errors
+```python
+# Run agent for 24 hours
+# Send 10,000 tasks
+# Check logs: Zero "too many concurrent requests" errors
+# Check AC server logs: All requests accepted
+```
+
+---
+
+## Next Steps
+
+1. **Review this design** - Confirm it addresses your concerns
+2. **Suggest any changes** - Especially threshold values
+3. **Approve for implementation** - We can implement in 1-2 days
+4. **Deploy to staging** - Test with real workload
+5. **Production rollout** - Gradual deployment
+
+---
+
+## Questions for You
+
+1. **Backpressure threshold**: Is 80% a good trigger point, or would you prefer 50%/90%?
+2. **AC server limit**: Is 2 concurrent the hard limit, or can it go higher?
+3. **Queue sizes**: Are 100 items per queue sufficient, or do you need larger buffers?
+4. **Executor pool size**: Should default be 10, or would you prefer 20/50?
+
+---
+
+**Prepared by**: Claude Code
+**Based on feedback**: Your two critical points about infinite polling and concurrent errors