fix(ai): improves handling of llm context window limit

[mvilanova]

Summary of Changes to AI Service Module

Key Improvements

1. Model Token Limit Management

• Added get_model_token_limit() function that returns maximum token capacity for different LLM models
• Included support for latest models (GPT-4o, Claude 3.5/3.7 Sonnet)
• Implemented a safety buffer parameter (default: 5%) to prevent hitting absolute limits

2. Dynamic Token Truncation

• Updated truncate_prompt() to use model-specific limits instead of fixed values
• Enhanced function signature to accept the model token limit as a parameter
• Improved logging to show the actual token limit being applied

3. Smarter Prompt Processing

• Modified prompt processing in signal and incident summary generation
• Each function now dynamically calculates appropriate token limits based on the model in use
• Applied consistent approach across different generation functions

Benefits

• Model-Specific Handling: Each model gets appropriate token limits based on its capabilities
• Safety Margins: Buffer prevents errors from edge cases or slight miscalculations
• Maintainability: Centralized token limit management makes updates easier
• Future-Proofing: Simple to add new models as they become available

These changes provide a more robust approach to managing context windows across different LLM models while maximizing available context space.