feat: improve token counting and add brevity pattern

[PatrickRuddiman]

This pull request introduces a new brevity pattern to handle context overflow scenarios, improves token estimation accuracy, and refactors the codebase to support these changes. The most important updates include defining the brevity pattern, implementing token estimation using TiktokenSharp, and enhancing the chunk summarization logic in OpenAIService.

New Pattern for Context Overflow:

• Constants/FabricPatterns.cs and Constants/PatternNames.cs: Added a new constant BrevityPattern to define the pattern used for summarization when context overflow occurs. [1] [2]
• patterns/brief_chunk_summary/system.md: Introduced a new pattern file that outlines the purpose, steps, and output for the brevity summarization process.

Token Estimation Improvements:

• Services/TokenHelper.cs: Added a new TokenHelper class using TiktokenSharp for accurate token estimation, with fallback logic for cases where encoding fails.
• WriteCommit.csproj: Included TiktokenSharp as a new dependency for token estimation.
• Services/SemanticCoherenceAnalyzer.cs: Updated the EstimateTokenCount method to use TokenHelper for better accuracy.

Enhanced Chunk Summarization Logic:

• Services/OpenAIService.cs: Introduced logic to handle context overflow by re-chunking summaries and using the new BrevityPattern for condensed summarization. Added a constant MaxContextTokens to define the token limit. [1] [2]

Documentation Update:

• README.md: Documented the new brief_chunk_summary pattern and its usage for handling overflowing contexts.

[Copilot]

Pull Request Overview

This PR implements a new brevity pattern for handling context overflow scenarios, refactors token estimation to improve accuracy using TiktokenSharp, and enhances the chunk summarization logic in the OpenAIService.

• Introduces a new constant and pattern file for brevity summarization.
• Implements token estimation improvements through a dedicated TokenHelper class and integrates it into other services.
• Refactors OpenAIService to re-chunk messages when context tokens exceed the limit.

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
patterns/brief_chunk_summary/system.md	Adds a new pattern file describing the brevity summarization process.
WriteCommit.csproj	Adds dependency to TiktokenSharp for enhanced token estimation.
Services/TokenHelper.cs	Implements token estimation using TiktokenSharp with fallback logic.
Services/SemanticCoherenceAnalyzer.cs	Refactors token estimation to use the new TokenHelper for better accuracy.
Services/OpenAIService.cs	Enhances chunk summarization logic with re-chunking of messages for context overflow.
README.md	Updates documentation to include usage of the brevity pattern.
Constants/PatternNames.cs	Adds a constant for the new brevity pattern.
Constants/FabricPatterns.cs	Adds a constant for the new brevity pattern.

[Copilot]

Since systemPrompt remains unchanged throughout the method, consider caching its token estimate once rather than repeatedly recalculating it to improve maintainability.

Suggested change

	var currentTokens = TokenHelper.EstimateTokens(systemPrompt, model);
	var currentTokens = systemPromptTokens;

[PatrickRuddiman]

this may not be needed; ill do more testing