Python: Twelve Labs Video AI Integration

[wbrennan899]

Motivation and Context

This change adds video intelligence capabilities to the Agent Framework, enabling agents to understand and interact with video content.

Why is this change required?
The framework currently lacks native support for video understanding. Users who want to build agents that can analyze video content must implement custom solutions or use external tools.

What problem does it solve?
It solves the problem of video analysis by providing a complete integration with Twelve Labs' Pegasus APIs, which use multimodal foundation models to understand both visual and audio content in videos.

What scenario does it contribute to?

• Content moderation for user-generated videos
• Educational content extraction from lecture videos
• Video library search and discovery
• Compliance and audit workflows
• Marketing and competitor analysis
• Interactive training video Q&A systems

Description

This PR adds a new Python package agent-framework-twelvelabs that integrates Twelve Labs' Pegasus video intelligence APIs into the Agent Framework.

What is Twelve Labs?
Twelve Labs provides video understanding APIs powered by multimodal foundation models. Unlike traditional video processing that relies solely on transcription, Twelve Labs' Pegasus model analyzes both visual and audio content to understand what's happening in videos.

Implementation approach:

The package follows established framework patterns (similar to OpenAI and Anthropic integrations) and provides:

1. TwelveLabsClient - Async client with chunked upload support, retry logic, and error handling
2. TwelveLabsTools - Collection of 8 AI functions that agents can call:
   • Upload and index videos (from files or URLs)
   • Interactive Q&A with video content
   • Generate summaries
   • Create chapter markers with timestamps
   • Extract highlights
   • Get video metadata
   • Delete videos
   • Batch process multiple videos
3. VideoProcessingAgent - Pre-configured agent specialized for video tasks
4. VideoExecutor - Workflow executor for batch processing scenarios
5. VideoMiddleware - Progress reporting and logging for long operations

Key features:

• Async-first design for optimal performance
• Support for videos up to 5GB and 60 minutes
• All FFmpeg-compatible formats (MP4, AVI, MOV, MKV, WebM)
• Automatic chunked upload with progress tracking
• Type-safe implementation using Pydantic models
• Comprehensive error handling with custom exception hierarchy

Integration options:

• Add tools to existing agents using TwelveLabsTools
• Use pre-configured VideoProcessingAgent
• Direct client access via TwelveLabsClient for custom workflows

The implementation is fully modular and doesn't modify the core framework.

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the Contribution Guidelines
• All unit tests pass, and I have added new tests where possible (23/23 tests passing)
• Is this a breaking change? No

[adityamandaleeka]

@wbrennan899 Thanks for checking out Agent Framework and putting this together!

This looks like a cool idea (and the product looks impressive!), but it’s probably not something we’ll take into the main repo. We’re still in the early days and figuring out the right balance of what belongs here, but for now we’re trying to keep this repo focused on the core framework pieces.

You can of course host this in your own repo and publish it as packages that integrate with Agent Framework... that’s actually how we’d love to see the ecosystem grow.

It’s awesome to see more folks experimenting and extending the framework :).

[wbrennan899]

@microsoft-github-policy-service agree company="TwelveLabs"

[adityamandaleeka]

Closing this PR per my earlier comment. Thanks again!

[james-le-twelve-labs]

@adityamandaleeka We are hosting our integration in our own forked repo here: https://github.com/james-le-twelve-labs/microsoft-agent-framework/tree/twelvelabs-pegasus-integration

Can you clarify what you mean by "publish it as packages that integrate with Agent Framework"? We would love to create developer enablement assets to showcase how to build video agents that can search and understand video content using Microsoft Agent Framework!

@wbrennan899 Thanks for checking out Agent Framework and putting this together!

This looks like a cool idea (and the product looks impressive!), but it’s probably not something we’ll take into the main repo. We’re still in the early days and figuring out the right balance of what belongs here, but for now we’re trying to keep this repo focused on the core framework pieces.

You can of course host this in your own repo and publish it as packages that integrate with Agent Framework... that’s actually how we’d love to see the ecosystem grow.

It’s awesome to see more folks experimenting and extending the framework :).