Skip to content

Commit 3f2236c

Browse files
wbrennan899wbrennan899
committed
feat: Add Twelve Labs Pegasus video intelligence integration
 Enables AI agents to understand and analyze video content using Twelve Labs Pegasus 1.2. Key capabilities: - Upload and index videos from files or URLs - Interactive Q&A about video content - Generate summaries, chapters, and highlights - Batch process multiple videos - Full async support with progress tracking Includes VideoProcessingAgent and TwelveLabsTools for easy integration with existing agents and workflows. Requires TWELVELABS_API_KEY environment variable.\
1 parent 8b4aa1e commit 3f2236c

File tree

16 files changed

+3931
-1
lines changed

16 files changed

+3931
-1
lines changed

‎python/.env.example‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,3 +29,9 @@ ENABLE_OTEL=true
2929
ENABLE_SENSITIVE_DATA=true
3030
OTLP_ENDPOINT="http://localhost:4317/"
3131
# APPLICATIONINSIGHTS_CONNECTION_STRING="..."
32+
# Twelve Labs
33+
TWELVELABS_API_KEY=""
34+
TWELVELABS_DEFAULT_INDEX="default"
35+
TWELVELABS_MAX_VIDEO_SIZE="5000000000" # 5GB in bytes
36+
TWELVELABS_CHUNK_SIZE="10000000" # 10MB in bytes
37+
TWELVELABS_RATE_LIMIT="60" # requests per minute

‎python/packages/twelvelabs/LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2025 Microsoft Corporation
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎python/packages/twelvelabs/README.md‎

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
# Twelve Labs Integration for Microsoft Agent Framework
2+
3+
Add video intelligence capabilities to your agents using Twelve Labs Pegasus 1.2 APIs.
4+
5+
## Features
6+
7+
- 🎥 **Video Upload & Indexing** - Upload videos from files or URLs
8+
- 💬 **Interactive Q&A** - Chat with video content using natural language
9+
- 📝 **Summarization** - Generate comprehensive summaries
10+
- 📑 **Chapter Generation** - Create chapter markers with timestamps
11+
- ✨ **Highlight Extraction** - Extract key moments and highlights
12+
- 📊 **Batch Processing** - Process multiple videos concurrently
13+
14+
## Installation
15+
16+
### From Source (Development)
17+
18+
```bash
19+
# Clone the repository
20+
git clone https://github.com/microsoft/agent-framework.git
21+
cd agent-framework/python/packages/twelvelabs
22+
23+
# Install in development mode
24+
pip install -e .
25+
```
26+
27+
### Requirements
28+
29+
- Python 3.10+
30+
- `twelvelabs>=1.0.2` (automatically installed)
31+
- `agent-framework>=1.0.0b251001` (automatically installed)
32+
- Valid Twelve Labs API key
33+
34+
## Quick Start
35+
36+
Set your API key from [Twelve Labs](https://twelvelabs.io):
37+
38+
```bash
39+
export TWELVELABS_API_KEY="your-api-key"
40+
```
41+
42+
### Option 1: Add Video Tools to Your Existing Agent
43+
44+
```python
45+
from agent_framework import ChatAgent
46+
from agent_framework.openai import OpenAIChatClient
47+
from agent_framework_twelvelabs import TwelveLabsTools
48+
49+
# Add video capabilities to any agent
50+
tools = TwelveLabsTools()
51+
agent = ChatAgent(
52+
chat_client=OpenAIChatClient(), # or any chat client
53+
instructions="You are a helpful assistant",
54+
tools=tools.get_all_tools() # Adds 8 video functions
55+
)
56+
57+
# Now your agent can process videos
58+
result = await agent.run("Upload and analyze video.mp4")
59+
```
60+
61+
### Option 2: Use Pre-configured Video Agent
62+
63+
```python
64+
import asyncio
65+
from agent_framework import ChatMessage
66+
from agent_framework.openai import OpenAIChatClient
67+
from agent_framework_twelvelabs import VideoProcessingAgent
68+
69+
async def main():
70+
# Create agent
71+
agent = VideoProcessingAgent(
72+
chat_client=OpenAIChatClient(
73+
api_key="your-openai-key",
74+
model_id="gpt-4"
75+
)
76+
)
77+
78+
# Build conversation with message history
79+
messages = []
80+
81+
# Upload video
82+
messages.append(ChatMessage(role="user", text="Upload video.mp4"))
83+
response = await agent.run(messages)
84+
print(f"Upload: {response}")
85+
messages.append(ChatMessage(role="assistant", text=str(response)))
86+
87+
# Ask about the video - agent maintains context
88+
messages.append(ChatMessage(role="user", text="What do you see in this video?"))
89+
response = await agent.run(messages)
90+
print(f"Analysis: {response}")
91+
92+
# Generate chapters
93+
messages.append(ChatMessage(role="user", text="Generate chapter markers for this video"))
94+
response = await agent.run(messages)
95+
print(f"Chapters: {response}")
96+
97+
asyncio.run(main())
98+
```
99+
100+
### Option 3: Direct Client Usage (No Agent)
101+
102+
```python
103+
from agent_framework_twelvelabs import TwelveLabsClient
104+
105+
# For custom workflows without an agent
106+
client = TwelveLabsClient()
107+
108+
# Upload video
109+
metadata = await client.upload_video(
110+
url="https://example.com/video.mp4" # or file_path="video.mp4"
111+
)
112+
113+
# Chat with video
114+
response = await client.chat_with_video(
115+
video_id=metadata.video_id,
116+
query="What products are shown?"
117+
)
118+
119+
# Generate summary
120+
summary = await client.summarize_video(video_id=metadata.video_id)
121+
122+
# Generate chapters
123+
chapters = await client.generate_chapters(video_id=metadata.video_id)
124+
125+
# Generate highlights
126+
highlights = await client.generate_highlights(video_id=metadata.video_id)
127+
```
128+
129+
## Configuration
130+
131+
```bash
132+
export TWELVELABS_API_KEY="your-api-key"
133+
```
134+
135+
Additional configuration options are available through environment variables or `TwelveLabsSettings`.
136+
137+
## Available Tools
138+
139+
These 8 AI functions are added to your agent when using TwelveLabsTools:
140+
141+
- `upload_video` - Upload and index videos from file path or URL
142+
- `chat_with_video` - Q&A with video content
143+
- `summarize_video` - Generate comprehensive video summaries
144+
- `generate_chapters` - Create chapter markers with timestamps
145+
- `generate_highlights` - Extract key highlights and moments
146+
- `get_video_info` - Get video metadata
147+
- `delete_video` - Remove indexed videos
148+
- `batch_process_videos` - Process multiple videos concurrently
149+
150+
The agent can automatically call these tools based on user requests.
151+
152+
## Video Requirements
153+
154+
### Supported Formats
155+
- All FFmpeg-compatible video and audio codecs
156+
- Common formats: MP4, AVI, MOV, MKV, WebM
157+
158+
### Technical Specifications
159+
- **Resolution**: 360x360 minimum, 3840x2160 maximum
160+
- **Aspect Ratios**: 1:1, 4:3, 4:5, 5:4, 16:9, 9:16, 17:9
161+
- **Duration**: 4 seconds to 60 minutes (Pegasus 1.2)
162+
- **File Size**: Up to 5GB (configurable)

‎python/packages/twelvelabs/agent_framework_twelvelabs/__init__.py‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# Copyright (c) Microsoft. All rights reserved.
2+
3+
"""Twelve Labs Pegasus integration for Microsoft Agent Framework."""
4+
5+
from ._agent import VideoProcessingAgent
6+
from ._client import TwelveLabsClient, TwelveLabsSettings
7+
from ._exceptions import (
8+
AuthenticationError,
9+
FileTooLargeError,
10+
InvalidFormatError,
11+
ProcessingFailedError,
12+
RateLimitError,
13+
TwelveLabsError,
14+
UploadTimeoutError,
15+
VideoNotFoundError,
16+
VideoNotReadyError,
17+
VideoProcessingError,
18+
VideoUploadError,
19+
)
20+
from ._executor import BatchVideoExecutor, VideoExecutor
21+
from ._middleware import VideoUploadProgressMiddleware
22+
from ._tools import TwelveLabsTools
23+
from ._types import (
24+
ChapterInfo,
25+
ChapterResult,
26+
HighlightInfo,
27+
HighlightResult,
28+
SummaryResult,
29+
VideoMetadata,
30+
VideoOperationType,
31+
VideoStatus,
32+
)
33+
34+
__version__ = "0.1.0"
35+
36+
__all__ = [
37+
# Main classes
38+
"VideoProcessingAgent",
39+
"TwelveLabsClient",
40+
"TwelveLabsSettings",
41+
"TwelveLabsTools",
42+
"VideoExecutor",
43+
"BatchVideoExecutor",
44+
"VideoUploadProgressMiddleware",
45+
# Types
46+
"VideoMetadata",
47+
"VideoStatus",
48+
"VideoOperationType",
49+
"SummaryResult",
50+
"ChapterResult",
51+
"ChapterInfo",
52+
"HighlightResult",
53+
"HighlightInfo",
54+
# Exceptions
55+
"TwelveLabsError",
56+
"VideoUploadError",
57+
"FileTooLargeError",
58+
"InvalidFormatError",
59+
"UploadTimeoutError",
60+
"VideoProcessingError",
61+
"VideoNotReadyError",
62+
"ProcessingFailedError",
63+
"VideoNotFoundError",
64+
"RateLimitError",
65+
"AuthenticationError",
66+
]

0 commit comments

Comments
 (0)