Skip to content

Commit f380270

Browse files
gurvinder-dhillongurvinder-dhillon
authored
Merge pull request #1 from auto-browse/feature_gemini_support
Feature gemini support
2 parents ceefad7 + f042ba8 commit f380270

14 files changed

+144
-63
lines changed

‎README.md‎

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -61,29 +61,38 @@ If you're using Auto Browse alongside an existing Playwright setup, you must upg
6161
Auto Browse requires environment variables for the LLM (Language Model) configuration. Create a `.env` file in your project root:
6262

6363
```env
64+
# For OpenAI (default)
6465
OPENAI_API_KEY=your_openai_api_key_here
65-
LLM_MODEL=gpt-4o-mini # Optional, defaults to gpt-4o-mini
66+
LLM_PROVIDER=openai # Optional, this is default
67+
AUTOBROWSE_LLM_MODEL=gpt-4o-mini # Optional, defaults to gpt-4o-mini
68+
69+
# Or for Google AI
70+
GOOGLE_API_KEY=your_google_key_here
71+
LLM_PROVIDER=google
72+
AUTOBROWSE_LLM_MODEL=gemini-2.0-flash-lite # Or other supported model
6673
```
6774

6875
You can find an example configuration in `example.env`.
6976

7077
### Environment Variables
7178

72-
| Variable | Description | Default |
73-
| ---------------- | ------------------------------ | ------------- |
74-
| `OPENAI_API_KEY` | Your OpenAI API key (required) | - |
75-
| `LLM_MODEL` | The LLM model to use | `gpt-4o-mini` |
79+
| Variable | Description | Default |
80+
| ---------------------- | ----------------------------------- | ------------- |
81+
| `LLM_PROVIDER` | LLM provider to use (openai/google) | `openai` |
82+
| `OPENAI_API_KEY` | Your OpenAI API key | - |
83+
| `GOOGLE_API_KEY` | Your Google AI API key | - |
84+
| `AUTOBROWSE_LLM_MODEL` | The LLM model to use | `gpt-4o-mini` |
7685

7786
## Supported LLM Providers
7887

7988
Currently supported:
8089

8190
- OpenAI (gpt-4o-mini and compatible models)
91+
- Google AI (Gemini models)
8292

8393
Coming soon:
8494

8595
- Anthropic Claude
86-
- Google Gemini
8796
- Local LLMs
8897
- Meta Llama
8998

@@ -200,7 +209,6 @@ import { Given, When as aistep, Then } from "./fixtures";
200209
aistep(/^(.*)$/, async ({ page }, action: string) => {
201210
await auto(action, { page });
202211
});
203-
204212
```
205213

206214
#### Setup Requirements

‎docs/environment-setup.mdx‎

Lines changed: 17 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -5,13 +5,11 @@ description: "Configure Auto-Browse with different LLM providers and environment
55

66
# Environment Setup
77

8-
Auto-Browse supports various Language Model (LLM) providers to power its AI capabilities. This guide covers the setup for each supported provider.
8+
Auto-Browse supports multiple Language Model (LLM) providers to power its AI capabilities. This guide covers the setup for each supported provider.
99

10-
## OpenAI Configuration
10+
## Supported Providers
1111

12-
OpenAI is the default and currently supported LLM provider.
13-
14-
### Setup Steps
12+
### OpenAI (Default Provider)
1513

1614
1. Get your OpenAI API key:
1715

@@ -24,16 +22,27 @@ OpenAI is the default and currently supported LLM provider.
2422

2523
```env
2624
OPENAI_API_KEY=your_openai_api_key_here
27-
LLM_MODEL=gpt-4o-mini # Optional, defaults to gpt-4o-mini
25+
LLM_PROVIDER=openai # Optional, this is default
26+
AUTOBROWSE_LLM_MODEL=gpt-4o-mini # Optional, defaults to gpt-4o-mini
2827
```
2928

30-
### Model Options
29+
#### Model Options
3130

3231
OpenAI models supported:
3332

3433
- `gpt-4o-mini` (default)
3534
- Compatible GPT models
3635

36+
### Google AI
37+
38+
Configure your environment:
39+
40+
```env
41+
GOOGLE_API_KEY=your_google_key_here
42+
LLM_PROVIDER=google
43+
AUTOBROWSE_LLM_MODEL=gemini-2.0-flash-lite # Or other supported model
44+
```
45+
3746
## Upcoming Provider Support
3847

3948
The following LLM providers will be supported in future releases:
@@ -43,15 +52,7 @@ The following LLM providers will be supported in future releases:
4352
```env
4453
ANTHROPIC_API_KEY=your_anthropic_key_here
4554
LLM_PROVIDER=anthropic
46-
LLM_MODEL=claude-3 # Example model
47-
```
48-
49-
### Google Gemini
50-
51-
```env
52-
GOOGLE_API_KEY=your_google_key_here
53-
LLM_PROVIDER=gemini
54-
LLM_MODEL=gemini-pro # Example model
55+
AUTOBROWSE_LLM_MODEL=claude-3 # Example model
5556
```
5657

5758
### Local LLMs

‎docs/installation.mdx‎

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -74,16 +74,25 @@ Auto-Browse requires configuration through environment variables for the Languag
7474
1. Create a `.env` file in your project root:
7575

7676
```env
77+
# For OpenAI (default)
7778
OPENAI_API_KEY=your_openai_api_key_here
78-
LLM_MODEL=gpt-4o-mini # Optional, defaults to gpt-4o-mini
79+
LLM_PROVIDER=openai # Optional, this is default
80+
AUTOBROWSE_LLM_MODEL=gpt-4o-mini # Optional, defaults to gpt-4o-mini
81+
82+
# Or for Google AI
83+
GOOGLE_API_KEY=your_google_key_here
84+
LLM_PROVIDER=google
85+
AUTOBROWSE_LLM_MODEL=gemini-2.0-flash-lite # Or other supported model
7986
```
8087

8188
### Environment Variables Reference
8289

83-
| Variable | Description | Default | Required |
84-
| ---------------- | -------------------- | ------------- | -------- |
85-
| `OPENAI_API_KEY` | Your OpenAI API key | - | Yes |
86-
| `LLM_MODEL` | The LLM model to use | `gpt-4o-mini` | No |
90+
| Variable | Description | Default | Required |
91+
| ---------------------- | ----------------------------------- | ------------- | ---------- |
92+
| `LLM_PROVIDER` | LLM provider to use (openai/google) | `openai` | No |
93+
| `OPENAI_API_KEY` | Your OpenAI API key | - | For OpenAI |
94+
| `GOOGLE_API_KEY` | Your Google AI API key | - | For Google |
95+
| `AUTOBROWSE_LLM_MODEL` | The LLM model to use | `gpt-4o-mini` | No |
8796

8897
## TypeScript Configuration
8998

‎docs/quickstart.mdx‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,13 +17,22 @@ npm install @auto-browse/auto-browse
1717

1818
## Basic Setup
1919

20-
1. Create a `.env` file in your project root with your OpenAI API key:
20+
1. Create a `.env` file in your project root:
2121

2222
```env
23+
# For OpenAI (default)
2324
OPENAI_API_KEY=your_openai_api_key_here
24-
LLM_MODEL=gpt-4o-mini # Optional, defaults to gpt-4o-mini
25+
LLM_PROVIDER=openai # Optional, this is default
26+
AUTOBROWSE_LLM_MODEL=gpt-4o-mini # Optional, defaults to gpt-4o-mini
27+
28+
# Or for Google AI
29+
GOOGLE_API_KEY=your_google_key_here
30+
LLM_PROVIDER=google
31+
AUTOBROWSE_LLM_MODEL=gemini-2.0-flash-lite # Or other supported model
2532
```
2633

34+
Auto-Browse supports multiple LLM providers, with OpenAI as the default.
35+
2736
## Simple Example
2837

2938
Here's a complete example showing form automation:

‎example.env‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,4 @@
1+
LLM_PROVIDER=google
12
OPENAI_API_KEY=OPENAI_API_KEY
2-
LLM_MODEL=gpt-4o-mini
3+
AUTOBROWSE_LLM_MODEL=gemini-2.0-flash-lite
4+
GOOGLE_API_KEY=GOOGLE_API_KEY

‎package-lock.json‎

Lines changed: 26 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎package.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@
3939
"license": "MIT",
4040
"dependencies": {
4141
"@langchain/core": "^0.3.43",
42+
"@langchain/google-genai": "^0.2.1",
4243
"@langchain/langgraph": "^0.2.60",
4344
"@langchain/openai": "^0.5.0",
4445
"@modelcontextprotocol/sdk": "^1.8.0",
@@ -48,8 +49,8 @@
4849
"zod": "^3.24.2"
4950
},
5051
"devDependencies": {
51-
"@types/node": "^20.0.0",
5252
"@types/dotenv": "^6.1.1",
53+
"@types/node": "^20.0.0",
5354
"@typescript-eslint/eslint-plugin": "^6.0.0",
5455
"@typescript-eslint/parser": "^6.0.0",
5556
"eslint": "^8.0.0",

‎src/auto.ts‎

Lines changed: 2 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,16 @@
11
import { test as base } from '@playwright/test';
22
import { AutoConfig } from './types';
33
import { sessionManager, context } from './browser';
4-
import { ChatOpenAI } from "@langchain/openai";
54
import { createReactAgent } from "@langchain/langgraph/prebuilt";
65
import { HumanMessage } from "@langchain/core/messages";
7-
import dotenv from 'dotenv';
6+
import { createLLMModel } from './llm';
87
import {
98
browser_click, browser_type, browser_get_text, browser_navigate, browser_snapshot,
109
browser_hover, browser_drag, browser_select_option, browser_take_screenshot,
1110
browser_go_back, browser_wait, browser_press_key, browser_save_pdf, browser_choose_file,
1211
browser_go_forward, browser_assert
1312
} from './tools';
1413

15-
// Load environment variables
16-
dotenv.config();
17-
const openai_llm_model = process.env.LLM_MODEL || 'gpt-4o-mini';
18-
19-
const openai_model = new ChatOpenAI({
20-
modelName: openai_llm_model,
21-
temperature: 0,
22-
});
23-
2414
// Extend base test to automatically track page
2515
export const test = base.extend({
2616
page: async ({ page }, use) => {
@@ -31,7 +21,7 @@ export const test = base.extend({
3121

3222
// Initialize the LangChain agent with more detailed instructions
3323
const initializeAgent = () => {
34-
const model = openai_model;
24+
const model = createLLMModel();
3525

3626
const prompt =
3727
`You are a web automation assistant. When given a natural language instruction:
@@ -67,8 +57,6 @@ const initializeAgent = () => {
6757
return { agent };
6858
};
6959

70-
71-
7260
// Main auto function that processes instructions
7361
export async function auto(instruction: string, config?: AutoConfig): Promise<any> {
7462
console.log(`[Auto] Processing instruction: "${instruction}"`);

‎src/llm.ts‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
import { ChatOpenAI } from "@langchain/openai";
2+
import { ChatGoogleGenerativeAI } from "@langchain/google-genai";
3+
import dotenv from 'dotenv';
4+
5+
// Load environment variables
6+
dotenv.config();
7+
8+
export function createLLMModel() {
9+
const provider = process.env.LLM_PROVIDER || 'openai';
10+
const model = process.env.AUTOBROWSE_LLM_MODEL || 'gpt-4o-mini';
11+
12+
if (provider === 'google')
13+
{
14+
return new ChatGoogleGenerativeAI({
15+
model: model
16+
});
17+
}
18+
19+
return new ChatOpenAI({
20+
modelName: model,
21+
temperature: 0,
22+
});
23+
}

‎src/tools/browser_go_back.ts‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,11 @@ import { context } from '../browser/context';
55

66
/**
77
* Schema for going back in browser history
8+
* Includes dummy property to satisfy Gemini's API requirement for non-empty object properties
89
*/
9-
const goBackSchema = z.object({});
10+
const goBackSchema = z.object({
11+
_: z.string().optional().describe('No parameters required for this operation')
12+
});
1013

1114
export const browser_go_back = tool(
1215
async () => {

0 commit comments

Comments
 (0)