feat: add Tavily Extract as configurable WebFetch backend alongside Jina#5
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces Tavily as an alternative WebFetch provider alongside Jina, adding configuration options, updating dependencies, and implementing the readpage_tavily method. Feedback on these changes suggests ensuring consistency with the Jina backend by passing and respecting the runtime_deadline parameter in readpage_tavily to enforce the agent's runtime budget. Additionally, it is recommended to add a type check to verify that the first element of the Tavily API results is a dictionary before accessing its keys, preventing potential runtime errors.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| response = self.readpage_tavily( | ||
| url, | ||
| start_line=start_line, | ||
| end_line=end_line, | ||
| max_chars=max_chars, | ||
| ) |
There was a problem hiding this comment.
Pass the runtime_deadline parameter to readpage_tavily to ensure consistency with the Jina backend and respect the agent's runtime budget.
| response = self.readpage_tavily( | |
| url, | |
| start_line=start_line, | |
| end_line=end_line, | |
| max_chars=max_chars, | |
| ) | |
| response = self.readpage_tavily( | |
| url, | |
| start_line=start_line, | |
| end_line=end_line, | |
| max_chars=max_chars, | |
| runtime_deadline=runtime_deadline, | |
| ) |
| def readpage_tavily( | ||
| self, | ||
| url: str, | ||
| *, | ||
| start_line: int = 1, | ||
| end_line: Optional[int] = None, | ||
| max_chars: int = DEFAULT_WEBFETCH_MAX_CHARS, | ||
| ) -> str: | ||
| tavily_api_key = os.getenv("TAVILY_API_KEY", "").strip() |
There was a problem hiding this comment.
Add the runtime_deadline parameter to readpage_tavily and check the remaining budget before proceeding with the Tavily API call. This ensures the tool respects the agent's runtime limit, matching the behavior of the Jina backend.
def readpage_tavily(
self,
url: str,
*,
start_line: int = 1,
end_line: Optional[int] = None,
max_chars: int = DEFAULT_WEBFETCH_MAX_CHARS,
runtime_deadline: Optional[float] = None,
) -> str:
remaining = self._remaining_budget_seconds(runtime_deadline)
if remaining is not None and remaining <= 0:
return "[WebFetch] Failed to read page: agent runtime limit reached."
tavily_api_key = os.getenv("TAVILY_API_KEY", "").strip()| results = result.get("results", []) | ||
| if not results or not results[0].get("raw_content"): | ||
| return "[WebFetch] Failed to read page: the provided webpage content could not be accessed. Please check the URL or file format." |
There was a problem hiding this comment.
Add a check to ensure results[0] is a dictionary before calling .get() on it. If the API returns an unexpected structure or None in the results list, this prevents a potential AttributeError.
| results = result.get("results", []) | |
| if not results or not results[0].get("raw_content"): | |
| return "[WebFetch] Failed to read page: the provided webpage content could not be accessed. Please check the URL or file format." | |
| results = result.get("results", []) | |
| if not results or not isinstance(results[0], dict) or not results[0].get("raw_content"): | |
| return "[WebFetch] Failed to read page: the provided webpage content could not be accessed. Please check the URL or file format." |
1 participant
Summary
Adds Tavily Extract as a selectable parallel backend for the
WebFetchtool, alongside the existing Jina provider. A newWEBFETCH_PROVIDERenvironment variable (jina|tavily, defaultjina) controls which backend is invoked at runtime.What changed
agent_base/tools/tool_web.py: Addedreadpage_tavily()method toWebFetchthat usesTavilyClient.extract(urls=[url])for content extraction. Modifiedcall()to dispatch based onWEBFETCH_PROVIDERenv var. Existing Jina paths (jina_readpage,html_readpage_jina,readpage_jina) are untouched..env.example: DocumentedWEBFETCH_PROVIDERandTAVILY_API_KEYas optional env vars.requirements.txt: Addedtavily-python>=0.5.0.Files changed
agent_base/tools/tool_web.py.env.examplerequirements.txtDependency changes
tavily-python>=0.5.0torequirements.txtEnvironment variable changes
WEBFETCH_PROVIDER(optional, default:jina; values:jina|tavily)TAVILY_API_KEYreference in.env.example(required whenWEBFETCH_PROVIDER=tavily)JINA_KEYunchangedNotes for reviewers
extract()response'sraw_contentfield is passed through the existing_format_page_content()for consistent line-range and max_chars formatting.TAVILY_API_KEYguard mirrors the existingJINA_KEYguard pattern.Automated Review
TavilyClient.extract(urls=[url])withraw_contentextraction) is correct, error handling is consistent with the existing codebase, and the Jina path is fully preserved. Two minor issues exist: (1)tavily-python>=0.5.0is added torequirements.txtin this diff, but it was already added by the prerequisite unit (web-search-serper-additive), creating a potential duplicate on merge. (2)TAVILY_API_KEYis added to.env.examplehere, despite the migration plan noting "TAVILY_API_KEY shared with web-search unit; no duplicate addition needed" — same duplicate risk. Neither issue blocks correctness, and neither was added to the sensitive env masking list (appropriately deferred to the prerequisite unit).