[Feat] Walmart scraper: product + reviews across API, chat, and MCP

[CREDO23]

Adds Walmart as a first-class native scraper, matching the other platforms end to end.

• Two verbs: walmart.scrape (products/search/category → prices, ratings, sellers, variants, availability) and walmart.reviews (full paginated public review history).
• Fully wired: REST, chat subagent + routing, MCP tools, per-product/per-review billing, playground, marketing page, and docs.
• Validated: 31 unit tests green, plus live E2E/smoke and manual chat testing.

High-level PR Summary

This PR adds Walmart as a first-class native scraper platform alongside Reddit, YouTube, Amazon, and others. It implements two main capabilities: walmart.scrape for extracting product data (prices, ratings, sellers, variants, availability) from product pages and search results, and walmart.reviews for paginated customer review history. The implementation includes a complete scraper engine that extracts data from Walmart's server-rendered Next.js JSON state, REST API endpoints with per-product and per-review billing, chat agent integration with routing and subagent system, MCP tools for external agent integration, comprehensive test coverage (31 unit tests), documentation, and marketing pages. The scraper uses residential proxies with anti-bot detection and retry logic to handle Walmart's Akamai and PerimeterX protection.

⏱️ Estimated Review Time: 30-90 minutes

💡 Review Order Suggestion

Order	File Path
1	surfsense_backend/app/proprietary/platforms/walmart/README.md
2	surfsense_backend/app/proprietary/platforms/walmart/schemas.py
3	surfsense_backend/app/proprietary/platforms/walmart/url_resolver.py
4	surfsense_backend/app/proprietary/platforms/walmart/next_data.py
5	surfsense_backend/app/proprietary/platforms/walmart/parsers.py
6	surfsense_backend/app/proprietary/platforms/walmart/fetch.py
7	surfsense_backend/app/proprietary/platforms/walmart/scraper.py
8	surfsense_backend/app/proprietary/platforms/walmart/__init__.py
9	surfsense_backend/app/capabilities/walmart/scrape/schemas.py
10	surfsense_backend/app/capabilities/walmart/scrape/executor.py
11	surfsense_backend/app/capabilities/walmart/scrape/definition.py
12	surfsense_backend/app/capabilities/walmart/reviews/schemas.py
13	surfsense_backend/app/capabilities/walmart/reviews/executor.py
14	surfsense_backend/app/capabilities/walmart/reviews/definition.py
15	surfsense_backend/app/capabilities/walmart/__init__.py
16	surfsense_backend/app/capabilities/core/types.py
17	surfsense_backend/app/capabilities/core/billing.py
18	surfsense_backend/app/config/__init__.py
19	docker/.env.example
20	surfsense_backend/.env.example
21	surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/walmart/tools/index.py
22	surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/walmart/agent.py
23	surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/walmart/description.md
24	surfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/walmart/system_prompt.md
25	surfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.py
26	surfsense_backend/app/agents/chat/multi_agent_chat/constants.py
27	surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/system_prompt/prompts/identity/private.md
28	surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/system_prompt/prompts/identity/team.md
29	surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/system_prompt/prompts/kb_first.md
30	surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/system_prompt/prompts/routing.md
31	surfsense_mcp/mcp_server/features/scrapers/platforms/walmart.py
32	surfsense_mcp/mcp_server/features/scrapers/__init__.py
33	surfsense_backend/app/routes/__init__.py
34	surfsense_backend/tests/unit/platforms/walmart/test_parsers.py
35	surfsense_backend/tests/unit/platforms/walmart/test_flows.py
36	surfsense_backend/tests/unit/capabilities/walmart/test_registry.py
37	surfsense_backend/tests/unit/capabilities/walmart/scrape/test_schemas.py
38	surfsense_backend/tests/unit/capabilities/walmart/scrape/test_executor.py
39	surfsense_backend/tests/unit/capabilities/walmart/reviews/test_schemas.py
40	surfsense_backend/tests/unit/capabilities/walmart/reviews/test_executor.py
41	surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py
42	surfsense_backend/scripts/e2e_walmart_scraper.py
43	surfsense_web/content/docs/connectors/native/walmart.mdx
44	surfsense_web/content/docs/connectors/native/index.mdx
45	surfsense_web/content/docs/connectors/native/meta.json
46	surfsense_web/content/docs/connectors/index.mdx
47	surfsense_web/content/docs/how-to/mcp-server.mdx
48	surfsense_web/lib/connectors-marketing/walmart.tsx
49	surfsense_web/lib/connectors-marketing/index.ts
50	surfsense_web/lib/playground/catalog.ts
51	surfsense_web/lib/playground/platform-icons.tsx
52	surfsense_web/public/connectors/walmart.svg
53	surfsense_web/app/(home)/mcp-server/page.tsx
54	surfsense_web/components/homepage/hero-section.tsx
55	surfsense_web/components/homepage/persona-paths.tsx
56	surfsense_web/components/seo/json-ld.tsx
57	surfsense_web/lib/chat/example-prompts.ts

[vercel]

@CREDO23 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel.

A member of the Team first needs to authorize it.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: bafb9786-779a-4344-affb-c9568a434878

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}