@@ -1338,13 +1338,15 @@ def web_crawl_md(
13381338 than this value, it will be aborted with a 408 status code. Maximum allowed
13391339 value is 300000ms (5 minutes).
13401340
1341- url_regex: Regex pattern. Only URLs matching this pattern will be followed and scraped.
1341+ url_regex: Regex pattern. Only URLs matching this pattern will be followed and scraped. An
1342+ automatic prefix scope in the form ^<starting URL> follows a redirect of the
1343+ starting page.
13421344
13431345 use_main_content_only: Extract only the main content, stripping headers, footers, sidebars, and
13441346 navigation
13451347
1346- wait_for_ms: Optional browser wait time in milliseconds after initial page load for each
1347- crawled page . Min: 0. Max: 30000 (30 seconds).
1348+ wait_for_ms: Browser wait time in milliseconds after initial page load for each crawled page.
1349+ Defaults to 3500 (3.5 seconds) . Min: 0. Max: 30000 (30 seconds).
13481350
13491351 zdr: Set to enabled to bypass shared caches and omit request and response content
13501352 from retained usage logs. Requires zero data retention to be enabled for your
@@ -3576,13 +3578,15 @@ async def web_crawl_md(
35763578 than this value, it will be aborted with a 408 status code. Maximum allowed
35773579 value is 300000ms (5 minutes).
35783580
3579- url_regex: Regex pattern. Only URLs matching this pattern will be followed and scraped.
3581+ url_regex: Regex pattern. Only URLs matching this pattern will be followed and scraped. An
3582+ automatic prefix scope in the form ^<starting URL> follows a redirect of the
3583+ starting page.
35803584
35813585 use_main_content_only: Extract only the main content, stripping headers, footers, sidebars, and
35823586 navigation
35833587
3584- wait_for_ms: Optional browser wait time in milliseconds after initial page load for each
3585- crawled page . Min: 0. Max: 30000 (30 seconds).
3588+ wait_for_ms: Browser wait time in milliseconds after initial page load for each crawled page.
3589+ Defaults to 3500 (3.5 seconds) . Min: 0. Max: 30000 (30 seconds).
35863590
35873591 zdr: Set to enabled to bypass shared caches and omit request and response content
35883592 from retained usage logs. Requires zero data retention to be enabled for your
0 commit comments