feat(deep_crawling): allow configuring batch_size and dispatcher

[yashikam19]

Related discussion: #2245

Summary

Adds two constructor parameters to BFSDeepCrawlStrategy and
BestFirstCrawlingStrategy for finer concurrency control during deep crawls:

1. batch_size (BestFirstCrawlingStrategy only) — previously a hardcoded
   module constant (BATCH_SIZE = 10) governing how many URLs are pulled from
   the priority queue per round before being handed to arun_many(). Now a
   constructor parameter, defaulting to the same value (10), so nothing changes
   for existing callers.
2. dispatcher (both strategies) — previously neither strategy forwarded a
   dispatcher to its internal arun_many() calls, so callers had no way to
   supply their own (e.g. SemaphoreDispatcher, or a MemoryAdaptiveDispatcher
   with a custom RateLimiter/CrawlerMonitor) - only CrawlerRunConfig.semaphore_count
   was reachable, which controls concurrency count but not dispatcher behavior.
   Now optional and forwarded when set.

Both changes are additive and opt-in — dispatcher is only passed to
arun_many() when explicitly provided, so the call shape (and any existing
test doubles built against it) is unchanged for the default case.

List of files changed and why

• crawl4ai/deep_crawling/bfs_strategy.py - added dispatcher param, forwarded
  in both _arun_batch and _arun_stream.
• crawl4ai/deep_crawling/bff_strategy.py - added batch_size and dispatcher
  params; batch_size replaces the module-level BATCH_SIZE constant at its
  three usage sites; dispatcher forwarded in _arun_best_first.
• tests/deep_crawling/test_deep_crawl_dispatcher_batch_size.py (new) — covers
  batch_size defaults/override/actual round-size behavior, and dispatcher
  default-omitted vs explicitly-forwarded behavior for both strategies.

How Has This Been Tested?

• pytest tests/deep_crawling/ tests/general/test_deep_crawl*.py: 80 passed,
  0 regressions
• New test file (10 tests) specifically covering the two new parameters.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes