Repository navigation
fix(sessions)!: Never reuse a retired session - #2348
Conversation
Persist retirement independently of error scores and cover session and pool restoration. BREAKING CHANGE: Retired sessions cannot become usable again after mark_good().
vdusek
left a comment
There was a problem hiding this comment.
A three more comments, regarding comparision with Crawlee JS
|
Thanks for the careful review, @vdusek! I addressed the five comments in 261dab6:
All 49 session tests, full repository lint, and the migration Markdown lint pass. The full type check passes targeting Linux; native Windows reports three unrelated multiprocessing fork-type errors in tests/unit/_utils/test_system.py. Both commit hooks passed using an external Linux-targeted type configuration with the same repository rules. Happy to adjust anything further. Thanks again! CI note: the new factory-review approval lookup failed because GitHub returned HTTP 502, while the AI review job passed. GitHub requires repository admin rights to rerun the failed job, so I could not retry it from my fork. Could you rerun that lookup when you have a chance? |
Description
A session retired after a blocked request could become usable again, because
mark_good()lowered its error score below the limit. Retirement is now tracked separately from the error score and is permanent, including after the session pool is saved and restored. The behavior matches Crawlee JS v4.Sessionkeeps aretiredflag, exposed as theSession.retiredproperty and saved inSessionModel. Saved state without the field loads withretired=False.is_usablechecks the flag.is_blockedstill depends only on the error score.mark_good()does nothing on a retired session, so its error score and usage count stay as they are.retire()addsmax_error_scoreto the error score once. Calling it on a session that is already retired changes nothing.mark_good()ormark_bad()finds a session expired, at its usage limit, or at its error score limit, it only sets the flag and adds no extramax_error_score.retire()last.Issues
Closes: #2312
Testing
mark_good()on a retired session, repeatedretire()calls, automatic retirement at each limit, JSON round trips, loading older saved state, pool lookups, and saving and reopening the pool.tests/unit/sessionsandtests/unit/crawlers/_basic: 211 passed, 1 existing skip.ruff check,ruff format --checkandty checkpass.