perf(impit): optimize client and connection reuse across Sessions

[Mantisus]

Description

• Cookie handling has been moved entirely from impit to ImpitHttpClient. Clients are now reused per proxy instead of being recreated for every Session, so a single client keeps its connection pool alive across all of them. The result is higher performance than that of HttpxHttpClient.
• Redirects are followed one hop at a time, which is what makes per-session cookie handling possible. Header handling along the chain now follows the WHATWG Fetch algorithm.

Testing

• Added tests for extracting cookies from Set-Cookie headers into the jar and for building the Cookie header.
• Added tests for redirect handling in ImpitHttpClient.

[Copilot]

Pull request overview

This PR refactors ImpitHttpClient to reuse underlying impit.AsyncClient instances across sessions (per-proxy), while taking over redirect handling to process redirects hop-by-hop so per-session cookie jars can be applied and updated at each hop.

Changes:

• Moved cookie extraction / cookie header construction into SessionCookies, and integrated it into ImpitHttpClient on each redirect hop.
• Implemented WHATWG Fetch–style redirect behavior (method/body/header adjustments + cross-origin header dropping) and added a TooManyRedirectsError.
• Updated dependency constraints/lockfile and added unit tests for cookies and redirect semantics.

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
uv.lock	Bumps impit locked version to 0.13.2.
pyproject.toml	Raises minimum impit version to >=0.13.2.
src/crawlee/sessions/_cookies.py	Adds get_cookie_string() and extract_cookie_from_header() helpers on SessionCookies.
src/crawlee/http_clients/_impit.py	Adds hop-by-hop redirect handling, cross-origin/body-header logic, and client reuse per proxy.
src/crawlee/errors.py	Introduces TooManyRedirectsError for redirect limit enforcement.
tests/unit/sessions/test_cookies.py	Adds coverage for Set-Cookie parsing and Cookie header construction.
tests/unit/server.py	Adds test endpoints for redirect loops and method/body echoing.
tests/unit/http_clients/test_impit.py	Adds unit tests for session cookie isolation, redirect behavior, and redirect limits.

Suppressed comments (1)

src/crawlee/http_clients/_impit.py:335

• After switching to a counted redirect loop, add an explicit guard before following the next hop so that max_redirects truly limits the number of redirects followed (otherwise an endless chain can still be followed one hop too far).

            next_url = current_url.join(URL(location))
            if next_url.scheme not in _HTTP_SCHEMES:
                return response

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.