docs: add guide on building custom extensions by shixi-li · Pull Request #2106 · apify/crawlee-python

shixi-li · 2026-08-01T09:03:07Z

Description

Refs #1936. Adds docs/guides/extending_crawlee.mdx, a page that maps Crawlee's extension points.

The four extension points you listed in the issue each already document their own contract in the guide that owns them: crawlers in the HTTP crawlers guide, HTTP clients and storage clients in theirs, and browser plugins in the Playwright crawler guide (#2089). What is still missing is the map, so someone who wants to extend Crawlee has to already know which guide to open, and a third-party project has no single page to point its users at for "here is the interface this integration implements".

This page is that map. For each extension point it states when to subclass rather than configure, names the base class and what its contract covers, and links to the guide that goes deep. It deliberately does not restate those guides.

Extension points — a class diagram plus one section each for crawlers (AbstractHttpCrawler), HTTP clients (HttpClient), storage clients (StorageClient), and browser plugins (PlaywrightBrowserPlugin), each linking to its detailed guide.
Choosing an extension point — matches the layer to what actually differs in an integration, and says to prefer configuration over a subclass where the built-in class already exposes the knob.
Framing for third-party integrations, per the issue's rationale: they host their own guide and reference the contract here.

Notes

Placed as its own guide rather than a section inside an existing one, since it spans all four extension points. The sidebar picks it up automatically (autogenerated over docs/guides).
No new code examples: each linked guide already carries a runnable one for its own extension point, so duplicating them here would mean two copies to keep in sync.
Happy to adjust the placement, depth, or wording, especially if you would rather this live under a different section or carry a worked example of its own.

Testing

uv run poe build-docs — build succeeds, the page renders at /docs/next/guides/extending-crawlee, and the build reports no broken links or anchors for it (the broken anchors in the log are pre-existing ones on API pages).
Verified every link target resolves before building: the five doc ids (http-crawlers, http-clients, storage-clients, playwright-crawler, architecture-overview), the playwright-crawler-with-camoufox example, and each ApiLink class (AbstractHttpCrawler, BasicCrawler, HttpClient, StorageClient, PlaywrightBrowserPlugin, PlaywrightBrowserController, BrowserPool) against src/crawlee/.

AI assistance

Written with AI assistance (Claude). The scoping decision was mine to check first what each extension point already documents, which is why this is a map rather than four new sections; the browser-plugin quarter was covered by my earlier #2089.

Crawlee has four extension points, and each already documents its own contract in the guide that owns it: crawlers in the HTTP crawlers guide, HTTP clients and storage clients in theirs, and browser plugins in the Playwright crawler guide. What was missing is the map: a page that names the extension points, says what each contract covers, and points at the guide that goes deep. That page is also what a third-party integration can link to, which is the case apify#1936 was opened for: the integration hosts its own guide and references a stable statement of the interface it implements. Refs apify#1936

Copilot

Pull request overview

Adds a new documentation guide that serves as a “map” of Crawlee’s primary extension points (crawlers, HTTP clients, storage clients, and browser plugins), briefly describing when to subclass vs configure and linking to the deeper, existing guides for each contract.

Changes:

Add docs/guides/extending_crawlee.mdx with an overview of extension points and links to the relevant detailed guides/examples.
Include a high-level Mermaid class diagram summarizing the main extensibility surfaces.
Add a short “Choosing an extension point” decision checklist to help integrators pick the right layer.

Suppressed comments (1)

docs/guides/extending_crawlee.mdx:73

This section describes PlaywrightBrowserPlugin as the base for browser plugins, but the abstract contract is BrowserPlugin (with PlaywrightBrowserPlugin being the built-in Playwright implementation). Updating the wording avoids confusion about which class defines the core interface vs which class is the default implementation to configure/subclass.

Subclass a browser plugin when an integration launches browsers through an API other than the standard Playwright one. Configuration options on <ApiLink to="class/PlaywrightBrowserPlugin">`PlaywrightBrowserPlugin`</ApiLink> cover the cases where the standard launch API is enough, so reach for a subclass only when the launch path itself differs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

shixi-li · 2026-08-05T11:25:23Z

+BasicCrawler --> HttpClient : uses
+BasicCrawler --> StorageClient : uses
+BasicCrawler --> BrowserPlugin : uses


Good catch — fixed in 83a258b. BasicCrawler itself has no browser knowledge; the diagram now shows PlaywrightCrawler (a BasicCrawler subclass) using BrowserPool, which initializes and manages the BrowserPlugin instances, matching the description in the Browser plugins section below.

…BrowserPool

Mantisus

Thanks for the PR and the contribution. The direction is right, but the page needs further refinement.

In its current state the guide doesn't match the documentation style of the project, and that's what most of my comments are about.

A few things that run across the page rather than sitting in one place:

Sections open with "subclass X when...", so they assume the reader already knows what the component is. Our guides describe the component first, then get to the details.
Class names are given in prose instead of ApiLink, which we use everywhere else.
The contracts aren't named anywhere, even though the page description promises them.
Style details: we use contractions, we don't glue independent clauses with a semicolon, and we avoid both repeating the same opening across parallel list items and using bold to carry meaning.

The rest is inline. I think this needs another pass over the structure rather than line edits, happy to take another look once it's reworked.

Mantisus · 2026-08-05T20:01:21Z

+
+Crawlee is built around a small number of abstract base classes that you can subclass to plug in your own behavior. This guide is the map: it lists every extension point, states the contract each one defines, and links to the guide that covers it in depth.
+
+If you maintain a third-party integration, such as an alternative browser backend or a storage adapter, you can build it against these contracts and host the integration guide in your own project. This page gives your users a stable reference for the interface your integration implements.


Hard to understand: "you" is the integration author, then "your users" are someone else, and there are three possessives in a row. I'd also drop "stable", that's a promise of our versioning policy, not of this page.

Mantisus · 2026-08-05T20:07:02Z

+
+## Extension points
+
+Crawlee currently has four extension points.


This reads as exhaustive, and there are other ABCs users do subclass, e.g. RequestLoader, FingerprintGenerator, RenderingTypePredictor. These four are the main ones but not the only ones.

Mantisus · 2026-08-05T20:22:05Z

+
+import ApiLink from '@site/src/components/ApiLink';
+
+Crawlee is built around a small number of abstract base classes that you can subclass to plug in your own behavior. This guide is the map: it lists every extension point, states the contract each one defines, and links to the guide that covers it in depth.


The intro starts with the mechanics, but it never says why the reader is here in the first place. I think it would help to lead with the motivation.

Mantisus · 2026-08-05T20:33:37Z

+config:
+    class:
+        hideEmptyMembersBox: true
+---
+
+classDiagram
+
+class BasicCrawler {
+    <<abstract>>
+}
+
+class PlaywrightCrawler
+
+class HttpClient {
+    <<abstract>>
+}
+
+class StorageClient {
+    <<abstract>>
+}
+
+class BrowserPool
+
+class BrowserPlugin {
+    <<abstract>>
+}
+
+BasicCrawler --> HttpClient : uses
+BasicCrawler --> StorageClient : uses
+BasicCrawler --|> PlaywrightCrawler
+PlaywrightCrawler --> BrowserPool : uses
+BrowserPool --> BrowserPlugin : manages
+```


Three of the four sections describe a relation the diagram doesn't have: AbstractHttpCrawler, the per-storage clients StorageClient opens, and the controller new_browser() returns.

The diagram shows relations between components, but nothing about what is extensible, so it doesn't add any information for the reader.

Mantisus · 2026-08-05T20:47:36Z

+Subclass a crawler when you need a parsing strategy or a request-handler context that the built-in crawlers do not provide.
+
+For HTTP-based crawling, <ApiLink to="class/AbstractHttpCrawler">`AbstractHttpCrawler`</ApiLink> is the base class. A custom crawler supplies a parser that turns an HTTP response into your parsed type, a context type that exposes that parsed data to request handlers, and the crawler class that ties the two together. Everything else, including retries, concurrency, session management, and storage, is inherited from <ApiLink to="class/BasicCrawler">`BasicCrawler`</ApiLink>.
+
+See [HTTP crawlers guide](./http-crawlers) for a worked example built on `selectolax`, and [Architecture overview](./architecture-overview) for how crawlers relate to the other components.


I'd expect the section to start with what the component is and what it does, then move on to what you subclass. It opens with "subclass a crawler when...", so it assumes the reader already knows.

The section is focused on the HTTP crawler and parsers, and BasicCrawler is mentioned only in passing, though it's the key class for building crawlers.

Mantisus · 2026-08-05T20:55:17Z

+### Storage clients
+
+Subclass a storage client when you want Crawlee's storages to be backed by a system that is not covered by the built-in memory, file system, SQL, and Redis clients.
+
+<ApiLink to="class/StorageClient">`StorageClient`</ApiLink> is the base class. It is a factory: it opens the per-storage clients for datasets, key-value stores, and request queues, and those clients implement the actual create, read, update, and delete operations.
+
+See [Storage clients guide](./storage-clients) for the interface, a custom client example, and how clients are registered and resolved.


Same as with the crawlers, I'd expect the section to start with what the component is and what it does, then move on to what you subclass.

he built-in clients are enumerated in prose, they should be ApiLinks to the corresponding classes. Same on L73: DatasetClient, KeyValueStoreClient and RequestQueueClient have neither links nor descriptions, though that's where the actual work is. StorageClient itself is only three methods that return them.

shixi-li · 2026-08-06T03:46:04Z

Thanks for the detailed review — that was exactly the kind of feedback the page needed. Reworked the structure rather than line-editing, as you suggested. Every section now opens with what the component is and what it does, then names its contract, then gets to what you subclass.

Point by point:

Intro (L9/L11). Now leads with the motivation — the cases where Crawlee doesn't do what you need and forking is the wrong answer — before any mechanics. Dropped the possessive pile-up and the "stable" promise; the third-party paragraph is one sentence and no longer mixes "you" and "your users".

"Four extension points" (L15). Reworded to say these are the main ones and explicitly names RequestLoader, FingerprintGenerator, and RenderingTypePredictor as also extensible.

The diagram (L51). Redrawn to show what the sections actually describe: AbstractHttpCrawler and PlaywrightCrawler under BasicCrawler, the three per-storage clients StorageClient opens, and the BrowserController that new_browser() returns. The extensible classes carry <<abstract>>, and the contract methods that matter (create_dataset_client / create_kvs_client / create_rq_client, new_browser) are on the boxes, so the diagram now says something about extensibility rather than just relations.

Crawlers (L59). Starts from BasicCrawler and what it owns (queue, fetch, context, retries, concurrency, sessions, storage) and why that makes it the base. The parser contract is named method by method: parse, parse_text, select, is_matching_selector, find_links.

Storage clients (L75). Starts with what a storage client is and why swapping it is useful. StorageClient is described as the three factory methods it is, and DatasetClient, KeyValueStoreClient, RequestQueueClient now have ApiLinks plus a line each on what they actually do, since that's where the work is. Built-in clients are ApiLinks throughout.

Contracts. Each section now names its contract explicitly — that was promised in the page description and missing before.

Style. Class names go through ApiLink everywhere. Contractions used. No semicolons joining independent clauses. No bold carrying meaning. The "Choosing an extension point" list items no longer share an opening — each starts from the thing that differs in your integration.

Verified locally: all 23 ApiLink targets resolve to real classes in src/crawlee, the Mermaid diagram renders, and every internal doc link points at an existing page.

Mantisus

Thanks for the rework, this is a big step up.

One thing that runs across the page. We don't use a spaced hyphen to join clauses. We split them into separate sentences instead.

Mantisus · 2026-08-06T09:39:44Z

+---
+config:
+    class:
+        hideEmptyMembersBox: true
+---
+
+classDiagram
+
+class BasicCrawler {
+    <<abstract>>
+}
+
+class AbstractHttpCrawler {
+    <<abstract>>
+}
+
+class PlaywrightCrawler
+
+class HttpClient {
+    <<abstract>>
+}
+
+class StorageClient {
+    <<abstract>>
+    create_dataset_client()
+    create_kvs_client()
+    create_rq_client()
+}
+
+class DatasetClient {
+    <<abstract>>
+}
+
+class KeyValueStoreClient {
+    <<abstract>>
+}
+
+class RequestQueueClient {
+    <<abstract>>
+}
+
+class BrowserPool
+
+class BrowserPlugin {
+    <<abstract>>
+    new_browser()
+}
+
+class BrowserController {
+    <<abstract>>
+}
+
+BasicCrawler --|> AbstractHttpCrawler
+BasicCrawler --|> PlaywrightCrawler
+AbstractHttpCrawler --> HttpClient : uses
+BasicCrawler --> StorageClient : uses
+StorageClient --> DatasetClient : opens
+StorageClient --> KeyValueStoreClient : opens
+StorageClient --> RequestQueueClient : opens
+PlaywrightCrawler --> BrowserPool : uses
+BrowserPool --> BrowserPlugin : manages
+BrowserPlugin --> BrowserController : returns
+```


The diagram is inconsistent. You are specifying methods only for StorageClient and BrowserPlugin.
Also, HttpClient is actually implemented in BasicCrawler, not in AbstractHttpCrawler.
The text mentions AbstractHttpParser, but it is not shown in the diagram.

Mantisus · 2026-08-06T09:42:33Z

+
+## Conclusion
+
+Each extension point is a small abstract class with a documented contract, and everything above it keeps working once you implement one. If you're building a third-party integration, these contracts are also the stable surface to write your own documentation against.


"stable surface" on its own is just an assertion. If we say it, it should say what backs it: the contracts are these abstract classes, and they only change with a major release.

Mantisus · 2026-08-06T09:43:01Z

+
+Each extension point is a small abstract class with a documented contract, and everything above it keeps working once you implement one. If you're building a third-party integration, these contracts are also the stable surface to write your own documentation against.
+
+If anything here is unclear or you hit a case the contracts don't cover, open an issue on [GitHub](https://github.com/apify/crawlee-python) or join our [Discord](https://discord.com/invite/jyEM2PRvMU).


Suggested change

If anything here is unclear or you hit a case the contracts don't cover, open an issue on [GitHub](https://github.com/apify/crawlee-python) or join our [Discord](https://discord.com/invite/jyEM2PRvMU).

If you have questions or need assistance, feel free to reach out on our [GitHub](https://github.com/apify/crawlee-python) or join our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping!

Mantisus · 2026-08-06T09:44:24Z

+
+## Extension points
+
+The four extension points below are the main ones, and they're where most integrations plug in. They aren't the only abstract classes you can subclass - <ApiLink to="class/RequestLoader">`RequestLoader`</ApiLink>, <ApiLink to="class/FingerprintGenerator">`FingerprintGenerator`</ApiLink>, and <ApiLink to="class/RenderingTypePredictor">`RenderingTypePredictor`</ApiLink> are extensible too.


The first half says these aren't the only ones, then the list itself reads closed: three items joined with "and", nothing marking them as examples. AbstractHttpParser and BrowserController are subclassable too and aren't there, so it can't be read as complete. I'd expect something that presents them as examples.

Mantisus · 2026-08-06T09:49:26Z

+A crawler drives the whole run: it takes requests from the queue, fetches each one, builds the context object your handler receives, and manages retries, concurrency, sessions, and storage along the way. <ApiLink to="class/BasicCrawler">`BasicCrawler`</ApiLink> implements all of that and stays agnostic about how a page is fetched or parsed, which is what makes it the base every other crawler builds on.
+
+For HTTP-based crawling, <ApiLink to="class/AbstractHttpCrawler">`AbstractHttpCrawler`</ApiLink> adds the fetch-and-parse layer on top. Extending it means supplying a parser that implements <ApiLink to="class/AbstractHttpParser">`AbstractHttpParser`</ApiLink>: `parse` turns an <ApiLink to="class/HttpResponse">`HttpResponse`</ApiLink> into your parsed type, `parse_text` does the same for a string, `select` and `is_matching_selector` apply selectors to it, and `find_links` extracts the URLs used for link enqueuing. You then pair that parser with a context type that exposes the parsed data to handlers, and a crawler class that ties the two together.
+
+See the [HTTP crawlers guide](./http-crawlers) for a worked example built on `selectolax`, and the [Architecture overview](./architecture-overview) for how crawlers relate to the other components.


Browser crawlers still aren't mentioned. The section goes from BasicCrawler straight to AbstractHttpCrawler, and the diagram shows PlaywrightCrawler under BasicCrawler with nothing in the text about it. Extending a browser crawler is a real case, we do it ourselves with StagehandCrawler.

Mantisus · 2026-08-06T09:58:18Z

+
+A browser plugin is what launches browsers for <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>. The crawler never launches one itself: it goes through <ApiLink to="class/BrowserPool">`BrowserPool`</ApiLink>, which initializes the plugins it's given, forwards browser context options when creating pages, and manages each browser's lifecycle.
+
+The contract is <ApiLink to="class/BrowserPlugin">`BrowserPlugin`</ApiLink>. Its `new_browser` launches a browser and returns a <ApiLink to="class/BrowserController">`BrowserController`</ApiLink>, which is what the pool then drives to open pages and tear things down. Reach for a subclass when the launch path itself differs from the standard Playwright one, since <ApiLink to="class/PlaywrightBrowserPlugin">`PlaywrightBrowserPlugin`</ApiLink>'s configuration options already cover the cases where it doesn't.


I'd keep both levels here. BrowserPlugin is the contract when the browser's launch and lifecycle are too specific for PlaywrightBrowserPlugin to cover, and PlaywrightBrowserPlugin is what most integrations should extend. Right now the page names only the base, so the common case is missing and it isn't clear why the two exist.

Mantisus · 2026-08-06T10:00:27Z

+- Data needs to live somewhere Crawlee doesn't support yet - implement <ApiLink to="class/StorageClient">`StorageClient`</ApiLink> and its three per-storage clients.
+- Browsers need to be launched through a different API - implement <ApiLink to="class/BrowserPlugin">`BrowserPlugin`</ApiLink> and hand it to <ApiLink to="class/BrowserPool">`BrowserPool`</ApiLink>.
+
+When more than one fits, pick the narrowest. A custom HTTP client works with every HTTP crawler, so it's easier to maintain than a crawler subclass that hard-codes the same transport.


"pick the narrowest" is good, but the narrowest choice is often no subclass at all, and the page no longer says that. Configuration covers a lot of these cases: PlaywrightBrowserPlugin's options, passing an http_client to any crawler, or parsing with a third-party library inside an HttpCrawler handler.

Mantisus · 2026-08-06T10:02:32Z

+- The response format is one no built-in crawler parses - subclass <ApiLink to="class/AbstractHttpCrawler">`AbstractHttpCrawler`</ApiLink> with your own parser.
+- The transport differs, but parsing doesn't - implement <ApiLink to="class/HttpClient">`HttpClient`</ApiLink> and pass it to any HTTP crawler.
+- Data needs to live somewhere Crawlee doesn't support yet - implement <ApiLink to="class/StorageClient">`StorageClient`</ApiLink> and its three per-storage clients.
+- Browsers need to be launched through a different API - implement <ApiLink to="class/BrowserPlugin">`BrowserPlugin`</ApiLink> and hand it to <ApiLink to="class/BrowserPool">`BrowserPool`</ApiLink>.


Two things beyond the style.
The first bullet answers only for HTTP, so someone who wants a custom browser crawler has no line here.
The conditions aren't parallel either: two describe a state, two describe a requirement.

vdusek requested a review from Mantisus August 4, 2026 09:28

Mantisus requested a lite review from Copilot August 4, 2026 09:35

Copilot started reviewing on behalf of Mantisus August 4, 2026 09:35 View session

Copilot AI reviewed Aug 4, 2026

View reviewed changes

docs: route the browser edge in the extension-points diagram through …

83a258b

…BrowserPool

Mantisus suggested changes Aug 5, 2026

View reviewed changes

docs: rework the extension-points guide around component contracts

9150dc8

Mantisus suggested changes Aug 6, 2026

View reviewed changes


		Crawlee is built around a small number of abstract base classes that you can subclass to plug in your own behavior. This guide is the map: it lists every extension point, states the contract each one defines, and links to the guide that covers it in depth.

		If you maintain a third-party integration, such as an alternative browser backend or a storage adapter, you can build it against these contracts and host the integration guide in your own project. This page gives your users a stable reference for the interface your integration implements.


		## Extension points

		Crawlee currently has four extension points.


		import ApiLink from '@site/src/components/ApiLink';

		Crawlee is built around a small number of abstract base classes that you can subclass to plug in your own behavior. This guide is the map: it lists every extension point, states the contract each one defines, and links to the guide that covers it in depth.


		## Conclusion

		Each extension point is a small abstract class with a documented contract, and everything above it keeps working once you implement one. If you're building a third-party integration, these contracts are also the stable surface to write your own documentation against.


		Each extension point is a small abstract class with a documented contract, and everything above it keeps working once you implement one. If you're building a third-party integration, these contracts are also the stable surface to write your own documentation against.

		If anything here is unclear or you hit a case the contracts don't cover, open an issue on [GitHub](https://github.com/apify/crawlee-python) or join our [Discord](https://discord.com/invite/jyEM2PRvMU).

	If anything here is unclear or you hit a case the contracts don't cover, open an issue on [GitHub](https://github.com/apify/crawlee-python) or join our [Discord](https://discord.com/invite/jyEM2PRvMU).
	If you have questions or need assistance, feel free to reach out on our [GitHub](https://github.com/apify/crawlee-python) or join our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping!


		## Extension points

		The four extension points below are the main ones, and they're where most integrations plug in. They aren't the only abstract classes you can subclass - <ApiLink to="class/RequestLoader">`RequestLoader`</ApiLink>, <ApiLink to="class/FingerprintGenerator">`FingerprintGenerator`</ApiLink>, and <ApiLink to="class/RenderingTypePredictor">`RenderingTypePredictor`</ApiLink> are extensible too.


		A browser plugin is what launches browsers for <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>. The crawler never launches one itself: it goes through <ApiLink to="class/BrowserPool">`BrowserPool`</ApiLink>, which initializes the plugins it's given, forwards browser context options when creating pages, and manages each browser's lifecycle.

		The contract is <ApiLink to="class/BrowserPlugin">`BrowserPlugin`</ApiLink>. Its `new_browser` launches a browser and returns a <ApiLink to="class/BrowserController">`BrowserController`</ApiLink>, which is what the pool then drives to open pages and tear things down. Reach for a subclass when the launch path itself differs from the standard Playwright one, since <ApiLink to="class/PlaywrightBrowserPlugin">`PlaywrightBrowserPlugin`</ApiLink>'s configuration options already cover the cases where it doesn't.

Uh oh!

Conversation

shixi-li commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Contents

Notes

Testing

AI assistance

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mantisus left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shixi-li commented Aug 6, 2026

Uh oh!

Mantisus left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shixi-li commented Aug 1, 2026 •

edited

Loading