fix: Respect container memory and CPU limits from cgroups

[Mantisus]

Description

get_memory_info and get_cpu_info reported the host machine even when the crawler ran under a container limit, because /proc/meminfo is not namespaced. The autoscaler sized its budget from host RAM and kept scaling until the container got killed, and a container pinned to two cores read the load of the whole machine as idle. Both now come from the cgroup of the process, with nothing to configure.

Two decisions worth a look:

• The control files are resolved through /proc/self/mountinfo and /proc/self/cgroup instead of assuming /sys/fs/cgroup, which is what makes this work under --cgroupns=host.
• Several cgroups can limit the same process, as they do under Kubernetes. The tightest limit is the ceiling a budget has to fit under, and the utilization is taken from the level closest to running out.

Without a limit, and outside Linux, the host values are used as before. A crawler in a container whose limit is below the host RAM now gets a smaller max_memory_size, so its concurrency drops.

Issues

• Closes: Memory metrics ignore container memory limits (no cgroup awareness) #2095
• Closes: Crawlee increasing concurrency until it dies #1535

Testing

• Add new tests with a fake cgroup filesystem

[Copilot]

Pull request overview

This PR makes Crawlee’s CPU and memory metrics container-aware on Linux by reading cgroup v1/v2 limits/usage (instead of relying solely on host-wide /proc/meminfo / psutil), so the autoscaler sizes budgets and CPU utilization against the resources actually available to the running process.

Changes:

• Add a new cgroup discovery + metrics reader (src/crawlee/_utils/cgroup.py) that resolves controllers via /proc/self/mountinfo + /proc/self/cgroup and reads the tightest applicable limits.
• Update get_memory_info() / get_cpu_info() to prefer cgroup-scoped totals/usage when limits apply, falling back to host metrics otherwise.
• Add comprehensive unit tests with a fake cgroup filesystem, and document autoscaling behavior under resource limits.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
tests/unit/_utils/test_cgroup.py	Adds unit tests covering cgroup v1/v2 discovery and limit/usage semantics, plus system.py integration behavior.
src/crawlee/_utils/system.py	Switches CPU/memory “system-wide” metrics to cgroup-aware readings when applicable; keeps host fallbacks.
src/crawlee/_utils/cgroup.py	Implements cgroup controller discovery and reads memory/cpu limits + usage from the most relevant hierarchy levels.
docs/guides/scaling_crawlers.mdx	Documents how autoscaling respects cgroup CPU/memory limits when running under container/systemd constraints.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[vdusek]

If I am correct, all the tests run against a fake cgroup filesystem. Is there a way to cover this end-to-end? e.g., running a crawler in a container with --memory and/or --cpus set, and asserting get_memory_info() and get_cpu_info()? Or have you tested it manually? Also, we should make sure we cover both --cgroupns=private and --cgroupns=host.

[Mantisus]

If I am correct, all the tests run against a fake cgroup filesystem. Is there a way to cover this end-to-end?

That's right. To add the test, we'll probably need to add a new job to CI. With my permissions in the repository, I won't be able to test it properly.

Or have you tested it manually?

Yes, I tested it locally with systemd and docker

(crawlee) mantisus@Kleimor:~/repos/apify/crawlee-python$ docker run --rm $MOUNTS $IMG $RUN
configuration
  available_memory_ratio : 0.25
  max_used_memory_ratio  : 0.9
  max_used_cpu_ratio     : 0.95
  memory_mbytes          : None
  system_info_interval   : 0:00:01
limits read from the cgroup
  memory      : None
  cpu quota   : None
  cpu set     : 20
latest memory snapshot
  current_size            : 60.27 MB
  max_memory_size         : 4.88 GB
  system_wide_used_size   : 2.63 GB
  system_wide_memory_size : 19.53 GB
  is_overloaded           : False
latest cpu snapshot
  used_ratio              : 0.216
  max_used_ratio          : 0.95
  is_overloaded           : False
system status over the last 6 cpu samples
  cpu        : 0.00 of 0.40 allowed   ok
  memory     : 0.00 of 0.20 allowed   ok
  event_loop : 0.00 of 0.60 allowed   ok
  client     : 0.00 of 0.30 allowed   ok
  is_system_idle: True

(crawlee) mantisus@Kleimor:~/repos/apify/crawlee-python$ docker run --rm -m 512m --cpus 1 $MOUNTS $IMG $RUN
configuration
  available_memory_ratio : 0.25
  max_used_memory_ratio  : 0.9
  max_used_cpu_ratio     : 0.95
  memory_mbytes          : None
  system_info_interval   : 0:00:01
limits read from the cgroup
  memory      : MemoryLimit(limit=536870912, working_set=84348928)
  cpu quota   : 1.0
  cpu set     : 20
latest memory snapshot
  current_size            : 60.19 MB
  max_memory_size         : 128.00 MB
  system_wide_used_size   : 80.98 MB
  system_wide_memory_size : 512.00 MB
  is_overloaded           : False
latest cpu snapshot
  used_ratio              : 0.996
  max_used_ratio          : 0.95
  is_overloaded           : True
system status over the last 6 cpu samples
  cpu        : 0.74 of 0.40 allowed   OVERLOADED
  memory     : 0.00 of 0.20 allowed   ok
  event_loop : 0.00 of 0.60 allowed   ok
  client     : 0.00 of 0.30 allowed   ok
  is_system_idle: False

(crawlee) mantisus@Kleimor:~/repos/apify/crawlee-python$ docker run --rm -m 512m --cpus 6 $MOUNTS $IMG $RUN
configuration
  available_memory_ratio : 0.25
  max_used_memory_ratio  : 0.9
  max_used_cpu_ratio     : 0.95
  memory_mbytes          : None
  system_info_interval   : 0:00:01
limits read from the cgroup
  memory      : MemoryLimit(limit=536870912, working_set=85512192)
  cpu quota   : 6.0
  cpu set     : 20
latest memory snapshot
  current_size            : 60.32 MB
  max_memory_size         : 128.00 MB
  system_wide_used_size   : 82.48 MB
  system_wide_memory_size : 512.00 MB
  is_overloaded           : False
latest cpu snapshot
  used_ratio              : 0.723
  max_used_ratio          : 0.95
  is_overloaded           : False
system status over the last 6 cpu samples
  cpu        : 0.00 of 0.40 allowed   ok
  memory     : 0.00 of 0.20 allowed   ok
  event_loop : 0.00 of 0.60 allowed   ok
  client     : 0.00 of 0.30 allowed   ok
  is_system_idle: True

(crawlee) mantisus@Kleimor:~/repos/apify/crawlee-python$ docker run --rm -m 512m --cgroupns=host $MOUNTS $IMG $RUN 
configuration
  available_memory_ratio : 0.25
  max_used_memory_ratio  : 0.9
  max_used_cpu_ratio     : 0.95
  memory_mbytes          : None
  system_info_interval   : 0:00:01
limits read from the cgroup
  memory      : MemoryLimit(limit=536870912, working_set=89137152)
  cpu quota   : None
  cpu set     : 20
latest memory snapshot
  current_size            : 60.34 MB
  max_memory_size         : 128.00 MB
  system_wide_used_size   : 85.98 MB
  system_wide_memory_size : 512.00 MB
  is_overloaded           : False
latest cpu snapshot
  used_ratio              : 0.200
  max_used_ratio          : 0.95
  is_overloaded           : False
system status over the last 6 cpu samples
  cpu        : 0.00 of 0.40 allowed   ok
  memory     : 0.00 of 0.20 allowed   ok
  event_loop : 0.00 of 0.60 allowed   ok
  client     : 0.00 of 0.30 allowed   ok
  is_system_idle: True