Skip to content

Account for CPU used by child processes #378

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sawenzel wants to merge 2 commits into
base: dev
Choose a base branch
from
+67 −20
Open

Account for CPU used by child processes #378

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
068142c
Account for CPU used by child processes
sawenzel Jul 25, 2026
2d32eac
Account for children in the context switches too
sawenzel Aug 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions include/Monitoring/Monitoring.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,11 @@ class Monitoring
/// \param enabledMeasurements vector of monitor measurements, eg. PmMeasurement::Cpu
void enableProcessMonitoring(const unsigned int interval = 5, std::vector<PmMeasurement> enabledMeasurements = {PmMeasurement::Cpu, PmMeasurement::Mem, PmMeasurement::Smaps});

/// Stops process monitoring and transmits the final measurement. Idempotent;
/// call it explicitly where destructor timing is not guaranteed, e.g. on a
/// DPL device's RUNNING->READY transition.
void finalizeProcessMonitoring();

/// Flushes metric buffer (this can also happen when buffer is full)
void flushBuffer();

Expand Down
7 changes: 6 additions & 1 deletion include/Monitoring/ProcessMonitor.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,9 @@ class ProcessMonitor
/// 'getrusage' values from last execution
struct rusage mPreviousGetrUsage;

/// 'getrusage(RUSAGE_CHILDREN)' values from last execution
struct rusage mPreviousGetrUsageChildren;

/// Retired-instructions hardware counter (perf_event_open, Linux only);
/// -1 when unavailable (high perf_event_paranoid, container seccomp, or no PMU).
int mInstructionsFd = -1;
Expand All @@ -128,7 +131,9 @@ class ProcessMonitor
std::vector<Metric> getSmaps();

/// Retrieves CPU usage (%) and number of context switches during the interval
std::vector<Metric> getCpuAndContexts();
/// \param force ignore the 1s minimum interval and report no percentage;
/// for the final measurement, whose delta no later call would pick up
std::vector<Metric> getCpuAndContexts(bool force = false);

std::vector<Metric> makeLastMeasurementAndGetMetrics();
};
Expand Down
10 changes: 9 additions & 1 deletion src/Monitoring.cxx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,13 +130,21 @@ void Monitoring::addBackend(std::unique_ptr<Backend> backend)
mBackends.push_back(std::move(backend));
}

Monitoring::~Monitoring()
void Monitoring::finalizeProcessMonitoring()

@ktf ktf Aug 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small remark. This is not thread safe. You probably want a:

bool expected = true;
if (!mMonitorRunning.compare_exchange_strong(expected, false)) {
  return;
}

It should not matter for DPL (where the .stop will always be called on the main thread) however I do not know if there are other usages of this elsewhere. Given at some point it was advertised that Monitoring is thread safe, I do not know if that's an hard-requirement.

{
if (!mMonitorRunning) {
return;
}
mMonitorRunning = false;
if (mMonitorThread.joinable()) {
mMonitorThread.join();
transmit(mProcessMonitor->makeLastMeasurementAndGetMetrics());
}
}

Monitoring::~Monitoring()
{
finalizeProcessMonitoring();
flushBuffer();
}

Expand Down
65 changes: 47 additions & 18 deletions src/ProcessMonitor.cxx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ ProcessMonitor::ProcessMonitor()
mPid = static_cast<unsigned int>(::getpid());
mTimeLastRun = std::chrono::high_resolution_clock::now();
getrusage(RUSAGE_SELF, &mPreviousGetrUsage);
getrusage(RUSAGE_CHILDREN, &mPreviousGetrUsageChildren);
#ifdef O2_MONITORING_OS_LINUX
setTotalMemory();
#endif
Expand Down Expand Up @@ -99,6 +100,13 @@ void ProcessMonitor::init()
{
mTimeLastRun = std::chrono::high_resolution_clock::now();
getrusage(RUSAGE_SELF, &mPreviousGetrUsage);
getrusage(RUSAGE_CHILDREN, &mPreviousGetrUsageChildren);
// The aggregates cover one monitoring period: monitoring stopped and started
// again reports the new period, not both blended together.
mCpuPerctange.clear();
mCpuMicroSeconds.clear();
mVmSizeMeasurements.clear();
mVmRssMeasurements.clear();
}

void ProcessMonitor::enable(PmMeasurement measurement)
Expand Down Expand Up @@ -167,31 +175,45 @@ std::vector<Metric> ProcessMonitor::getSmaps()
return {{pssTotal, metricsNames[PSS]}, {cleanTotal, metricsNames[PRIVATE_CLEAN]}, {dirtyTotal, metricsNames[PRIVATE_DIRTY]}};
}

std::vector<Metric> ProcessMonitor::getCpuAndContexts()
std::vector<Metric> ProcessMonitor::getCpuAndContexts(bool force)
{
std::vector<Metric> metrics;
// RUSAGE_SELF does not see work done by reaped children (e.g. an external
// event generator forked by o2-sim), so every counter below sums the two.
struct rusage currentUsage;
struct rusage currentUsageChildren;
getrusage(RUSAGE_SELF, &currentUsage);
getrusage(RUSAGE_CHILDREN, &currentUsageChildren);
auto timeNow = std::chrono::high_resolution_clock::now();
double timePassed = std::chrono::duration_cast<std::chrono::microseconds>(timeNow - mTimeLastRun).count();
if (timePassed < 950) {
if (timePassed < 950 && !force) {
MonLogger::Get(Severity::Warn) << "Do not invoke Process Monitor more frequent then every 1s" << MonLogger::End();
metrics.emplace_back("processPerformance");
return metrics;
}

uint64_t cpuUsedInMicroSeconds = currentUsage.ru_utime.tv_sec * 1000000.0 + currentUsage.ru_utime.tv_usec - (mPreviousGetrUsage.ru_utime.tv_sec * 1000000.0 + mPreviousGetrUsage.ru_utime.tv_usec) + currentUsage.ru_stime.tv_sec * 1000000.0 + currentUsage.ru_stime.tv_usec - (mPreviousGetrUsage.ru_stime.tv_sec * 1000000.0 + mPreviousGetrUsage.ru_stime.tv_usec);
double fractionCpuUsed = cpuUsedInMicroSeconds / timePassed;

double cpuUsedPerctange = std::round(fractionCpuUsed * 100.0 * 100.0) / 100.0;
mCpuPerctange.push_back(cpuUsedPerctange);
// CPU time (user + system) of one snapshot, in microseconds
auto cpuMicros = [](const struct rusage& usage) {
return (usage.ru_utime.tv_sec + usage.ru_stime.tv_sec) * 1000000.0 + usage.ru_utime.tv_usec + usage.ru_stime.tv_usec;
};
uint64_t cpuUsedInMicroSeconds = (cpuMicros(currentUsage) - cpuMicros(mPreviousGetrUsage)) +
(cpuMicros(currentUsageChildren) - cpuMicros(mPreviousGetrUsageChildren));
mCpuMicroSeconds.push_back(cpuUsedInMicroSeconds);

metrics.emplace_back(Metric{cpuUsedPerctange, metricsNames[CPU_USED_PERCENTAGE]});
metrics.emplace_back(Metric{
static_cast<uint64_t>(currentUsage.ru_nivcsw - mPreviousGetrUsage.ru_nivcsw), metricsNames[INVOLUNTARY_CONTEXT_SWITCHES]});
metrics.emplace_back(Metric{
static_cast<uint64_t>(currentUsage.ru_nvcsw - mPreviousGetrUsage.ru_nvcsw), metricsNames[VOLUNTARY_CONTEXT_SWITCHES]});
// A child's CPU time appears all at once when it is reaped, so the delta of a
// forced (final) measurement is not a rate over the interval: absolute time only.
if (!force) {
double fractionCpuUsed = cpuUsedInMicroSeconds / timePassed;
double cpuUsedPerctange = std::round(fractionCpuUsed * 100.0 * 100.0) / 100.0;
mCpuPerctange.push_back(cpuUsedPerctange);
metrics.emplace_back(Metric{cpuUsedPerctange, metricsNames[CPU_USED_PERCENTAGE]});
}
uint64_t involuntaryContextSwitches = (currentUsage.ru_nivcsw - mPreviousGetrUsage.ru_nivcsw) +
(currentUsageChildren.ru_nivcsw - mPreviousGetrUsageChildren.ru_nivcsw);
uint64_t voluntaryContextSwitches = (currentUsage.ru_nvcsw - mPreviousGetrUsage.ru_nvcsw) +
(currentUsageChildren.ru_nvcsw - mPreviousGetrUsageChildren.ru_nvcsw);
metrics.emplace_back(Metric{involuntaryContextSwitches, metricsNames[INVOLUNTARY_CONTEXT_SWITCHES]});
metrics.emplace_back(Metric{voluntaryContextSwitches, metricsNames[VOLUNTARY_CONTEXT_SWITCHES]});
metrics.emplace_back(cpuUsedInMicroSeconds, metricsNames[CPU_USED_ABSOLUTE]);

#ifdef O2_MONITORING_OS_LINUX
Expand All @@ -212,6 +234,7 @@ std::vector<Metric> ProcessMonitor::getCpuAndContexts()

mTimeLastRun = timeNow;
mPreviousGetrUsage = currentUsage;
mPreviousGetrUsageChildren = currentUsageChildren;
return metrics;
}

Expand Down Expand Up @@ -262,14 +285,20 @@ std::vector<Metric> ProcessMonitor::makeLastMeasurementAndGetMetrics()
}
#endif
if (mEnabledMeasurements.at(static_cast<short>(PmMeasurement::Cpu))) {
getCpuAndContexts();

auto avgCpuUsage = std::accumulate(mCpuPerctange.begin(), mCpuPerctange.end(), 0.0) /
mCpuPerctange.size();
// forced: no later call would pick up a delta the rate guard discards here
auto lastCpuMetrics = getCpuAndContexts(true);
std::move(lastCpuMetrics.begin(), lastCpuMetrics.end(), std::back_inserter(metrics));

// A process that ends before the first periodic measurement has no
// percentages at all (the forced one contributes none), and averaging an
// empty vector would give NaN.
if (!mCpuPerctange.empty()) {
auto avgCpuUsage = std::accumulate(mCpuPerctange.begin(), mCpuPerctange.end(), 0.0) /
mCpuPerctange.size();
metrics.emplace_back(avgCpuUsage, metricsNames[AVG_CPU_USED_PERCENTAGE]);
}
uint64_t accumulationOfCpuTimeConsumption = std::accumulate(mCpuMicroSeconds.begin(),
mCpuMicroSeconds.end(), 0UL);

metrics.emplace_back(avgCpuUsage, metricsNames[AVG_CPU_USED_PERCENTAGE]);
metrics.emplace_back(accumulationOfCpuTimeConsumption, metricsNames[ACCUMULATED_CPU_TIME]);
}
return metrics;
Expand Down