[Bug] Prometheus metrics format bug

[cui2022]

Bug Type (问题类型)

None

Before submit

• 我已经确认现有的 Issues 与 FAQ 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)

Environment (环境信息)

3 pd
3 server
5 store

Expected & Actual behavior (期望与实际表现)

I pulled the latest version of the HugeGraph source code, compiled and deployed it, and then deployed a Prometheus + Grafana monitoring environment.

During the monitoring test, I found that the metrics exposed by hugegraph-server (/metrics) could not be scraped correctly by Prometheus. After checking the logs, I found that some metric names contained the illegal character ~, which does not comply with the Prometheus metric naming specification. As a result, Prometheus failed to parse the metrics data, causing the server monitoring metrics collection to fail.

I located the issue in the metric name processing logic and modified the related code to correctly replace the illegal characters in metric names. After recompiling and redeploying HugeGraph Server, the metrics format was fixed, and Prometheus was able to successfully scrape the server monitoring metrics.

public static String replaceSlashInKey(String orgKey) {
    return orgKey.replace("/", "_");
}
->
public static String replaceSlashInKey(String orgKey) {
    return orgKey.replace("/", "_").replace("~", "_");
}

I also modified the following code section. However, I am not sure whether this change has any effect. What I can confirm is that the code above definitely needs to be modified, because I first modified the code below, recompiled and redeployed the project, but the issue still occurred. After modifying the code above, recompiling and redeploying HugeGraph Server successfully resolved the problem.

public static String replaceDotDashInKey(String orgKey) {
return orgKey.replace(".", "").replace("-", "").replace("/", "").replace("$", "");
}
->
public static String replaceDotDashInKey(String orgKey) {
return orgKey.replace(".", "").replace("-", "").replace("/", "").replace("$", "").replace("~", "_");
}

Vertex/Edge example (问题点 / 边数据举例)

Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

[cui2022]

pd and store monitor always normal, only server was affected. At the same time, I noticed that there were approximately 20 problematic metrics, and all of them were related to the Cache module.

[dosubot]

Hi @cui2022! I'm Dosu and I'm helping the HugeGraph team.

Your analysis looks correct. The replaceSlashInKey and replaceDotDashInKey methods in MetricsUtil.java currently do not handle the ~ (tilde) character, which is indeed not a valid character in Prometheus metric names. This is the same class of bug that was previously fixed in PR #2462, which added sanitization for / and $ characters after they were found to break Prometheus scraping.

Regarding your two changes:

• replaceSlashInKey — This is the one you confirmed fixes the issue. It's used directly in the Prometheus format writing logic in MetricsUtil.writePrometheusFormat() [1], so adding ~ replacement there would catch tilde characters that slip through.
• replaceDotDashInKey — This method is also called during Prometheus format conversion for gauges, histograms, meters, and timers [2]. Adding ~ replacement here as well is a good defensive measure for consistency, even if replaceSlashInKey already catches the cases you encountered.

It would be great if you could open a PR with this fix — it would be a straightforward and well-scoped contribution. You might also consider making the sanitization more robust by using a regex to replace any character that doesn't match the Prometheus metric name spec ([a-zA-Z_:][a-zA-Z0-9_:]*), which would prevent similar issues with other unexpected characters in the future.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[cui2022]

I am not very familiar with the proper process for contributing to the project, such as how to evaluate whether a code change is appropriate and how to submit my modifications.

Based on my current progress, after completing the testing phase, I plan to continue running HugeGraph in a production-level environment. During this process, I will keep documenting any issues I encounter, along with their corresponding solutions and potential fixes.

I would be very glad if my findings, bug fixes, or improvements could contribute back to the HugeGraph community. I would also appreciate any guidance on the recommended contribution workflow and review process.

[imbajin]

Hi @cui2022, thank you for the detailed investigation and for validating the behavior in a real deployment. This is a valuable production finding, and we would be very happy to welcome your PR.

After checking the current master source, the problem is small and well scoped, but we recommend fixing it through one shared Prometheus-name boundary instead of adding another character-specific .replace() call.

What happens in the current code

There are currently two partial conversion helpers:

• replaceDotDashInKey() is used by GET /metrics for gauges, histograms, meters, and timers. The Cache metrics registered by GraphManager.registerCacheMetrics() reach this path.
• replaceSlashInKey() is used by GET /metrics/statistics for REST-path statistics.

The reported Cache metrics therefore directly depend on replaceDotDashInKey() in current master. Changing only replaceSlashInKey() cannot change the Cache gauge names emitted by /metrics. The difference observed during deployment may be related to the scraped endpoint or to which rebuilt Server artifact was deployed, so please verify the URL and JAR/image once more.

The deeper issue is that the two helpers each know about only a few characters. Every newly encountered character would require another patch:

MetricRegistry / ServerReporter keys       REST statistics paths
                 \                              /
                  \                            /
                   v                          v
             partial replacement helpers (current)
                   |   '.', '-', '/', '$', ...
                   |   '~' is missed
                   v
           invalid unquoted metric name
                   v
          Prometheus rejects the scrape

Recommended small and unified fix

Replace both partial helpers with one method, for example sanitizePrometheusMetricName(), and use it at every point where HugeGraph writes a Prometheus metric name:

private static final Pattern INVALID_PROMETHEUS_NAME_CHAR =
        Pattern.compile("[^a-zA-Z0-9_:]");

public static String sanitizePrometheusMetricName(String originalName) {
    String sanitized = INVALID_PROMETHEUS_NAME_CHAR.matcher(originalName)
                                                   .replaceAll("_");
    if (sanitized.isEmpty()) {
        return "_";
    }
    if (Character.isDigit(sanitized.charAt(0))) {
        return "_" + sanitized;
    }
    return sanitized;
}

The intent is:

1. Keep letters, digits, _, and :.
2. Replace every other character, including ., -, /, $, and ~, with _.
3. If the result starts with a digit, prefix _, because the first character has a stricter rule.
4. Compile the regex once as a static Pattern, rather than recompiling it for every metric.
5. Apply the conversion only while producing Prometheus output. Do not rename the internal Cache or registry keys.

Examples:

Original name	Prometheus name
org.apache.Cache.schema~DEFAULT-hugegraph.hits	org_apache_Cache_schema_DEFAULT_hugegraph_hits
/graphs/1~x	_graphs_1_x
9cache.hits	_9cache_hits
valid_name:total	valid_name:total

Then replace all calls to both old helpers with the unified method:

MetricRegistry / ServerReporter keys       REST statistics paths
                 \                              /
                  +---- sanitizePrometheusMetricName() ----+
                                      |
                         one sanitized local name
                                      |
                         +------------+------------+
                         v            v            v
                       # HELP       # TYPE       sample

In the current code this covers:

• all four metric-family loops in MetricsUtil.writePrometheusFormat();
• gauges, histograms, meters, and timers in MetricsAPI.baseMetricPrometheusAll();
• metric names in MetricsAPI.statisticsProm().

This remains a small change: one shared helper, replacement of the existing call sites, deletion of the two old helpers, and regression tests. It avoids both duplicated logic and a future sequence of one-character patches. It also deliberately does not expand this PR into unrelated histogram, label, or exposition-format refactoring.

Regression tests

Please add a focused MetricsUtilTest under the existing unit-test package and register it in UnitTestSuite, because this repository runs unit tests through that suite.

The test should cover at least:

• ., -, /, $, and ~ are converted to _;
• a valid name remains unchanged;
• a name beginning with a digit is prefixed with _;
• a Cache-like name containing ~ is rendered without ~;
• the same sanitized name is used in # HELP, # TYPE, and sample lines.

For the rendering test, create a fresh MetricRegistry, register a gauge with a Cache-like name containing ~, call MetricsUtil.writePrometheusFormat(), and assert against the resulting text. This isolates the regression without depending on a running cluster or global registry state.

Run the focused test and basic checks with:

mvn test -pl hugegraph-server/hugegraph-test -am \
  -P unit-test -Dtest=MetricsUtilTest
mvn editorconfig:format
mvn clean compile -Dmaven.javadoc.skip=true

Please also include your successful Prometheus/Grafana scrape result in the PR description as end-to-end evidence.

Contribution workflow

The complete guide is in CONTRIBUTING.md. The shortest workflow is:

# 1. Fork apache/hugegraph on GitHub and clone your fork
git clone https://github.com/<your-github-name>/hugegraph.git
cd hugegraph

# 2. Add the official repository and branch from the latest master
git remote add upstream https://github.com/apache/hugegraph.git
git fetch upstream
git switch -c fix/server-prometheus-metric-name upstream/master

# 3. Implement the helper, update all call sites, and add the tests
git add <changed-files>
git commit -m "fix(server): sanitize Prometheus metric names"
git push -u origin fix/server-prometheus-metric-name

Open a PR from your fork to apache/hugegraph:master with:

• title: fix(server): sanitize Prometheus metric names;
• Purpose of the PR: close #3142;
• Main Changes: explain the shared output-boundary sanitizer and affected endpoints;
• Verifying these changes: list the exact test commands and the successful scrape result;
• Doc - No Need, unless you also change user-facing documentation.

GitHub will run CI and contributor/CLA checks. If CI fails or reviewers request changes, update the same branch and push again; the PR updates automatically. Multiple review rounds are normal, and maintainers will help you refine the change.

Your plan to continue documenting production issues, reproduction steps, and verified fixes is exactly the kind of contribution that benefits HugeGraph. Issues, tests, documentation, and code fixes are all welcome. Thank you again, and we look forward to your PR!