It’s not always possible to modify the alerting rules configured in a prometheus instance, as they might be provided by a third party dependency like the prometheus-operator helm chart. This can lead to the issue that one gets alerted on false positives for a certain timeseries. In order to silence these alerts one could deduce from the alertmanager documentation that it’s possible to write an inhibiton rule matching these timeseries using the provided labels.

An inhibition rule mutes an alert (target) matching a set of matchers when an alert (source) exists that matches another set of matchers. Both target and source alerts must have the same label values for the label names in the equal list.

This cloud be such an inihibiton rule for suppressing CPUThrottlingHigh alerts for specific containers.

inhibit_rules:
  - source_match_re:
      alertname: CPUThrottlingHigh
    target_match_re:
      alertname: CPUThrottlingHigh
      container: (config-reloader|...)
    equal: ['pod', 'namespace']

But due to a change which prevents alerts from inhibiting themselves this doesn’t work and doesn’t appear in the logs. Fortunately it’s still possible to suppress such an alert by routing them to an empty receiver. Even though the alertmanager documentation doesn’t mention it an empty receiver is configured by only setting the name.

route:
  routes:
    - receiver: /dev/null
      match_re:
        alertname: CPUThrottlingHigh
        container: (config-reloader|...)
    ...

receiver:
  - name: /dev/null
  ...