It’s not always possible to modify the alerting rules configured in a prometheus instance, as they might be provided by a third party dependency like the prometheus-operator helm chart. This can lead to the issue that one gets alerted on false positives for a certain timeseries. In order to silence these alerts one could deduce from the alertmanager documentation that it’s possible to write an inhibiton rule matching these timeseries using the provided labels.
An inhibition rule mutes an alert (target) matching a set of matchers when an alert (source) exists that matches another set of matchers. Both target and source alerts must have the same label values for the label names in the equal list.
This cloud be such an inihibiton rule for suppressing CPUThrottlingHigh alerts for specific containers.
inhibit_rules: - source_match_re: alertname: CPUThrottlingHigh target_match_re: alertname: CPUThrottlingHigh container: (config-reloader|...) equal: ['pod', 'namespace']
But due to a change which prevents alerts from inhibiting themselves this
doesn’t work and doesn’t appear in the logs. Fortunately it’s still possible to
suppress such an alert by routing them to an empty receiver. Even though the
alertmanager documentation doesn’t mention it an empty receiver is
configured by only setting the
route: routes: - receiver: /dev/null match_re: alertname: CPUThrottlingHigh container: (config-reloader|...) ... receiver: - name: /dev/null ...