fix: label mismatch for alertmanager_notifications_failed_total #3599
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cr7258
force-pushed
the
fix-label-mismatch
branch
from
d8744f2
to
37d0dac
Compare
| @@ -44,7 +44,7 @@ | |||
| ( | |||
| rate(alertmanager_notifications_failed_total{%(alertmanagerSelector)s}[5m]) | |||
| / | |||
| rate(alertmanager_notifications_total{%(alertmanagerSelector)s}[5m]) | |||
| on (integration) group_left rate(alertmanager_notifications_total{%(alertmanagerSelector)s}[5m]) | |||
There was a problem hiding this comment.
I'd suggest to ignore the reason label since it's the only one that's different between the 2 metrics. It will change a bit the alert condition in case of multiple reasons (e.g. the sum for all reasons might be above the threshold while the individual reasons are below). But the current threshold value is low enough that it shouldn't be an issue in practice.
Suggested change
| on (integration) group_left rate(alertmanager_notifications_total{%(alertmanagerSelector)s}[5m]) | |
| ignoring (reason) group_left rate(alertmanager_notifications_total{%(alertmanagerSelector)s}[5m]) |
There was a problem hiding this comment.
Thanks for your suggestion. Fixed.
cr7258
force-pushed
the
fix-label-mismatch
branch
from
37d0dac
to
f5c69b9
Compare
Signed-off-by: chengzw <[email protected]>
cr7258
force-pushed
the
fix-label-mismatch
branch
from
f5c69b9
to
aff09c2
Compare
Closed
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since Alertmanager 0.26.0 version, a new label
reasonwas added in thealertmanager_notifications_failed_totalmetric to indicate the type of error of the alert delivery.As a result, the original alert rules are broken because labels are mismatched between
alertmanager_notifications_failed_totalandalertmanager_notifications_total metrics.Prometheus requires samples with exactly the same labels to get matched together when performing calculations. docs
Use the
ignoringvector matching keyword to ignore the newreasonlabel to allow for matching between series with different labels.