Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[mixin/alerts]: Enable configuring job prefix for alerts to prevent clashes with metrics from Loki/Tempo. #9659

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

mtweten
Copy link
Contributor

@mtweten mtweten commented Oct 17, 2024

What this PR does

This enables the ability to configure a job prefix for alerts in the mimir-mixin. Currently, some alerts just use a regex that matches any job prefix (e.g. .+/ or .*/). This can cause a problem because certain alerts will start triggering because they will match metrics for not just mimir, but also loki/tempo (if present).

As a specific example, GossipMembersTooHigh and RingMembersMismatch will start firing if Tempo metrics are being collected, since the current alerts will match not only on job=mimir/ingester, but also job=tempo/ingester

Typically, users would configure this alert prefix to the namespace mimir is deployed to, e.g. "mimir/".

This is a non-impactful change as it's a new configuration option which defaults to the previous prefix, .*/. However, it does change GossipMembersTooHigh from the prefix .+/ to .*/, but this shouldn't be relevant. Every other alert using this pattern was already using .*/ for the job prefix.

Which issue(s) this PR fixes or relates to

Fixes #5260

Checklist

  • Tests updated.
  • Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
  • about-versioning.md updated with experimental features.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request mixin
Projects
None yet
Development

Successfully merging this pull request may close these issues.

PrometheusAlerts should filter by namespace (conflicts with cortex)
2 participants