Prometheus Docs Recommend Bizarrely Low Cardinality

[sbaynham-gridwise]

Doc: https://prometheus.io/docs/practices/instrumentation/

This documentation recommends keeping cardinality low, which is good advice, but its definition of "low" is bizarre.

As a general guideline, try to keep the cardinality of your metrics below 10, and for metrics that exceed that, aim to limit them to a handful across your whole system. The vast majority of your metrics should have no labels.

If you have a metric that has a cardinality over 100 or the potential to grow that large, investigate alternate solutions such as reducing the number of dimensions or moving the analysis away from monitoring and to a general-purpose processing system.

I have never seen a metric with cardinality below 10 in the wild, A simple latency metric that uses default prometheus bucketing and runs on 9 hosts will hit 100. This gives new engineers a very incorrect idea of what prometheus is capable of and I think it should be changed.

[sbaynham-gridwise]

By contrast, this presentation about cardinality from promcon: https://promcon.io/2019-munich/slides/containing-your-cardinality.pdf

discusses cardinalities in the 10s and 100s of thousands when talking about limitations