Pruning doesn't work when multiple groupings

[pbadenski]

Based on my understanding of https://github.com/Haufe-Lexware/pushgateway-pruner/blob/master/index.js#L145 metrics are pruned based on a "randomly selected" (ie. first found that is not "job") label. This doesn't work when metrics were reported for multiple groupings. Prometheus pushgateway requires exact match on groupings when deleting the metric, ie.

${PUSHGATEWAY_URL}metrics/job/${job}/${labelName1}/${labelValue1}/${labelName2}/${labelValue2}/.../${labelNameN}/${labelValueN}

As a sidenote.. I am actually not entirely sure if this bug is possible to be fixed easily... I went through pushgateway's documentation and source code, and it seems to me that is not possible over HTTP API extract the list of metrics with their respective groupings. Important: identifying labels is not enough, as not all labels have to be part of the grouping (grouping is some subset of the label set). I believe that it is GetMetricFamiliesMap method in pushgateway which has that information. Grep through the codebase reveals that it doesn't seem to be exposed in any different way than through the UI (https://github.com/prometheus/pushgateway/blob/master/handler/status.go#L85)

[pbadenski]

Seems that it is the case that groupings information is not easily available via an endpoint from pushgateway. I added an issue for this in pushgateway that can be tracked here: prometheus/pushgateway#184

[DonMartin76]

Would you mind outlining the use case, e.g. give an example how to reproduce this? I admit I have been assuming a mapping job+instance --> grouping when I implemented this. The main use case here was to remove statistics for a specific job for "dead" instances. But as soon as I have grokked your use case, I would of course be happy to help out.

[pbadenski]

I'm glad you asked this question :) It made me realise that we can make do without multiple groupings.

Still let me give you the initial context. We track metrics for a fat client ui that runs in user's browser. We initially chose groupings to be "environment + user id + tab id". I know changed the grouping to be "tab id" which is a uuid and uniquely identifies each open instance of the app even across multiple user's tabs. It also works nice with the pruner :)

[DonMartin76]

OK. Let me know if you see something we can change to support you better.