Skip to content

Remove the two deprecated Summary metrics #599

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
FZambia wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Remove the two deprecated Summary metrics #599

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
979516b
Remove the two deprecated Summary metrics
FZambia Jul 29, 2026
3073e25
Refine duration histogram buckets for the latency range Centrifuge ru…
FZambia Jul 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 5 additions & 6 deletions _examples/native_histograms_otel/readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,12 @@ Centrifuge metrics (native histograms)
→ stdoutmetric exporter (prints OTel JSON)
```

Centrifuge exposes both a Summary and a Histogram for the two duration
metrics that have historically been Summaries — `command_duration_seconds`
and `survey_duration_seconds`. When `EnableNativeHistograms` is on:
The two duration metrics that were historically Summaries —
`command_duration_seconds` and `survey_duration_seconds` — are now exposed
only as Histograms, under their `_histogram` names. When
`EnableNativeHistograms` is on:

- The Summaries are no longer exposed (no-op internally; absent from output).
- The companion `_histogram` metrics switch to native (sparse, exponential)
schema.
- Those `_histogram` metrics switch to native (sparse, exponential) schema.
- The bridge translates native histograms to OTel `ExponentialHistogram` —
the high-fidelity form most OTel-native backends prefer.

Expand Down
27 changes: 6 additions & 21 deletions config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -443,30 +443,15 @@ type MetricsConfig struct {
ClientLabels []string
// EnableNativeHistograms switches every Histogram instrument in the
// package to Prometheus native (sparse, exponential) schema with no
// explicit buckets exposed, and stops exposing the legacy Summary
// counterparts of dual-instrument metrics. Designed for OpenTelemetry
// export via the client_golang Prometheus bridge — native histograms
// map to OTel ExponentialHistogram, and dropping Summaries keeps OTel
// Summary (which most backends treat as second-class) out of the
// pipeline.
// explicit buckets exposed. Designed for OpenTelemetry export via the
// client_golang Prometheus bridge, where native histograms map to OTel
// ExponentialHistogram.
//
// Centrifuge exposes both a Summary and a Histogram for the two
// distribution metrics that have historically been Summaries:
// - {ns}_client_command_duration_seconds (Summary) +
// {ns}_client_command_duration_seconds_histogram (Histogram)
// - {ns}_node_survey_duration_seconds (Summary) +
// {ns}_node_survey_duration_seconds_histogram (Histogram)
// Both are constructed and observed unconditionally.
//
// Default is false: today's behavior is preserved (Summaries still
// exposed; companion Histograms additionally exposed with classic
// explicit buckets — see Histograms section in metrics docs for the
// exact bucket lists).
// Default is false: Histograms are exposed with classic explicit
// buckets — see the Histograms section in the metrics docs for the
// exact bucket lists.
//
// When set to true:
// - Summaries are no longer exposed (the underlying instrument
// becomes a no-op so cached observers continue to satisfy
// prometheus.Observer without nil-checks).
// - All Histograms switch to native (sparse, exponential) schema
// with no explicit buckets.
// - Text-format Prometheus scrapes lose _bucket series on every
Expand Down
187 changes: 58 additions & 129 deletions metrics.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,30 +80,24 @@ var (
)

type metrics struct {
messagesSentCount *prometheus.CounterVec
messagesReceivedCount *prometheus.CounterVec
actionCount *prometheus.CounterVec
buildInfoGauge *prometheus.GaugeVec
numClientsGauge prometheus.Gauge
numUsersGauge prometheus.Gauge
numSubsGauge prometheus.Gauge
numChannelsGauge prometheus.Gauge
numNodesGauge prometheus.Gauge
replyErrorCount *prometheus.CounterVec
connectionsAccepted *prometheus.CounterVec
connectionsInflight *prometheus.GaugeVec
subscriptionsAccepted *prometheus.CounterVec
subscriptionsInflight *prometheus.GaugeVec
serverUnsubscribeCount *prometheus.CounterVec
serverDisconnectCount *prometheus.CounterVec
transportOutgoingCloseCount *prometheus.CounterVec
// commandDurationSummary holds the legacy Summary by default; when
// EnableNativeHistograms is true it is a no-op (the Summary is not
// exposed). The companion commandDurationHistogram below always carries
// the real observations, with native schema when the flag is on.
commandDurationSummary prometheus.ObserverVec
messagesSentCount *prometheus.CounterVec
messagesReceivedCount *prometheus.CounterVec
actionCount *prometheus.CounterVec
buildInfoGauge *prometheus.GaugeVec
numClientsGauge prometheus.Gauge
numUsersGauge prometheus.Gauge
numSubsGauge prometheus.Gauge
numChannelsGauge prometheus.Gauge
numNodesGauge prometheus.Gauge
replyErrorCount *prometheus.CounterVec
connectionsAccepted *prometheus.CounterVec
connectionsInflight *prometheus.GaugeVec
subscriptionsAccepted *prometheus.CounterVec
subscriptionsInflight *prometheus.GaugeVec
serverUnsubscribeCount *prometheus.CounterVec
serverDisconnectCount *prometheus.CounterVec
transportOutgoingCloseCount *prometheus.CounterVec
commandDurationHistogram *prometheus.HistogramVec
surveyDurationSummary prometheus.ObserverVec
surveyDurationHistogram *prometheus.HistogramVec
recoverCount *prometheus.CounterVec
recoveredPublications *prometheus.HistogramVec
Expand Down Expand Up @@ -136,15 +130,15 @@ type metrics struct {
commandDurationSubRefresh prometheus.Observer
commandDurationUnknown prometheus.Observer

broadcastDurationHistogram *prometheus.HistogramVec
pubSubLagHistogram *prometheus.HistogramVec
pingPongDurationHistogram *prometheus.HistogramVec
broadcastDurationHistogram *prometheus.HistogramVec
pubSubLagHistogram *prometheus.HistogramVec
pingPongDurationHistogram *prometheus.HistogramVec
brokerPublishSuppressedCount *prometheus.CounterVec
mapBrokerPublishSuppressedCount *prometheus.CounterVec
mapBrokerRemoveSuppressedCount *prometheus.CounterVec
mapBrokerCleanupLag *prometheus.GaugeVec
mapBrokerCleanupRemoved *prometheus.CounterVec
mapBrokerCleanupErrors *prometheus.CounterVec
mapBrokerCleanupLag *prometheus.GaugeVec
mapBrokerCleanupRemoved *prometheus.CounterVec
mapBrokerCleanupErrors *prometheus.CounterVec

redisBrokerPubSubErrors *prometheus.CounterVec
redisBrokerPubSubDroppedMessages *prometheus.CounterVec
Expand All @@ -171,27 +165,27 @@ type metrics struct {

config MetricsConfig

transportMessagesSentCache sync.Map
transportMessagesReceivedCache sync.Map
commandDurationCache sync.Map
replyErrorCache sync.Map
actionCache sync.Map
recoverCache sync.Map
unsubscribeCache sync.Map
disconnectCache sync.Map
messagesSentCache sync.Map
messagesReceivedCache sync.Map
tagsFilterDroppedCache sync.Map
transportMessagesSentCache sync.Map
transportMessagesReceivedCache sync.Map
commandDurationCache sync.Map
replyErrorCache sync.Map
actionCache sync.Map
recoverCache sync.Map
unsubscribeCache sync.Map
disconnectCache sync.Map
messagesSentCache sync.Map
messagesReceivedCache sync.Map
tagsFilterDroppedCache sync.Map
brokerPublishSuppressedCache sync.Map
mapBrokerPublishSuppressedCache sync.Map
mapBrokerRemoveSuppressedCache sync.Map
pubSubLagCache sync.Map
sharedPollHandlerCache sync.Map
sharedPollResultCache sync.Map
sharedPollChannelCache sync.Map
sharedPollPublishCache sync.Map
nsCache *otter.Cache[string, string]
codeStrings map[uint32]string
pubSubLagCache sync.Map
sharedPollHandlerCache sync.Map
sharedPollResultCache sync.Map
sharedPollChannelCache sync.Map
sharedPollPublishCache sync.Map
nsCache *otter.Cache[string, string]
codeStrings map[uint32]string

// Cache for client label combinations: maps cache key -> {labelValues, cacheKey}
// This allows sharing pre-computed label data across all clients with the same label values
Expand Down Expand Up @@ -375,50 +369,6 @@ func buildClientLabelsCacheKeyFromMap(labelNames []string, labelsMap map[string]
return b.String()
}

// dualObserver fans observations out to both a Summary and a Histogram
// observer. Used when a metric is exposed as both instrument types — the
// Summary preserves existing {quantile="..."} dashboards while the Histogram
// supplies the histogram_quantile()- and OpenTelemetry-friendly form. When
// EnableNativeHistograms is true the Summary side is a no-op, so only the
// Histogram records data.
type dualObserver struct {
summary, histogram prometheus.Observer
}

func (d dualObserver) Observe(v float64) {
d.summary.Observe(v)
d.histogram.Observe(v)
}

// noopObserverVec implements prometheus.ObserverVec with all no-op methods.
// Assigned to a Summary accessor when EnableNativeHistograms is true so the
// Summary side of a dual-instrument metric is not exposed and contributes no
// observation cost. Callers that cache observers via WithLabelValues do not
// need nil-checks — they get a noopObserver that silently drops Observe()
// calls.
type noopObserverVec struct{}

func (noopObserverVec) Describe(chan<- *prometheus.Desc) {}
func (noopObserverVec) Collect(chan<- prometheus.Metric) {}
func (noopObserverVec) WithLabelValues(...string) prometheus.Observer { return noopObserver{} }
func (noopObserverVec) With(prometheus.Labels) prometheus.Observer { return noopObserver{} }
func (noopObserverVec) GetMetricWith(prometheus.Labels) (prometheus.Observer, error) {
return noopObserver{}, nil
}
func (noopObserverVec) GetMetricWithLabelValues(...string) (prometheus.Observer, error) {
return noopObserver{}, nil
}
func (noopObserverVec) CurryWith(prometheus.Labels) (prometheus.ObserverVec, error) {
return noopObserverVec{}, nil
}
func (noopObserverVec) MustCurryWith(prometheus.Labels) prometheus.ObserverVec {
return noopObserverVec{}
}

type noopObserver struct{}

func (noopObserver) Observe(float64) {}

// nativeHistogramOpts returns opts unchanged when native is false. When true,
// it enables Prometheus native histogram schema with no explicit buckets —
// the metric exposes only _count, _sum, and the native histogram chunk.
Expand Down Expand Up @@ -527,24 +477,17 @@ func newMetricsRegistry(config MetricsConfig) (*metrics, error) {
Help: "Number of channels with one or more subscribers.",
})

if config.EnableNativeHistograms {
m.surveyDurationSummary = noopObserverVec{}
} else {
m.surveyDurationSummary = prometheus.NewSummaryVec(prometheus.SummaryOpts{
Namespace: metricsNamespace,
Subsystem: "node",
Name: "survey_duration_seconds",
Objectives: map[float64]float64{0.5: 0.05, 0.99: 0.001, 0.999: 0.0001},
Help: "DEPRECATED — use survey_duration_seconds_histogram. Will be removed in future releases. Survey duration summary.",
}, []string{"op"})
}
m.surveyDurationHistogram = prometheus.NewHistogramVec(nativeHistogramOpts(prometheus.HistogramOpts{
Namespace: metricsNamespace,
Subsystem: "node",
Name: "survey_duration_seconds_histogram",
Help: "Survey duration histogram. Use for histogram_quantile() and OpenTelemetry export.",
// Surveys are node-to-node round trips, so millisecond resolution is
// the right scale, but the 1ms..5ms step had the same 5x gap as the
// command histogram and is filled in for the same reason. Additive
// only - existing boundaries are unchanged.
Buckets: []float64{
0.001, 0.005, 0.010, 0.025, 0.050, 0.100, 0.250, 0.500,
0.001, 0.002, 0.003, 0.005, 0.010, 0.025, 0.050, 0.100, 0.250, 0.500,
1.0, 2.5, 5.0, 10.0,
},
}, config.EnableNativeHistograms), []string{"op"})
Expand All @@ -563,25 +506,21 @@ func newMetricsRegistry(config MetricsConfig) (*metrics, error) {
Help: "Number of messages received from broker.",
}, []string{"type", "channel_namespace"})

if config.EnableNativeHistograms {
m.commandDurationSummary = noopObserverVec{}
} else {
m.commandDurationSummary = prometheus.NewSummaryVec(prometheus.SummaryOpts{
Namespace: metricsNamespace,
Subsystem: metricClientCommandDuration.Subsystem,
Name: metricClientCommandDuration.Name,
Objectives: map[float64]float64{0.5: 0.05, 0.99: 0.001, 0.999: 0.0001},
Help: "DEPRECATED — use command_duration_seconds_histogram. Will be removed in future releases. Client command duration summary.",
}, m.buildMetricLabels([]string{"method", "channel_namespace"}))
}
m.commandDurationHistogram = prometheus.NewHistogramVec(nativeHistogramOpts(prometheus.HistogramOpts{
Namespace: metricsNamespace,
Subsystem: metricClientCommandDuration.Subsystem,
Name: metricClientCommandDuration.Name + "_histogram",
Help: "Client command duration histogram. Use for histogram_quantile() and OpenTelemetry export.",
// Centrifuge command latencies live in the microsecond range - a couple
// of hundred microseconds is normal - so the ladder starts at 25us
// rather than 100us, and the 1ms..5ms step is filled in. Without the
// former the median is interpolated from zero on a fast node; without
// the latter p99 lands in a 5x-wide bucket and reads far high. Every
// boundary that existed before is still here, so le= queries and
// recording rules keep working.
Buckets: []float64{
0.000100, 0.000250, 0.000500,
0.001, 0.005, 0.010, 0.025, 0.050, 0.100, 0.250, 0.500,
0.000025, 0.000050, 0.000100, 0.000250, 0.000500,
0.001, 0.002, 0.003, 0.005, 0.010, 0.025, 0.050, 0.100, 0.250, 0.500,
1.0, 2.5, 5.0, 10.0,
},
}, config.EnableNativeHistograms), m.buildMetricLabels([]string{"method", "channel_namespace"}))
Expand Down Expand Up @@ -955,10 +894,7 @@ func newMetricsRegistry(config MetricsConfig) (*metrics, error) {

makeCommandObserver := func(method string) prometheus.Observer {
labels := buildCommandLabels(method)
return dualObserver{
summary: m.commandDurationSummary.WithLabelValues(labels...),
histogram: m.commandDurationHistogram.WithLabelValues(labels...),
}
return m.commandDurationHistogram.WithLabelValues(labels...)
}
m.commandDurationConnect = makeCommandObserver(labelForMethod(protocol.FrameTypeConnect))
m.commandDurationSubscribe = makeCommandObserver(labelForMethod(protocol.FrameTypeSubscribe))
Expand All @@ -984,7 +920,6 @@ func newMetricsRegistry(config MetricsConfig) (*metrics, error) {
m.numSubsGauge,
m.numChannelsGauge,
m.numNodesGauge,
m.commandDurationSummary,
m.commandDurationHistogram,
m.replyErrorCount,
m.connectionsAccepted,
Expand All @@ -1002,7 +937,6 @@ func newMetricsRegistry(config MetricsConfig) (*metrics, error) {
m.transportMessagesReceivedSize,
m.tagsFilterDroppedCount,
m.buildInfoGauge,
m.surveyDurationSummary,
m.surveyDurationHistogram,
m.pubSubLagHistogram,
m.broadcastDurationHistogram,
Expand Down Expand Up @@ -1098,10 +1032,7 @@ func (m *metrics) observeCommandDuration(frameType protocol.FrameType, d time.Du
if !ok {
baseLabels := []string{frameType.String(), channelNamespace}
labelValues := m.appendClientLabels(baseLabels, c)
observer = dualObserver{
summary: m.commandDurationSummary.WithLabelValues(labelValues...),
histogram: m.commandDurationHistogram.WithLabelValues(labelValues...),
}
observer = m.commandDurationHistogram.WithLabelValues(labelValues...)
m.commandDurationCache.Store(labels, observer)
}
observer.(prometheus.Observer).Observe(d.Seconds())
Expand Down Expand Up @@ -1506,9 +1437,7 @@ func (m *metrics) incActionCount(action string, ch string) {
}

func (m *metrics) observeSurveyDuration(op string, d time.Duration) {
seconds := d.Seconds()
m.surveyDurationSummary.WithLabelValues(op).Observe(seconds)
m.surveyDurationHistogram.WithLabelValues(op).Observe(seconds)
m.surveyDurationHistogram.WithLabelValues(op).Observe(d.Seconds())
}

type tagsFilterDroppedLabels struct {
Expand Down
Loading
Loading