Add alerts for high 5xx error rates - #1168
Open
amar-gds-ndl wants to merge 3 commits into
Open
Conversation
…izing Fixes harvest-gather crash-loop by increasing memory limits: - Production: 1Gi → 3Gi memory limit, 512Mi → 1Gi requests, 250m → 500m CPU requests - Integration: 1Gi → 2Gi memory limit, 512Mi → 1Gi requests, 250m → 500m CPU requests - Staging: 1Gi → 2Gi memory limit, 512Mi → 1Gi requests, 250m → 500m CPU requests Health probes already enabled on all environments. This addresses the underlying resource exhaustion causing 116+ restarts since April 2nd.
Implements PrometheusRule resources for monitoring 5xx error rates: - CKAN (ckan.rules): Alert when 5xx rate > 20% for 5m (warning), > 50% for 2m (critical) - Datagovuk find (datagovuk-find.rules): Alert when 5xx rate > 20% for 5m (warning), > 50% for 2m (critical) - Datagovuk app (datagovuk-app.rules): Alert when 5xx rate > 20% for 5m (warning), > 50% for 2m (critical) Alerts are evaluated every 30 seconds and will fire when thresholds are exceeded. Severity levels: warning for > 20%, critical for > 50%.
Removed helm template variables from string annotations that were causing linting errors. Changed description fields to use static text instead of template rendering.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added alert rules to watch for 5xx errors on the main data.gov.uk services.
We don't have automatic alerts when services start returning errors. We need to know right away when something breaks.
The following are changed:
I have now added alert rules that watch 5xx errors:
Alert rules
When 5xx error rate goes over 20% and stays there for 5 minutes = sends warning alert
When 5xx error rate goes over 50% and stays there for 2 minutes = sends critical alert
This way we catch problems early.
Uses Prometheus to count errors every 30 seconds. If threshold is hit, alert fires.