feat(yandex-cloud): add metric, log, instance, balancer tools and a generic API reader by nowhere-in-space · Pull Request #4990 · Tracer-Cloud/opensre

nowhere-in-space · 2026-08-13T21:25:45Z

Part of #4605

Describe the changes you have made in this PR -

Second PR of the Yandex Cloud series. #4947 landed the credential and REST client layer but shipped no tools, so a connected folder could be verified and then not read. This adds the first tool families and the generic reader that covers everything else. Most of these tools are ported from my working plugin, opensre-yc-plugin, where they have been running against real Yandex Cloud infrastructure.

Curated tools (they hard-code their own paths and resolve the host through the endpoint registry added in #4947):

Package	Tools
`integrations/yc_monitoring`	`query_yc_metrics`, `list_yc_metrics`
`integrations/yc_logging`	`read_yc_logs`, `list_yc_log_groups`
`integrations/yc_compute`	`list_yc_instances`, `get_yc_instance_diagnostics`
`integrations/yc_network`	`get_yc_lb_health`

Generic reader in integrations/yandex_cloud/tools: find_yc_api resolves a path by plain words ("security groups", "certificates"), and execute_yc_operation reads it. That is what makes services without a dedicated tool reachable - container registry, DNS, KMS, Lockbox, YDB and the rest - without a tool per service.

api_index.json. Yandex generates its REST API from the protos in yandex-cloud/cloudapi, and every method carries its REST binding as a google.api.http option. build_api_index.py extracts only the get: bindings, which is why the index is also the allowlist: the client refuses anything but GET, and a write path is never in the file to begin with. The file records the cloudapi commit and build date, so "is this stale" is answerable without rebuilding and diffing.

Coverage. The index carries 937 read endpoints across 68 services, and every service in it resolves to a reachable host (asserted in the tests). Everything SRE-shaped is in there - compute, vpc, all managed databases, managed kubernetes, both load balancers, serverless, container registry, dns, iam, kms, lockbox, certificates, ydb, storage, backup, quota-manager, resource-manager, data transfer, audit trails, cdn. What the index does not carry is what Yandex exposes with no GET binding at all: model inference (ai-*), data-plane endpoints that read a payload rather than a resource (kms-crypto, lockbox-payload, *-data), and the two reads that take a request body - Monitoring's metric read and Cloud Logging's entry reader, which are exactly what the curated tools above cover. So a read either has a dedicated tool or is one find_yc_api call away; nothing readable is unreachable.

post() returns to the client. It was removed in #4947 as unused, which was correct at the time. Monitoring's metric read takes a request body and is the one read here that a GET cannot express; it is the only caller.

Read-only throughout. When an investigation concludes something needs changing, the tools report the exact yc command for an operator rather than attempting it.

One optional dependency. Cloud Logging is the only Yandex Cloud read with no REST endpoint - the entry reader is gRPC-only - so Yandex's generated stubs ship as the yandex_cloud_logs extra, in the same shape as the existing kafka, postgresql and azure_sql extras: imported inside the function that needs them, with a failure that explains the install. Listing log groups is plain REST and works without it. A test asserts the install hint names an extra this project actually declares, so a message pointing at something uninstallable fails the suite rather than reaching a user.

A folder does not scope a nested collection. The executor decided to send folderId from "is this a collection", but a folder scopes only the collections directly under the version. /compute/v1/instances/{id}/operations is already scoped by the instance in its path, and Yandex rejects the extra parameter with the same bare 404 it gives a single-resource read - so operation histories, cluster hosts, node groups and disk operations all answered as if the resource did not exist. Confirmed against the live API (no folderId returns the history, folderId 404s, pageSize is fine either way) and fixed.

The commits after the first two fix defects found auditing this port, both of which would have shipped silently. api_index.json was in neither the wheel nor the frozen binary - package-data and the release manifest both glob only **/SKILL.md - and the loader swallowed the resulting error, so find_yc_api answered "no endpoints" on any packaged install. It is now packaged, covered by the wheel validator, and the loader says which file it could not read. Separately, the balancer tool called :targetStates where this repo's own proto-derived index (and the application-balancer call ten lines below) spell it :getTargetStates; target health came back empty against a real cloud, and the test had stubbed the misspelling, pinning the bug.

A folder with more balancers or instances than one page holds now says so instead of implying the list is whole: get_yc_lb_health reports complete: false, and a name-filtered list_yc_instances with further pages warns that the match was local to the page. "No unhealthy target" must never be a guess.

Not in this PR: reading audit events. The audittrails service itself is in the index, so trails are listable, but the events land in a sink rather than an API, so surfacing them needs its own design discussion.

Demo/Screenshot for feature changes and bug fixes -

pr2demopro.mp4

Explain your implementation approach:

The problem is coverage against tool budget. Yandex Cloud exposes more than 900 read endpoints across nearly 70 services, and only 32 tool schemas reach the model per turn, shared with every other configured integration. A tool per service is therefore impossible, and picking a dozen services by hand means the agent reports "cannot read that" for everything else.

So the split is deliberate: a small set of curated tools where the shape of the answer matters - a metric series has to be summarised, an instance list has to say which ones are stopped, a balancer has to name the unhealthy targets - and one generic reader for the long tail, where returning the raw resource is the right answer.

On the optional dependency: vendoring the generated stubs or hand-writing a protobuf client was possible but means carrying generated code that Yandex already publishes. Making it a hard dependency would put grpcio in every install for a read most users never make. The extra keeps both out of the way.

Alternatives I considered. Generating the endpoint list at build time was rejected because it adds a network dependency on cloudapi to CI; the file is 230 KB, comparable to snapshots already in the tree. Trimming the index to only the services with curated tools defeats its purpose, which is precisely the long tail. Enforcing read-only with a regex on operation names, as the AWS integration does with boto3, was not available here because there is no shipped catalogue to match against - extracting get: bindings from the protos gives the same guarantee from the authoritative source.

Key components: api_index.py loads and searches the index, ranking collection endpoints above single-resource reads because an agent without an id yet needs the list; build_api_index.py regenerates it, including an alias table for the six services where the registry hyphenates and the protos do not, which had silently dropped 34 endpoints; each family's tools convert the raw payload into the narrowing answer described above.

Edge cases covered by tests: a path containing .. or a scheme is rejected, an unknown service is refused before any request, folder scope is applied only to endpoints that accept it, the folder is discovered from instance metadata when it is not configured, an unreadable serial console does not fail the diagnostics call, and an unknown aggregation is rejected without a call going out.

Code Understanding and AI Usage

Did you use AI assistance (ChatGPT, Claude, Copilot, etc.) to write any part of this code?

No, I wrote all the code myself
Yes, I used AI assistance (continue below)

If you used AI assistance:

I have reviewed every single line of the AI-generated code
I can explain the purpose and logic of each function/component I added
I have tested edge cases and understand how the code handles them
I have modified the AI output to follow this project's coding standards and conventions

Checklist before requesting a review

I have added proper PR title and linked to the issue
I have performed a self-review of my code
I can explain the purpose of every function, class, and logic block I added
I understand why my changes work and have tested them thoroughly
I have considered potential edge cases and how my code handles them
If it is a core feature, I have added thorough tests
My code follows the project's style guidelines and conventions

…neric API reader The Yandex Cloud integration could authenticate and verify a folder but could not read anything from it. This adds the first tool families and the generic reader that covers the rest of the API. Curated tools hard-code their own paths: - monitoring: query metric series, list metric names and labels - compute: list instances, read serial console output for diagnosis - network: report unhealthy network and application balancer targets Everything else is reached through execute_yc_operation, which resolves a path via find_yc_api against an index generated from Yandex's own protobuf definitions. Only get: bindings are extracted, so the index doubles as the allowlist that keeps the reader GET-only; build_api_index.py regenerates it and records the cloudapi commit it came from. The client regains post(): Monitoring's metric read takes a request body, which is the one read in this set that a GET cannot express. Tools are registered for discovery, classified for Sentry coverage, and held to a seventeen-schema ceiling so later families stay inside the per-turn budget.

Cloud Logging is the one Yandex Cloud read with no REST endpoint: the entry reader is gRPC-only. The stubs therefore ship as an optional extra, following the same shape as the kafka and azure_sql extras - imported inside the function that needs them, with a failure that explains the install rather than reading as a broken integration. Listing log groups is plain REST on the management host and works without the extra. Only reading entries needs it. The reader host is separate from the management one, entry reads are limited to five per second, and retention is 31 days, so the client throttles, honours Retry-After, caps filter expressions, and flags a window that reaches past retention. The install hint is checked against the extras this project declares, so a message naming something unavailable fails the suite rather than reaching a user.

…states path Two defects found while auditing the port against the plugin it came from. The endpoint index was in no distribution artifact. `package-data` lists only `**/SKILL.md` under integrations, and the release manifest globs the same, so `api_index.json` was absent from both the wheel and the frozen binary. The loader swallowed the resulting OSError and returned an empty index, which the tool reports as "no endpoints" - so `find_yc_api`, the entry point the workflow guidance routes to, answered as if Yandex exposed nothing. It now ships, the wheel validator covers it, and the loader logs which file it could not read. The network balancer tool called `:targetStates`. The proto-derived index in this repo spells the binding `:getTargetStates`, as does the application balancer call ten lines below it, so target health came back empty against a real cloud. The test stubbed the misspelling and therefore pinned the bug. Also, guidance and tests that were carrying their own small lies: SKILL.md pointed the model at tools from families this tree does not ship yet; `list_yc_metrics` was the only tool here with no execution coverage; and the schema text that stops the model writing PromQL had nothing holding it in place.

…ng it is whole Two reads answered as if they had seen everything. `get_yc_lb_health` took the first page of each balancer type and dropped the rest without a word. "No unhealthy targets" is the one answer that must never be a guess, so the tool now reports `complete: false` and points at the `type` filter and the generic reader for the remainder. `list_yc_instances` matches a name fragment locally, because Yandex's own filter compares names for equality and has no substring form. That makes the match local to the page just fetched, so an instance on a later page reads as absent. It now says so when a filtered read has more pages behind it. Also records why Monitoring asks for `gapFilling: NULL`: PREVIOUS would carry the last value forward, and a service that stopped reporting would come back as a flat healthy line.

Reading an instance's operation history returned a bare 404, as did every other collection nested under a named resource: cluster hosts, node groups, disk and balancer operations. A large part of the API answered as if the resource did not exist. The executor decided to send folderId from "is this a collection", but a folder scopes only the collections directly under the version. A nested one is already scoped by the resource named in its path, and Yandex rejects the extra parameter the same silent way it rejects one on a single-resource read. Verified against the live API: /compute/v1/instances/{id}/operations returns the history with no folderId and 404s with it, while pageSize is accepted either way - so only the folder is withheld and paging still applies.

github-actions · 2026-08-13T21:25:55Z

Greptile code review

This repo uses Greptile for automated review. Before merge, aim for Confidence Score: 5/5 with zero unresolved review threads — see CONTRIBUTING.md.

Run a review — add a PR comment with:

@greptile review

Give it ~5-10 minutes (sometimes longer) for results, then fix feedback and re-trigger until you reach Confidence Score: 5/5.

Optional: automate with the greploop skill.

nowhere-in-space · 2026-08-13T21:26:03Z

@greptile review

greptile-apps · 2026-08-13T21:29:51Z

Greptile Summary

The PR adds Yandex Cloud tools for metrics, logs, compute instances, and load-balancer health, plus an indexed generic reader for other read-only API operations.

Packages and validates a generated catalogue of Yandex Cloud read endpoints.
Adds curated investigation tools and optional Cloud Logging gRPC support.
Walks application load-balancer routes, backend groups, target groups, and target states to report unhealthy targets.
Updates tool discovery, packaging, documentation, and integration tests.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the previously reported application load-balancer traversal and aggregation gaps are fixed, and the empty-log guidance no longer names an unavailable tool.

Important Files Changed

Filename	Overview
integrations/yc_network/tools/yc_lb_tool/init.py	Adds complete network and application load-balancer health collection, including HTTP and gRPC route traversal and partial-result reporting.
integrations/yc_logging/tools/yc_logs_tool/init.py	Adds Cloud Logging reads and group listing with accurate empty-result guidance and actionable supported fallbacks.
integrations/yandex_cloud/api_index.py	Loads and searches the packaged read-only endpoint index used by the generic API tools.
integrations/yandex_cloud/tools/yc_operation_tool/init.py	Executes allowlisted generic Yandex Cloud read operations with path and parameter handling.
integrations/yc_monitoring/tools/yc_metrics_tool/init.py	Adds curated metric query and metric-discovery tools.
integrations/yc_compute/tools/yc_instances_tool/init.py	Adds instance listing and diagnostics with pagination and partial-diagnostics handling.
platform/packaging/release_manifest.py	Includes the generated Yandex Cloud API index in release artifacts.
pyproject.toml	Declares package data and the optional Yandex Cloud Logging dependency group.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Agent[Investigation agent] --> Curated{Curated tool?}
  Curated -->|Metrics| Monitoring[Yandex Monitoring]
  Curated -->|Logs| Logging[Yandex Cloud Logging]
  Curated -->|Instances| Compute[Yandex Compute]
  Curated -->|Balancer health| LB[Load balancers]
  Curated -->|Other resource| Lookup[find_yc_api]
  Lookup --> Index[Packaged read-only API index]
  Index --> Execute[execute_yc_operation]
  Execute --> REST[Yandex Cloud REST API]
  LB --> Router[HTTP router]
  Router --> Route[HTTP or gRPC route]
  Route --> Backend[Backend group]
  Backend --> TargetGroup[Target group]
  TargetGroup --> Health[Target states]

_{Reviews (6): Last reviewed commit: "fix(yandex-cloud): follow gRPC routes wh..." | Re-trigger Greptile}

CodeQL flagged three new alerts, all in yc_logging. The high-severity one is a false positive in a test - a `host in url` assertion reads to CodeQL as incomplete URL sanitization even though it only checks which host the tool called. Made it exact: parse the URL and compare the hostname, which is both CodeQL-clean and a stronger assertion. The client carried a `logger` that nothing used - removed it and its now-unused import. And the throttle's last-read timestamp was a bare module global whose reassignment reads as write-but-never-used, because the value is consumed on the next call; moved it onto a small dict so the read and write are unambiguous. No behaviour change - the rate limiting is identical.

nowhere-in-space · 2026-08-13T22:12:59Z

@greptile review

…dead tool reference Two review findings, both real. get_yc_lb_health summarised unhealthy targets from network balancers only. The application balancer stored its raw target states under a different key that the aggregation never read, so a failing application backend left unhealthy_targets empty - the one answer that must not be wrong. Both kinds return the same getTargetStates shape, so a shared normaliser now feeds both into the summary; the raw application response stays for the detail the flat view drops. The empty-Cloud-Logging guidance still told the agent to read managed-database logs with read_yc_db_logs, a tool this PR does not ship. Reworded to say those logs are not readable yet and to fall back to metrics and cluster state.

nowhere-in-space · 2026-08-13T22:30:10Z

@greptile review

…balancer does not serve The application balancer reused the network balancer's :getTargetStates action, but that verb does not exist for it: its target states live under a nested path keyed by backend group and target group (/apploadbalancer/v1/loadBalancers/{id}/targetStates/{backend_group_id}/{target_group_id}), which needs the balancer's backend-group graph walked first. The old call 404s against a real cloud, so application target health was never real. Rather than ship a request the API rejects or fabricate health from it, the tool now lists application balancers with their status and carries a pointer to the real nested path, reachable through execute_yc_operation. Network balancers keep full per-target health, which does follow the :getTargetStates contract. The output docs say which is which so unhealthy_targets is not read as covering both.

nowhere-in-space · 2026-08-13T22:48:51Z

@greptile review

…h the backend graph The application balancer keeps target states behind a nested path that the network balancer's :getTargetStates action does not reach: /apploadbalancer/v1/loadBalancers/{id}/targetStates/{backend_group}/{target_group}. Getting there means walking the balancer's graph - listener to HTTP router to route to backend group to target group - and only then reading targetStates, where health is reported per zone (a target is unhealthy only when every zone fails its active health check). get_yc_lb_health now walks that graph and normalises application targets into the same shape as network ones, so a failing application backend reaches the same unhealthy_targets summary. Single-resource reads on that path pass page_size=None, since Yandex answers a target-state read carrying a stray pageSize with a bare 404. Shapes captured from a live application balancer.

nowhere-in-space · 2026-08-13T23:24:11Z

@greptile review

…get health The backend-group walk read only http routes, so an application balancer that routes over gRPC never reached its backend group, and its unhealthy targets were left out of unhealthy_targets. A route names its backend group under its own protocol key, so both http and grpc are now inspected. The backend group itself already handled http, grpc and stream backends.

nowhere-in-space · 2026-08-13T23:52:27Z

@greptile review

nowhere-in-space added 5 commits August 13, 2026 22:29

greptile-apps Bot reviewed Aug 13, 2026

View reviewed changes

Comment thread integrations/yc_network/tools/yc_lb_tool/__init__.py Outdated

Comment thread integrations/yc_logging/tools/yc_logs_tool/__init__.py

github-advanced-security AI found potential problems Aug 13, 2026

View reviewed changes

Comment thread tests/integrations/test_yc_logging.py Fixed

Comment thread integrations/yc_logging/client.py Fixed

Comment thread integrations/yc_logging/client.py Fixed

Navigation Menu

Uh oh!

feat(yandex-cloud): add metric, log, instance, balancer tools and a generic API reader - #4990

feat(yandex-cloud): add metric, log, instance, balancer tools and a generic API reader#4990
nowhere-in-space wants to merge 10 commits into
Tracer-Cloud:mainfrom
nowhere-in-space:yc-tools

nowhere-in-space commented Aug 13, 2026

Uh oh!

github-actions Bot commented Aug 13, 2026

Uh oh!

nowhere-in-space commented Aug 13, 2026

Uh oh!

greptile-apps Bot commented Aug 13, 2026 •

edited

Loading

Important Files Changed

Flowchart

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nowhere-in-space commented Aug 13, 2026

Uh oh!

nowhere-in-space commented Aug 13, 2026

Uh oh!

Uh oh!

nowhere-in-space commented Aug 13, 2026

Uh oh!

Uh oh!

nowhere-in-space commented Aug 13, 2026

Uh oh!

Uh oh!

nowhere-in-space commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

nowhere-in-space commented Aug 13, 2026

Describe the changes you have made in this PR -

Demo/Screenshot for feature changes and bug fixes -

Code Understanding and AI Usage

Checklist before requesting a review

Uh oh!

github-actions Bot commented Aug 13, 2026

Greptile code review

Uh oh!

nowhere-in-space commented Aug 13, 2026

Uh oh!

greptile-apps Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Flowchart

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nowhere-in-space commented Aug 13, 2026

Uh oh!

nowhere-in-space commented Aug 13, 2026

Uh oh!

Uh oh!

nowhere-in-space commented Aug 13, 2026

Uh oh!

Uh oh!

nowhere-in-space commented Aug 13, 2026

Uh oh!

Uh oh!

nowhere-in-space commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

greptile-apps Bot commented Aug 13, 2026 •

edited

Loading