Skip to content

Commit 43519f4

Browse files
committed
update readme
1 parent dde32b6 commit 43519f4

1 file changed

Lines changed: 59 additions & 11 deletions

File tree

README.md

Lines changed: 59 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77

88
[Quick Start](docs/quick-start.md)[Documentation](docs/)[Examples](docs/examples)[Twitter](https://twitter.com/hyperionhq)[Website](https://hyperionhq.co)[Blog](https://hyperionhq.co/blog)
99

10-
**⚡ <5µs Overhead • 2-Layer Caching • Smart Routing • Budget Controls • Spend Forecasting • Key Scoping • Real-time Analytics**
10+
**⚡ <5µs Overhead • 2-Layer Caching • Smart Routing • Budget Controls • Spend Forecasting • PII Redaction • Key Scoping • Real-time Analytics**
1111

1212
**LLM caching and orchestration for teams who can't afford a single millisecond of overhead.**
1313

@@ -31,6 +31,19 @@ What you get beyond just proxying:
3131

3232
---
3333

34+
## Table of Contents
35+
- [Quick Start](#quick-start)
36+
- [Performance](#performance)
37+
- [Capabilities](#capabilities)
38+
- [Use Cases](#use-cases)
39+
- [Architecture Context](#architecture-context)
40+
- [Documentation](#documentation)
41+
- [Community & Support](#community--support)
42+
- [Admin Tools](#admin-tools)
43+
- [License](#license)
44+
45+
---
46+
3447
## Quick Start
3548

3649
```bash
@@ -40,7 +53,7 @@ cp .env.example .env
4053
docker compose up -d --build
4154
```
4255

43-
Make a request through Hyperion using the official Python SDK:
56+
Make a request through Hyperion using the official [Python](https://github.com/hyperion-hq/hyperion-python) or [TypeScript](https://github.com/hyperion-hq/hyperion-typescript) SDKs. Here is an example using Python:
4457

4558
```python
4659
from hyperion import HyperionClient
@@ -88,18 +101,53 @@ All overhead is measured as pure gateway dispatch time (request receive → rout
88101

89102
## Capabilities
90103

91-
### ⚡ Multi-Layer Response Caching
92-
- **L1 (Redis) - ~0.5ms:** Sub-millisecond exact-match. Byte-identical requests never touch the external internet. They are served directly from RAM before the upstream connection is even initiated.
93-
- **L2 (Semantic) - ~15ms:** Vector similarity on prompt embeddings via Qdrant. Catches conceptually equivalent queries that differ in phrasing ("What is the speed of light" vs "How fast does light travel").
104+
### Cache-First Latency Reduction
105+
- **L1 exact-match cache:** Redis serves byte-identical requests in the lowest-latency path, often without touching the upstream provider at all.
106+
- **L2 semantic cache:** Qdrant-backed vector search catches meaning-level duplicates to cut both latency and token spend on repetitive traffic.
107+
- **Scoped cache safety:** Cache entries can be isolated by tenant, team, and project so higher hit rates do not come at the cost of data leakage.
108+
109+
### Smart Routing And Reliability
110+
- **Cost-aware model routing:** Hyperion routes traffic to the cheapest capable provider in real time instead of forcing every request through your most expensive model.
111+
- **Health-based failover:** If a provider is slow, degraded, or unavailable, requests can be transparently redirected to a healthy fallback.
112+
- **OpenAI-compatible endpoint:** Keep one client integration while Hyperion handles multi-provider orchestration underneath.
113+
114+
### Budgets, Keys, And Policy Controls
115+
- **Scoped API keys:** Issue keys with model allowlists, quotas, and per-key governance instead of sharing root provider credentials.
116+
- **Budget enforcement:** Reject requests before they hit the model once a key, project, or tenant crosses its configured spending limit.
117+
- **PII redaction and policy controls:** Sensitive data can be scrubbed before requests reach cache or upstream providers, and stricter routing paths can be enforced for sensitive workloads.
118+
119+
### Observability And Spend Visibility
120+
- **Real-time analytics:** Track cost, latency, cache hit rates, and provider health in the dashboard.
121+
- **Spend forecasting:** Monitor burn rate and projected budget exhaustion before overruns happen.
122+
- **Audit-friendly telemetry:** Centralized request and spend data gives you one place to understand how traffic is being served across providers.
123+
124+
---
125+
126+
## Use Cases
127+
128+
### High-Volume Support And FAQ Traffic
129+
When thousands of users ask the same question in slightly different ways, Hyperion turns that repetition into low-latency cache hits instead of repeated model calls. That cuts both response time and token spend without changing your application code.
130+
131+
### Multi-Tenant SaaS AI
132+
If you serve many customers from one AI platform, Hyperion gives each tenant isolated keys, scoped cache boundaries, and separate budget controls. You get shared infrastructure efficiency without cross-tenant leakage or uncontrolled spend.
133+
134+
### Internal Copilots And Team Tools
135+
For engineering assistants, support copilots, internal search, and workflow automation, Hyperion becomes the control layer in front of every model call. It lets you standardize access, enforce model policy, and see where money and latency are going across the company.
136+
137+
### Cost-Controlled Agent Workloads
138+
Agents can burn budget fast through loops, retries, and repeated tool decisions. Hyperion puts hard spend limits, routing controls, and cache-aware request handling in front of those flows so agent systems stay useful without becoming financially unpredictable.
139+
140+
### Privacy-Sensitive AI Applications
141+
In healthcare, finance, enterprise search, and other sensitive environments, Hyperion can redact PII before prompts ever reach cache or upstream providers. That makes it a practical privacy perimeter for teams that need stronger control over what leaves their infrastructure.
94142

95-
### Intelligent Routing
96-
An ML classifier runs per-request to assign complexity scores. Simple queries route to cheap models. Complex reasoning gets heavier models. No configuration—it adapts to your traffic dynamically.
143+
### Multi-Provider Reliability And Failover
144+
If your product depends on AI being available all the time, Hyperion gives you one endpoint in front of multiple providers with health-aware routing and failover. Instead of wiring fallback logic into every app, you centralize reliability in the gateway.
97145

98-
### Budget Enforcement & Security
99-
Hard ceilings at the key level. When a limit is hit, traffic stops. Upstream credentials are encrypted at rest (AES-256-GCM). Keys are never logged or stored in plaintext. PII filters scrub sensitive data before it reaches the cache or any upstream.
146+
### Latency-Sensitive User Experiences
147+
For chat, autocomplete, copilots, and real-time workflows, a few hundred milliseconds matters. Hyperion improves the fast path with exact-match caching, semantic reuse, and smarter provider selection so your application feels consistently responsive under real production traffic.
100148

101-
### Observability
102-
ClickHouse captures every request at throughput. The dashboard surfaces cost, latency, cache hit rates, and provider health in real time.
149+
### Centralized AI Governance
150+
When multiple teams use different models and providers, governance gets messy fast. Hyperion gives you one layer for API key management, budgets, routing policy, observability, and forecasting so AI usage is easier to scale and easier to control.
103151

104152
---
105153

0 commit comments

Comments
 (0)