Last month at Portkey, we crossed 2 BILLION total requests processed through our platform. To think, we just started a year ago when this number was at 0!

We're truly humbled to be your production partners and that drives us to continue innovating and making Portkey better.

In July & August, we did 3 major releases: Guardrails, Conditional Router, and Tracing.

Check out this release note to dive deep into everything that's new at Portkey.

We released an improved flow for adding and editing your Virtual keys on Portkey! Head over to the Virtual Keys page and let us know what you think.

We're thrilled to announce full interoperability of Audio routes between OpenAI and Azure OpenAI APIs.

This update brings the comprehensive suite of Portkey Gateway & Observability features to all your Text-to-Speech, Speech-to-Text, and Translation requests across both OpenAI and Azure OpenAI routes.

What This Means for You:

✅ Enhanced Reliability: Automatically route Audio requests to Azure when OpenAI API experiences issues.

✅ Improved Scalability: Tackle rate limiting by load balancing requests across multiple instances.

✅ Global Compliance: Direct requests to specific Azure regions to meet data residency requirements.

...and that's just the beginning!

Let's see an example:

Creating the fallback config

Let's create a fallback config that calls OpenAI in case there's failure from Azure OpenAI API

Making a request

Explore docs here:

 Text-to-Speech 

 Speech-to-text 

🚀 You can use Portkey with Tembo to conduct vector searches on Postgres Use multiple LLM providers to create embeddings - without changing your setups

 Link to docs 

Get incredible inference speed with the Cerebras API: 1800 tokens/sec on the Llama 3.1 8B model. And you can easily route to the Cerebras API using Portkey:

Explore docs: https://docs.portkey.ai/docs/integrations/llms/cerebras 

You can now ensure that your OpenAI outputs are strictly following a supplied JSON schema. Portkey natively supports OpenAI's new Structued Outputs mode with our SDKs & APIs.

Just start sending the JSON schema in the response_format param, and Portkey will log all your requests and calculate cost, tokens, latency for them.

Portkey now seamlessly integrates with major agent frameworks like LangChain, CrewAI, AutoGen, and others, making your agent workflows production-ready.

With Portkey’s native integration, you can effortlessly incorporate our Gateway, Guardrails, and Observability suite into your favorite agent frameworks, and take them to production with confidence.

 Browse Portkey Agents docs here.

We are open sourcing our AI Guardrails 🚧 framework on the AI Gateway! Now, on Portkey, you can synchronously run guardrails on your requests AND orchestrate them based on guardrail verdicts.

Portkey offer 50+ guardrails built in - everything from detecting hallucinations and retrieval accuracy to detecting PII, PHI, toxicity, and more.

 Browse Portkey Guardrails docs here. 

Reach out to us on Discord to enable Guardrails for your account!

Last Thursday, the Anthropic API was unstable/down for almost 12 hours:

We saw that orgs who had setup fallbacks for their Anthropic requests, they did not face any failures — 99.86% of their requests succeeded, and only 0.14% failed.

These users had setup fallbacks to route their requests to (1) OpenAI, (2) Azure, (3) Gemini, and (4) a bunch of hosted Llama models.

There are some key learnings from the 0.14% requests that failed though:

1. Some users hadn't configured fallbacks for the 529 status code (which had spiked the most that day)
2. A few had improperly set up fallback targets (expired keys, non-existent targets)
3. In rare cases, even the fallback target failed (pro tip: always have multiple options!)

Check out our fallback documentation to protect your app from going down with an LLM API failure again → Portkey Fallback Docs 

Last month, the Portkey team, in partnership with Postman, organized the first #LLMsinProd event in Bengaluru.

We had some of the brightest AI engineers, leaders, researchers, PMs, and founders come together to discuss the best methods of productionizing Gen AI systems.

• Portkey's co-founder Rohit shared how the AI Gateway pattern is becoming a crucial middleware for shipping enterprise-ready Gen AI products.
• AI assistant architectures are evolving! Rajaswa & Nitish from Postman shared how they are using a custom agent architecture that allows for controlled agency, fine-grained execution, and easy extensibility.
• DSPy is democratizing LLM prompting & optimization unlike anything else! Ganaraj is using DSPy at his e-com firm to normalize product attributes across millions of items, as well as for translating Sanskrit texts to English.

We wrote a detailed note about these latest trends and what they mean for your org.

 Read the article here.