Announcements

Stay informed about Portkey's newest releases, features, and improvements

  1. August at Portkey: 2 Billion Requests, Guardrails, Tracing, and More

    Announcement
    B2

    Last month at Portkey, we crossed 2 BILLION total requests processed through our platform. To think, we just started a year ago when this number was at 0!

    We're truly humbled to be your production partners and that drives us to continue innovating and making Portkey better.

    In July & August, we did 3 major releases: Guardrails, Conditional Router, and Tracing.

    Check out this release note to dive deep into everything that's new at Portkey.

    Like

  2. ✨ New Virtual Key Modal

    Improvement

    We released an improved flow for adding and editing your Virtual keys on Portkey! Head over to the Virtual Keys page and let us know what you think.

     

    CleanShot 2024-09-04 at 09.35.12@2x

    2

  3. Route Your Audio Requests Between OpenAI & Azure OpenAI

    Improvement

    We're thrilled to announce full interoperability of Audio routes between OpenAI and Azure OpenAI APIs.

    This update brings the comprehensive suite of Portkey Gateway & Observability features to all your Text-to-Speech, Speech-to-Text, and Translation requests across both OpenAI and Azure OpenAI routes.

     

    What This Means for You:

    Enhanced Reliability: Automatically route Audio requests to Azure when OpenAI API experiences issues.

    Improved Scalability: Tackle rate limiting by load balancing requests across multiple instances.

    Global Compliance: Direct requests to specific Azure regions to meet data residency requirements.

    ...and that's just the beginning!

     

    Let's see an example:

    Creating the fallback config

    Let's create a fallback config that calls OpenAI in case there's failure from Azure OpenAI API

    Fallback-Config

    Making a request

    Fallback-from-OpenAI-to-Azure-OpenAI

     

    Explore docs here:

     Text-to-Speech 

     Speech-to-text 

    Like

  4. OpenAI Structued Outputs

    New Feature

    You can now ensure that your OpenAI outputs are strictly following a supplied JSON schema. Portkey natively supports OpenAI's new Structued Outputs mode with our SDKs & APIs.

    CleanShot 2024-08-26 at 09.32.29@2x

    Just start sending the JSON schema in the response_format param, and Portkey will log all your requests and calculate cost, tokens, latency for them.

    Like

  5. Bring Your Agent to Production with Portkey

    Announcement
    hero-small

    Portkey now seamlessly integrates with major agent frameworks like LangChain, CrewAI, AutoGen, and others, making your agent workflows production-ready.

    With Portkey’s native integration, you can effortlessly incorporate our Gateway, Guardrails, and Observability suite into your favorite agent frameworks, and take them to production with confidence.

     Browse Portkey Agents docs here.

     

    Like

  6. Open sourcing Guardrails on the Gateway

    Announcement

    We are open sourcing our AI Guardrails 🚧 framework on the AI Gateway! Now, on Portkey, you can synchronously run guardrails on your requests AND orchestrate them based on guardrail verdicts.

    Portkey offer 50+ guardrails built in - everything from detecting hallucinations and retrieval accuracy to detecting PII, PHI, toxicity, and more.

     Browse Portkey Guardrails docs here. 

    Reach out to us on Discord to enable Guardrails for your account!

    3

  7. Anthropic's 12-hour outage

    Last Thursday, the Anthropic API was unstable/down for almost 12 hours:

    image

    We saw that orgs who had setup fallbacks for their Anthropic requests, they did not face any failures99.86% of their requests succeeded, and only 0.14% failed.

    These users had setup fallbacks to route their requests to (1) OpenAI, (2) Azure, (3) Gemini, and (4) a bunch of hosted Llama models.

    There are some key learnings from the 0.14% requests that failed though:

    1. Some users hadn't configured fallbacks for the 529 status code (which had spiked the most that day)
    2. A few had improperly set up fallback targets (expired keys, non-existent targets)
    3. In rare cases, even the fallback target failed (pro tip: always have multiple options!)

     

    Check out our fallback documentation to protect your app from going down with an LLM API failure again → Portkey Fallback Docs 

    Like

  8. LLMs in Production - Scaling DSPy, New Agent Architecture, AI Gateway, and more

    Community

    Last month, the Portkey team, in partnership with Postman, organized the first #LLMsinProd event in Bengaluru.

    image

    We had some of the brightest AI engineers, leaders, researchers, PMs, and founders come together to discuss the best methods of productionizing Gen AI systems.

    • Portkey's co-founder Rohit shared how the AI Gateway pattern is becoming a crucial middleware for shipping enterprise-ready Gen AI products.
    • AI assistant architectures are evolving! Rajaswa & Nitish from Postman shared how they are using a custom agent architecture that allows for controlled agency, fine-grained execution, and easy extensibility.
    • DSPy is democratizing LLM prompting & optimization unlike anything else! Ganaraj is using DSPy at his e-com firm to normalize product attributes across millions of items, as well as for translating Sanskrit texts to English.

    We wrote a detailed note about these latest trends and what they mean for your org.

     Read the article here.