From 0 to 2Bn+ requests on Portkey’s AI Gateway!

Listen to Rohit Agarwal as he shares lessons from building a billion-scale AI Gateway, in conversation with Shaw Talebi from The Data Entrepreneurs

Why do 75% of AI projects get stuck in the proof of concept stage? Rohit shares how Portkey is helping 600+ teams overcome the production challenges in AI deployment.

 📺 Watch here 

With Pydantic & Zod support, you can now enforce Gemini models to return responses that match exactly the JSON schema you define. No more unexpected outputs! 🎯

Currently Supported Models:

➡️ Gemini 1.5 Pro

➡️ Gemini 1.5 Flash

 Link to Docs 

This was a long standing demand from many users: The ability to organize prompt templates inside folders.

This is available to EVERY org now!

• Create multiple folders as you need
• Move any prompt from one folder to another

Try it out now!

Open WebUI (45k+ stars) is one of the most beloved open-source ChatGPT clones.

Now you use Portkey with Open WebUI natively. Connect to 250+ LLMs (ChatGPT, Claude, Ollama) through a single interface in Open WebUI.

Track costs, monitor token usage, measure response latency, and collect user feedback - all through 40+ on Portkey's dashboard.

Learn more at Portkey docs 

vLLM is a fast and easy-to-use library for LLM inference and serving, with State-of-the-art serving throughput.

Portkey now integrates seamlessly with vLLM.

Run Llama, Mistral, Qwen, and other open-source LLMs with Portkey to:

1. Save $$$s on your GPU costs using semantic caching
2. Full stack observability (logs, tokens, latency)
3. Built-in reliability features: load-balancing, fallbacks, etc.
4. Prompt management & versioning
5. Guardrails to protect your data

and much more...

 Link to docs 

🎉 Exciting news!

Portkey now supports Nvidia's Triton Inference Server.

Big Thanks to Tapan Sharma for contributing to Portkey's open-source AI gateway with the integration.

Triton is NVIDIA's inference server that makes AI model deployment lightning-fast and scalable across multiple GPUs and CPUs.

 Link to docs 

Set any message to be cached by just toggling the Cache Control setting in the UI.

We're excited to share that MindsDB now integrates natively with Portkey!

Easily connect your databases, vector stores, and apps to 250+ LLMs with enterprise-grade monitoring and reliability built-in.

Transform your existing data infrastructure into advanced, AI-powered systems today! 🚀  

 Link to docs 

We've added support for Google's latest Gemini models (1.5-Pro & 1.5-Flash) following their recent production updates.

Key Improvements:

• 50% reduced pricing on 1.5 Pro (for prompts <128K)

• Increased rate limits: 2x higher on 1.5 Flash & 3x higher on 1.5 Pro

• Performance gains: 2x faster output generation & 3x lower latency

Get Started:

•  Google Vertex AI x Portkey Integration 
•  Google Gemini x Portkey Integration 

Have you always thought of contributing to Portkey but did not find the right time? Hacktoberfest is your opportunity! We are giving away Airpods Pro & Portkey swag to to community contributors this month.

Read this blog for more info: https://portkey.ai/blog/portkey-hacktoberfest-24/