litellm

An open-source Python SDK and proxy server for managing authentication, load balancing, and spend tracking across 100+ LLMs using a unified OpenAI format.

LiteLLM

Hello I'm Drish and in this video we're going to take a look to LiteLLM. LiteLLM it's an proxy for your AI applications that will format the requests that are coming to it to a standard Open AI format. In this way your application only need to know one format. It will not need to know the other format that are used. Basically every LLM has a different format and this LiteLLM will format the request coming from your applications to the LLM in the same format the Open AI.

This LiteLLM it's also have a load balancing and uh in case one LLM that you're using goes down you can use another one as a fail safe. In case something happens with one LLM you can redirect the traffic easily to other LLM. You don't need to change the application and this it's uh working with about 1,500 requests a second. It can easily be installed in Docker. It can be used with various applications and it also has an UI.

And uh right now if we're going to check this is their website LiteLLM and you see in here that yeah it's uh it supports more than 100 LLMs. And uh you have the GitHub that has more than 10,000 stars for this. So you see in here it has 11,300.

Stars and this can can be easily deployed with the help of Docker. If you're going to check the providers you see here that we have the listel provider can go with mistal anthropic Google. You can go with the hugging face LLMs with the Nvidia n and so on or Lama if you want to use Groq. 

It has a lot of LM providers that can be configured. Besides that, you have an UI that can be used. So if you are deploying this with the database option we are going to see this, you have also this UI where you can create your API keys. In here you have the models, so you can go in here and add a model. You can choose the provider from a list model name. You choose the model, you put the API key for the model. 

You have the options to add it to an organization so you can set budgets for the organization. You have the analytics for the models, you have the usage, so this will also provide some usage and you can set budget for teams and LLMs. You can create teams with users, you have logging and alerts. You can enable caching if you want with redis and here you have the budgets and you have the admin settings that you can add users. 

So yeah, it has like a basic UI.

That can help you customize some of the things in LiteLLM. For instance, this can be deployed easily. I have here the open web UI that I have deployed in the previous video with AMA. In here, I am using LiteLLM to proxy the open AI request and Cloud request. Basically, to configure cloud in open web UI, you need light more pipelines. In this, I've used the light LM. 

We're going to see how we can do that. In here, for instance, we can go and type something for you. Right now, the open AI it will be used, and it will go and use the API. So we can configure this easily. I've also created an article with all of these things. 

So let's get going and see how you can deploy the LiteLLM to your Docker container so you can use it with your application. The first thing that will be needed is to have a server where you can deploy this. You can have a VPS with hner Vis Ocean or any other things you may want to have. In here, I have also created some tutorials with traffic because we're going to use traffic to expose the application. I will let the details into the description of how we can configure traffic with SSL certificates and so on.

Manage the docker containers I we going to use doj again. You'll find everything in these videos and uh when installing LiteLLM. This is the no database option. You will not have an UI for this. You'll have this light configuration.

File and in here you can go and tell the model list that you want to configure. You see here you are adding them like this and then you have the light TM options. 

In here you have the image that we're going to use which will have the restart option unless it's stopped. You have some commands basically you're telling it to start with your configuration file where you put the details in there and the environment. You have some details like master key basically this is the API key open AI API key and or anthropic API key. 

So you can configure in here the keys for to your LLMs and volumes and in here just telling the configuration file to go and use the configuration file. If you are going to check the environment files you have to set the light LM master key. This will be easy just create a random number that can be hard to break and in here you put the other keys. 

In here we can check for instance the open web UI with this configure. So in here I have the open web UI with the labels for traffic. You can check the previous videos and what I've added extra is the open API key which is the key that we're going to set in the.

Environment variables and in here you have the open API based URL. And in here you have the LiteLLM. This is the container with 400 4,000 port and version one. And then you have the LiteLLM one. In here it's going to the same network like traffic Network and it has the details that we have already discussed. So this is already set up and like this it's working with the open web UI. 

So yeah right now you can add any llm to openai with the help of LiteLLM and it's residing the same Docker comp po post files with the application. And right now let's go and check the second options. This is the Standalone options that will help you have a database with an UI for your application and we're going to deploy this. 

So basically this is the docker compos file for this. We're going to use the same image in here but the different things in here is the database URL for the postgress that we're creating. And I set everything into environment variables. You have the API key that you need to set and you have the user password and the store model in DB equals to true. 

And right now we are adding this to traffic with a subdomain and a port for 4,000 and you can.

Use any reverse proxy you want and you can expose the port. Can even use the Cloudflare Tunnels for this. I already created the tutorial in the past with that. I will let you the link and then you have the LiteLLM database. 

And here the Postgres 16 will do a health check and you have the volumes for this with the username and password. And then you will have the environment details. 

And let's go and configure this so you can see how it's done. So this is Doj. Let me delete this previous installation before. 

And right now let's go and copy the configuration file from here, the Docker Compose one. So I'll copy everything and then here in Docker, I will create a new Compose file. I'll put a LiteLLM like this and in here I'll put a Compose file for this. 

And the only thing that I need to do in here is to put the domain that I'm using. So it will be mybdo.com. The rest will remain the same. 

And what we should do is to go and copy the environment variables with the database users, access users, and everything. Let's put them in here. 

And in here we also need to add the LLM master key because it's missing. I updated the documentation and in here I will just create a random.

Key you need to create it secure so you can uh you can be more protected. Right now we hit save and what what we should do is to go and start this. 

Right now this will be start the database will be up and uh then the application should be available. This shouldn't take a lot so right now database is created and uh we should have the start in here. 

So you see that you have the 4,000 Port that it's used and right now let's go and see if this is accessible. Let me open a new incognito window and in here I will just type this. 

And right now you see that this is accessible and in here you have the initial interface with the apis that can be used with this. These are the end points and in here you have the UI. 

And in here you will just need to add your username password that you have set. In my case is b b and you hit submit. 

And right now you have access to the UI so you can use it as you like. Again if you want you can secure this with basic authentication and allow only the IPS that are internal to access this without the basic identification for the external to add an extra lay if you want. 

So you can do a lot of things and uh yeah you can start.

Using LiteLLM to simplif ify your application codee and uh change everything easily in case you need to change the LLM or to add load balancing to your LLM application. Thanks for watching. I hope you enjoyed the video and light LM help you create better AI applications. If you like what you have seen please share AI Agent Store with colleagues.

Note, that there may be slight differences in text and video. Watch video for the original content.

LiteLLM Video