agency-swarm

An open-source AI agent orchestration framework designed to automate and streamline AI development processes through collaborative agent swarms.

Agency Swarm

Imagine if you could have an AI agent that could run your entire life for you. What if you could just say, "Hey Sky, what's my schedule for today?" 

Your schedule for today: 5:30 a.m. to 6:30 a.m. meditation and journaling, 6:30 a.m. to 12:00 p.m. deep work, 11:30 a.m. to 12:30 p.m. meeting with Helen, Elizabeth, Wayne, Arseny, and Merlin Ravens, 2:00 p.m. to 3:30 p.m. workout. 

Okay, awesome. Can you check all of the emails I have unread in my mailbox? You have 10 unread emails. The latest ones include "Need more details" or "Want to address any of these now?" 

Can you please open that website that the client mentioned in one of the emails? The website has been opened. 

Okay, cool. And can you tell me what's on this website? On the website, you see an e-commerce store with a variety of kitchen products. 

Is there anything else you'd like to do? Yes, can you please tell the project scoper agent to create a project scope for this potential client? 

The project scoper agent has created a detailed project scope for the e-commerce client specializing in kitchen products. It covers objectives, key deliverables, requirements, timeline, collaboration, and success metrics. 

Would you like to review any specific section in more detail? Uh, no. Can you just save this project scope to a local file, please? 

The project scope has been saved to a local file named "Projector Scope Ecommerce Kitchen Products."

Txt so that's how it works. I've been saying for months now that the way we interact with technology is going to change soon. Today we are introducing agency form voice in interface which allows you to interact with your agents in a more natural way. But the best part about this project is how simple it is to customize yourself. So all it takes to add your tools and agents is literally just dropping them into either the tools or the agency's directory and that's it. 

And for those who do not know how to create tools or agents themselves, we of course also have the cursor rules file which will allow you to create all of the tools and agents for you. So let's Dive In. 

Okay so first huge shout out to indev Dan. This project was fored from his PC python realtime API assistant repo. However, with this repo there were a few significant challenges. Specifically, it made it incredibly hard to add tools, especially if you're a beginner. All of the tools in the repo are inside the tools.

P directory and in order for you to add a tool you would have to like create and modify code in like three different places. Because at first you would have to like create a function then you would have to create the schema then you would also have to upload it to your agent. So that's definitely challenging for not just beginners but even for pros. 

Which is why in our repo you can do this with a consistent B tool class interface that you can extend with your own capabilities. And then all you need to do is basically just drop it into the tools that trory and this assistant will instantly have access to that tool. So as you can see the code is much cleaner if you know how to use my framework. You know how convenient it is to use penic for tools. 

So the same principles apply here. Additionally, it made it incredibly hard to add agents. In fact, agents in this repo are not really agents they are just basically tools. So as you can see here it has like a tool that calls another lolm and it has a specific prom structure and then just returns the response from this lolm. 

But in ouro again you can just simply drop your agents into the agency's folder and then instantly this voice interface will allow you to interact with these agents. And it can not only interact you know with the CEO of your agency. So for example if you drop like an agency here with the COO agent on top it can still interact with other sub agents in this agency. 

And lastly you can't use assistance API here while of course in Agency for all of the agents are built with assistant. So it means that you have code interpreter and file search out of the box. So that's basically what we've done here. 

And now let me show you how to actually use it how to add your own agents and tools if you already have them. And then later we'll also do a step-by-step tutorial on how you can create new tools and agents with cursor. All right let's dive in. So the first step would be to clone the repo if you run into any.

Challenges here just ask Char gbt and then open it in cursor. After that the next step would be to install the requirements. So for this project to run you need to install UV, which is a modern package manager. I really like this choice to be honest and you can install it with just a single command. 

So you can either use Mac OS or Linux command. So just run this in your terminal and this will install UV. After that you would need to copy the n.

Sample file so just copy this n.

Sample file and here add your own open I key and also rename it into justn. So you need to copy this and rename it into n and then add your API key. Then you can also modify personalization and this actually another thing that we modified from the PC repo. In the PC repo it was hard to modify the instructions for your agent while here you have access to the assistant instructions. So you can easily know add your own information about know what you do about yourself and the tasks that you want this agent to perform and how.

Okay after that install Port audio. So just simply run this command Brew install Port audio in your terminal and finally just run the UV sync command which is like pep install basically in UV like this. You can already use this default assistant with some of the tools that don't require authentication. However, if you want to use tools that use Google cloud apis like for example the tool to check your calendar or the tool to check your emails and of course you can also build more tools on top of this with other Google apis, you would have to get the Google Cloud credentials.

So in order to do that we first need to create a Google Cloud project. So go to console cloud.google.com and here just select your project. If you don't have a project just create one it's not that hard. And after that you would need to enable two API. So first of all you would search for Gmail API in the search bar here on top and then initially here you will see an activate button or something. So just click on that button and then wait until the API is enabled. After that do the same for the Google Calendar API. 

Then click manage and here under o off consent screen you would need to configure your app. So just name it as the name of your project like for example voice interface and then when creating this app so I already have this app created but I think it will go through the same process here. So just click save and continue here on add or remove Scopes you would.

Need to add the Scopes from the the readme file. So here just copy all of these links right here and then add them under manually add Scopes. Just add it like this and then click add to the table. 

So do this for all of the Scopes. I've already done this before so I'm not going to do all of that. And then just click save and continue again. 

In your test users, just add email addresses of everyone who's going to be using this application and then click save and continue again. 

Okay, now what we need to do is just get the credentials file. So go to credentials right here and then just click create credentials here on top. 

Select all off client ID for the application type. Select desktop app, name it again as the name of your project and click create. 

Awesome! So after that click download Json and this will download the credentials.

Json file to your local machine and you would just need to rename it into simply credentials. Json. So like I have right here initially it's going to have like a very long name but you just need to name it credentials. Json and place it into the top level directory of voice interface repo. So yeah that's basically it for setup. This is literally all it takes. Now you can use it with the default tools. 

So we've added some example tools for you that you can run out of the box and to do that just simply run the final command which is UV run Main. Hey Sky how are you? Hello how can I help you today? Awesome so Sky is online and yeah feel free to play with this. Even like this it can be really helpful. I'm actually going to be using it as a COO. You can use it for a lot of different purposes actually just by modifying The Prompt. 

But if you want to take it a step further what you can do is of course also create your own tools and agencies. So if you want to add agencies as I said it's very simple. You will just like drop them into the directory. So let me show you for example let me take the agency that I developed on the previous video which is the content creation agency. So let me copy that and then go back here and just paste it into the agency's file. 

So now as you can see the content creation agency is here but what you need to do is also just change the UTS. So you can just select all the code and say change to relative Imports. And yeah that's the smartest way to use AI ever. As you can see I added three thoughts this is why open I chain this model for $100 million. 

So now we can run this again and this voice assistant will have access to this agency. So I actually got an error because the T package is not installed. So I just forgot to install the packages from the previous repot. So just make sure to copy the requirements.txt file also put it somewhere in the root directory and then just run UV people install dasr requirements.

Txt so this will install all of the packages from the requirements. Then you can run the UV main command again. 

Hey Sky, so what's the performance on my Channel today? The performance metrics for your YouTube channel today are as follows: one 5 AI Trends you must be prepared for by 2025 16,21 15 views. Two why structured outputs by open AI change everything 13,000 524 views. 

All right, amazing thank you thank you Sky. Okay, now let me actually show you how to build new agents in Tools in this repo. The process is extremely similar to what I have already showed you in my previous video where we built an agency from scratch using cursor without writing a single line of code. Here the folder structure is just a bit different but essentially the process is the same. 

So the first step will be to open cursor composer with command shift I and if you're on Windows just replace it with control. Then I'm going to ask it to create a new tool that will check all the unread messages in my slack because as a COO I also want to you know get a quick overview without having to go through all the messages myself. 

Okay, so very simple prompt you can play with the models. I'm going to use GPT 4 all for this one. Please create a tool that will check all my unre messages in slack. 

So as you can see it now adds a tool but here it actually added it into incorrect folder. So this can happen. I mean we're still working on the cursor rules file for this repo so hopefully by the time you use this it's going to be fixed. But if this happens just tell it to place it in the correct folder and the correct folder for this would be the tools folder inside the SRC voice assistant. 

So inside tools you would place tools that the voice assistant can access directly and inside agencies you would place any tools that the sub agents can access on its behalf. So let me now just tell cursor to place this tool under Tools in SRC Voice Assistant. 

Okay, so as you can see now the path is correct SRC Voice.

Assistant tools check and messages. So let's accept the changes. Let's go now check our new tool. So as you can see, it requires also Slack SDK. So if you need to install any additional package, you will simply do this with UV by running UV add and then package name. 

Okay cool, so let's see if these quickly lines disappear. And now the package is installed. So let's now run this tool to test it. I previously already added this Slackbot token into the N file. But again, if you don't know how to do this, simply ask cursor. It provides some pretty accurate instructions, but it's going to depend on the API that you want to use. 

Okay now let me run this file and see if it works. So as you can see, it says Channel not found because for some reason it decided to use like a channel ID in the parameters of the tool. So I'm actually just going to tell it to check all channels and output and messages at a time. 

All right, so this looks better. Now let's run again. So again, the process is very simple. You just tell cursor what to modify and then you run this again and test. And then if there are any errors, just add the error message back. You can select it like this and then just press command I. So this will add the error message into the composer and then you can just tell it to fix this thing. 

Okay, so the arrow is fixed. But now, as you can see, it's checking like all of the channels. So let me just tell it that I want to only check channels which I am a member of. Right awesome, so there is actually a way to do this. As you can see, sometimes you just need to hint these models on what EX exactly you want because this is a much simpler way than checking all channels at a time. 

Okay cool, so now it works. Total under messages is zero because I had some coffee this morning. And after this, you don't have to run any other commands. All you need to do is just run UV run Main and instantly the agent will have access to your tools. 

Hey Sky, can you please check the latest messages and Slack from.

Me here are the latest messages from the Artery test Slack channel. A few moments later all right so that's basically it. This is how you can create your agents and tools again without writing a single line of code. The process for creating agents in the Streer is the same. Just adjust the prompt and make sure it places it into the correct folder. 

So in conclusion after working with this realtime API I found it extremely similar to assistants. Honestly I'm not sure why OpenAI decided to create like a whole different API for this because the only difference between real time API and assistance is that assistance we have threads and in real time we have conversations. 

And obviously you can store the state of your agent on OpenAI site and you can't use the file search and code interpreter. So I'm really hopeful that in the future OpenAI will bring this API directly in assistance and this will streamline the process for us even further. 

So when this happens we'll probably add a new method in agency's form something like demo Voice or demo real time and this will allow you to instantly run your assistants using voice and interact with them in a more more natural way. 

So yeah that's it for this video. Let us know in the comments what you're building and share the AI Agent Store with colleagues.

Note, that there may be slight differences in text and video. Watch video for the original content.

Agency Swarm Video