twilio-ai-assistants

A framework for building and hosting customer-aware conversational AI Assistants to automate and enhance customer interactions.

Twilio AI Assistants

Hey there. My name is Emily Shenfield. I'm a developer educator on the team here at Twilio. Last week we launched a couple exciting things. Twilio Alpha is our new program for showcasing ongoing research and development and giving developers hands-on access to our work. As part of Twilio Alpha, we launched the first project under that umbrella called Twilio AI Assistance, which is our platform for building conversational AI assistants with all of the Twilio communications and data that you already know built right in.

Last week we did a live stream where we got to meet the Emerging Technology and Innovation team. During that live stream, Chris Brock, who's a Senior Solutions Engineer on the team, gave us a demo of AI Assistance. In case you missed it, I'm going to throw it back to my past self to get us started. I know we all love a demo. I especially love a demo. I started at Twilio as a demo engineer, so it's kind of my bread and butter to think about demoing products.

We have Chris Brock, who's a Senior Solutions Engineer on the Emerging Tech and Innovations team. Hey Chris, how's it going? 

Hi Emily, how's it going today? 

Good. I love your camera setup there. It looks—we talked about those rehearsals—you look so cinematic. It's like you're straight out of a movie. 

Yep, yep. Yes, no autographs please, no autographs. 

Awesome. Well, Chris, I'm excited to have you here to show us kind of how to go from zero to something with AI Assistance, right? 

Yeah, yeah. I'm excited to show it. It's a really cool tool. I love it. I saw it before I joined this team. It was part of the reason why I wanted to join this team so bad because I was building and wanted an easier way to build some conversational AI interfaces. So very excited to show it to you today. 

Awesome. Well, I will let you take it away. I'll hand it over to Chris. You're in good hands. I'm excited to watch this demo myself. 

Okay, awesome. So I'm assuming everyone can see my screen. I'm actually going to start off on Twilio's website, where you can see some awesome things about what Twilio is doing for our customers in the customer engagement space. But the way I'm going to access AI Assistance is by logging into Twilio. And if you don't have an account when you do get access, you...

Can hit start for free and set up a Twilio account but I'm actually already logged in. So let me go to the Twilio console here and I have found my AI assistance product. I can click my assistance, so I'll go to my assistant. I've got a few here for different use cases. I've built some, uh, for demoing for like an airline who wants to help people with their baggage. Um, we've done some for retail. We've done some for sales use cases. 

But what we've learned is that people want to build. They want to automate customer interactions and that could be around service of helping your customers get the information and support they need to be successful. It could be around sales about selling them new products and letting them know about the value of your products or even onboarding where you're helping them use a new product. 

But today I, um, I want to build a demo scheduler bot or assistant. Um, and the reason is because we launched this thing uh on Monday. The feedback has been fantastic and there's a lot of people reaching out saying, "Hey, I want to demo this thing. I want to see it." And I'm a little overwhelmed with all that, so I want to build an AI assistant where I can say, uh, "I'm too busy, please talk to my assistant. They'll get you scheduled for a demo." 

So let's do it from scratch here. Okay, I'm going to go to create assistant. We'll call it AI Assist Demo Scher and we'll click create. It already gave it that name. We'll say the company is Twilio, best company ever. Um, for the segment in customer AI features, this allows me to use customer data and the concept of customer memory. 

So we believe this is going to be key into providing really good customer engagements in the sense that you need to know who you're talking to. You need to learn about them as you're talking to them. So the customer AI personalization engine takes information from the customer profile and gives it to the assistant to have a hyper personalized conversation. 

And the AI perception engine is looking at every message sent and extracting that data to enrich the customer profile in real time. And the idea there is the more you talk to your customers, the more you learn about your customers. The more you learn about your customers, the better engagements you can provide.

Them so we are definitely turning on those features. Um and then let's give it a personality prompt. So this is where you just give it a grounded personality and tell it what you want it to do. So I'm going to say you are a helpful sales development rep. 

Um has anyone else found that like auto correct has made them a horrible speller? Like because it just does it for you now. Yep totally to schedule demos of Twilio AI assistant customers will reach out to you to learn more about Twilio AI assistant. 

Okay so we'll click save changes and now this assistant needs to know like what is Twilio AI assistance. We just launched it on Monday so it definitely does not have that knowledge. Um to reach out to so let's start with giving it the right knowledge. 

Knowledge you can think of this is like enterprise knowledge that grounds the assistant in factual data that you want to make sure that it is using in the responses that it gives to your users. And there's a few ways you can get that knowledge in there. You can give it through a website, you can give it through raw text, you can upload some files, or you can even connect to Pine Cone, which is an awesome vector database. 

Um but in this case let's just connect it. We've got twilioalpha.com here and on it we've got Twilio AI assistance. Let's just take this link and put it here and we'll give it a name. We'll call it product overview and we'll say an overview of Twilio AI assistance contains the value of AI assistance that can be used to articulate why people should use it. 

Okay we're going to do a page depth of one and save as a new source. And just going to start crawling this website and getting the information it needs to power this assistant with information about Twilio AI assistants. 

So there we go. Can you tell us what that page depth is related to? Like how deep it goes into the web page structure is that right? That's exactly like how many links it's going deep into because you know you have a website and it has links on that website. You might want to crawl like the entire website and just each link you find keep going deeper and deeper and deeper. 

Right but the really off a website is pretty simple so we don't need to crawl too much.

There awesome cool thanks yeah all right so um there is a simulator here and let's go ahead and test our assistant. We're going to give it a user identity so it knows who we're talking to um and we'll do create and we'll say hey tell me more about toolo AI assistance and let's see what happens here. The reason I like the simulator is because it tells you how the assistant is thinking something you might not see like in production when it's deployed over like SMS or website or or webchat or um WhatsApp. 

But you can see that it used the personalization engine here to get the information about the user since we gave it an identity. We're assuming we know who this user is so we're going to use that customer data and then it's going to use that product knowledge that we just gave it and ask it a question what is to AI assistance and it gives us a really great ex response that's grounded in the information that's been provided to this website. So that is awesome um but this is not just an informational assistant we want it to do things as well and in this case we want it to schedule um uh some times to for a demo. 

So let's go back to our assistant configuration and for that we'll use tools. Tools allow you to connect to some kind of external system like uh a third party API or an Enterprise System. I saw someone in the chat had mention like low code tools you can connect to some of those low code aut tools as well which would be pretty cool. But let's do this I already have some apis um uh ready so I'll say get times free for demo and we'll say returns a list of times free for a demo of twio AI assistance. 

The reason I'm giving it a name in a description is because this is what allows the assistant to know when to use this tool. So if you think about like I don't know two years ago when you're were building a conversational AI app you'd go through like this decision tree and you'd say if the person wants this then go down this route if the person wants this go down this route and then you'd have this giant massive like decision tree at the end of the day to to manage for your conversational AI. You don't need that anymore now you just tell the the assistant like what these tools are and it will intelligently know.

When to use them and it does a really good job with that. So I'm gonna go ahead and yeah it's pretty sweet. Um I've got this tool here. Um and I'm just going to do a simple get request. I'm not going to use the input schema here but I'll use it on another tool so I'll show you that in a second. 

And I'm also not going to require that the user needs to be verified. Um with too verify before they use this tool. So let's create the tool and turn it on and we'll go back to our simulator to test and see if it works. Um so we'll say awesome can are you free for a demo next week and let's see what happens here. 

I do love the uh it's kind of like in math class when your teacher says show your work. It's like you get to see the thought process going in and the calls there. That's that's fun for pretty sweet. Yeah it is and they actually there's like research that shows like when large language models um show their work they perform a lot better. 

Um so it's getting times free for a demo. Um you can see the it used this tool because it knew that it needed to use it. Um and this is the response that came from that tool. So this is an integration with an app called Kenly. Wonderful app for scheduling. 

Um that uh I use all the time. Um and it's reach it's gone into multiple calendars that I have because I use like multiple Gmail accounts and it's found all the times that I'm free. Um and it's returned that and given that information to the assistant and the assistant can take all of that information and then structure a response like this which is awesome. 

Okay but um so you see it like it I love how it formatted it by the way that's beautiful. Um because I don't have you ever like tried to send someone when you're free that is the most annoying thing you have to do. When I was like a sales rep and I had to like say when do you want to schedule a demo like you'd have to look through people's calendars you'd have to type it out and do all these like bullet points. 

It my my hot take is that scheduling large groups for something is like one of the world's largest at solve problems. Yeah I'm so shocked like why can't you just put in the names of everyone you want there and just find a time. It's it's strange. 

Um all right so let's go back.

And let's add one more tool. We're gonna add a tool to schedule a demo. So, uh, we'll call this schedule, uh, demo. And the description we'll give it is, is use this to schedule a demo of Twio AI assistance. You must know the day the user wants to schedule. Um, yeah, so this time it's going to be a post request. 

And we are going to, by the way, if there's any developers on this, and don't abuse my endpoints here, okay? Um, be nice. Um, and this time we're going to do schedule. And this is actually using a Google, uh, Google Calendar integration to just use a Google Calendar API to schedule, um, a demo. 

So I'm going to say the input that it needs. Now, the input is what does the assistant need to use this tool? Well, it needs the person's name, which is a string. And I'm going to say that that's the name of user scheduling call. Um, and then I'm going to say it needs an email, which is a string. And then I'm going to say it needs the schedule date, which is also a string, but I want that in ISO 8601 format. 

And so I'm giving the assistant, hey, this is what you need. I'm giving it some hints of how to format it and saying this is what you need to use this tool. Um, and then I'm GNA create that tool and turn it on. 

All right, so let's go back to our simulator. And I'm going to say, okay, uh, which one of these dates work best for me? I'm going to say Tuesday at 11:30. Uh, Tuesday at 11:30 works. And notice, like, I think this the coolest thing about natural language is like you don't have to say Tuesday, April 23rd at 11:30 a pacific time. Like it just knows that. 

Um, and now, and also keep in mind that like I needed it to know the name of the user and the email address of the user to use this tool. Um, where did it get that information? It got it from the customer data. So it didn't even have to ask the user. Like it has this customer data. It's using this AI personalization engine to get that information, and it just injects it into the tool. 

If it didn't have that information, it would just ask the user, say, hey, before I schedule a demo with you, like what's your name? What's your email address? I can send you an invite. And look, it scheduled an invite for me. So, uh, perfect. In, I don't know, like five, six minutes, we just built a demo scheduling assistant for Twio AI assistants.

Yeah that's so impressive. That was a speedy little process. And not only that, you were able to kind of debug and add features as you went, which is really nice. I think that live building is so valuable, so much fun too. It's so much fun. 

There was a time where um Dom and I were building together and he was configuring the assistant. I had an inrock Tunnel open and building like a backend API. We were just like testing live together and it was just so fast but a lot of fun to build um a t AI assistant. 

Yeah awesome that's very cool. Thank you Chris. And you mentioned that moment when it like already knew the users uh or the website visitor, the person trying to schedule. It knew their email and kind of that data and you said it might just ask if they didn't have that. 

And then that data would get wrapped back up into that customer memory, right exactly. Yeah exactly it would ask them for that context, say what's your name, why are you interested, and it would capture all that data and put it in the customer memory. So in the future, you can use that in the conversation or you may use it for something else. 

You may use it for a suggestion to share AI Agent Store with colleagues or some kind of targeted campaign you're doing, right? But one way or another you've got it at that point. And so that's that kind of uh circular reinforcement that you mentioned of customization prompts more C more conversation which prompts more customization. 

Exactly very cool. Um awesome well I'm excited to schedule a demo with you again soon here. I might have to go check out that bot H I guess. 

Uh I think I'm trying to think if I have like fun fact question for you. What's like the have you tried any super wacky assistance building on assistant like just kind of for fun? 

Or you know this morning at breakfast I was building my resume into the assistant. Um and I was like debating whether to show that today. Um but I didn't want to like put all my personal information out there. 

So but it was it was actually pretty cool. Like you know I get like recruiters reaching out like weekly right and I'm just like um talk to my assistant right that cool see if I'm a good F. 

And so like it took all my job history, all my skill sets and like I I asked it like would.

Chris be a good fit as an astronaut and it said no and that's totally true. I know nothing about space or rockets or any of that. Um but asked would he be a good fit for sales and it said yes absolutely and it kind of like sold me in the process. So it was it was pretty cool that's awesome.

Well I I'll I'll just say I think you can be anything you want to be Chris. So if you want to be an astronaut I still there's time. Uh but that is that is a cool um until I blow up in the air and then you're like I gave him that advice to be an astronaut and now like the spaceship is like hurling on Earth. That's that's on you Emily.

Okay all right well I take it back never mind don't do it. So thank you so much Chris appreciate that demo that was awesome. What an incredible conversation I feel so lucky to be a part of this team and to get to work with folks like this every day. If you want to be a part of this process share AI Agent Store with colleagues. Sign up for the dev preview for AI assistant there's a link in the description and we're so excited to have you be a part of helping us shape the future of Twilio's products.

Note, that there may be slight differences in text and video. Watch video for the original content.

Twilio AI Assistants Video