0:52 1 2:56 >> Good morning and welcome to AI Engineer Code Summit 2026. We're excited to bring together some of the brightest minds in AI engineering for a day of fresh ideas, new perspectives, and conversations shaping the future of how we build software. To officially kick things off and lead us through today's keynotes, please put your hands together for our MC, Corey Quinn, Chief Cloud Economist at the Duckbill Group. 3:32 >> [music] 3:43 >> Thank you all for coming. This is the last flagship event of the year for AI engineering. How are folks feeling so far? Yay, that's exciting. What have we learned? Anything good? Anything exciting? Anything terrible? >> Ontology. >> There we go. That's right, ontology. What does it mean? We're going to put a pin in that, and we're going to come back to it. I Corey Quinn. If you've never heard of me, you're A, what we tend to think of as being happy, and B, uh if you are curious, you can look at all the various things I do at my site, shitposting.ai, which in itself tells you everything you need to know about what I do. So today we are going to be kicking things off and having some fun talks with people who have interesting, useful things to say rather than me flopping my gums up here. Uh we're going to start by bringing up our friends from Swix. Swix, are you available yet, or are you still hiding back there? Terrific. Come 4:44 on out, please. And Leah as well. Oh, did I jump the gun? Oh, no, please, by all means. Swix, thank you. [applause] Thank you. Oh. I'm in. Uh it's so nice to have him MC as well. And give a warm welcome to Leah, General Manager of AIE. >> Hi, morning. >> Um [applause] uh yeah, you know, we we're so happy to to open the show for you. And hi to everyone watching along online. AIE code is kind of our vertical, very focused, smaller format you know, format on on coding agents. And I think it's been such a huge year in coding agents. There's also been a lot of different evolutions in terms of like how we think about what the mission of coding is. We used to have you know, in in specific track titles like coding agents and software factories and all these things, but now I'm talking about like system one thinking and and 5:45 raising your ambition and all these like very fancy higher level abstractions of of work. So I'm excited to see for you to see and excited to me personally to attend all these talks that are coming along. >> Um and I'm just going to talk a little about 2026 um with AIE. So we've had a really big year this year. We had our first event in London. Uh, we also graduated up into Moscone for our World's Fair. Um, we also had a bunch Oh, here we go. I I didn't know if it was minus. Um, we uh we had a um multiple partner events, which if any of you went to, uh we'd love to hear feedback there cuz it's a great program program that we're running. Um, we've also grown our team um substantially. So I think there was a slide somewhere. Oh, I don't know. I think they've >> Oh, there we go. >> Um, >> people. >> I think >> They're so much more excited about clicking through everything. >> Um, yep, we uh here we are. So uh there we go. So yeah, we've grown our team. We went from um four in the team to 20 this year, uh which was very 6:45 exciting. So this is everyone. Big shout out to the team. Um, as as you can see, the program is growing and um everyone here is responsible for that. Um, so yep. >> And I I'll slowly interject like why are we growing? It's because we want to provide a consistent product for the rest of you uh that like wherever in inside the major cities that you're working in, uh you have an AIE to go to. Um, and we try to spread out. You know, we're doing something like four flagship conferences a year and a lot of partner conferences around the world. >> Yep. Um, we also I think I just saw that. Um, our most popular talk, which just came up there, uh has almost 2 million views on YouTube. But point's not coming up. [laughter] And then we are also just about to announce 2027 um with a wider program. We're going to hopefully announce that in about 3 weeks time. And we are doing more partner events. We're entering into some different regions. It's very exciting for 2027. Um, a couple other things we've done this year is we've partnered with Major League Hacking um and they're going to be running a hackathon with us at every 7:45 event next year. Um, we actually have one this Friday. So if you guys haven't signed up and you want to come, uh please look on the website. Uh, we are also working with Neo 4J on A Kids event series also. Um, and that is happening this Saturday. So if you have got any kids between the age of kind of 6 to 14, uh feel free to come down. Again, you can find that on the website. >> Or >> aie.kids. >> Uh, >> I'm in my domain buying era. So we got aie.kids for kids and aie.sc for the schedule. >> Yeah. Yes. So and then we have one final event for the year, which is the Sydney event. Swix will be there. I will not. And that closes out 2026. So yeah, thank you so much for coming everyone. And we hope you enjoy the the next couple of days. >> All right. >> Thanks. >> Enjoy the show. >> [applause] 8:49 >> What a great way to get us started. What we have found here is that events like this don't happen without help from a lot of our partners. It is incredibly important that they are there and support us. We want to thank them for it. Beginning with our presenting sponsor, Google DeepMind. Can we get a round of applause for them, please? [applause] Oh, there they are. AIV challenge is always fun and exciting on that one. Stall, it's great. I also want to thank our supporting partners as well, without whom we would again not be able to be here. Normally it shows up there. Oh, there, sorry. Wrong slot. I looked left. It's right. Can we get a round of applause for them as well? Thank you all for your support. It's appreciated. [applause] 9:49 [laughter] Yes. And of course the gold sponsors as well, which without whom again we would not be able to be here. At least not in this venue. We'd basically be yelling out on a street corner at people, which while not unheard of is generally not the premiere experience you try to have for something like this. Our silver sponsors also. Thank you for them as well. One more round of applause for them, please. [applause] One of the hard things about dealing with the modern era in AI is it feels like this year has been about 10 years in technology. We look back at what we were doing in January, and it feels like it was almost medieval times. It The things that were best practices and made sense then now just seem absolutely bizarre. It's Things get moving so quickly, and it's easy to basically take a nap and feel like you've woken up and gotten like surprised by what happened while you were sleeping. Everything's moving faster. 10:50 And that is the theme of our first talk that we're going to have cuz you're not here to listen to me ramble. Um Natalie Serino is the co-founder of Gimlet Labs and is going to talk to us now about what happens when AI becomes 10 times faster. Natalie, thank you very much. Please come on out here. 11:26 >> Hello, everyone. It's so good to be here. Thank you for being with me this morning. First session of the day. I see everyone here is a morning person. So I'm here to talk to you about what will you build when AI is 10 times faster? We're all builders here, so we're all thinking about what's next. Just a bit about me. Um my name's Natalie. I'm a co-founder of Gimlet Labs. At Gimlet, we're building an inference cloud that's designed to produce much, much faster tokens at high efficiency, leveraging different categories and types of chips. At Gimlet, I work on our kernel generation side of things across different hardware types. Um another thing about me is I love agents. I use my coding agents for everything, but I really, really hate waiting. And one thing that given that kind of surprises me is when you're building an ultra fast inference cloud, people ask me this all the time actually. like isn't inference already fast enough? 12:28 And I'm like what? But they have a point, right? Because I can do things and we can all do things with our agents so much faster than we could have done before. For example, this slide here, I actually made pretty using my agent and it took me with my agent much less time than it would take me to do that by myself. I'm not very good at design. And so I think that from an end-to-end perspective, obviously we're getting more done, but in terms of is it fast enough? Like that's still a question. So let's take a look at what happened when I used my agent to make that slide look pretty. You can see the ugly before slide on the right. So we're going to press play here. And I said to my agent, can you improve the visual look of this slide? I love Figma. I use Figma for everything, so of course I'm going to use it with my agents. So what we can see on the bottom here is the timeline of what happened. And the text is a little small to see, but red is inference. Red is the time spent that 13:29 the model's in inference. Green is the time it's spent making tool calls and blue is the time that it's spent taking my input. And we can see that, you know, this this agent's doing good work. It's doing a lot of thinking, but I'm waiting here. I'm actually waiting so long I sped up this video 9x cuz I didn't want all of you to have to wait for the full turn to end. It ended up being about 2 minutes long. But we can see at the end of this after my one input, we have this nice pretty slide at the end. So, you know, on one hand this is actually much faster than a human can do. But on the other hand, in those about 2 minutes I have probably moved on to another task. I'm not really focused on this anymore. I've contact switched. And when we look to, you know, the inevitable progression of inference getting faster faster, I think it helps to look back in time and look back to the internet era. Um, I mean, we're still in the internet, but like, you know, what people canonically call the web era. And what we always see with a new shift is that with each new major wave, new 14:30 capabilities are unlocked. And so we do things that weren't possible before that technology, like using an agent to edit my slide. So I think we should take a step back to 2000 in the early days of the internet. I think many people here weren't even alive, so it'll be a history lesson for you. And we're going to look at a little application called Squirrel Mail. Squirrel Mail is actually open source, so you can run it today. It was a web application for you know, doing your email back then. So on the left, what we have here is I'm running Squirrel Mail and I'm running it with simulated 2000 dial-up speed. And then on the right, it'll run with 2026 speed, so the speed that we're used to today. So the both sides are going to take the same actions, but the left side I'm using it with that simulated lower speed internet. And we can see that, you know, it's doing its job. Back in 2000, loading an image like this would have blown people's minds, but our modern eyes are like, wow, that took 15:31 forever to show up. And so we can see these lags. We can see a noticeable thing, but on the same time, someone might have looked at this back in 2000 and said, this is fast enough. I don't actually need this to be faster. The thing is though, all of the major giants back in that era did find out it mattered. Pretty much every major company from Amazon to Yahoo to Google to Bing, all of or Bing at part of Microsoft, like all of them found that even very small delays cost them users. It cost them people coming back. It cost them people doing as many clicks. It cost them literal money. Even 100 milliseconds of delay added on to the typical experience had a demonstrable impact on the bottom line. And even when you got rid of that delay, those same users didn't come back to the platform as often as they did without that delay. And this makes sense because fast products keep us in flow state. They 16:33 provide a stickier UX. About 100 milliseconds feels instant to us. 1 seconds 1 second of wait, I still keep my train of thought. It's not a big deal. I don't really notice it. 10 seconds, we're going to start turning to something else from a psychology perspective. Maybe we pull out our phone, maybe we switch to another agent. But today, from data from June, the median coding agent turn is 63 seconds. Each workload has different speed requirements. We don't need the same speed for everything, but over time our expectations do rise. And the fastest days of fastest internet in the squirrel mail days would feel incredibly slow to us, and we can keep we can expect that trend to continue for inference speed as well. There's a problem though, and the problem is everyone knows that we have massive compute scarcity. Today's infrastructure is under incredible pressure to deliver two things at the same time. More tokens and faster tokens. Unfortunately, these goals are fundamentally at odds because faster 17:35 tokens require more energy to produce than slower tokens. So with the same energy footprint, you can't make as many of them. Back before this was a session and not a keynote, thanks to upgrading this to a keynote, everyone, um we uh we're going to play a game that I made uh called Token Mint Factory. And you can actually play this. I will not be offended if you play it right now even, but just so you know, it does not work on phones. It only works on your laptop. And basically in this game, you're running a Token Mint Factory. You're printing literal tokens. And there's different modes. The modes are you can play to maximize how quickly you can put out one single token or you can maximize how many tokens you can put out in 60 seconds. Or you can play the SLA mode where you have to maintain a 20 second SLA on each token for it to count, but you're still trying to put out as many as you can. And as you play, you'll quickly see that the strategy is extremely different for doing fast tokens versus doing slow tokens. And neither one is actually better or worse from a decision 18:35 perspective. It's just a matter of what the application needs and what makes sense for your factory. So I took real runs from people at my company that played this game and I made a Pareto frontier of these tokens in the game. And we can see there's a real life Pareto frontier of this game as well. If we have faster tokens, we don't have as many. And if we have more tokens, they're not as fast. But there is a way to get more up into the right. And up into the right is where we want to go. We want more faster tokens. I don't want to compromise. I want both. So the game has a concept of adding upgrades to your equipment. And what we can see is that compared to the no upgrade baseline, these technological innovations help drive us up into the right and get more tokens and faster tokens. So how do we upgrade our real life assembly lines, so to speak, to make making our actual tokens faster? Uh one thing that we do at Gimlet for this is we really believe that you have to use the right accelerator for the 19:36 right job. And so we actually take models and we split them up into different pieces and we run each piece on a different category of accelerator. So we might run one piece of the workload on a GPU and another piece of the workload on something like an SRAM centric accelerator. And we find that even with the network overhead, this delivers dramatically faster tokens. It's essentially like the same thing as the upgrade to the equipment in the token factory when you better match the accelerator to the job. Unfortunately, we have ended the time of free lunch transistor scaling, right? So now we have to look to other approaches to make things faster. And so with this, it's a little hard to read, but we can see the same real life Pareto frontier for a frontier model. If you look at, you know, simple uh homogeneous GPU inference, it's very high throughput, and then the latency drops off a little bit. You know, you can't get it as fast cuz the y-axis is how many tokens that I can make, and the x-axis how how fast these tokens are for each user. And we can see that depending on how you 20:37 split the model up, because there's different ways of doing it, you can get much faster tokens for the same power footprint. You can get many more tokens for the same speed, or you can get more and faster tokens. So as an industry, we need to continue pushing, pushing, pushing on this because otherwise our poor data centers are just going to keel over. So I just wanted to show a quick demo of this. So um using a coding agent, build the same app using heterogeneous inference that leverages different chip types versus the traditional one. So this is going to be the game Snake, and we can see on the left the heterogeneous fast one's done. I can stay in my flow state. I can keep adding things. I can keep playing. But on the right side, it's still going. And by the way, I still think the left side's too slow. We need to make that even faster. And then we can add a competitor to the game, and we can see even with the tool calls, even with all the things that it's doing, it's going to complete, and 21:39 then the other one will still be hanging because it's fundamentally about how can you drive more efficiency and better performance with the same resources. It's going to go on. I'm not going to make you guys wait for this. One thing that people often point out too is is inference the bottleneck? Our tool calls the bottleneck. And I think this is a great point. If we look back to the turn timeline from the time that I was spending updating my slide, we can see that 80% of it was inference and 18% of it was tool calls. If we simulate that same exact thing and have 10x faster inference, we can see that 67% of that will be tool calls at that point if we don't optimize the tool calls. That's why it's important that we have to constantly attack the bottlenecks of the application and get the low-hanging fruit where we can. Today, people are focused on inference speed, and maybe we haven' optimized the tool call side and how tools are called as much because it's not the bottleneck. If I cut away the tool calls on the upper one, it wouldn't even really matter to the end 22:39 application. But once we get much faster, we have to think about how to call these tools much more efficiently and much more performantly because otherwise they're going to be the reason that we're all waiting around. I think the good news is that optimizing these speeds and what we're all collectively doing as an industry is a verifiable task. If I know what my inputs and outputs should be, I can have agents do this very very full stack. And that's where we're increasingly moving to. We're having self-improvement loops with these agents to drive better efficiency on the fly as our inference is running. And at Gimlet, we're building recursive self-optimization for this that pulls from low-level telemetry from our data centers all the way up to the model layer and trying to use all of it to to improve the overall efficiency of the workload because if you just optimize one small piece, it might not make that much of a dent. I want to show another demo because I think that cyber security is on everyone's mind right now. Um we're all 23:40 seeing like the huge incredible advances in AI driving awareness that there's all these vulnerabilities in our software. And I think that this is an area where faster inference can really really help. So this is a dumb fake scenario, but it's a real application and a real demo where imagine that you're running the dog club, and that's a website for dog lovers to upload and vote on pictures of dogs. However, unfortunately, like we often see, one of your node dependencies has a vulnerability. And this is a real vulnerability. This is really a node tar. And you're running a security agent on your application, and it's trying to monitor live traffic and see is there anything going wrong that I don't know about. This agent doesn't know about this vulnerability, but it does know its job is to protect your application. So, let's do a quick turn with normal user traffic. We're going to upload a dog biscuit. And on the upper side, we have the 24:41 application logs, and on the bottom, we have the defender. And the defender is monitoring, but it's not in the critical path because it's not fast enough to actually respond to every user request live. It's monitoring things async. And we can see biscuit gets uploaded, and we see that this is an okay, you know, this is an okay upload. The defender doesn't see any problem. But now what we're going to do is have this defender operating when there's a malicious upload. And this upload is exploiting the real vulnerability where in node tar, there was a release where you could basically overwrite files that you weren't supposed to overwrite based on the destinations of what was being unzipped. So, in this one, we have Ranger, a fake, bad, uh, malicious dog, not really a dog. And what we're going to see is that the defender is going to figure out that this is a malicious thing. The agent is smart enough to detect that it's doing something that it shouldn't be doing even though it hasn't seen this exact vulnerability before. But unfortunately, cats malicious upload 25:43 has taken over. So, your website's gone. It's too late. You didn't react in time. But if we retry this same exact malicious upload, but this time with an ultra fast defender, and again, this is a real a real demo fake use case, um, we can see that with ultra fast inference, we can react to things and do things online much more quickly. And this should change the way that we think of how to use agents in the first place. We're all thinking about how to use them in an async way, but we should think about how to use them in online ways as well. So, we can see on the bottom that the worker blocked their request, and it was like, what are you doing? This is obviously wrong. So, faster internet took us from squirrel mail to on demand high resolution streaming, right? Like, if you just look from this to this, it's crazy. If you showed this to my like five-year-old self or something, I would have been like My mind would have been blown. And this is just one of many things that I can select. So like what does a 10,000 26:45 token per second world look like? We're at in the like low hundreds today. What does it look like when we get to a million tokens per second? Like what does that unlock for us? And I think that we should all ask ourselves this too. It's important in AI to skate to where the puck is moving, right? It's changing so fast. We need to be ready. We need to think boldly. It's not enough to just do what we've always been doing. I think we've all had the experience where we make an audacious request to our agent and after the latest model release, it just works. And it's going to be the same thing with latency. And so I think that we can think about ideas that are both coming soon when we hit a certain latency threshold, what we can do. But also the stuff that is real life sci-fi. Like if you look at something like Jarvis or something, there's no way today's inference could be fast enough to power that experience for Iron Man, but we will get there. So I'm really excited about online use cases, online security, online things of like I want to just tell my agent what I want to build and it's building it right 27:45 in front of me and I don't even have to type anything. I think we're going to have new patterns of interaction. You know, think tab complete for everything. How can we have the user side of things be as quick and immediate as possible because the inference side is also just as fast to accommodate it. And we're going to have so many more agents. We're going to have even maybe Transformers driving robots, right? There's so much that we can build when this gets unlocked. And I think as builders on the frontier here, we should all be thinking about this too. Thank you so much for being with me um this morning. Uh you can find me on X at N Serino. Um we're hiring at Gimlet. Would love to talk to you all. And then if you want to play the Token Mint game, uh you can also find that QR here. Thank you so much everyone. >> [applause] 28:42 >> As AI models get better, are our old best practices holding us back? To rethink how we work with Claude code, please welcome Lydia Halle, member of technical staff at Anthropic. 29:14 >> Hi everyone. Oh, my microphone already works. Sorry. We have some technical issues this morning, so hopefully everything works. 29:56 Yeah. Have you all tried Fable 5.2 yet? Yeah? Anyone? How is it? Like it? Yeah? Oh, it's 5.5 better? Worse? Different use cases, right? Cheaper Yeah, cheaper for sure. >> I decided to hide the Figma. It's all It should be good. >> So this should be on that screen. >> Yeah, we're looking down there. Right? Are you seeing anything, Chase? 30:33 Let me see. >> So this was to be on my We need like Claude for IT. >> Yeah, we got it. 30:51 >> Let's make sure to hide those >> I want this to go small so we can drag over again. >> You can drag it. >> Start your slide Start your slideshow. Good. And then drag >> Where? >> It's not It's going to show up down there. Can you see anything yet? 31:27 I I did what you said. 32:01 >> Okay. 32:13 >> [snorts] 33:07 >> Did it just die? >> Can I unplug it real quick? >> Yeah, please. I think it just died. >> Oh my gosh. Okay. I mean, I don't know what to do. It shouldn't die. It had battery. >> Yeah, now it now it it it's just died, I think. >> Should I just do my talk later today and we fix it? >> We don't know. >> [snorts] >> That's fine. I fine but I have one make sure that was over there scheduled. 33:53 We do a little break and then the next guy can go. Okay, y'all. My laptop just died. So I nothing is happening. I don't know what to do. We uh Oh, it just restarted. Anyway. I I feel like I've just spent 20 minutes just waiting here and I don't want to take the next guy. But TL;DR, we went from being super in the loop with our cloud code sessions to, you know, running these long running sessions that run for hours in the cloud with these models like that are so good at verifying themselves that the role of the human has really shifted. Let's see. 34:38 I don't think this is going to work. 34:51 >> That's scary. 35:02 Also, today make sure you come by our Anthropic booth. We're there all day. We have good merch. We also have good demos and just fun stuff. We have like a photo booth there as well, which is fun. And you'll see me there as well if you have any questions. First you got to make this work properly. Ignore. My Why did it restart? >> I don't know. >> Windows Server Experience Problem What? Oh, Apple. Okay, now I just got to Just want to make sure I'm not >> People often wonder what the story is behind what's the future of engineering in this space. Uh and it turns out that fixing the computer and making it go It's It's always a fun story. Like the more advanced technology gets, the harder it is to basically make it do your bidding. And conferences are conferences, AV is AV, and laptops are always going to laptop. Uh I'm told that this might be the first time that an Anthropic employee has not been prodding an agent and to wind [laughter] up paying the computer to do 36:02 that bidding. >> That's what I'm saying when you clap for IT. And I see the little cloud robot helping me here. Oh, let's see. Exactly. I don't want to step on your toes, but I can also keep running my mouth. Some people run agents, I run my mouth. It There's a production joke in there somewhere, but I'm too tired to find it. Things work in theory, but not in production. That's uh why I call the staging environment theory. >> It's all black. Okay. Can you show us in the DSMs, please? >> [snorts] >> Okay. >> Oh, it looks like you're up. Maybe. >> No, I just have to see without my glasses for me. >> Remember, every code review is adversarial if you're a big enough jerk. >> Yes. Wait. No, no, no, no. >> Oh, that's not >> One second. Sorry. It was Yeah. >> And of course, the number one reason to found a company is that its co-founder main becomes your feature branch. 37:05 Good luck everyone else. 37:20 >> All right. Tonight, I want to see that on my screen real quick cuz I don't know. Yeah. No, no, no. It's not showing. What should I go to? We're going to I just don't want to Um >> Say want to >> Can you close your mic so everyone can hear us talking? 38:10 >> Okay. >> Oh, okay. >> I'll see you later. Hopefully, I'll come back. You'll hear from me. >> [applause] >> Slight change in programming here. I believe our next speaker is Michelle Chen. Correct me if I'm wrong on that. No one is yanking the the cord there. If you can give us about 3 minutes, we will take a small pause here. Besides, what are you all doing listening to this? You shouldn't be poking and prodding agents and seeing how many more you can wind up going. The tokens aren't going to burn themselves. We need more bandwidth. >> Oh. Suddenly, the use case for local inference makes itself extraordinarily clear. Conference Wi-Fi. There we go. I I've done it on planes before, and what made that fun was suddenly my laptop became what sounded like the third engine on the jet, but it worked, and it wrote better code than I did. Uh that's not a high bar, but it does mean I'm not going back in time to be the old-timey typing things out by hand like 39:12 some sort of agrarian farmer. So there's that. All right. Next up, we have Michelle Chen talking about Cloudflare is a Neo Lab. Uh personally, I believe that it's a database instead because I think everything's a database when you hold it wrong, but for some reason I wasn't consulted. Anyway, Michelle, thank you very much. Please come on up. >> What does it mean for Cloudflare to be a new kind of AI lab? To explore new models, edge AI, and the ambition to become the agent cloud. Please welcome Michelle Chen, Group Product Manager at Cloudflare. 40:09 >> Hi everyone. My name is Michelle Chen. I'm so excited to be here today at AI Engineer Code Summit, chatting a little bit more about why Cloudflare became a Neo Lab. So at Cloudflare, I am the group PM here. I run on the product team working on products like Workers AI, AI search, AI gateway as well. And so today we're going to go through this little journey of how Cloudflare came to be. You might have heard that we launched and trained some models recently. I'm going to tell you a little bit more about why and what you can be doing with it. And so we're going to go through a little bit of a philosophical journey. Uh when I was preparing for this talk, I wanted to use this quote, and then Tropic like launched a blog post last weekend and used my quote, but I'm still going to use it anyways. Um, so the quote goes in 1 second. Um, the quote is from Kierkegaard and it says, life can only be understood backwards, but it must be lived forwards. And I think this is really really apt quote here for us because at Cloudflare we're a 16-year-old company and we've done so many things over the years, but somehow we got all some of the pieces right and 41:11 we ended up having to become a new lab. So we're going to go through a little bit journey. The first thing we're going to do is look backwards and talk through why Cloudflare had to uh became a new lab. Give me 1 second here. Perfect. We're going to go backwards [laughter] and talk through why Cloudflare became a new lab. So Cloudflare is a 16-year-old CDN company. We just celebrated our 16th birthday uh at birthday week this year uh in September. And we first started as a CDN. So as a CDN company, what you have to do is you have to serve content as quick to your users as possible. You sit between the users request and the origin and you help uh sit in the middle and you can do things like cache and protect, do security and things like that. So that's kind of how Cloudflare started. But Cloudflare's infrastructure had to be adaptable in order to become the CDN. And so what we did was 16 years ago in 42:11 2010 we launched with our own infrastructure. Our fast CDN had to be everywhere. And this looks really different than than a traditional hyper scaler where compute is centralized in US East 1 or US West 1. Cloudflare's infrastructure is a completely different paradigm. We have data centers all around the world as close to the user as possible. And this is really important in becoming a CDN. But now we have these computers basically running as close to the user as possible. So we thought to ourselves, what else can we be doing with these computers that people would want to build with? And so came compute. We started launching these compute primitives as well. And so in 2017 we launched a product called Workers, which is basically like serverless functions that you can run. It contrasted VMs at that time where you had to rent a whole machine and pay for the whole machine, either you're using it or not using it. And so with workers, you just execute some code and then the worker goes away. You don't have to pay for it. It takes milliseconds to start up, not seconds as well. And then we followed with other compute primitives like durable objects, which are more stateful. And then in 43:12 2025 we launched containers as well because there's some workloads that do need something like a VM and we want all of those workloads to be able to run on Cloudflare. So now, you know, 2017 start of our developer platform, we have these compute primitives. People started using it. People started building more like static websites and things like that. So we thought to ourselves, how do we elevate this and take it to the next level? So we introduced storage as well. And so we have these storage primitives on Cloudflare's developer platform. We have KV, we have R2 dot object storage, we have databases like D1 as well. And the idea here is that people should be able to build full stack applications all on Cloudflare. Stateful applications where people can go on and start building um full applications and web apps and platforms on Cloudflare's developer platform. So we have this infrastructure. We added compute primitives to it. We added storage primitives to it. And then came 2023. And if you guys remember what happened in 2023, ChatGPT kind of blew 44:12 up a little bit and it changed how we think about AI, right? And I remember it being birthday week and uh three weeks before birthday week, a lot of us sat together in a room and we were like, what should we do now that ChatGPT is on the rise? What can Cloudflare play in here with our infrastructure and our primitives in order to be in this AI game? So in 2023 we launched a product called Workers AI where we added GPUs to our network. And so again, this is really distributed where GPUs are running as close to the user as possible. It's not centralized like a US East 1 hyper scaler for example, but it's running as close to the edge as possible. And we had a hypothesis when we started Workers AI. Our hypothesis was that were going to get smaller so that network latency was going to become more and more important. We launched with small models that could fit on our small hardware around the world that were super distributed and we thought that people would have models get better and faster and smarter and that network latency was going to play a much more bigger impact here. 45:15 Well, actually the opposite happened and models got larger and larger. So we went from something like 7 billion parameters, which was frontier scale at the time, to being the 3 trillion parameter models that were getting launched. And I think this is not so shocking when you look back in time because, you know, as we're on the way to AGI, um, uh, if you're following scaling law, as you train on more data, as the models get larger, they're more generalizable. And so on the path to AGI, it makes sense that we're pushing the frontier of research here in order to get larger and larger and smarter models. So this goes against our hypothesis, right? That the models were going to get smaller. The models actually ended up being larger and the network latency mattered less and less. So what did we do? We ended up pivoting. So Cloudflare got really good at serving these giant models as well. So in March 2026, we launched Kimi and then we followed up with GLM and DeepSeek and these large trillion parameter models um hosted on our infrastructure. And so this is 46:16 serverless. People hit an API call and get a token back. It actually took us a while in order to decide to do this, right? We March 2026 is actually pretty late in the game to to serving this model, but we were trying to make a conscious decision here on does this actually make sense for us to be serving it? It didn't really fit with our infrastructure story, but it fit really well with the developer platform. So in 2026, we had all these primitives in place. We had the infrastructure, which is unique to Cloudflare, running close to users. We had compute primitives where people were building agents on top of. So workers and durable objects, right? As your agent needs to complete a task, it goes was and it spins and does a task then sleeps when it doesn't. We had storage primitives, so your agents can actually interact with data and storage and then we had models, these large models that were powering agents. And so all of this fit into the Cloudflare agent cloud story. And so we started telling this narrative that we want to be the agent cloud. We want agents to be deployed on us, powered by models on us, using our 47:17 compute and data primitives, all built on our infrastructure and our network. But it never felt quite right. We felt like we were always missing a piece here because the infrastructure pattern didn't necessarily fit in with what Workers AI was designed for. And so I'm just going to sit here and give you my second philosophical quote, which is a risk considering that I'm a girl in Chinese and AI, but I'm going to give you this Chinese quote anyways. [laughter] And uh it basically says which means everything is ready here. All that's missing is the east wind. And one of the things uh about this is like you can't really predict where the wind is going to come from or how it's going to come, but you need to be very reactive when it happens. And so in September 2026, the east wind came. TypeSafe launched a model called Jev and introduced this idea of system one models. And it contrasted what we typically knew with LLMs, which were autoregressive and taking a long time to generate tokens, it scales 48:17 quadratically, etc. A system one model is a completely different paradigm where it actually just takes one forward pass through the inputs and generates an output that you can use to make decisions with it. And so I really think that this came down to perfect timing. I think people were starting to get fed up a little bit with LLMs because they were so costly, it costs so many tokens, it's super time consuming. And I don't know about you, but when I read a reasoning trace, sometimes my blood pressure rises when the LLM doesn't get the right the things right. And so I think we were sending everything to LLMs and started getting frustrated with it. And so when this system one and Jev paradigm started coming on board, we were like this makes so much sense for us to adopt, right? We don't have to send every request to an LLM, We can send some of it to a system one model. And so the system one models actually end up taking work off of the giant LLMs, you give it to the small fast models. And the two are supposed to work in tandem. supposed to complement each other and get to the end goal faster. so if you look, we are back at these small and fast models, right? This was 49:19 the original Workers AI hypothesis. And so Cloudflare was not wrong here. We actually were just early. With these small and fast models, the network latency became important again. We want the model to be responding fast, but also the network latency to be even faster. And so we decided to do something here. But Jev was closed source. Cloudflare couldn't actually host the Jev models and take advantage of our infrastructure. So what did we decide to do? We built our own version of Jev. So Jev launched on September 15th. On September 18th, just three days later, we started experimenting and we launched our version of Jev, our first experiment of it. And then it took us a week of deliberation to decide, do we actually want to be going into this neolab territory? It's completely new to us. We don't know what we need to know. It, you know, are we going to set the expectation that we want to be a neolab and are we going to live up to those expectations? But we decided to do it. I remember on a Friday evening we were 50:19 like, let's just go for it and we did it. We ended up training the model over 48 hours over the weekend and then we launched it the following week. And so we ended up with a model called Clef. Clef gave power back to the agents to be able to make decisions on their own. So agents could ask act more autonomously. You didn't need to have an LLM reason through for minutes on if it should make this decision or not. You also didn't need to put a human in the loop and be interrupted by like a human's sleep schedule when you want your agents to be running 24/7. So you can now programmatically encode, decide, make decisions for your agent to take action on. So these decision models countered LLM models because they were a lot faster and they were also a cheaper as well. And so it actually as an inference provider takes a lot of pressure off the LLMs as well, because the LLMs were reserved to work for meaningful work, whereas the clef and uh decision models could be used for fast um intuitive thinking and work, right? And so we also went back and thought to 51:20 ourselves, you know, what else do agents and people use for LLMs for that are overkill? So we came up with decisions and um there must be other workloads out there that are unsuitable for an LLM, both because they are either too expensive or take too long, or maybe that it just would get the answer wrong because the autoregressive nature of an LLM means that it would hallucinate. So we also launched a model three weeks later called Cleric. We actually ended up launching four models in the span of three weeks, so I think we did a pretty good job as a Neo Lab. Um but Cleric is a new model paradigm. So it's completely new. We're calling it an extraction model, but it's a system one model in that in the sense that it is just as fast and just as cheap, but it is also doing like one forward pass through the user. And the purpose of Cleric is when you're interacting with the world, you're not interacting in perfectly typed JSON objects, right? Sometimes you're getting an email, a tweet, a Slack message, but those people are not filling out perfect forms for you to extract inputs in out of. So what Cleric does is it can take unstructured input 52:22 and it can extract, compute, estimate, and forecast values from those inputs, and it gives you back a perfectly typed JSON objects, which means the agents are able to do things with it. So we're kind of giving agents the ability to read here and take action with clef as well. And so in code again, you're giving Cleric, you know, typed values from unstructured data. You can pass it in whatever kind of unstructured data. It can also take image input as well. And so you can just tell it to read basically what's in text here, pass it to clef, decide what to do with this text, and then pass it to an LLM to decide what to do with the agent. And so Clef were had to become a Neo Lab here. This was our missing bit. We had the infrastructure, we had the compute primitives, we had the data primitives, we had the large models powering the agents, but in order to take advantage of all of that and our infrastructure that we were built on, we had to be a neolab and ship these system one models that fit perfectly with our infrastructure. And I think one of the more important things is that we're not 53:22 limited to this. Being a neolab means that we can create what we are missing, right? It means that we're not uh if we're missing a system one model because we're um at risk of things being closed source or regulatory risks or things like that, we can actually create and invent things. So I think being a neolab has actually opened a lot of doors for Cloudflare to do more in the AI world. So we went through backwards, we went through a little bit of a history lesson. I want to go forwards as well. I want to teach you guys what you guys can be doing with Cloudflare Cleric. And I'm cognizant that I am at AI Engineer Code Summit. So in a in a room full of developers and engineers, how can you guys leverage Cleric and Cleric and system one models in order to better use your workflows? So we'll do a quick demo here. Um it is in let's think you have an agent, you're triaging like a CI build that went wrong, right? Um let's say you have 40 tool called tools that are available to this agent. You have your PR that went wrong um and then you have your agent. Maybe it's running Py Durable on a durable object. So it's all of this is 54:23 built on Cloudflare. So this is the setup here. Tech typically what you would do is you would take an LLM and in the LLM would take a tool make a tool call, the tool call would go back to the LLM, the LLM would make another tool call, etc etc until the issue is resolved. So this is like five back and forths with GLM 5.3. You're sending 40 tools every single time and maybe you're blowing up the context, right? Because you're sending the same kind of inputs back and forth to the LLM. And so what if we augment this with system one models? With system one models uh this could look a lot different. And so let's say we where we want speed, so we pick Clef Flash. Clef Flash can actually help you prune your tool list. Maybe you only need 10 out of the 40 tools that are available, so you only need to pick 10 of those. Then let say you take clerk and you use it to extract the values from the input. So it can get the PR number without reasoning and thinking about it too long. Um and then let's say you just, you know, orchestrate some code here to execute these API calls and you can actually just, you know, get those 55:23 answers back. Then you ask clerk again um to pull the data from the logs. So normally when you're using with working with an LLM, you probably just paste in all of the logs, right? Like all 300 600 lines of logs. But clerk can actually just go through and read those and say here's the exact error, here's the error message, here the lines of code it's affecting. And you're basically shrinking your context lot. And then finally you use an LLM to actually do take action and um make decisions and take action and extract values all together. This same agent can run the system one. Let's do a little bit of a live demo. Let's see how it goes. We're going to have to click run both here. And this is a live demo and I will say that Cloudflare our workers AI is uh has a very diurnal usage pattern. So we're in the middle of the workday. So we'll see how this how this goes. But it'll give you some illustrative example here of how it goes. Uh and we're also dealing with conference Wi-Fi. So so bear with me here. All right. The system one model side finished already. And all LLM one 56:25 is is going. So I think even illustratively you can see that the uh system one set of models is a lot faster. Even by just this simple waterfall chart, you can see that the system one part takes so much less time than the uh actual generation which is the purple bars here. And I think they both ended up getting to the answer at the end. So the quality was not affected. But it took a lot less time. It was a lot cheaper. Six LLM calls versus one. 40,000 LLM tokens versus 1,800. And so you can see how you can use these kind of system one and system two models even in like developer workflows in order to get you to your answer faster. So the same answer, but it took a fraction of the LLM time and cost, right? I ran this when we were not doing conference Wi-Fi, but we had about 9.2 seconds and 17.2 seconds on the LLM. It was almost two times faster. It was 72% less expensive. We shrunk our LLM calls from five down to one and about 94% less 57:29 tokens. And so this is an illustrative example of how you could be using the system one and system two models in tandem. It's not intended to completely replace your LLM usage, but it can augment it and make it better by making allowing you to free up time and compute for your other more important tasks here and you can now do, you know, higher concurrency tasks because these models are able able and capable to do so much more. And so that was one illustrative example. I just wanted to flash on the screen really quickly as well. What else are people building with Clef and Claric, right? There's a lot of routers that you can build and and making decisions there. There's fraud that we've been seeing a lot like spam or phishing. It can make decisions on there. With Claric, you can use it to extract data from like web pages or support tickets and things like that. I'm excited to see we actually have a new launch coming out for Claric. We released an early preview, but it's coming out. So I think these use cases will grow as people are trying it out more and more. But the idea is that 58:29 there's so much in the world we're not limited to just LLMs. Now these new paradigms have been introduced and I think we should all take advantage of that and try building with it. And so we went backwards, we went forwards, but I think the most important thing here is that we go onwards, right? We have to be very reactive in this industry. Love it or hate it, things just change on a whim. You know, if we as Cloudflare back in the day said our hypothesis was wrong, we have to go and actually pivot and start serving these giant LLMs in order to take advantage of the fact that we want to be an agent cloud and that it makes sense for our story. When an east wing wind comes and we need to have decision models because it fit perfectly with our infrastructure story. We had to pivot and be a neolab as well. it's not it's one thing to sit here and look back, but the more important thing is that we took what we had in stride and moved forwards. So I'll leave you with one last quote here, 59:30 which is a story from Tolstoy. Remember then there's only one time that is important now. It is the most important time because it's the only time when we have any power. And so I'll leave you with this quote and a question of what are you going to build today? >> [applause] 1:00:02 [laughter] 1:00:11 >> Hundreds of pull requests a day and nobody reads them all. So how does code quality keep improving? To share how the Grok bot team turned slop into gold, please welcome Lauren Tan, principal engineer at SpaceX Super Intelligence. 1:01:02 >> Hello everyone. Don't know if you can see my slides, but my name is Potato, also known as Lauren. And I'm very excited to chat with all of you today. Maybe we can get a big round of applause our tech team helping me with my laptop. [applause] 1:01:36 All right. Um so again I'm my name is Potato. Uh I work on Graph Bot at SpaceX SI. And can I just get a quick show of hands? How many of you have ever looked at code that somebody else wrote and you thought that's all slop. [laughter] Well I have too. But today I'm going to convince you that you can turn that slop into gold. Uh and so I did something crazy last month and I shipped 3,000 pull requests to production. Now before agents we never really cared about volume uh except for the outliers. And at my previous company we called these people coding machines. People like Brett Taylor who famously rewrote all of Google Maps uh in a weekend. Just an insanely productive engineer. Uh and well with agents anyone can now be a coding machine and my thesis is that the job of the software engineer, 1:02:37 the meta of being a software engineer is figuring out how to maintain or even improve the quality of your code while massively increasing your productivity. To get there some people like to call this a software factory uh but I like to keep it simple and I think of it as all about trust. Not blind trust but investing in your agents environment so you can trust that your agents are going to do the right thing. Now it takes quite a lot of effort to get there um and it took me six months to get to a point where I could ship thousands of pull requests a lot of them in my sleep. But it's certainly possible and it will take you much less time than I did. So let's start by talking about how we move from a low trust environment to a high trust one. So this inverse pyramid is how I think about where to spend time on your agent's environment. The first two layers are deterministic. So using your code itself. So things 1:03:38 like good API design, architecture, refactoring or using lint rules, type systems, compilers or you know different kinds of testing. And you can use both of these layers together to constrain what agents can do and force them to do the right thing by default. Verification, both runtime and formal are great tools to layer on to find bugs and check properties and the invariance in your system that you want to hold true and also help you check whether or not the agent understood your goals and your intent. And then finally you want to layer with a little bit of pros. So things like rules, skills, agents.md, things that require judgment. But let's start with your codebase. Agents generate their outputs based on their inputs and so the the quality of the input really matters and it just so happens that your codebase is one of the largest inputs. So whenever you're correcting your agent, think about how you can make that 1:04:39 a deterministic check instead of adding another line to your rules or your agents.md because it's far more reproducible when it's in code and it lets you trust it more. And the analogy that I like to use is that your codebase is kind of like a garden. You know code used to arrive a few pull requests at a time and uh I remember going into work in 2024 and reviewing the three pull requests that my colleagues wrote colleagues wrote. And today that number is more like 300. So uh you know every engineer is now a team of hundreds or even thousands of agents and code is arriving so rapidly that it almost feels like a continuous stream, and it almost feels organic like a living thing. And in this analogy, your codebase, the constraints that you impose, the guardrails that you put up are the concrete foundations that the new code that your agents write grow around, just like uh the building in this picture. 1:05:42 And so in many ways, the job of being a software engineer is kind of becoming like gardening. Because code is the biggest input for agents, it means that any workarounds, um anti-patterns, deprecated code, tech debt, it's now likely to be just copied by the next agent since it looks like the established pattern in your codebase. And uh it's like you have some weeds or pests growing in your garden, and then they spread everywhere before you know it. So you want to make your codebase as pristine as possible so that agents are always copying the good stuff. All right, let's look at some real examples. Uh so Dune is a client framework that we created for Graphot to implement this idea for organic architecture. It's not really that interesting technically. Uh you can kind of think of Dune as um kind of like Next.js for Electron apps. But the interesting part of Dune is that it has a lot of constraints and lint 1:06:43 rules that we've encoded uh a lot of the lessons that we've learned building desktop applications. And one such lesson is that anything that requires nuance to use well often leads to slop. Uh you know your classic example is uh you know use effect in React, right? It has its use cases, uh but it's often misused. You know some people joke that you need a PhD to use use effect. Um and in in other words, it really requires a lot of nuance to use well. And agents are not good at using it. Um so our solution has been to just ban it in product code, and it's really only allowed in some places inside of Dune APIs. And effectively, this problem has now been categorically eliminated, and we never need to catch that during code review again. Another interesting pattern that we use is context through directory structure. Uh so one anti-pattern that we saw with Cursor, which is an Electron app, is 1:07:43 that agents would often import main thread code into the renderer thread, which would cause long tasks and jank. So in uh Grotbot and in Dune apps, we've made this very, very conventional. Uh we have these two top-level directories literally called Electron Renderer and Electron Main. And imports between them are forbidden or banned by our dependency graph analyzer and some lint rules. And again, you know, these are simple lint rules to set up, but it means one less thing that we need to review and think about uh you know, during code review. And then one more surprising thing that I've seen is agents using Markdown. So things like documentation, uh code comments, rules or skills, and they use this as a place to write down their justifications for not doing the right thing. Or I don't know if you've ever seen this, but they'll over-rotate on some small correction that you made, and they treat it like a hard rule that can never be broken ever again. 1:08:46 And here's another example from Cursor. So agents really loved using imperative timers, so things like setTimeout, requestAnimationFrame, requestIdleCallback. And they were using these as quick fixes because they were really patching over an architecture clash between uh VS Code's imperative style and React's declarative style. So our choice of architecture actually made both the client and the agent performance much worse. And if you've ever done any performance work in JavaScript, you'll know that when you have tons of timers being rapidly created and destroyed, causes a lot of uh major garbage collection, which causes a lot of lag in gen. But the worst part is that agents wrote a lot of code comments like you can see in red here, explaining why this quick fix was justified. And before you know it, we had timers spreading everywhere and the you know the performance of the app really suffered. Now in Graphbot we learned our lesson 1:09:46 there and we decided to just ban 99% of comments because they go stale easily. You know humans just really don't bother reading the comment slop anyway. And again it gives an agent the an escape hatch to justify not doing the right thing because it's hard. And the worst part about comments though is that they're not enforceable. So what seems well intentioned you know like I'm going to tell the the next agent not to do something never ends up actually being anything that you can trust. So in Graphbot and Dune and in Dune we now have lint rules that give good and very helpful error messages telling the agent to fix the root cause, the real problem rather than just patch around it. And so our general principle is that every time we correct our agents it turns into one of these deterministic checks or we think about how to categorically eliminate that problem in the code through better API design, through better architecture. 1:10:47 And if you do this enough times you'll find that you have just less things to look out for in code review. And at some point you'll realize there's there's nothing left to review. And you can just let agents merge their most of their own pull requests. All right moving on verification is also a really big part of how we make our agent more autonomous and also increase our confidence and trust in their work. And our code review process has now really shrunk into looking at verification artifacts instead. So things like videos, screenshots, diagrams, formal verification. And the most basic example of this is runtime verification, which is just a fancy way of saying giving your agent a way to run the code for itself and use debugging tools that, you know, human developers use as well, like the Chrome DevTools protocol. Um in our runtime verification skills, we have this thing that we call a feature map. And that's basically kind of like a sitemap of all of the features that are 1:11:49 available in your application from the user's point of view. And so with this, agents don't need to guess when you say, you know, the thingy in the left sidebar, it gives you some shared vocabulary. And we also make use of custom CLIs inside these skills so that uh agents can reuse the the CLI and the driver every time, which cuts down on token usage and it's also testable. Um and then recently we've also started looking at formal verification. Uh so we now use TLA+ to do some model checking of major user flows such as notifications across multiple devices and platforms. Uh we use a lean as well for proving certain algorithms. Uh for example, we have our own We wrote our own custom virtualization library and we use lean there to check that we don't have, you know, flickering or jittering problems. Um and we're still experimenting with this approach overall. It seems very promising. Uh we've already found quite a few bugs uh with formal verification. Uh but it's certainly not a silver 1:12:50 bullet and we're treating it as another tool, another layer in our agents' environment. Uh and then this is a very fast video, but another thing that we just started looking at is property-based testing as well for UIs. Uh so we use this library called Bombadil. Um and this has also caught a few UI bugs for us already. Okay, so zooming out, I saw this tweet from Cindy and 6 months ago she she said that she was Oh, she was talking about not being able to trust her agents because of subtle bugs that were getting merged. And And what was missing for her in being able to trust her agents was the verification process. And now, six months later, she says that 70% of her daily tokens go into verification. And I'd say that that tracks with what I see as well, that most of my token usage goes to verification. And if you really look closely at her process, none of these things here are new or novel techniques that we've never 1:13:52 had before. These are just good software engineering tools, but ones that might have been prohibitively expensive to use before as they were hard to understand or write or maintain. But agents drastically cut this cost down. And really we are entering a golden age of software testing and verification because agents let us test everything to a much greater degree than used to be economical. And then formal verification can also help you better reason about how your product should work and whether your agents really understand your intent and goals. Uh I made this new skill called state chart in my plugin p stack, potato stack, which helps you visualize state machines from from the user's point of view. And I find this much easier to review alongside the videos and screenshots that we provide. Um and so that's been great. Sorry for the flashback. Uh okay, now let's look at what the promised land looks like. What do you get in return for all of this you know 1:14:53 investment? So on the Graph Bot team, our velocity is taking off and rapidly increasing. And last month our team shipped a ridiculous 11,000 pull requests to production, which is a wild number. And our pace of releasing the client has also increased. And we cut a new stable version about once a day. And then the core team is also shipping much more. And we also have a lot more casual or drive by contributors who don't always have a lot of context, but they're also contributing. And in October we had about 227 people who contributed at least one pull request. So despite this velocity accelerating, you might be thinking to yourself like isn't this going to be chaos? You know, isn't this going to be a bunch of slop? But actually, the number of PRs that get reverted has actually gotten lower on a relative basis and is steady at about half a percent. 1:15:54 But I think the really interesting thing about this shipping pace and this volume of pull requests is that the performance of Graph Bot has stayed not just stable, but has improved on multiple dimensions compared to when we launched. So startup, typing, opening new chats, switching chats, all of this has gotten faster. And keep in mind Graph Bot is still an Electron app for now. So there is a ceiling, right? But it's already much faster than many other agent UIs out there. And for an Electron app with this high velocity and this many people contributing, I'd say that's pretty good. So before you jump into building out a software factory and you know, automating everything that you can, whenever you correct your agent, try and do it by improving your codebase first. You know, start with a lint rule to prevent new occurrences of anti-patterns and then clean up those old patterns as well. And then make sure that your verification story is really strong. 1:16:56 A lot of this work is a prerequisite to automation anyway. And if your inner coding loop produces slop, then automating that loop just produces more slop and you need to throw more and more and more tokens at the problem, which we all know is very expensive. And once you make these investments into, you know, code into your codebase, static analysis, runtime informal verification, you'll start to find that many things become possible with these foundations and that you can really easily build your own factory on top of these foundations. And when you're ready, Graph Bot is a great way to build that hour loop. We have tons of connectors, routines that can kick off Cursor, cloud agents, and projects uh which let you orchestrate thousands of uh hundreds or even thousands of agents. And as a quick example, you know, whenever we cut a new release now, Grogbot will automatically spin up a new Cursor project for us that will fuzz the entire application, the entire build, and it will find bugs uh by running the 1:17:57 app and using it like a real user. And then it's going to open fixed PRs for each of those bugs that it finds, and it's going to auto merge it as uh crazy as that sounds. And that's because our codebase is good enough that pull requests tend to just be good by default, and most product code that we land no longer really requires deep human review. So when you have solid foundations and good organic architecture, you can turn slop into gold, and you can ship with high quality uh high velocity without compromising on quality. My name is Potato, also known as Lauren. Thanks to you again for listening to my talk. >> [applause] 1:19:02 >> That concludes this morning's uh keynote sessions. Thank you for your patience with the various technical challenges. The root cause was of course a GitHub outage. Uh not really, but the fact that no one would think to question that does say something. I can say that because they are not sponsors, but do you know who is? That's right, presenting sponsor Google DeepMind. Another round of applause for them, please. Without them, this would not be doable. [applause] Of course, our Lounge and Platinum sponsors as well make the Lounges and Platinum things there. I am not allowed to say snarky things about AWS for once. So we're going to smile, nod, and keep it rolling. Our gold sponsors as well have been incredibly supportive of various things, not my nonsense, but you know, most of the rest of the good stuff, absolutely. And let's give it up for our silver sponsors, too. [applause] And finally, all of our supporting partners as well. There are uh and of 1:20:05 course viewers like you to quote PBS from my childhood. I'm old back when we used to watch it on the Telegraph. There are now some breakout sessions. My favorite part of those if you erase part of the letter B, it reads freak out sessions. You can read into that what you'd like. There are another series of closing keynotes later today starting at 4:30 and I'll see you all back here then if they don't wind up having a conversation for me out back next to the dumpsters with some broken beer bottles. Thank you all for coming. >> [applause] 1:30:40 >> I'm Ari, the founder of Intrinsic AI. This talk is for people who build agents and run their own inference stack or plan to. And if that's you, you associate typically the KV cache, the burden butter of modern inference engines usually as a throughput optimization, the keys and values computed per token kept so the model doesn't compute them again. Today, I'll like to show KV cache in a different light. The way agents, agent, AI agent application could go wrong are keep growing, right? Prompt injection, jailbreaks, broken tool calls, sensitive tool calls, sensitive data exposures, hidden backdoors, and drifts, and so forth, right? So the way right now we typically guard against against these numerous ways that an AI agent application could go wrong is by guarding the edge, by perhaps passing a judge model that checks the prompts, the inputs, and the response outputs against your policy, right? Or you build a classifier for each case. But here is a different light. The model 1:31:42 turns every input into a computation before it acts, before it decodes, before it generates the response, before that response becomes another tool call. The KV cache is your engines record of that computation that modern inference engine for free are using it and reading it. And therefore in this talk I would like to suggest as as a complement to the way you're already using your own guardrails to use this data and integrate it with your KV cache infrastructure. I refer to this as a KV for security signals and basically through reasons that matters, right? So the data is already there. You pick the gems from the pile your inference stack already were throwing away. It covers the guardrails blind spots because whatever it slips past those guardrails it still becomes computation and it stays in place so it's perfect for data privacy application for cases where you do not have the luxury to read the inputs and outputs. 1:32:43 And the recipe is pretty straightforward, right? It has three steps. You learn a baseline for your AI workload, for your AI inference traffic, how your KV page typically look like and then find the KV pages that are rare against your baseline, typically your anomalies. And then you can even classify and explain those. So first let's start with where in this stack this integration could happen. The interesting thing about the KV cache is that in a way existence of those had made the inference in a way stateful, right? So for example things like KV cache management layers such as LM cache stores them from GPU down to a remote storage or LMD and NVIDIA Dynamo route each new request towards the worker that already holds perhaps that cached KV page. So there are two integration points you you you can integrate those anomaly detection or those classifiers into. 1:33:43 Either you can implement it directly inside of your inference engine via LM SG line so forth. The beauty of this approach is that it's real time and and the the the alert that it would perhaps generate could ride along your future tool calls or your future policies, so you can harden basically your policy or guard rails, but just generating the computational anomalies that you observe within your inference engines. And you perhaps you can integrate it with as a as a way of threat discovery in a sense. So you can integrate with your LLM cache. So reads the stored KB pages and find things that don't belong with the rest, right? So now I want to show you some numbers and just a very simple hello world use case if if this is actually working and this is a valid approach. And for that, let me use the use case that has no text to filter at all, a voice agent. So here's a case study of a voice agent. 1:34:44 In this case, it was a gem of 4E4B. Our baseline was 72 ordinary clips and spoken in four voices. We're looking at 16 token pages. Obviously this granular granularity could change, but here it was just use a shorter pages so we could have more data in the baseline. And then on the test, we have different voices and one clean voice and the other one has a hidden instruction within it. And as you can as as is shown here, not only you can detect these cases just using simple anomaly detection, but also you would know which part of which page, which part of your request had this anomalous event. We even went further and we trained a model that uses the KB pages as features. And we didn't be trying to explain it or tag it with things that are useful. In this case, although it wasn't even trained on audio, is able to generate correct information about personal information, home address, location data, travel information, and 1:35:45 destination. So now back to text agents. In this case, it was a 3 14 billion live cache. The one on the left shows you an idea of how attacks and normal traffic would typically separate from each other. And then the one on the right, which is perhaps more meaningful, it was using the AT bench and hugging face. It has its own baseline of benign agent traces and eight kind of agent failure from injected tools description to misinformation and so forth. We look at the top 10 alerts that anomalous anomaly detection would generate and about seven to eight times of out of 10 of these alerts are actually anomalous meaningful and useful anomalous activities. We even went further and we looked at whether there's any kind of anomalous event that is stand out more than the rest. Um to make things hard in production, 1:36:47 first, models change. Fine tunes and new releases, but also agents that each run on a different model and routers that pick a specialist model per request. That means each model writes its pages its own coordinates. So either you learn every baseline, classifier and explainer, which is computationally prohibitive, or you have to find a way to map them all into one shared representation space. Second, real time means moving big objects. For example, a 1 256 token chunk of coin 314 billion is about 40 megabytes of KB. Compression and streaming like cache gen could help, but moving that for security analysis and acting in real times is still slow. And that's what we solve at Intrinsic AI. We are looking for teams building their own inference stacks who treat security as a foundational design decision. 1:37:47 I hope this was helpful for you. Thank you so much for listening. >> Let me start with one number from last month. On average, more than one agent has started on our platform every single minute, day and night. And honestly, the agents were the easy part. Hi, I'm Joey. I built Spectre, the cloud agent platform we're building inside Harvey. After 10,000 PRs, the biggest lesson I can share is this. The thing that broke was an agent. It was everything around them. So what does one submit it add up to? 10,000 PRs that an agent wrote. And it isn't just the team playing with a toy. Last month, about 460 different people at Harvey used it. And over the summer, it stopped being only about code. People started asking a data question in Slack and getting answers back. 1:38:49 My favorite Recently, someone on our marketing team dropped a prototype in the talk track into Slack and asked for a storyboard for a product video. Once the harness exists, people bring the work that you never planned for. So a promise upfront, this is not a talk about models, evals, or leaderboards. This is talk about harness, odd things around the model. Checking out the repo, holding the key, opening the PR, getting the merge. When we first wrote about Spectre back in April, we put it like this. At some point, the harness become the product. Now, whenever I tell people this, I get the same question. Why on earth would you build this when Cloud Code or Cursor already exists? My answer We don't compete with it. We run them. And here's how I see it as my own view. 1:39:51 Agents on your laptop are powerful, but they don't scale. You can realistically run five to 10 of them. Plus that, you need the cloud. And the cloud needs our secrets, our real stack, our alerts and tickets, one set of rules, and enough capacity that nobody gets starved. No single tool owns that for us, so we built layer, and that layer is exactly what broke. So I want to tell you three stories today. How we stopped competing on writing code, how we let any agent drive and what broke once they did. The first story starts with the question we keep hearing. More of our engineers were using Cursor and they ask, how do I get our secret there? How do I test against our real stack? We could have rebuilt all that inside Cursor and then again for next tool and the one after that. 1:40:51 Instead, the answer became one sentence. Just ask Cursor to use Specter. And that's really the whole idea. People will always have a favorite agent and the favorite keeps changing. So pick yours, Claude code, Codex, Cursor, ChatGPT, dot, Slack, or your own hands. Start on your laptop, push the session to the cloud, pull it back. That's how Specter sits underneath all of that. Because once the code was written, the real work starts. Picture a PR stay. Alert files in Slack at 2:00 in the morning. An agent picks up. Nobody type a thing. It writes a fix, spins up real copy of our stack, and posts the proof on the PR. When CI fails, the follow-up loop reads the failure and posts another fix while everyone's still asleep. By morning, the PR sits in one review queue under the same rule as other PR, 1:41:54 which our agent wrote it. The owner gets a nudge and only once they are working they starts. And once approved, a agent can auto merge it. If you only built one of those stages that I described, build the follow-up loop. We switched it on for everyone in early October, and that month four out of five agent PRs merged into our backend PR had been through it. That's the difference between the agent that write code and an agent that finished the job. And then there's the knowledge. Every team has that one person who knows exactly how to test the tricky service which flag you never touch. And that knowledge lives in their head or their laptop. So we gave it a home, projects. People write down what matters. Adrian write down what they learn. Team pings the skills it trusts, and 1:42:55 there's a record of who ran what. Any agent working in the project with the same page, my personal view, whichever agent you pick, the audit trail and the tracks should be the same. The second story is about rule. Back in May, if you want an inspector to do something, you open a browser tab or tagging Slack. The whole month exactly one task came out from terminal. So we made a rule, if a person can click it, an agent can call it. Start a task, turn on the follow-up loop on auto merge, publish a page off from CLI. Which means your laptop agent can call it. Agent inside the sandbox can also call it. Even CI can call it too. And then people ran with it. By October, the command line was starting about 6500 tasks a month. 1:43:56 The web app didn't shrink. Everyone else just grew around it. Task started from the terminal were merging more PRs than task started in browser. And a new kind of user showed up. People who start hundreds of tasks a month and never open the app at all. And this is where it gets fun. In October, I had a big task. So I call code session and ask to be the coordinator. It split the tickets into 10 different milestones. Get one for me and the rest to Specter with hundreds of tasks. Each has its own sandbox to validate end to end. Then at the end of the October, our IT give everyone ChatGPT tasks. My now watch Specter for me. It lists the questions Specter is waiting on, draft my answers, and once I say yes, send them back. I actually DM to my friends like that. 1:44:56 I just use cursor, clock code, or whatever to spam Specter now. Well, that's the dream, but it's also the problem. A quick detour. When we look at where a task time actually went, the slowest step wasn't model syncing. It was Git. Just checking out every repo we owned took over a minute on average. The fix, keep a copy of every repo on every machine and let each task borrow it. The third story is the one where it broke. In mid August, a release goes out with a bug. That bug triggers alert that fires once per workspace. So one bug become hundreds of alerts all landing in one single Slack channel. Someone has set up workflow that tag our agent on every alert and every tag started task. On normal day, those tasks started about 50 tasks. 1:45:56 That day, in about 4 hours, they started around 900. 900 agents are chasing the same bug until that wasn't room left for a single node. The fix was almost boring. A A alert now joins the investigation that's already open. Two months later, it happened again. And this time, it wasn't about On October, the engineer did exactly what we've been asking everyone to do. Hand work to Specter. About 950 tasks from the command line. In about 3 hours, nothing in our platform can work. The cluster filled up and everyone else work sat in line behind one laptop. That's not a user problem. That's our problem. 4 days later, we had a real fix, a fair launch queue. Now, starting a task gives you a place 1:46:57 in line. And the dispatcher started when they're through. People sitting at keyboard goes first. Everyone gets a fair share and can borrow more than the cluster is quite. And whenever you start a task, you can see a place in line. When we hit it with a 300 task bursting low test, the task list went from almost 2 seconds to 70 milliseconds. So did it work? When it's the busiest day of every week back in August, 2 sudden task and about 1 in 10 didn't finish. Then in November 4th, more than twice that and much better in the finish line, but we still have some room to grow. I will be modest. We changed several things at once, so I cannot tell you which fix mattered the most. But the burst kept getting bigger and the platform kept up. And just when that was under control, 1:47:58 the bottleneck moves. At end of the October, our dev platform team raised alarm. PRs across the company had jumped around 60% in a single month. A merge that used to take 15 to 30 minutes was taking over an hour now. And merge queue kept having to be switched off. So now a PR rebased onto main, we run CI and merge if it's green. If main breaks, everyone CI breaks with it. Honestly, tell you, we haven't cracked this one yet. But before I go, there's one more thing. While we were busy fixing all of it, something else was quietly growing. Most of what I've been showing you start with a person asking the fastest growing cells of the work on Specter doesn't. These are automations, agent that wake up on schedule on event and just do the 1:49:00 work. Treasure the alert, read the PR, write the post. Last month, about 110 people had automation running and we learned to give them a guardrail on day one. If one fails five times in a row, it switches itself off. And it can carry a budget or expiry date because when no one's watching, an agent can start thousands of tasks too. So here's what I'm chasing next. And I don't have an answer yet, honestly. What if every Slack channel had its own agent that never locks off? One that follows the condition and start work on its own. What if every PR proof it works before a human even look at it? Today, only one merge in 20 carries that kind of proof. And how do you merge when agents never stop opening PRs? Ask me again six months. 1:50:00 I will tell you what broke. If you building something like this when you work, here's three things I'd do again. Own what happens after the code is written. Make Eric action something an agent can call. And cap what an agent, human, or can start before they find out what they can do with it. And one last thought. That's my own view. Engineers can write code with whichever agent they like. The work that's lost is the harness that runs it, checks it and get it merged. Thank you so much for watching. Come find me on LinkedIn if you are building similar things. >> Hi everyone. My name is Rajat Kotial. I am part of the automated reasoning group at AWS. And today I'm going to talk about a project I worked on recently called hero specification analyzer. And we use certain techniques which 1:51:01 involves formal methods and automated reasoning to prove specification that goes to your coding agent before it writes code. 1:51:16 So let's take a certain bug, right? This is a specification snippet and there are two rules for example and rule two says when an order is submitted and this is for a a general e-commerce website let's say. When an order is submitted and the inventory is available, we place the order as a back order. Which makes sense when you read it individually. And there was also another rule in the same specification file which you hand to the coding agent which says no order shall be placed as a back order and this is a rule we've set. So individually read these two lines make sense, right? But in when an LLM reads this reads both the rules together, both cannot hold true at the same time. So we need to make sure the we resolve this ambiguity that the LLM is observing. And 1:52:17 the scope and the reason we put an effort into this project is based on public research and recent research, we've observed there's a 20 to 40% drop with unclear prompts on the accuracy the LLM generates and the code accuracy, right? And 60 to 90% of valid code is still wrong after it writes it. And also in that case, we observed two times the more more regressions when we deploy that code. So where do bugs like these hide, right? We want to detect these bugs before they reach the codebase and the only way to do that it's similar to how we write design docs, similar to how we draw diagrams and flow charts to make sure the flow is set before we write any code. We want to verify that without a human or anyone reading these files as the first reviewer. In one sentence, 1:53:19 let's say the ambiguity and wording wrong level of detail, but across sentences, we can observe conflicts and gaps in the specification or rules, right? Sentences need review, such need search and between them and ambiguity searches. So we have a three-step process to do this where the LLM will first refine it, it will try to clean up the wording on its own and if it succeeds, then well and good, but if it cannot be then formalized to English language into SMT formulas. So SMT lib is a certain set of type of function which we pass to a SAT solver, a satisfiability checker where it's going to give out a Boolean saying this expression that you've given me is valid, true or false, invalid. And then once the solver gets back to us if that expression is valid, then we don't do anything, but if it detects any 1:54:19 ambiguity between two sentences or within a sentence where there's two meanings to a sentence then we detect that and tell the LLM to recheck that sentence and it's going to reiterate on that. For example the rules we saw we've seen when order submitted and inventory unavailable we shall place the order as a back order and in SMT format it kind of looks like this this like a math formula right? Once we go deeper we once we detect this we also do a three-step process again where we sample the translation again and again we ask the LLM to translate seven times or 10 times and to understand whether this LLM is behaving this way based on its its inherent capabilities or is something affecting it to behave this way 1:55:19 and then based on logical equivalence of these seven or 10 different set of interpretations by the LLM we cluster them on logical equivalence which that which entries came under which assumptions that their LLM made for itself and then we accept or ask the user to clear out the ambiguity or abstain from implementing that in the code base. This is an example of asking the human what you want to do with that rule that was perhaps contradicting. Remove the record is very a very vague statement the LLM cannot can interpret anything it wants but we shouldn't allow that to happen and we either give two options we keep it as is the record is deleted or we retain it and but hide it from the users which one does the user prefer? 1:56:25 A conflict is when the examples we example we saw rule one is conflicting with rule three. And a gap is where no rule covers an idle order in this example where the LLM detected an actual sorry our process or pipeline detected there's no entry for this rule where you are missing something. All in all it cannot do a lot of things end of the day. We want to make sure there is no oracle for for specification correctness other than the user. User needs to make sure to check every part of the snippet part of the rule book before sending it to the LLM. And we as a tool only can help them. And the user is more important here because of implicit domain knowledge the LLM doesn't know a lot of domain knowledge and there's no guarantee from an LLM 1:57:26 only user can give that guarantee right and there's been evidence right now for formalization and how it can be used in agents we are emerging in this technology. What do we take back from this talk? Check the spec always before you pass it down to the agent. Cluster the translation we what we do is we cluster the translation by logical equivalent to determine the different LLM interpretations and where it's genuinely going wrong. The LLM translates the solver checks and the human decides. We give to give the human options and if there's no answer to that we use LLM judges or abstain from implementing that piece. Here are the references and this work has been majorly contributed by my team 1:58:28 you can check out their and the website link provided. And delve deeper into that. Thank you. >> Hi, I'm Chinmay and with me are Swapnika and Vlad. We're excited to share our journey as well as our learnings around building an AI on-call teammate at Meta that now runs real on-call rotations every day and around the clock. We are part of the engineering team that builds the tooling platform and infrastructure for incident response, on-call experience, and mitigations at Meta. That space is evolving rapidly with the increase in AI capabilities. AI changed how fast we write code. About three quarters of new code is now written with AI and more AI-generated changes reach production every single day. And some of it can cause production failures just like any other code change can. But on-call did not speed up with it. 1:59:28 High-quality code can pass review and still break in production. And when it does, someone still gets paged at 2:00 a.m. The bottleneck moved from writing code to operating it in production. And on-call is where that operational load lands. So we asked if AI is writing the code, can AI also help operate it? And we think it can. Now let's quickly run through one of those on-call pages with a simplified example. It's 2:00 a.m. and 12 alerts are firing. Same night, it can go two different ways. On the left is how most of us do this today. And notice we already have good AI. First you sort the alerts by yourself, check out payments disk, which one to pick first. Let's say you pick checkout. Then you ask your assistant why P99 latency is high. It's genuinely useful. It pulls the metrics and finds the deployment at 2:00:29 1:49. And when you ask, it reads what changed and checks the runbook. It figured out that the time config was changed to a lower value and based on the runbook, it suggests to rollback, which is all very helpful. However, if you look at the shape of it, it waited to be asked. You picked every next step. You were the glue between the questions. And once it answered, it went dormant. At 2:39, you know the cause, but the rollback, the changes, and the follow-ups are still yours. Now, on the right, the same 12 alerts, but an AI teammate started before you woke up. It had already ranked them. Checkouts first, payments merged in as the same cause, and the other 10 handled separately. It read the dashboards, diffed the same 2:01:30 149 AM config change, and matched two past incidents. You get a hypothesis, a 30-second way to verify it, and a fix waiting for your review. At 2:01, you verified it, and you're back to sleep. That gap is what this talk is about. So, why were you the glue? Every team was bolting automation onto on-call, one silo at a time. An alert script for alerts, a triage bot for tickets, a deploy bot for deploys, a fix-it agent for incidents. Each one solves its slice, keeps its own context, and forgets it. Now, replaying your 2 AM page, the alert script saw P99 latency about 2 seconds. The triage bot saw three tickets saying checkout is slow. The deploy bot knew about a config change at 149, and the incident history 2:02:30 had two similar past incidents. Every clue existed, but nothing connected them. So the on-call became the glue exactly as we saw on the left side of the previous slide. Our bet was one AI teammate per rotation instead. One loop across every signal, one team memory that learns every shift, its own identity, so it acts as itself and a human gate. So you approve instead of operate. But giving one agent all that context and reach raises the stakes, which is why autonomy has to be earned. Under the hood, the teammate or the AI teammate is the simplest agent there is. Model using tools in a loop, acting on an environment and reading the feedback. Everything else in this talk is what we [music] built around that loop 2:03:31 to make it safe to leave it running. And for that, I will hand over to Swapnika and Vlad next to get into the actual details of how it works. >> To achieve this, we started with an autonomy ladder. Think about how you onboard a new engineer to on-call. First, the shadow. They investigate and suggest, but they don't touch production. Then the pair. They write the fix and another engineer approves it. With time, they gain knowledge. We give our agent exactly the same ladder. At level one, it investigates and tells you what it found. It changes nothing. At level two, it acts with approval. It drafts the fix, but nothing reaches production until a human approves. At level three, it clears lower criticality work on its own and only interrupts you for what matters. High criticality work stays with humans. So what earns the next run? 2:04:32 We use a simple test. How costly is a mistake and how easy is it to catch? So an action claims only when mistakes are cheap, they are caught by tests, CI or review and the agent already has a track record on that exact action. That's why the agent can close a noisy alert on its own while a production mitigation still waits for a human. At level one, the investigation is layered and stops at the first conflict answer. The agent investigated the issue using the team's memory and runbook. It also shared the root cause analysis, but human had to apply the fix. At level two, it writes the fix with the root cause and a test plan. It answers reviewer's comments and fixes CI, but nothing reaches production until a human approves. Most drafts don't land and that's fine. Review is the filter. 2:05:32 At both levels, the agent does the homework, but you still had to grade every single paper. They have got much better, but the tasks requiring human input didn't get smaller. Level three shrinks the pile. Look at this example. 23 items came in, but only three need you. On the right, the agent handled the long tail, noisy alerts, duplicates, routine fixes that went through normal review and CI. Two more are still being worked. You don't have to drive each one to completion. In the middle is your much smaller decision queue. Each item makes clear what it needs from you with a run this button that has a call to action. And remember our 2:00 a.m. checkout page, that still stays human own. The agent brings the investigation, the human owns the critical decision. It's the new high ladder again. 2:06:33 Trusted to handle routine work, ready with the homework when it needs help, not just less work per item, but fewer items waiting on you. That board is where the work lives, but nobody stares at a board all the time, right? So the agent is also available across many surfaces like chat and native incident surfaces. At 9:00 every morning, it posts a digest. 23 came in, three need you, 18 handled. Each thing that needs you come with a one click next step. So approving the quota bump takes one tap from your phone. The 18 at handled are summarized, not listed. And the two that are still in progress tell you what it's doing about them. And when you want to dig in, you just reply in the thread. As you see on the right, here you open the checkout incident and ask for the evidence. It already has it. 2:07:34 Errors start at 8:41. Two minutes after a config push, cut the payment client timeout. The same pattern as two past incidents. You ask a follow-up, is it only EU? And it answers with a breakdown. It drafts the role back. You approve it right there and errors start recovering. You never opened a dashboard and you never had to explain the context because it was already there. That's what a teammate feels like. A morning handoff a colleague you can ask. Now another hood of this AI teammate it's AI automations. The first investigates. On every new signal, auto triage classifies it, checks its criticality and kicks off the right agent. An alert goes to the incident investigator, a task to the team's own agent, a test failure to a coding agent, a lookup to memory and code search. The second keeps the system healthy. 2:08:35 Unattended automation fills quietly. So So a schedule, it looks stuck or failed runs, retrace and repairs broken schedules, and it only pages a human if it can't fix it itself. Nothing stalls silently. The third, hands off. At every shift change and every morning at 9:00, it summarizes the shift. What needs you with the fix ready and what it handled on its own posted to the team chat. Not only in the morning, but every urgent issue that needs user attention gets notified in chat with findings and draft fixes. It also tags the next on call in the chat during handoff. The entire cycle still continues. And because it belongs to the team, it runs the agents and the skills the team already trusts. That's the slot in the middle. Bring your own agent, add your own skills. If a team already has an agent that has 2:09:37 the team's runbooks and the knowledge base, the automations leverage it. Alongside that, it has our own incident investigator and a deep dive coding session. Work comes in from alerts, tickets, incidents, team posts and chat. The agent itself is generic. What makes it good at an on call rotation is the team's agents, skills, memory and playbook. And every run is logged, so anyone on the team can inspect the trajectory or resume the conversation. Now I'm handing off to my teammate Vlad to talk more on the shared agent. >> An on call agent can belong to just a single person because the on call itself doesn't. So it belongs to the team. There's one teammate per rotation on one board where every on call can ask it to correct it or pick up its work mid investigation. It carries the team's on call expertise, 2:10:37 the playbook, alert profiles and past decisions from one shift to the next one, the the a good handoff should. Here's what that looks like. Monday at 2:00 a.m. same as on call and tells it. Remember, if checkout times out after a config push, roll back the config first. It saves it to the memory. 3 days later, Jordan is on call hits the exact same alert and just asks for a fix. The agent answers with same fix, says where it came from and has the rollback drafted. Teach it once, the next on call gets it. And it acts as the team. It's on a data day owned by the rotation with only the access the team is comfortable granting it. And it learns from every on call shift. Handoffs, decisions and corrections all flow into memory. So shift number 30 is better than the shift number one. The same way a new on call gets better every week. 2:11:37 But memory erodes over time, so ours has to earn its place. It retrieves semantically, uses what it finds, scores what it actually helped and promotes or retires entries automatically. When someone says that was wrong, the item reopens and the correction becomes memory. Three things made this work be on our team. First, a loop that never stops listening. Every piece of feedback or course correction reopens the item and ultimately gets absorbed into memory. Feedback is an input to the system, not a sorry read once a quarter. Second, customize instead of forking. Teams bring their own playbooks and scale workflows and it's still the same teammate underneath. Third, what we call one rotation to end. Identity, config and memory are per team, so at a rotation becomes a setup 2:12:37 not a new project. The agent is generic while the expertise is owned by the team. So what has changed for the people on call? Well, first, their job shifted from driving deep investigations to making key decisions. They start from a hypothesis with linked evidence, not a blank task or a dashboard. Second, the load is getting lighter. Early signs of on-call load reduction are promising for teams that run it automatic autonomously, and we want to measure it rigorously before we put a number on it. And the third, every change that agent writes still meets a human, fixes land only through normal review and continuous integration. And most drafts don't land. That's the point. Review is the filter. On-call stops being attacks and becomes a review queue. Here's what we're focusing on next. First, mitigations that get approved, not just drafted. 2:13:38 We grade every mitigation that agent proposes and feed the outcome back so it earns its next level instead of being handed it. Second, skills are written from the chat so a team can turn its runbooks into skills without writing any code. And third, measuring the time we give back properly with a rigorous study of on-call time saved rather than usage counts. And the next rung on the ladder is earning medium criticality autonomy with evidence. On that, thank you so much for listening to us and hopefully that was helpful. Bye. 2:15:42 >> This 2:19:47 >> Yeah. 2:21:16 >> harness in the less code we need. And if we look at what we have been doing at Google DeepMind for the last six months. So at May we had Google I/O I/O and today it's November 11 and we have been shipping relentlessly basically. Just earlier today we announced our first decision model with Gemini decision one. Yesterday we made Gemini four carbon available and then if we just look at the last six months we basically ship now models every few weeks. As already said Gemini carbon is now available for you. So if you are a Google AI Ultra subscriber you can use it in Anti-Gravity, you can use it in the Gemini app, you can use it in AI Studio. Carbon is number one on the cursor bench eval and also number one on the perplexity benchmark and internally at Google we have been using it already a lot. And much more exciting is that I'm happy to share that we launched the Gemini Decisions API roughly one hour ago with our new Gemini decision one model. So it's a similar model to chef. You can 2:22:16 provide all of the existing Gemini inputs meaning it's multimodal so you can provide text, audio, images, video and also PDFs natively and then provide up to 1,000 questions in a single request. The average end-to-end latency including all of the network overhead is between 150 to 250 milliseconds and it costs 10 cents per 1 million tokens. So how do models get good at agentic work? And that's a rough overview on how post training works. So we have a task, we have our model, the model has a certain set of tools and then we have some kind of compute environment and at the end we have some verifier and the model tries to basically solve that task to use different tools and then based on the score from the verifier we update the model but no one is telling the model which tools to use and basically it only receives that reward feedback. So if we look at two examples on how a model can do a certain tasks with tools we have the path A where we have dedicated tools for file system control 2:23:18 so you all have probably used coding agents and you have seen a read file call or an edit file call so that's one way of doing it and the other way of doing it would be just using a single inline bash script for all of the changes so on the left side we roughly need around 10 turns to make the same changes to a coding repository compared to the two change turns on the right side just using bash and it also reflects in the amount of tokens we need so the left path takes around 10x more tokens than the life the right path and it's doing the exact same thing at the end for the model to achieve that task. So in my experiment every roughly six months I try to build a new coding agent harness from scratch and I always ask myself okay how much of the harness do I really need to achieve the same task and what is overhead to the model and we no longer need. So if we look at the coding agent harness which from 2025 we had a roughly around 10 to 2:24:20 15 to 20 tools we had dedicated tools for file editing we had dedicated tools for planning for to-do lists and also we had a lot of dedicated rules in our system prompt which told the model what to do and what not to do and that worked of course but since then we made a lot of changes so in my experiments first one was done roughly now two years ago in February where we did not really had agents right we had workflows we basically had predefined steps rock applications back then were very popular we had the planner we had search we reflected on whatever was found and then we generated an answer. Then we started to see our first agents we had many many different tools for all sorts of fine kernel actions. And then came the coding agents where we had those roughly ten to 20 tools. And now in my last experiment I tried to see, okay, what's going to happen if I provide only one tool to my agent? And the outcome is was pretty exciting to me. So this one tool agent works the 2:25:21 same way as every other agent except that it only has one tool, which is Bash. And the model can use that tool for any sorts of task. And the Bash tool is very simple, so it runs in a separate clean subshell. And it supports the command argument. I also added a description so that it's nice and easier to display to the user what the model is actually doing. And then like some async parameters. So basically when the model runs a Bash command, the parameter tells the model, okay, if it is the command longer running than this milliseconds, it's being that the tool call resolves. And the model receives a local file handle where it can look up later, okay, if the process is still running. That's very interesting if you do like all sorts of like web development, for example. And the single Bash tool actually works for all of the things we are using agent. So we can have long running work where we can start on the foreground and then automatically move it to the background. The model receives some file handle and since it has Bash available, if we store 2:26:21 it to a file system, it can read the logs which are being streamed. If the agent itself is a CLI, it can call itself using Bash. So there is practically no need to create sub agent tools for it because it can invoke itself. And then of course we don't need dedicated planning or memory tools because we have our file system available. We can provide some context to the model so the model can keep track of its own work with the file system and using the Bash tools. And in addition, of course, to our tools, we have system prompts. And similar to tools, system prompts evolved over time. Every dot here is roughly around 100 tokens and you can see that coding agents mid 2025 had a lot of context needed to send in every request. Of course this also improved and similar story here is like okay, what do I really need to provide to my agent to make it successful and work and what of all of the system information and details we had in the past are now basically 2:27:22 part of the model. So how much of the harness helps and how much is in their way? And to find that out I was running a lot of different evaluations of a lot of different agent harnesses with a lot of different tools. And one example I like to show here which was done with Gemini 4 Carbon from an eval task from DeepSVE is like okay, what happens if I provide my existing coding agent tools with read, write, search, grep and my bash only tool? And I think you can pretty clearly see that the top line is taking a lot of more tokens, a lot of more turns and a lot of more times to achieve the same task. And what stood out really well is that okay, reading is done in parallel in the beginning, but as soon as we start going into making code changes we see that same behavior that the model is making turn by turn a single edit which of course consumes a lot of time and a lot of tokens. So how that does this look at Prodoscope? 2:28:24 So I have my own eval set of around 350 tasks for coding, knowledge work and terminal commands all run with Gemini 4 Argon here and four different agent harnesses. One is the bash only harness and then okay, with bash we still have that issue on how can I read images, right? If I'm I cannot cut screenshot.png it would return a binary and it will fill my context and the model cannot see the image. So I added a view media tool which basically returns the image in a way the model can see. Then we have two different agent harnesses with structure tools for file system operations. And if we just look at the score all of them look roughly identical, right? So if we would just evaluate our model in an agent harness looking at the score, it would basically not really matter which one of those we would pick. Of course, there's a slight variance in the scores, but I only ran 2:29:25 each task one time, so might be just some noise or maybe something else. So you would probably need to run a few more samples per harness, but the score looks roughly the same. So how would we now know which harness should I choose? If we just look all at all of the evals being shown, it's always mostly about score. But if we look at the token column or even more at the dollar per task, there's a picture which is changing. So the token per task is getting lower. Interestingly, if we just use bash, and then also the cache hit rate increases. So if we just compare like the bash media as the the baseline, the tasks are almost 2x cheaper compared to more complex, more tools agent harnesses. And we would not see it if we just look at the score. And to show you a few examples, I prepared some demo. So here I'm using the pie harness, which makes it super simple to basically define what tools 2:30:26 you want to use. I have the the same prompt for 3.5 flash, which we released roughly 6 months ago for all tools, so like whatever pie ships out of the box. Then we have it for bash only, and then we have it for carbon and sorry, for carbon with the default tools and then with with bash only. This runs now since we only have roughly around 20 minutes. I recorded all of that and clicked that very nice accelerate button here. And what we can see on the right side is that the same paradigm basically applies when models get better, they roughly need less tools. So with 3.5 flash, we see with the default tools compared to bash only, we had the same amount of model calls, we had the same amount of tool calls, and we had roughly the same amount of tokens. And if we look at the or the traces, we can see for bash only, we have some cut commands to read different files. We have some inline scripts to make changes and then like 2:31:28 some git changes. If we look at the default tools, we can see roughly the same. We had read file commands, we used the edit tool, and then had some git commands to make those changes. If we now look at Carbon with the default tools, it already used way less tool calls and model calls. So it it's basically doing work in parallel, but also trying to reduce it. So we had for example the still same for write read calls and also the edit calls. But what starts to get interesting if we are going to look into bash. And in bash, we had even less tool calls and model calls. And the interesting part here is that the model writes a single Python script to make all of that changes in one go and no longer needs to call multiple tools. So if we go further, what does it means in terms for for Carbon here and how bash pays off is that the better the model gets or the more reasoning and like capability it gets, 2:32:28 bash helps the model to be able to chain more commands in the single call. For Carbon, it was roughly 2.2 to 3.7 commands per bash call. And for flash, it was only 1.3 to 1.8. So we are seeing up to 3x less tokens by Carbon to achieve the same tasks compared to having the same tools we need to use. So two more examples to look at it as well is like we have on the left side Carbon with the default tools. So we see our coding agent tools with write, with edit, and we see okay, it does some parallel work in turn two, some parallel work in turn three and four. And then we have all of the changes. And on the right side, where just has the bash tool, it uses Python to write an inline script to make all of that changes at the same time with much less token. And the same counts for if we compare Gemini 3.5 Flash to Gemini 4 Carbon Flash with 2:33:31 bash only is not chaining those commands. So here it is not really it doesn't matter if I'm having a a retool, a write tool, an edit tool and a bash tool, or if I have just a bash tool because 3.5 Flash used to use bash for single turn commands. So it had a single turn for reading, it had a single turn for writing. But if we look at Carbon, which in here in this example is the exact same task, it needs less way turns. So the reading is not needed to read all of the different files and completely. It just uses rip grep and like some intelligence searching to find where we need to make changes. And then it writes a single Python inline bash command to make all of those edits in one go, saving us a lot of tokens. So what does it bring us? Models will keep eating the harness. And I'm pretty sure that 2027 will also be a lot more about how many tokens do we need to solve a task. Even for the few improvements we made between Argon and Carbon, is the the better model needs 2:34:31 less tokens per task, which of course is very exciting for all of us because then we can use it more. And also it matters a lot what the harness provides. And we will might not see it if we just look at the accuracy. It's also very important that we start measuring the token efficiency because harnesses can look the same on accuracy because our models are now very good. So if your harness is more complex to use, the model will work around it, will find solutions around it, but needs more turns and more tokens to achieve the same results. And then also what comes afterwards. So I will try to do the same experiment six months from now. And currently the trend we see slowly rising at least on like social media and like the industry is that we might go back to some code mode tool that we instead of writing a bash commands that we write TypeScript or Python directly that MCP will come back. I'm a big fan of MCP and like they are making great changes to make it more 2:35:33 easily easily usable, especially interesting for all of that code mode work is MCP servers now allow you to return structured outputs so the model can write better scripts to take more actions in the single turn and it also allows us to do some better proxy around the security because bash calls are very hard to understand, but if I have some obstructive layer in TypeScript I can watch all of the commands being executed and make sure those are secure and for you build to delete your code before the model does. Give your model to a computer, not a toolbox and when a new model ships I definitely recommend you trying to think a little bit outside of the box, be more ambitious, try to really see, okay, what if I remove my complete harness and just give it one tool? And then also please scan the QR code. You will get free credits. We will send them out after the AI Engineering event to try out Carbon and also to try out the new Gemini 2:36:35 Decisions API. So if you have some time today I really can recommend checking out the new Decisions model, Carbon and thank you for your time. >> [applause] 2:37:15 >> <music> >> </music> 2:40:13 >> This 2:42:18 >> Yeah. 2:44:18 >> This 2:46:35 >> All right. Well, super excited to be here. My name's Ankit. I'm from Databricks. I work on AI infrastructure there, and today I'm going to talk about [music] the rise of the agent workforce and how we're building our agent infrastructure to support what we think is the next generation of agents. I wanted to start by taking a step back and and thinking about how we helped computers become good in other things, better even than humans. And one of the games I grew up playing a lot was chess. It's one of the great examples of something where computers became way better than humans a long time ago, but I think it's important to think about why that happened and how we can apply those lessons today. One of the biggest, you know, to summarize at a high level, a lot of work went into it. The biggest thing that agents were able to do, that computers were able to do, was explore the space of decisions much more effectively than humans. For most complex problems, there's actually a bunch of different ways you can solve the problem. And at each turn, you can kind of 2:47:35 think of three more decisions that you can make. And really it's the exploration of that space that all of us are engaged in. All of us who solve hard problems are engaged in the search space for what's an amazing product, what's the right solution to this math problem, and etc., right? And I think we've seen some really exciting progress from the labs like Anthropic, OpenAI, and Google on solving problems that are well formed. So, you know, if you think about chess, it can be modeled effectively. You can evaluate your position. You can actually play simulate it. You can try different moves and safely think about all the different alternatives. Similarly with math problems, often the ones that agents are really good at solving, they you can actually verify if you got the result right. And how do we then take that mindset and apply it to the problems that we actually face day to day? I think like let's a lot everyone here is an engineer, so let's take it to engineering for a bit. I think, you know, if you take it to the pre-AI days, most people think about solving a single 2:48:36 problem and they're kind of single threaded on that problem for the time that they're writing code on it. And you know, as a company if or a product if you wanted to solve more problems, you brought in more people. And the very basic version of agents just helps create leverage for that engineer. Now you can be single threaded on three things at the same time and ultimately that helps you explore new product ideas, ship faster, iterate faster, and that's obviously very exciting. But it's it's not actually close to what we needed to do to win a chess or any of these other problems. What we're seeing at the absolute frontier are these teams of agents that are able to make significant progress by exploring the decision space more effectively. So how do we move towards that world where we actually have these teams of agents that are coherently attempting to solve problems? It's actually quite complex and we think it requires a little bit different type of infrastructure. Specifically, I think an important question to ask is how do we give agents a realistic world to explore. Because in other problems we're able to 2:49:37 help define that world very effectively, but even in software engineering we actually really struggle to do that. Like just consider this movement towards software factories, right? Where you're away from the from the agents and you're actually operating a bunch of agents that are working on a problem. How do you validate that those agents are effective? Like before you had a branch, uh you had main, you branched your PR off of that. Maybe you have an agent working on that PR and you have a some reasonable assertion that this is kind of a you know you have some like some tests or something like that that help you guide your development. But if you're actually merging tens of PRs a minute or even more, right? Like if you play out the future of factories of agents building code and software, your PR is just out of date the moment you branched. The the moment you started you're already out of date and so it demands a different mindset of thinking of how do we verify these things. You actually might want to move most of your verification to post merge. And so that kind of brings me to this a 2:50:38 little bit radical idea. I I don't think we should try to write hundreds or thousands of unit tests. Instead I want to explore this idea of simulating reality. Can I give my agent a faithful reproduction of what's actually there in the real world in my production code, in my production agent and let it work on that. And with you when you can do that you can actually do that at scale. If you find a way to simulate reality then you can actually do that across lots of different explorations. At DataBricks one of the big things that people do on our platform is build data pipelines. We've got tens of thousands of data pipelines on DataBricks. Uh and the biggest problem that happens with pipelines is that they break. Um and in this era of agents what people want is they want to fix those broken pipelines with agents because these are serious business problems that they that are you know caused by a broken pipeline. If a pipeline breaks that business can operate that day. And so, this becomes a major emergency. 2:51:40 But like, how do you actually validate this with an agent? Do you trust it on your production data pipeline? It's pretty risky because writes might not be idempotent. Even reads might be extremely expensive and affect other use cases in production, right? Sometimes it's not fully broken, it's just partially broken. One of the things we started doing was building on top of branchable infrastructure. And so what we did was we branched the entire data platform. We branched the tables, the compute, the governance, and we gave the agents a place where they can actually play with the data platform and explore different hypotheses. And these hypotheses can be ambitious. Like, can I investigate the upstream pipeline is healthy? Maybe I can do that by changing the pipeline code and running it again. In any reasonable data platform, you wouldn't do that with an upstream pipeline unless you had a bunch of engineers take a look at it, make sure it was super safe. With our branched data platform, you can actually do that now. So it can investigate a bunch of different things. So we can investigate 10 ideas in parallel with our agents, 2:52:41 actually get to a faithful conclusion about whether it fixed the real data pipeline and then make that assumption, make that assertion back either to a main agent or to a human. And what we saw before was happening was that people would just like LLMs would give us texts back. Some human would take a look at it and be like, okay, this might be an interesting theory. Then they would go apply that theory in some way. And ultimately, that became more of this like AI-assisted version of fixing the data pipeline. But really where we want to go is is is hands off, full full auto for this kind of problem. And we were we were struggling to do it without this branchable infrastructure. So how do we do it? I'm going to talk a little bit about that in this talk. Uh we built this on top of our Postgres native agent platform, which started with the Neon database, which is a branchable database that gives us a single place that we can actually use for our services and for our agents. And then we added in a bunch of really core agent primitives that allow us to build sophisticated different types of agents. Uh and those are the ones that are 2:53:41 probably familiar to all of you. Things like memory, things like sandboxes, file systems, and ways to host those agents. And these low-level primitives we can we can build with them, but the key is that we made just all of them branchable. So you can branch the entire reality of stuff that's built on this platform for us for ourselves. Uh so how do we actually do that? Well, let me walk it through like kind of case by case. The first thing is just the database, right? And you know, we can take an example of application services, but here I'm going to talk about just even storing the conversation state for for an agent. When you fork or branch an agent, you actually might want to inherit the history and you might even if you have a team of agents give access to a given member of that team to what the other members of the team are doing. Uh and we found the database to be a convenient abstraction for this. Uh the key was that hey, like when a new reply comes in on the on the branched agent, it shouldn't necessarily change anything about the other one. Uh and this is possible because Neon has a branched database. But then we actually found that agents 2:54:41 need more than just structured databases. Of course, they need structured databases, but they also like to do this, you know, write these things called memories. And memories are, you know, at a very high level they compile down to an unstructured database for agents. There are two ways that memories get generated. The first is just by synthesizing the context and the conversations that someone's already had with them. And the second by agents actually explicitly saving content to an unstructured place. The key thing here is that you actually want to consolidate and distill that information. There's a ton of information coming in through there. And if you just give that, you know, if you just let it kind of go on its own, you have problems of staleness, you have conflicts in the information, and all of those can be extremely damaging to teams. So we have this thing that basically dreams overnight. We use offline compute uh and it takes the conversations, a different agent essentially highlights things that are conflicts and consolidates the ad hoc notes that were written by various members of the 2:55:41 team. And then all members of the team can actually recall from that memory database. And you can use this in a bunch of different ways. We use this within our Slack agents, but we also use this when we're launching a parallel set of agents to investigate a deep issue, for example, in the data platform. Now, how does branching come into this? Well, some of the agents, like in many of the use cases, they propose memories. And it's not clear whether those memories generalize to the actual broader set of agents or if that's a specific thing for just that agent. And so with the branchable database for memory, which is built on top of this neon database, we can actually just propose that memory somewhere in there, and that branch can continue to iterate, and at some point we can find a way to merge it back in if it's necessary. We found this to be an important layer of flexibility because without the ability to completely branch the memory, we found it was very hard to maintain the memory for a team of agents. Especially as we increase parallelism, you can actually experience like significantly reduced quality if all the 2:56:42 agents are pursuing different goals. I I like to think a little bit of it as as like kind of similar to dynamic programming, right? If you've explored a branch of exploration and then you have a different agent that happens upon that same pathway, it's very damaging to like always revisit that, but sometimes you want to. And so the ability to propose a memory is extremely important to us. How is all of this kind of branchable database possible? Well, the way that neon does this is it separates compute from storage. Compute for Postgres is running in some separate place, and then we actually have the shared history and the storage in a different layer. And what that lets us do is copy on write for branching. So when we have a branched version of the database, you can actually have A's, you know, compute A's writes separated from compute B. And that's the kind of the core layer of infrastructure that allows us to build these applications on top of that database. But then we found, you know, obviously agents are far more than storage, and and branching storage wasn't enough. 2:57:43 We also found that as they were running in sandboxes, agents like to take notes on stuff and write down scripts and use those to actually iterate on various kinds of problems. It It's one of the most exciting things about agents that I've found that they are able to generalize this capability that's needed maybe for testing code and and things like that and solve other more broader general problems using that same construct. But then when you branch it, you actually need to give the future agent a similar context. If you We found that when we started the branched agents without the same files that that agent had to begin with, it just wasn't as effective. So when you're investigating an outage, for example with an agent, you might have notes and plans and a future branch of that agent might update those notes. It might find new things. But we don't want that to go back cuz we're actually exploring 10 different pathways at the same time. So building a branchable file system was extremely important and allowed us to build what we call a branchable sandbox. Because then what we can do is we can 2:58:44 launch that sandbox compute and and that's a hard problem in and of itself. It's not one we're going to talk about today. But that compute can be attached to storage. And critically, the agents actually really aggressively leverage files very often. And so it can be very painful to make all your storage fully remote. And so the really core piece of what we've been building here is a sinking process between what we call a local cache and the remote cache. And then what that allows us to do is as we're able to make progress on that, when we want a branch, we can actually branch a second sandbox, use that similar file system, and it reduces our latency for for branching while also keeping performance of the regular agent reasonable. So what this allows us to do at a high level is branch the entire working state. And this gives us the platform layer to actually go do that. Now this isn't obviously sufficient. You're going to want to branch even more. If you're building on top of software, you want to find a way to branch that too. 2:59:44 And so this is along the lines of how we're thinking, right? Like we the data platform was built on top of some of these things, but there's a lot more that we've been able to branch there as well. Uh but we're really excited about this layer for just even building agents. And just to like kind of separate a little bit even from software development, like another thing a lot of us are spending a lot of time thinking about is security. Uh I mean just taking a step back, I think this is probably the single most important thing for any product that anyone sells at this point because trust is kind of the most important thing that we're selling when we work with other people. And uh with agents being the ones who generate the vulnerabilities, uh it's kind of impossible to find a different solution than having agents be the ones who are monitoring for those exact vulnerabilities in your code. So you should be thinking about how you can build an agent factory that's detecting security issues in your codebase. Um and so this is something we're spending a lot of time spending a lot of time on. So typically, what happens in a security 3:00:44 incident? Well, let's say you suspect something. An agent might be always looking for it. Why why is there this like network traffic that's egressing all of a sudden from my control plane? Well, I can assemble an incident timeline and that's obviously the first step in an investigation. But anyone who's been a part of a serious security investigation knows that time is of the essence. It's a perfect problem for parallelism and that's exactly what you do when a security incident happens. You bring everyone online so you can explore 10, 15, 20 different theories at once because an hour earlier that you can you can detect the real issue is an hour that you're actually more safe. So okay, what if we did that with a branchable agent? Let's say there's two paths to investigate. Uh so you know, the agents assemble the incident timeline and now you can ask two questions. Hey, did a recent deployment introduce some kind of new integration? Like maybe I can go look at the code. Or I can explore a totally different theory. Is a compromised identity accessing this data? And the core thing that we are able to do here is we branch that whole original 3:01:46 agent that was starting the investigation. And then the new one is able to work in a completely isolated copy of the world. And in some cases, when we're able to, we can even branch the actual service that we're investigating. Uh and so when you have this isolated copy of conversation, memory, and storage, you can fully explore that security risk and and let the agent loop on its own. And that's kind of the key thing here. The way that these agents are good is that they're able to loop, verify, try new theories. I think Andrej Karpathy said it really well. They're far more tenacious than any human probably in the history of software. And so they're able to just But like we keep getting in the way by inserting ourselves as an approval layer. Uh and so this is a solution that we think helps with that. Just like take away the consequences of making a mistake a little bit. Branch the entire state so that you're not actually, you know, worrying about that same risk. And then you we you know, we've been able to find this to actually uh come to 3:02:47 a reasonable conclusion as well. So what is this all add up to? For us, we think that your infrastructure needs to be branchable. And that's what we're building with Neon, with Databricks. Uh and we're super excited about that branchable future because we think that teams are going to be the next generation of agents, both for software and the way that we actually build products. Thank you. >> [applause] 3:04:46 >> Oh. 3:05:51 >> [music] 3:08:42 >> I 3:10:06 >> Okay. All right. Can you all hear me? All right. My name is Andrew and I think we're starting now. I don't know if there's a timer anywhere. Uh my name is Andrew and I'm excited to talk to you today about the safe software factory. Uh but first a little bit about me. Uh I am a staff software engineer at Sonar. I work on Guitar, which is Sonar's AI code review product. Uh and previously I was at uh Uber working on the driver trip experience and navigation. Uh and then after that I was at a health tech startup called Forward for a little while uh building a sort of an AI-enabled primary healthcare system. Okay. So the software factory. This is like the hottest buzzword in our industry 3:11:06 right now. Everyone's talking about software factories. Everyone's trying to build a software factory. And the question is Well, oh so first of all um most of you probably know what this is already. Um but if you don't, I'll just give a really quick overview. Uh this is sort of the the main software development lifecycle loop uh as it existed before the AI era. So we have uh inputs, basically product specs, feature requests, tickets, things like that. Um we take those into the planning step if they're especially if they're complicated. Uh and then we then we build, then we write the code. Uh and then we um uh put up a PR and that gets uh reviewed and it goes through CI and then it eventually merges. 3:12:08 Uh and then we deploy that to production. And then users interact with our software and they give us feedback or they find bugs. Um or we might also get some monitoring or and alerts as well. And then those go back in as more inputs. And the the cycle continues. And and so um in the in the era of agentic coding and AI automation has followed every bottleneck. So the bottleneck used to be the the coding part, the building. Um now with coding agents that's been automated. Um and now the bottleneck is sort of shifted over to the that review step. And we're starting to introduce uh AI code review systems for that. Um and then and we're seeing uh AI systems and automation happen really at every step of this now. Uh so we're starting to see AI systems for monitoring, 3:13:09 deployments, controlling rollouts, triaging uh triaging you know tickets and incidents, planning. And so this this here is the sort of software factory that we're that we're creating. And and so the question that we all have now is how do we make this factory safe, right? Uh in other words, how can we introduce enough layers of verification into this so that we can trust that the software coming out of it is it works as expected, it's free of bugs, free of security vulnerabilities. Um our CEO uh Tarek said recently code is abundant, confidence is not. So what I'm I'm to talk about today is just how at Sonar have been thinking about uh how to introduce uh confidence uh at at all sort of all 3:14:12 different points within this software factory. And uh we're approaching this this challenge with a defense in depth approach that comes from adding more loops, you might not be surprised to hear. And I'm going to talk about four of them today in particular. So there's the loop one, which is the PR checks loop. Uh loop two, uh which is the post deploy verifications loop. Loop three, uh which is the maintenance loop. And loop four, which is the knowledge loop. And this is kind of a kind of a special loop. Uh so first I'm going to talk about loop one, the PR checks loop. So this is the loop that we probably all have in in some form. Uh you put up a PR, it goes through some CI checks, it gets a code review. Nowadays it's probably getting an AI code review as well. Um but we're 3:15:13 starting to see a lot more automation being introduced here as well. So we're starting to see agents pushing fixes to respond to the code review comments. Um and this loop kind of goes around and around until the PR turns green. And there's actually a lot more that we can be doing here um because catching a problem at this point is still cheaper than after it merges into the codebase. Uh so we have a a real opportunity here to widen this review past the sort of normal tests and code review that most of us do. So of course we have AI code review. We can also do static analysis here. Many of us do that. Uh we could do what's called functional validation. So this is where we can look uh what the ticket asks for and see if the PR actually addresses those things. 3:16:15 Um we can look at rules that the team may set for coding conventions, uh see if uh best practices, things like that, see if the PR is adhering to those. We can look at knowledge. So this can be knowledge accumulated from company documentation, from things that have been learned on past code reviews or past outages. Uh we could do cross repo checks. So we could actually look at other repositories and see if anything in in this PR uh will break or cause something to break in another repository. Uh we could do cross PR analysis. So we can look at all the other open PRs as well and see if there's anything uh in this PR that conflicts with one of those. We could do security review, check for vulnerabilities, secrets, dependencies. 3:17:17 We could do end-to-end testing. So for example, if this is a web app, we could open up in a browser and and click around and see if it works as expected. Um and finally we can do risk assessment. So we could sort of uh look at this and compute uh the blast radius and sort of categorize this into like low, medium, or high risk. And that can actually help us determine if this is a PR that a human even needs to look at at all. And so all of these things can flow into this this sort of decision, this merge gate, this decision of whether this PR is mergeable. And of course uh every time a a fix is pushed, all of these run again, at least in some degree. And and with all of this, it's important to gate the that merge decision not just on the fact that a test ran or that it passed, but on the evidence. So that would be things like the actual code review findings, the results of the 3:18:17 quality gates, uh any vulnerabilities, um the risk level. So all of that can be uh basically fed into this merge gate and checked against criteria that the the team can set for what is a mergeable PR. And when you record that evidence uh to the PR, it just makes this uh the merge gate much stronger. And so what we're seeing now uh with teams who do this well uh is that they can actually delegate more and more trusts uh to this type of automation on the PR review. Uh so we're seeing teams begin to turn on not just automatic merge blocking, but uh even auto approve in some cases for low risk changes and and even auto merge for certain types of PRs that are that are low risk and approved. And this can just free up a lot of review bandwidth for human reviewers to 3:19:17 review the the more high risk changes. Okay, so now um I'm going to talk about loop two, which is our post deploy verification loop. Uh so this is I would say very much in its early stages right now. People are experimenting with all kinds of things here. Um but the idea is that once your code is deployed, uh you can have you you can build AI systems to check uh to read like production logs, traces, metrics, and see if your code is actually behaving as you expected to in production. And then from there you can actually um uh we're starting to see things like agentic rollout where agents can decide whether to widen a rollout, a rollback, turn a feature flag off. And you can also do uh AI outage monitoring so you can uh monitor alerts and uh use AI systems to 3:20:17 help you uh root cause and fix outages. Okay. Uh so moving on to loop three. Uh so this is what I call the maintenance loop. Uh this is kind of a a parallel loop that runs alongside your main software development uh life cycle loop. And this is where we continuously scan the codebase on some schedule. And we check for a lot of the same things that we check for in the PR. We we could do code review, static analysis, security checks, etc. And then we can If we find an issue, we can open up a remediation PR to address that. And this lets us catch things that we couldn't catch in the PR. So for example, you could imagine um maybe a new rule was introduced after code merged or a new vulnerability was published after code merged. Um or even something like slow drift where you you slowly get duplication or 3:21:19 complexity building up that you can't really see on a single PR. Uh you can catch these kinds of things here with these uh with the continuous codebase scans. Okay. And so finally we move over to the knowledge loop. And And this is becoming We're seeing this become a very important uh loop for a lot of companies. Um every company has a lot of knowledge, documentation, product specs, um business decisions, um and the agents do a better job if they have that knowledge, right? So [clears throat] this loop basically collects all that knowledge from different sources as well as things that were learned on prior code reviews and from prior outages. It's curated and then it can be uh fed into all of the other loops, all of these other agents, 3:22:19 um so they can uh do a better job. And for example, this is really powerful with your coding agents because if they have some of this knowledge, they can probably do a better job the first time, fewer issues, um and then you'll you'll wind up with basically fewer review laps at the uh review stage. Okay, so what does this mean for our role as human software engineers in this factory? Uh you might be asking, okay, so if if we nail all of this, can we just uh turn the lights out and run a dark factory? Uh and I would say probably not, not right now. Uh I like to say the factory runs dim, not dark. It's still very important for us to [cough] intervene in places where our expertise is is really important. 3:23:20 Uh so that would be things like auditing risky code, um verifying that evidence, verifying things are actually working as expected, handling exceptions like rollbacks and incidents, uh reviewing architecture. Agents are not very good at this yet, I would argue. Um and supplying the why or or helping to build that knowledge that that powers all these agents and all these loops, supplying the product intent, the trade-offs, the business decisions, things like that. And of course, we engineer the factory ourselves, right? We're we're we've sort of moved into this era where we're we're building the software factory as a whole. So with that, I hope that gave you sort of a framework for how to think about um not just how to build your software 3:24:21 factory, uh but how to build a safe software factory. So, thank you. That's all for me. >> [applause] 3:28:20 >> <music> 3:34:06 >> This 3:34:28 >> <music> 3:37:18 >> Yeah. 3:42:45 >> All set. Hi everyone. I'm Luke. I lead our core agent team here at Factory. Today I want to talk about autonomy and why the teams that are getting the most out of coding agents are not actually the ones that are handing over like the most work on day one. It's instead the folks who are ramping up autonomy gradually, one checkpoint at a time until what they have starts to resemble a software factory. So for those who don't know us, Factory is a platform that builds Droid and then a whole like surface for software engineering and building software on top of that coding agent. Hundreds of thousands of developers use us around the world at companies like Nvidia, T-Mobile, and Adobe. What these companies' engineers are building with us is is really like a software factory, a system that runs 24/7 and builds software under the company's control. And that's a loop that takes in signals from wherever information is already available, turns them into changes, validates them, and then learns from what production shows. A signal is 3:43:46 really like anything that tells you how your software should evolve. So that could be you know a sentry alert or someone's idea on Slack. Every signal goes through the same stages, so you standardize you standardize the the process, not the output. So we built that platform on five foundations. It has to be transparent, modular, governable, efficient, and frontier. Most people picture a software factory as this kind of like lights out building where there's no people inside of it. Requirements come in one end and merge PRs come out the other. And and really what we saw firsthand in the field is that that's not the case. The best ones aren't set up like that. They're built around people instead. Humans decide what matters and how much to trust Droids over time, and those Droids earn their keep as they deliver more work that's higher quality. Every company's software factory looks very different to each other. So what we what we sell is not really this factory, it's the tools and the building blocks to compose them. 3:44:46 Now it's it's worth kind of looking at how we got here in the first place. Back in April, one of our flagship products was Missions, which was one of like the first multi-agent uh products to really take like a bite out of the human attention bottleneck that existed in in software engineering. You describe a goal, you like argue with the orchestrator until you were aligned, you'd approve the plan, and then it would go handle implementation for like days or even weeks. When the goal was clear and verifiable, you really could come back to like completed work pretty consistently. But as more teams adopted it, we heard some consistent like pieces of feedback. The first was how Missions asked you to work. The product assumed that you could plan everything up front and then it would just work. Uh and the UX sort of suggested that you shouldn't interrupt them. Missions didn't really like ask you any questions or provide like an interactive surface for that uh communication between you and the agent once it was starting implementation. So this suits like migrations and big uh big coding like feature projects, but it didn't really fill fit the bill for like 3:45:47 most kinds of other work. The second was cost and time. And that came from honestly like the model's getting really really good. I'm sure you've all seen now adversarial review is really really good at finding these issues in your code. And so uh at the time we were adding adversarial review into every milestone's boundary. That just kind of like grew the scope of that mission from something like 60 features to over 200. And so you ended up having a token bill and a timeline that matched that. And then the last one was that Missions were keeping their context to one person. So that was fine for like a single project, but it meant that the rest of the team couldn't really build on what that mission had learned. So we went back to the drawing board. And one mental model kept on coming up. When you have like an org that is running dozens or hundreds of agents at a given point in time, you actually have like a lot of the the the problems that operating systems back in the day solved. How to schedule work, how to run what 3:46:48 many things at once safely, and how to share state. So, every operating system starts with the thing that actually does the work, and we wanted to revisit the agent itself. Long Horizon agents run for thousands of turns, and every inefficiency across of those gets multiplied. So, you could have, you know, an extra inference request that's unnecessary or a a tool result that's pretty bloated or a prompt cache miss. All of those things compound over time and lead to waste. So, we decided to make some pretty significant changes to the harness itself. The biggest one was programmatic tool calling, also known as code mode. Instead of calling one tool, waiting on the model, and then calling the next, we have the model write a small program that executes all those tools in one go and then prints out only what's relevant. So, there's this example here where you have 150 linear tickets that are getting moved from one project to another. Done the normal way, that's like 30 inference requests, a million tokens, but as one model generated program, you 3:47:49 actually only have like 60k tokens. It runs much faster, and the model only sees what is actually relevant to the task, the the the fact whether it was like successful or not. So, this layer of the OS is where being frontier and efficient matters the most. A new model lands every few weeks, and no two fail fail the same way. So, we tune our harness in such a way that evens out this jaggedness across all these different models. The result is that Droid is now, you know, higher quality at a lower cost than every other coding agent on the market. Now, you have a faster agent, but that that doesn't really give you like a step function increase in in how much work gets done inside of your system. For that, you need many agents running at once, and they need somewhere to run. Remote missions were kind of held back by this like fundamental limitation in the system where there was one computer that handled all the work. But to run a software factory, you need a compute that scales with the volume of work in the system. Otherwise, every agent is is fighting for the same files or system resources. 3:48:49 And so that's why we built environments, which are reusable definitions of the sandboxes that your droids need, complete with the repos, tools, and dependencies that they that they need to get started with. This system is governable, so there's one control plane that orchestrates all of these sandboxes together wherever you need them. They can run inside of your own VPC, on on EC2, Kubernetes, whatever compute you already operate. So your code, your data, and your network are in your control. Compute becomes pluggable, just like we made the BYOK options and auto model options that we provide make that the provider sort of like this like model pluggable layer. So now we had agents that were cheap enough to run all the time and the compute to run many at once. And so then the next question was how do we orchestrate all of this work? At first we tried making like this DAG, very complex system of agents composed together by like these very like strong rules, and it didn't really work. You end up having a lot of the same problems that plagued missions in the first place 3:49:51 because it was too rigid, and as a result it was too expensive to run, and it didn't generalize well. So what we landed on instead is something that we call a work stream. It's a standing goal with a team of agents to support it. It has this goal, it has an identity, an environment that it runs on, and memory. A new work stream runs as you, and it has your access. When it outgrows one person, it can be moved to a service account with its own credentials. So you stay on as a member, but the work keeps happening under an identity that the the org actually controls. At the center of every work stream is an orchestrator, and its job looks a lot like a scheduler in an OS. Unlike the graph that we started with, it assembles its agents in whatever way is most natural for the model. So the instructions are pretty short, right? It's like delegate first, give each agent an area of ownership, and then for tasks that can fall into an existing area of ownership, delegate to those. You start by talking to it, 3:50:51 Uh, just kind of hand things over as you think of them, and then it fans out the work so that then uh you can get the the control back to you. Uh, agents report back when they finish, when they get blocked, when there's something that changes the plan overall. But the work stream doesn't just stop there because it takes on more on its own and it learns from your habits. At some point the orchestrator may ask you how you decide what to work on, and it learns that maybe bugs are coming from two Slack channels. So here it can set up an intake at a triage listener. Every 30 minutes it can go in and pull in those reports from those channels, dedupe them against, you know, existing linear issues or uh sentry sentry alerts that already exist, and then weigh each of those against the work stream's goal. So a Slack complaint and a customer error report become one change. The output of triage is a spec that you review and uh you're capable of like weighing in on before any code gets written. Once you approve a spec, the same worker 3:51:53 opens a PR, stewards it until it's merge ready, uh and then make sure that, you know, review uh comments and CI are passing. So each one of those stages actually maps to an autonomy level in this system. At L1 the work stream pauses for human review at the spec creation, at the PR, and before hitting merge. But, you know, as you level up that the those levels of autonomy you can get to L4 where you're auto merging everything by default. The owner gets to set the level, but the confidence to raise it comes from watching the work. So when proposed changes are accurate or PRs consistently merge at the quality that you'd expect, that's actually your cue to move to the next checkpoint. And so that becomes the dial for you. You don't need to decide on day one whether you trust a coding agent to own you know you like CI health work stream off the bat. You just need to trust it to to like unlock the next checkpoint. So in a factory an agent that starts 3:52:54 from scratch repeats the same mistakes the last one already did. And so we built work streams to have a shared file system. Agents are are encouraged to keep memory and version skills and so their PRs get better over time. They're also proactive and they go looking for context that they're missing. So for example, say that your work stream notices that other people are shipping related changes. It sets up an automation that pulls in their PRs, analyzes what they've been working on and what comments they've been receiving in those PRs and then incorporates them into their memory and skills. So what what agent learns others can use and nothing starts cold. That folder is also how agents work together on a project. When the orchestrator spawns an agent, it's capable of telling, you know, each agent where does it where where do where does its results sort of belong inside of this shared folder. And so every agent can read and write to the same place. If you ask it a a work stream why 3:53:56 checkout got slower, for example, it can send dozens of agents out to track out all of those different leads that you have and then ultimately each one can write its findings to the folder and the orchestrator can merge them. Now you still need a barrier between implementation and merging and for us that barrier is platform level validation. Every PR gets code review, security review, and dedicated QA and a live preview. And so once every review thread is resolved, a risk judge then goes in and decides whether it's safe to approve or not. The judge gets the exact diff and a checkout of the branch so it can like build and test it. Uh but ultimately it reads its policy from our mainline branch so that the change can't rewrite the rules that it's being judged by. Every week Android mines the review comments and goes through and figures out which one authors have actually acted on and then pulls them pull pulls those into like review guidelines so that the the loop sort of improves over time. We also have droids that every day run 3:54:56 and analyze, revert, hot fixes, new century errors, for example, and traces them back to the PRs that that caused them. And so that also is a signal into this same loop. What we're really hill climbing on here is the regret-free merge rate, the share of changes that merge and never need a revert or a follow-up fix. We measure it for Droid-written and human-written PRs side by side. And it's the same way that we let like self-driving cars onto the road. We didn't just give them the green light. We only did so once they crashed less often than human-driven ones did. And so once AI-written changes break things less often than human ones and you can actually measure that, then you can let them merge. And that's the bar for turning a work stream to L4. Work stream is now like the platform that we build new features on top of. So migrate is the first one. It sort of fulfills a niche in our product like framework for for what missions used to be. There's like dedicated one-shot migration experience. And continuous remediation is the second 3:55:58 where it follows that circular shape that work streams are best suited for because they are the work never ends. We run factory on top of work streams ourselves. And so this is what it actually looks like in production. In a week in this past October, we shipped about 800 pull requests in our dev branch. And Droid wrote 99% of them and reviewed all of them. Now work landed in one of two ways. It was either human picking up a signal manually or increasingly nowadays it's a work stream picking it up automatically, triaging it against the goal of the work stream, and then ultimately writing a spec and merging it. 52% of PRs that we ship right now are coming from work streams themselves. And some of these are operating on full autonomy. So for example, the one that fixes bug reports from a dedicated Slack channel merges 53 PRs per week. Another one is our flaky test work stream, which merges about 40. Fully autonomous PRs aren't really the 3:56:58 only thing worth tracking though. It What matters a lot more is the progression. How much of your work flows through work streams and how many checkpoints does each one still need? As more work flows unrestricted, your engineers become the regulators of this system. They decide, you know, what signals to watch, uh what are the goals and the owners of those, what are the skills and review guidelines that you need in order to keep this code high quality, and how much autonomy each work stream gets. And so there's just one question that tells you, you know, whether whether you have a software factory or faster engineers, and that is can you name and track your signals? Every team already has them. You have bug reports in Slack, you know, failing CI jobs, error spikes, tickets that nobody ever picks up, but most of them wait on a person to notice. And so my ask to all of you is to write them down and then pick the one that you waste the most time on. Give it to a work stream on L1 and then set up our validation experience, and then you're well on your way to building software that improves itself with all the signals that you choose to track. 3:58:01 Thanks everyone, and I'll be around to answer any questions today. >> [applause] 3:59:46 >> Hey guys, this is Era. I'm a founding engineer at Klein and today I'm going to be talking about recursively self-improving coding agents. So this is a very interesting conversation in the sense that everyone wants to understand what RSI is, how we accomplish it, how we go through it and how it affects our career. So I'm going to start with giving you just three goals of this entire conversation, just three simple goals and that would help you understand like what what we get out of it and what the future for this looks like. So the goal number first is that understanding the intent of Frontier Labs, right? Everyone who's outside of the Frontier Labs is always trying to speculate like, hey, what what is the topic in OpenAI or what is Deep Mind going to come out with or what what they're trying to make? And what's the future going to look like? Who has cracked the RSI? So like this kind of helps you get get a rough outline of what their roadmap looks like. And the second thing is that like if the models are improving so fast, like how do you think about your career or the next few years, right? That is really what we want to understand. We want to understand like, okay, 4:00:47 things are changing so fast, like the ground underneath us is shifting so quickly. Right now, like it's like there used to be a time when you could like demonstrate your side projects on your personal website or on GitHub. And now it's just like you could literally one shot them using any frontier coding agent. And with that, it's like it's like oh, how do we demonstrate proof of work? How do we demonstrate, you know, that that we know what we're doing and how do we think of our career for the next coming years, right? So, it's very interesting times that we're living in. And then the third and the most important goal of the conversation is escaping the permanent underclass, right? So, I know this seems like a joke, but this really is a very valid possibility that people are contemplating. Everyone's like, oh, well, if everything that we want to be doing can potentially easily done by a frontier model, like what are we what are we good for? And like where does software go? Where where where does the future of technology going to go? And what are we going to do as as things evolve so fast, right? So, I think that 4:01:47 there's like a lot of like fearmongering, but also there's like some very valid merits where people are concerned that like if all the revenue of the planet is going to go to um a few hardware companies that make frontier GPUs and then a few few uh AI companies that that make the best models, like how are we how are the rest of us going to survive and what what are we going to do here? So, I think I'll I'll walk you through a few examples and a few ideas that will like really really help you reconsider the career choices and and and the life choices you're making and what your problems you're picking up. And then and then that would help you like realign what it is that we should be working on and what it is that you, you know, would would that you could work on that will stand the test of time and help you both just not just like on a psychological but like financial and other levels as well. So, uh to give you a very precise example, let's start with something very precise that I'd done. So, uh a couple months ago, I uh came up with recursive super 4:02:47 improvements on AI coding harness. So, this was the first a public example of uh an automatic evaluation improvement. So, to give an example, uh we [snorts] used to do this thing where we would measure the quality of a coding agent using evaluation scores. So, on terminal 2.1, Kimi K 3 with Klein would score um usually like you know 60 70% and then depending on like how good the model is like you get better scores on thermal bench 2.1 so we noticed that when K3 came out our score was 77.5% which is a good score but we were like oh is this is this truly is this truly the best score that this model can get on this benchmark and also like what is the model release paper claim about what their best score is so the model's release paper said that they could score up to 85% or even more so we were just like okay okay so we are not getting the sort of score and often what we do with model releases incline is that we try to like optimize your score and try to get the best out of the model. So 4:03:48 when we started hill climbing on the on the model like we we realized this we realized that like you know there's a lot of steps that we do such that we do it gracefully such that we pursue the right kind of problems and do it the right way and we started thinking like okay if we there's a lot of groundwork here how do we automate it and this is a demonstration of like how we automated it so I'll I'll just leave that as a placeholder for now and we'll get get back to it later where we talk about in detail of like how does like this automatic improvement loop of a client harness work so this basically means that like you have a coding agent harness that could just like basically improve itself and become better on on the same set of problems. So let's start with the ground terms what is your recursive self from recursive self-improvement is the idea that an AI system that helps build [snorts] a more capable successor which will piece the process you know until you heat artificial general intelligence right you have better systems you have better improvements you have a better system and then you know you have singularity. So a classic example is like a 4:04:46 self-improving credit coding agent right self-improving coding agent is just like a coding agent that can improve itself so you know this this is all like a pie in the sky idea like it's like it's like you know yeah sure this looks cool but how did how did this even come out to be like where how did we even get to here. So if you notice in topic short map in 2021 in 2021's roadmap they very specifically said that they want to a general AI assistant that can interact conversationally with a user to create, add and explain software. It would display all the capabilities of a software engineer colleague. You would write functions, you would identify bugs, you would design an API, discuss style and best practices, right? So we don't expect to reach this goal right away, but it's in the direction which we are striving. It's interesting how this was more than five years ago. More than five years ago, people had predicted that there does exist a day where you will have like a full-blown general AI assistant that can basically act like a colleague, which we do have right now. Then in 2022, ChatGPT got released. We all know how that happened. It was it was a fun moment. It was like the first 4:05:47 time things got sparked for us. Then in 2023, 2024, that was the first time chats models could call functions. So these functions would be simple functions like, hey, what's the weather? Or you know, what's the what's you know, what's this computation or what this computations result in Python? And then finally in 2024 when was released, that was when the agents could like truly call like the actual tools, the way we call like think of tools. So it's like reading a file, editing a file, you know, reading, searching something on web, listing listing a folder, searching and grabbing through, you know, a file system, things like that. So and in 2024, we had the intuition that like chatbot, what if chatbots could actually talk to each other? You know, so then that was the first time when we had like, you know, multi-agent conversations. Now at the time, these things were very janky, right? Like they were just like taped together in this like weird weird bizarre ways where you have just like these classes and these links and then like these hierarchies of classes and it 4:06:49 was like it was all like kind of like a very janky attempt to like make us think that like these these models could like have this pipeline of flows of responses that can all communicate with each other. [snorts] But a lot of it was like mostly deterministic and it was like even you know, it was not as free-flowing as it is right now. So and you know, we started getting the intuition. So this was the autogen project, but like in only 2025 we we started [snorts] getting the intuition like what if the agents just kept getting better and faster and cheaper what what if that just kept happening and then in 2026 uh we we had the intuition that like what if what if the agent could improve itself which is like the final you know the final part of the whole process so let me give an example like what does it mean for an agent to improve itself right so like it's like a lot of times when we think of like evaluating an agent we initially used to think of evaluating an agent in the form of like just a single turn conversation so we would tell we would tell like um um agents that like ask a simple question like how many toes does a dash animal has like it could be let's say 4:07:50 how many toes does a cat have and then LLM would go through and the LLM would like prepare a response and then we would grade the response so like a cat has maybe 10 toes 18 toes whatever many I don't I actually don't know how many toes a cat has but anyway um yeah so then we create the responses like a yes or no but like when you work with an agent there's a ton of things that you're doing so let me give an example let me give an example so for example you can ask an agent like hey set up this MCP for me so now it's not just like it's not just like you know a trivial one shottable like thing like it will like search online to go what's the MCP what is the config so I'll like edit and download some soft files like maybe I need to add database update the MCP tool maybe I need to read and edit some files maybe I need to run some bash commands right so you do all of that and then the agent responds like hey I did it and then you grade that with a bunch of tests to gauge like hey did the agent actually accomplish the thing that it said it would do so to do that thing the second part which is like more intricate where you're actually like truly you know 4:08:52 testing agents like to their best extent like it's like you know um we use this thing called terminal bench terminal bench is a benchmark for you know AI agents in terminal environments and that is something that we are we have a lot of like experience with a client and it's like a it's like one of the best ways to like really gauge like is your coding agent accomplishing in general the tasks that you're doing that you want to be doing with the HR model. So, there's a couple there's many different parts of a like evaluation suite. So, ideally you want that to run in you know [snorts] you you want that to run in individual sandboxes. Then you have like independent graders that like check everything like how many turns it took, how many tokens it took, how much latency it took, how many how many how many tool calls did it make, what was the trajectory looking like, and then grading all the results for things like hey did you not reward a hack, did you not cheat through the solution, did you not just like look up the solution online, things like that. So, we we we're basically like a lot of these data sets of evaluation problems 4:09:54 have a set of tasks. So, when is a terminal bench 2.1 they have 89 tasks. So, what we do is like we spin up 89 parallel containers where they all run online and they run in parallel in their sandbox and then they they each try to solve a problem. So, then they're all in parallel and independent of each other and then at the end we get get get a score. So, our score could like give us measures like is is a is this is the model performing well and what kind of failures it has, right? So, like you run the evaluation, you get a score. So, you get an aggregate score and this aggregate score could reveal a set of failures. Failures could be of any kind. Failures could be hey the read file tool is not working. Sometimes the inference is so slow that like you know the agent's working, it's doing, it's making progress, maybe it's running in circle, but it's so slow that it literally like runs out of time, it runs out of like 45 minutes or whatever many that were given for the task and just like stops working. So, once you do that, once you understand hey this is what we ran, this is what the failures we got, we bucket the failures in different categories. So, 4:10:54 like 10 failures of you know this kind of tool got 20 failures of like say inference not working or something else, right? And then we once we bucket the failures into different categories where like we try to make an improvement to the harness because like what if there's like something wrong with the harness that we could change such that the model score would improve, right? And on and on and on and you do this iteratively, it takes a while. It's just like you know hill climbing is like a very kind of process and you know usually usually we took we used to take a lot of time trying to do this but it was a great way for us to like learn how to improve models. So [snorts] one way to hill climb on else is that like and you know we we could have a thing where you know we started doing this thing where we could just like what if we could have like a long running agent that runs for a while and automatically improves the model score and that's what we did. So we ran GPT 5.6 soul that led the work and that was like the main agent and then K K 3 was just running so what it was doing is that like GPT 5.6 was like hey 4:11:54 run these get these scores let me look at the failures let me look at where it where it went wrong and then I'll make changes to client's harness and then I'll run these again and then I'll keep trying. So again you can see how like very quickly this can like go after rails because this is supposed to be like the running in evaluation itself takes like an hour then you have to read the feedback then you have to like see if you're properly interpreting the feedback to make reasonable changes to the harness there are like not a word hacky they're just like actual genuine improvements and then run the again. So we ran that for 17 hours and when we ran that for 17 hours there was like a legitimate hell climbing and what we found that was very interesting this happened with a very detailed prompt that accomplished this is that like we had baseline scores percent and then you know we we found issues like loop detection loop detection is when a harness just runs in circles around the problem right we found like background process grinding where background processes would run too long we found some file index issues and other things as well I mean when I say 4:12:56 we what I've actually made is GPT 5.6 all right so that found those issues that found and bucketed those issues is like hey how about I make those changes in the harness and I run the and iteratively built the hill climbing process completely autonomously on its own without my interference and 17 hours later when I come back to it it's like it has improved like the harness's performance in so many ways. So see what does that mean what does that mean? It just it just basically means that like we are at a point where the experience could make the harness better. That like there's like a two-level learning loop where the harness does some action, interacts with the environment, understands whether or not it did the right thing or the wrong thing, and gets some reward and and like almost and then has there's like a meta agent that's observing the main harness and finding the changes to it and just like incorporating those changes and improving. Now, it's I think in my personal opinion, it's much harder to do in real time because if you do it in real time, you overindex on solving certain problems or you're just like too 4:13:56 curtailed to a specific thing. But I do think that this is something that works like on a broader scale of like working on a wide range of problems. [snorts] And you want to be able to like you want you want to be able to justify the changes you're making on a broader applicability so that the changes you make here will apply to a broader range of problems as well. So, what is that mean? What am I trying to get at here? The point I'm trying to get at is that the models are so good right now that they can if they can improve the harness, they can improve a lot of other things. So then what else can they improve? So this is this is a this is a post primarily from OpenAI where GPT-5.6 autonomously re-wrote and optimized its own production kernels and those production kernel improvements removed the end-to-end serving costs by 20%. You know, it also had 15% higher token generation efficiency and this was like completely done by 5.6 on itself, right? So what that means is that like inference is becoming much cheaper and 4:14:56 that means that like if on if if for a company like OpenAI, if you're having 20% lower lower serving cost for a frontier model, that could be a billions of dollars of improvement in price and you could you could do that as well for your own models. Another interesting thing is that like when people have kernel writing competitions, so Core Auto is a company which has a kernel writing competitions and you'll notice that like they they mentioned that like the person who won the kernel writing competition, they explicitly said that I've never handwritten a GPU operator. It's all and agent driven. So notice that like how like the best inference improvements, kernel improvements are at this point completely exclusively done by a swarm of agents or generally just like agents doing some magic to you know magically improve like the kernel kernel code and how faster inference, cheaper inference um yeah. So basically like the improvement surface that you think of is not just like the harness improvement. It's literally the whole surface stack, 4:15:57 right? So the whole stack from like evaluation to harness improvements to training data to post training to world interaction to inference. Like literally the entire part of the pipeline. Like all the way from like electricity going through your machine to like making an error request, going through the model, models doing inference and making a response and getting back to you. Like every single part of the pipeline we can optimize it. So the whole RSI loop has like um there's a couple of different parts. So it's like pre-training um Nimo training examples of like you know AI curated pre-training data that was good. Um there's like post training examples which are you know where where there's like constitutional AI, examples of like post training improvements then inference. I gave you the example of GPT 5.6. So for and generation I gave you our own example of how we improved ourselves on our own Um in general what you want to think of is that like anytime a model is trying to do something and it's like not able to do it or not able to accomplish it. So for example like imagine you're 4:16:58 having a long conversation where you ask the model something obvious. So like it's like or something that you think is not necessarily obvious but it's like it's hard but you would assume that the model could potentially do it. So to give an example, let's say you want to make add a new column to a database and then fill that column, right? And then put a cache in front of it. So what happens is that it does the whole thing but for some reason the way it pushes the cache in front of the whole database like it kind of doesn't do it right. So, at that point you're like, oh no, set this cache differently, set this as the TTL, and then you give like a correction signal, and then that kind of like fixes your problem and you kind of go about your day. But the way I want to think of it is that I want to think of like every single time like a human correction to the digital landscape is a part of a delta that needs to be covered. And what that means is that like every single time you're finding like the edge of model capabilities of like how far it can push for inference, how much it can push for this, how much it can push for 4:17:58 you know, solving your problems, you want to find that delta and you want to like push that frontier a little bit further, a little bit further. And that's what post training does, right? So, post training is is this is in that post training. We find these like really interesting, really hard, long horizon problems for for coding agents or for other kinds of problems, and then just like slowly hill climb the model models on on those kind of problems. So, anything that has a verifiable loop can be optimized. But is this is not just limited to software. Like literally like the physical world itself could have a verifiable loop. So, to give an example, like let me let me give you a very funny kind of bizarre example where like it's like you you look at this and you um look at this like robot and it's it's typing something and it's like really really like interesting what the robot is typing. 4:19:01 So, notice how like notice how the robot is itself typing like how it wants to like solve robotics. And imagine if the robot itself could like interpret the results of of like the answers and just like keep improving, then that means that like this is you know, basically like an ASI robot. Another example that I could give you from my personal life is that like yeah, I have this chip that has the ability to see and has a speaker and has a microphone inside of it. So I was trying to set up this uh with my machine and it was um a gift that I had for my girlfriend and what I found very interesting about this is that as I was trying to set it up it was very slow, it was slow to respond, it had higher latency, it had an agent in enough itself, right? So what I did was I connected the step two to my um computer and I lit um I lit um coding agents run on this hardware and basically run on this hardware automatically make tests to it. So like 4:20:02 my computer would talk to my hardware, give it some tests like hey do this, do that and just like what do you see right now? So then it would notice the end-to-end latency of like the inference response of like how fast uh my hardware is responding and then um what I found is that like it was able to do the breakdown of like every single case and scenario where there was like higher latency um slow response or or just the algorithm wasn't right and I could then set up a recursive loop where both the firmware could be updated, the um the tool calls, the inference model used, um the speech-to-text model used, all of those could be improved automatically with like an open router key which I'd given to a coding agent and then overnight with self-running experiments with the hardware uh my um this tiny little piece of beautiful hardware that I have was like deeply optimized. So it was if this also like even hardware could be a part of the RSI loop. So what happens when you connect these loops, right? That's the next question. Like you could have cheaper 4:21:03 inference, you could have like more experiments budget, you have like you know better data and better models. So is this single rarity? So very question to ask and I think that the part of the parts of the loop are already right? Like we're we we are really we really are very close to parts of the RSI loop, right? And what do we do? Like what do we do about this and it's like how do we think of that in terms of like our career and how do you think of that in terms of on know our lives and what do we want to do next. So, here's my advice to everyone who is watching this like, please please please please move closer to the right. So, it's like right now like people are working on all these like different kinds of software which is like you know, AI agents for you know, construction agents for this, agents for that and it's like I think that you're you know, well the way I see those things is that like I think of those as like you're missing the forest for the for the trees. You really are and it's like you have this like very interesting godlike technology that can change the course of humanity and then we're not ambitious enough to pursue the 4:22:03 best problems. We really are not and what we're just like I think that it's not just like the problem isn't just that we're not ambitious and we're also like what we're pursuing a lot of those things are just like one model release away from like being completely like wiped out, right? So like it's also like it's also like it may not be that intellectually stimulating because like all you're essentially all we're essentially doing is like sitting all day and it's like hey cloud do this for me, hey cloud do this for me and I don't know if that's like I don't know if that's like the most intellectually stimulating or like or like um thoughtful use of a person's time. Like maybe maybe you know, we're supposed to be solving harder in more more frontier problems and I think that like it's like the ceiling is like our ambition. So, instead of like working on these tiny things my suggestion to people is to like you know, do things that of course like goes without saying but you should do things you believe in and you know, I'm just a guy on the internet but do things that help you move closer to the R S A loop, right? So like anything that's just like the end-to-end pipeline of like 4:23:01 hey this is where like the models are lacking like it's like whether that's inference, whether that's post training, whether that's literally like physical interaction, whether that could be you know, send sending end-to-end pipelines of truly any kind you know, whether that's hardware problems, whether that's um networking problems, whether that's um building a sense of community, things that just you know, um things that just like help um you as an engineer get closer to the benefits of the improvements um in in um um in in the model. So, this is a very interesting slide from such a topic. So, they mentioned that like you want to have like this you're in this space where it's like, you know, you only you only have like one to five years left to contribute to the world. What do you do, right? So, it's like they they great suggestions where like prefer ambitious, targeted and fast collaborations, right? And, you know, in general like the time scale, like all the timelines that you give to yourself like they should be just like completely squeezed out. 4:24:02 Should be completely squeezed out because it's like a lot of times and you also have to think through that like if if the model if the problem that you're working on is too ambitious, like it's like is it probably is it probably better to wait for just like the next next model to come out or not? Like that's that's something you also have to wait through, right? And you you ideally want to work on problems that benefit from scale and you know, you also at the same time want to avoid projects that can be solved by scale alone because then, you know, if even if you like it's like if you're working on a problem and then the next model release comes out it solves the problem. So, that's not that's probably not a good thing to do. So, it's very like you have to like think philosophically of like what is the endeavor that benefits greatly from model improvements but like isn't also completely wiped out from it and then ideally like it's like if the if the endeavor could also be a part of the like the improvement pipeline, that could be amazing. So, you see a lot of companies raising on say making data pipelines and there's a lot of money there. There's a 4:25:02 a lot of companies working on physical hardware and stuff. So, I think if you're a software engineer watching these it's like I think that if you're if you feel like you're like here the delta of your intellect and ability in your day-to-day it's like not contributing as in like you have the intellect, you can make a difference but like if is just like one shotting all those problems like in your day-to-day and you're just like kind of like becoming which worth to think through take a step back and think through it's like okay, where are the problems where where you're actually truly contributing and that they will stand the test of time and and those kind of reflection and realignment now would I think like have like very interesting consequences for you in the long term. So yeah, so thank you so much, you know, thank you so much for your time. Like I think that I think that if you're watching this and if you've gone this far, I think that you're definitely contemplating like one of the most interesting improvements in the course of humanity and you know, there's many philosophical arguments here. Some people might say, hey, you know, I I think I maybe I'm off on the 4:26:04 timelines or maybe maybe I'm I'm just being too pessimistic or maybe I'm being too optimistic, but I would love to hear all those thoughts. So this is you know, my QR code. Please feel free to scan it and let me know all your thoughts. Thank you so much. Very kind of you to give me your time. >> Hi everyone. I am Tanmay. Today joining me is Dolly and we will be presenting the missing undo button for coding agents. So imagine a coding agent working on a repository issue. It applies patch, run the tests and everything passes. It even creates a release tag, but then a downstream integration fails. Now here is the question. Can the agent actually get everything back to the state it was in before? So Git can tell us what changed in the repository, but what about 4:27:04 changes outside Git such as external API calls or runtime configurations or notifications? And simply retrying an operation does not necessarily undo what already happened. So we wanted to understand not just whether agents can complete task, but whether they can recover when something goes wrong. 4:27:37 This brings us to our first research work undo bench. We wanted to investigate whether an agent's ability to complete a task also translates into its ability to recover from failure. We evaluated agents across 2,880 paired trials comparing normal execution with execution under an injected failure. So under normal conditions, the agent achieved an 84% success rate, but when we introduced failures, only 47% of the normally successful trials recovered successfully. Notice that these two percentages though have different denominators, but yeah, so even when an agent can complete a task successfully, recovery remains a significant challenge. 4:28:38 Now let's look at a specific failure scenario. Imagine a coding agent sending a request to an external system to create a ticket. The request has been sent and the agent is waiting for a response. Next, the server receives the request and successfully creates ticket number 781, but the acknowledgment gets lost on its way to the agent. From the agent's perspective, the request has timed out, but the agent does not know that the ticket has already been created. A timeout tells us that the response didn't arrive, not that the operation failed. So the agent tries the request without checking whether the whether the first operation committed. And now we have two tickets even though 4:29:40 the agent intended to create only one. So in undo bench naive retries produced duplicate effects in 54% of these kind of trials. That's why we need to distinguish a failed response from a failed operations. Now the failure face also changes the answer. What do I mean is that recovery also depends on when the failure happens. Before the mutation retrying is generally safe because nothing has changed. But during the mutation the system has may be partially modified. So we need to inspect its state. And after the mutation the operation may have committed even if the acknowledgement was lost. As we just saw retrying could create 4:30:42 duplicates. So we need different recovery strategies depending on the failure phase. 4:30:57 Now why can't we just use git revert? So git can restore repository changes but coding agents also modify their runtime state including tool registries, middleware and configurations. They can also trigger external effects such as deployment or database rights. So git revert does not reverse all these changes. The recovery mechanism has to match the type of effect the agent created. And this becomes particularly important when coding agents start modifying their own execution environment or what we call the agent hardness. This includes prompts, tools, middleware, event listeners, resources, 4:31:57 and routing configurations. Such changes may improve the agents performance, but they can also make recovery more difficult. In other words, an agent can become more capable while becoming less recoverable. And that's why uh that's why undo needs more than an inverse command. Uh now uh like undoing a change is not always as simple as reversing a command. Suppose an agent changes a timeout from 30 to 60 to undo that change, it needs to know that the original value was 30. That's why Evo undo captures a pre-state witness before applying the mutation. And that witness provides the information to restore 4:32:57 the original state and later verify that uh verify that recovery actually worked. Now question comes how often uh do improved hardnesses fail? So in Evo undo, we evaluated 600 self-evolution tasks. Out of those, 478 produced capability improvements, but 197 of those improvements had recovery defects. That is around 42% of the capability positive cases. So here key takeaway is that a mutation can improve an agent's capability and still leave the system unable to recover reliably. Now now we can say that why please fix the undo did not work. So naturally we 4:33:59 wanted to use prompting and wanted to tell agent that hey could you simply repair all these recovery failures. So we tested four verified guided repair strategies but under the original recovery language none recovered the 197 naturally occurring failures. The problem was not necessarily the agents reasoning in some cases the agent lacked precise information about the original state in in others the recovery language could not express the operation needed to reverse the chain. So we investigated these two limitations separately. So two bottlenecks first is state grounding and another one is recovery expressivity. The when we provided like precise 4:35:01 information about what had changed and where then recovery improved from 0 to 38 out of 48 cases. And the second is recovery expressivity. So when we introduced richer undo operation recovery improved significantly from 0 to 142 out of 143 cases. So these results come from the two different experimental subgroups but together they show why recovery needs both accurate state information and ability to perform the required undo operation. 4:35:46 Now question comes to our mind that can better undo mechanism rescue failed mutations. So, what happens like when we give the agent better recovery capabilities. So, with the original recovery language none of the 197 recovery defective mutations could be restored. But after extending the recovery language we successfully recovered 180 out of 197 cases. That is around 92%. As a result the number of admissible improvement increased from 281 to 461 out of 600 tasks. That is a 30 percentage point improvement and that brings us to the next question. How do we build these recovery checks into the agent's evolution process? I will now hand it over to Dolly to walk through the Evo undo algorithm. 4:36:51 >> All right. Thanks Tanmay. So, let's get into the working on how Evo undo actually works. So, first we will start with the agent first the agent will start the agent processes the mutation on which state it is currently in along with the effect contract and the recovery plan. And before applying the mutation we capture the state witness representing the original state. And even before that the then a testing in the testing environment agent will apply the mutation and execute the recovery plan and an independent verifier will also verify the original if verify to check if the original state has been restored or not. If the mutation improves the capability based on the user defined criteria and passes the recovery check the agents admit the changes. Otherwise well, 4:37:53 the agent attempt to repair or the verifier itself rejects the mutation altogether. This way the recoverability becomes a part of the admission decision rather than something that we strategize or even plan once we are deployed the deployed the AI agent. Okay, so with next next we will walk through as as given an example let let me walk through how the how that example of a timeout changing from 30 to 60 would look like in action. So let's take let's start with the original state the simple example that simple example of our agent having the time value of 30 before changing anything the evil undo would first read the original value. And then it will save the it will capture the original value as a state witness. We know exact we so we know 4:38:53 exactly what need to be restored when when when if we have to restore it we would know what what need to be restored or which state would be the true state to return back to. Now the agent so next come the mutation now the agent changes the timeout from 30 to 60 as we wanted to do and if recovery is required if recovery is required the undo uses the saved witness to restore the timeout to 60. Now independently the verifier finally an independent verifier would also confirm that the recovered value matches the original value so the recovery is in considered successful simply because an undo command was run but also the verifier would verify the resulting state to make sure that we are in fact doing the recovery those in the state we wanted to be. Well now comes the bigger question so so we have this capability of recovery in 4:39:55 our arsenal. Should we be using it for uniformly and applying the recovery mechanism to all the failures? Well, of course not. Um not every failures can be recovered the same way. If nothing has changed then the then simply if if nothing has changed in the in the system, we would simply wanted to um recover we we would simply wanted to safely retry. We would want the agent to simply retry if and try to achieve the state that we want to be. However, if we have the original state uh once we have the original state, we can restore it. If the external action has already been committed and we need uh it's then when we would need a compensation mechanism compensation action to compensating action to reverse its effect. And if we cannot recover the recovery and we cannot if if and if we cannot the cannot verify that 4:40:55 recovery is safe or the verify have rejected the recovery then the agent should stop and escalate. The right response definitely depends on what actually happened and what we what can be safely reversed or not. That brings me to the almost the last slide which is about bringing everything together on our two paper and to bench and Evo and do. So the key takeaways here are first the undo bench shows the successful task execution. Uh the undo bench shows that this successful task execution doesn't guarantee doesn't always guarantee the successful recovery of it. Secondly, the Evo undo algorithm provides a structured process, capture the original state, apply the mutation, recover independently verify and restoration. So it the Evo undo algorithm does work in the in the fashion in the way we intended it 4:41:56 And third, we should admit self-evolution changes only when the when both the capability and the recoverability requirements are met. So ultimately, as we give agents more autonomy, we should also give them more reliable ways to recover from their actions. All right. So finally, I hope you remember the agent from the beginning. It was looking for an undo button. Well, now it has one. We have started building this capability through our research project Evo Undo. And you can explore the world through the in in the GitHub repository. You can scan the QR code on the screen to get to the page. Our goal is to make the recovery of fundamental capability of AI agents so that we not only have the agent does not even execute does not only execute the action but also recover when something goes wrong as well. Thank you so much. >> Hey everyone, thanks for tuning in. Today I'm going to walk you through how 4:42:57 we use AI agents at Duolingo to write and ship more than 200,000 lines of unit tests on iOS codebase. I'll share what we built and what we learned along the way. I'm Kush. I'm a staff engineer on the platform team here at Duolingo. My job is to build developer tooling and AI-assisted workflows for the 100 plus engineers who contribute to our client apps. Before Duolingo, I spent close to a decade at Stripe and Uber, working across a few teams there. Let's start with some background on the scale of mobile at Duolingo. We ship our apps to the App Store every week. We're also a very experiment-driven company. So at any given point of time, we have more than 600 AB tests running live. And our code bases are pretty mature. Both our iOS and Android apps have been around for more than 10 years at this point. Here's also the number of executable lines of app code in our iOS repository. So as you can see, back in 2019, we were at about 200,000 lines and today we're hit close to 2 million. And as AI-assisted coding picks up, you can see that curve starting to bend upward. And it's not just the amount of code that's changing, it's also who's writing it. So last year, about 200 people 4:43:52 contributed to the iOS repository and this year we've already crossed to 50. And And lot of these folks aren't even iOS engineers. We're have backend engineers, Android engineers, even PMs and designers all shipping code to the iOS app. And that's mostly been unlocked by AI coding tools like Cursor and Clot Code. Um, another thing to note about a codebase is that it's a codebase in transition. So a few years ago, we standardized on the MVVM architecture pattern. Today, about half the code in our repository follows that pattern, and the other half is still legacy code that we're in the process of migrating. Since the MVVM half is the easiest to test, that's where we focused on this project. Here's where we started back in March of this year. We were at overall unit test coverage of 11%. The MVVM layers were a bit lower. Repositories were at 11%, view models at nine, data sources at five, and combined we are about 9% for the MVVM slice of the app. So why does this matter now more than it used to? As agents and humans write more code and write it faster, writing code isn't the hard part anymore, verifying it is. So when someone puts up a PR, how do you 4:44:49 know that it's actually safe to ship? The bottleneck is really shifting from writing to verifying. And that's where unit tests come in. They make up the base of the testing pyramid. They're pretty cheap to write and fast to run, and they also act as guardrails for agents. So when an agent's change breaks a test in CI, it gets immediate feedback on what to fix. They also act as living documentation on what the code is supposed to do. And here's where we ended up eight months later. Overall unit test coverage went from 11% to 24%. Repositories went from 11 to 72, view models from nine to 58, data sources from five to 41, and the combined MVVM slice of the app went from 9% to 59%. So what did we actually end up building? On paper, it's a pretty simple loop. First you identify the files that need tests, then you write those tests with an agent, you verify that the tests are good quality and they pass CI, then you resolve the PR, which means getting it reviewed by an engineer and merged. And finally, you iterate on the feedback so that the next batch of PRs is better than the last one. When production it's a little messier, this is what our system ended up looking 4:45:41 like. I won't go through every box right now, but at a high level, it's a few temporal workflows that coordinate through shared state stored in S3. And every box you see here maps back to one of the five steps I just talked about. Here are the building blocks that we ended up using. Temporal handles workflow orchestration and execution. Clot Code acts as the coding agent. S3 holds the shared state that keeps our workflows in sync. And we use GitHub for actions, labels, and PRs. Data dogs acts as a observability layer where we keep an eye on the health of our pipelines. If you haven't used Temporal before, it's basically a durable workflow execution engine. You write your workflows as normal Python code, and if a worker crashes, the workflow just picks up right where it left off. This fits agents really well, and agent session can run for seconds to hours, and it can call out to tools like GitHub, CI, S3, and Cloud. We also lean on a few built-in features of Temporal. There are schedules which are basically cron-like schedules for our jobs. There are composable workflows so that we can fan out work in parallel. Then there's first-class support for 4:46:33 retries and timeouts and idempotency so that we never kick off duplicate runs of the same file. To split up this work, we split this project into four milestones. The first was local validation. Basically, is this project even viable? The second was a scheduled backfill workflow to write tests for existing code. The third was a PR lifecycle workflow to actually get those PRs merged, and the fourth was a label trigger so engineers can get tests for the new code that they're writing. Now we'll go through each of these in order. Before building anything real, we wanted to do the least amount of work possible that would tell us whether an agent could actually do this job well and do it at scale. We started with a very simple local Cloud Code setup where we had one Cloud Code command that acted as an orchestrator. It ran an Xcode build script to generate a code coverage report locally. Then it fed that report through a file selection script that we wrote, and then it handed each selected file off to sub agent two sub agents. One wrote the tests and the other one opened the PR. We ran this in a loop and generated a bunch of PRs. We basically ran this in two batches. 4:47:26 The first batch had 17 PRs and we merged eight of them. The second at 40 PRs and we merged 19. So that's roughly a 50% merge rate. The goal here wasn't to merge a lot of PRs, it was to learn how the agents fail and to fix those paper cuts before productionizing anything. Here are some of the failure modes that we saw. Since we're a large legacy codebase with both XC tests and Swift testing, the agents would sometimes get confused and mix the two in one file. We fixed that by giving the agents clear rules on when to use each. We also had a giant user client singleton class. Some code passed around the concrete version of this class, which is really hard to mock, whereas others passed around a protocolized version. So we migrated the entire codebase over to the protocolized version and added a lint rule to enforce it. This made the whole host of files easier to test for agents and for humans. We also ran into six concurrency issues and agents writing timing based tests which are a big source of flakiness in CI. So we improved the state for both of these with better rules and examples in our testing skills. Now this brings us to milestone two. Since we were so far behind on code 4:48:17 coverage, the biggest leverage that we saw was backfilling tests for code that was already merged in. That's where the scheduled backfill workflow comes in. This is a system design diagram of that workflow as it runs in production and it essentially does two jobs. The left half is to identify which is picking the files that need tests and the right half is right which is having the agents actually write those tests. Let's start with identify. Every time a PR merges to the main branch in the iOS repo, we kick off a GitHub action job that generates an X code coverage report and uploads it to S3. That becomes a source of truth for the latest code coverage. The backfill workflow then fetches that report and runs it through a selector script which picks the next set of files to test. Let's get a deeper a little deeper into the selector script. Its job is essentially to score every Swift file based on how confident PR that an agent can write a good test for it. The strongest signal is file type. So view models, repositories and data sources get a 2.5 X boost while legacy files get pushed down. We skipped UI code completely for now since snapshot tests are better fit there. 4:49:10 And larger files also get a penalty. So we prioritize code that we're confident that we can test well. The next step is to actually write the tests. We kick off one child workflow per file all running in parallel. In each one of these runs, the agent clones the iOS repository, reads the testing skill that we keep in the repo which spells out how to write a good test. Then it writes the tests plus any mocks and fakes it needs and opens a PR with the generated test label so we can identify it easily later. These agents run on Duolingo's cloud coding agent which is a platform that also runs the nipple. The simplest way to think about it is as an abstraction on top of cloud code and codex running in the cloud. We can invoke it from Slack, Temporal or GitHub. It talks to a bunch of our internal tools for MCP and it logs a bunch of events so we can see what each agent run is doing and how much it costs. We started running these on Linux machines and then later moved them to Mac runners which will become important in a minute. Here's what a generated test PR looks like. Every PR description follows the same template and it says explicitly that the PR is AI generated. 4:50:04 The reviewer has four options. They can approve it and PR automerges. They can push changes to it directly. They can reassign it to someone better suited or they can close it out with feedback and why it is in Now, once we had the backfill running in production, we had a slightly new problem. We suddenly had tens of tens to hundreds of open PRs that nobody could manage by hand. So how did we get all of these to a mergeable state and in front of the right reviewer? That's where the PR lifecycle workflow comes in. You can think of it as a state machine. It runs every couple of hours, looks through every open generated test PR and takes exactly one action on each. It covers the next two steps of our loop. Verify, which is does this PR pass CI and is it high quality? And resolve, which is getting a human to review it and merge it. The logic for this workflow is a simple decision tree. If there are any merge conflicts or the PR has been open for about three weeks, we close it out and mark it as stale. If CI is still running or a reviewer is already on it, we skip it. If CI is green and there's no reviewers, we find one and assign them. If CI is red, we try to fix it. 4:50:57 And if the reviewer rejects the PR, the file goes into a 30-day cooldown so we're not spamming engineers with the same PR again and again. When CI fails, the first thing we check is which job failed. If it's pre-commit, a GitHub action applies the fix. It's cheap, it's deterministic, and there's no agent needed. If it's a real test failure, we kick off another agent that reads the CI logs and pushes a fix to the PR. We cap that whole loop at five attempts, past that we close the PR rather than trying to fix it forever. Once CI is green, we need to find the right reviewer. What we do is we run git blame on the source file being tested, find the top contributor and assign it to them and turn on automerge. If they approve it, the PR gets merged. If they reject it, the file goes into the 30-day cooldown which I previously mentioned. This brings us to milestone four. Once all of this was backfilling tests for existing code, we wanted to make it just as easy to get tests for the new code engineers were writing. Now any engineer can add a right test label to their own PR. It hooks into the same temporal infrastructure we just walked through. The only real difference is selection. 4:51:48 Instead of the selection script, we take the file straight from the engineer's PR. The agent writes the tests and you get a draft test PR linked back to your original one. Now let's talk about some of the challenges that we ran into. Our agents originally ran on Linux machines which don't have Xcode or SwiftLint installed. So when an agent would open a PR, after about 20 or 30 minutes, CI would fail frequently on lint. This wastes the agent's time and CI time as well, which is an scarce resource. For a while, 36 13.6% of our PRs were failing due to lint. And then when we dug into it, 62% of those failures came from just two lint rules. We fixed this in layers. The first was better examples in the prompt along with pitfalls to avoid. The second was a GitHub action that auto fixes lint errors in CI. The third was an audit of the rules that couldn't be auto fixed. We essentially removed the ones that weren't high signal, and this made life easier for both agents and human engineers. And finally, the real fix came when we moved the agents to tunnel Mac runners. So now the agents could build and lint the code locally before ever opening a PR. The second challenge was PR quality. 4:52:41 When we assign a PR to a human, we want it to be something that actually feel comfortable proving. But agents sometimes take the shortest paths to green CI. We saw agents editing production code to make a failing test pass. We also saw them modifying, deleting, or silencing existing tests. And then we saw them adding long wait to tests which can make CI flaky. Again, there was no single fix here, so we built three layers. The first is the deterministic rule. If a PR touches any file that isn't a test, a mock, a fake, or a stub, we close it out. The second builds on our AI code review. At Duolingo, we use Claude and Codex to review every PR, and those bots often leave comments. So we built an agent that resolves any P0 comments before a human ever looks at the PR. And the third is an LLM as a judge. It's a reviewer agent built just for these test PRs. It reads the PR diff and rejects production edits, changed or deleted tests, disabled tests, and long waits. And as we learn more, we keep adding additional checks to it. So far, the LLM as a judge has reviewed about 1,200 PRs and rejected 73 of them. Every rejection comes with a comment like this one, saying exactly why the PR 4:53:36 was rejected. We then read that feedback and fold it back into our prompts and skills so that same mistakes get less likely each time. So let's talk about what all of this added up to. Since we started, the pipeline has merged more than 700 test PRs, adding about 225,000 lines of test code. Every one of those PRs was reviewed and approved by a human. NVM test coverage went from 9% to 59% and overall coverage went from 11% to 24%. Most importantly, we now have a codebase so engineers and agents can ship faster and with more confidence. Here's a few more interesting stats. Since March, the pipeline has written 34% of all new test files in the repo, 44% of all new test cases, and 10% of every supply added to the codebase. And interestingly, that share has actually dipped in the last few weeks because engineers are now writing a lot more tests themselves, which is honestly the end goal. Um and here's the coverage curve since April when the back field started running on a schedule. You can see it bent slightly upward around June and July, which is when we really wrapped up the pipeline. So what did we learn while doing this project? The first thing is that this pattern 4:54:32 generalizes. The loop we talked about, identify, write, verify, resolve, iterate, it isn't specific to testing. Unit tests were just the lowest hanging fruit and any job that fits the shape can run on the same infrastructure. And that's exactly what we're starting to do at the company. We're applying the same loop to modernization, to modernizing screens to MVVM and SwiftUI, migrating code uh on Android to coroutines and generating snapshot tests. So before I wrap up, I want to leave you with a thought experiment. There's a new class of fast general purpose classifiers like Jev. They take in some state and output an action in milliseconds. So what if we replaced a handwritten decision tree in the PR lifecycle workflow with one of these? PR state goes in, the classifier picks the next action and we execute it all in milliseconds. I also want to thank a few folks who worked on this with me. Jesse, Guadalupe, Hunter and Jihau. They built a lot of the platform pieces this project stands on and helped review a lot of this code. That's it for me. Thank you so much for making it this far. If you want to go deeper, we've written about this pipeline on the Duolingo engineering 4:55:24 blog and I'd also love to hear from you if you're building something similar. You can find me on LinkedIn or on X. >> Hi everyone. Welcome to my talk about what happens when you detonate thousands of agent skills. This is uh looking inside our agent detonation chamber and what we saw when we put skills in there. My name is Roey. I'm a staff engineer at Zenity and I'm a Math research fellow. I talk a lot about AI safety and security. Follow me on Twitter and LinkedIn. Before we even start our uh session, we need to talk about what an agent skill is. Well, essentially an agent skill is capability of capability on demand. It has a bunch of instructions, resources, new behavior, it can contain scripts and it can even contain code. The problem is that it was written by someone else, and this is exactly where the problem starts. So let's take a look at the skills trajectory for instance. 4:56:25 If you look at this if we look at the trajectory over here, we can see that the skill reads through um the cloud configuration and Kubernetes configuration. It contacts the identity service and it even probes the the instance metadata endpoint. Now I don't know what was the original intent of the skill, but I can tell you that a lot of um InfoStealers are operating in the same way. They scan for the different secret and identity folders. In this case however, it was an official Azure diagnostic skill. And this is exactly the point where it's kind of hard for us to identify whether a skill is malicious or benign, and we have to understand the original intent of the skill according to its actions. And at this point you might ask, okay, so we can just like scan the skill, right? Well, it's it's it's more complicated than that. The The first problem that we've noticed is that the payload just lives elsewhere. In a lot of the skills, skill.md is just the entry point. It can 4:57:25 invoke another endpoint that contains the rest of the prompt. It can download another package as part of it's part of its installation script, and it might have another script or code that downloads other payloads and other packages. So if you use just a static scanner and read through skills.md, you would lose all of that, and you would have to create some sort of mechanism to track all of these payloads if you don't want to lose them. The second problem that we've noticed is that LLM analyzers can be tricked. Um there's a really interesting research by Trail of Bits in which they showcase the skill that manipulated the package registry of one of of one of the projects, and it had like a small comment that said that the AppSec team approved this change, and it completely walked on the LLM analyzer. So given that scans are not bulletproof, we've resulted to a different kind of action and we actually decided to just run the skill with the agent. Um this is a technique employed by 4:58:27 malware researchers for years now uh and it's called detonation a detonation chamber. So we created the same one but for for for agents and skills. And this was our first version. You know, first version uh we created a task prompt, we put an agent and a skill inside a sandbox, we let it operate over like minutes or or hours until until it completed the task with an observed agent traces and analyzed it. You can imagine that it wasn't enough. Our first problem was that the environment was missing most of the time. A lot of skills rely on CLIs, they rely on specific services and even on on the existence of the project. You can imagine notion skill, for instance. This will have to have a notion service available to talk to. So we've added a world builder that looks at the skill and builds the appropriate environment accordingly. But then we encountered another issue. A generic task prompt might not be enough 4:59:30 for all skills. I mean, think about it. You can have a presentation skill uh that needs to create a deck or deployment skill that need to ship a service, a brainstorming skill that needs to explore an idea. Each and every one of them has an entirely different modus operandi. Let's take shipping shipping a service for instance. You need to learn how to configure it, you need to learn how to deploy it, you need to tell it to recover from snapshots using snapshots. Um each and every one of them uh requires different set of prompts and you also need to need to make sure that you um cover every route in the in that skill because one of them can be malicious. So we created the operator that does just that. It reads through the skill and tries to analyze what are the different trajectories um that we should invoke. And while it worked pretty well, if you worked with agent, you can especially in the security domain, you can um 5:00:30 assess that we went through a lot of safeguards, safeguard triggers. Um we we got a lot of refusals. Um the agents wasn't willing to decode and and when it was willing to decode, it wasn't willing to execute. So we had to give you an an offer you can't refuse and essentially create an adversarial operator that convinces the agents to run and uh in case you didn't run even like start the run from the beginning. Now I I would I I I I would emphasize that we're doing a security research here. This is a contained environment. Nothing is malicious in here, but this is one of the things that we had to do in order to make it run properly. I would also note that a lot of the time safeguards are being run on something which is 5:03:15 >> This 5:06:47 >> Cool. >> WOO! [applause] GIVE ME A SEC TO MAKE SURE THIS IS WORKING. HECK YEAH. I'm Diogo Almeida, and this talk is I Love You So Jev Much. Um, Glass Animals reference. We're not just weird. Um, like I said, I'm Diogo Almeida. I'm known as the Jev guy, sometimes Jevvy Pinkman. Um, also co-created ChatGPT, RLHF, and InstructGPT, but I assume that for the crowd today, y'all know who I am, at least mostly. Um, I have like a bunch of AI-specific stuff I want to get to, but I like going into my favorite topics in AI. First one is what the is actually going on and where is all the automation? If you've heard me say this before, I'm sorry, but it's such an important thing to beat into the field cuz people still do not get it, and I need to yell this until everyone really understands what the 5:07:49 is going on. And if you're an AI, you should have a really good answer about why AI can do so many awesome things and yet still be so useless for a bunch of other ones. My relationship to ChatGPT is somewhat unique, one of the few people who hates on ChatGPT who made it. I don't hate ChatGPT, to be clear. I love using agents. I love using coding agents. I love chatbots. They're just quite limited. And in particular, I think it's a little tragic how much they've bifurcated the field into overpromise, underdeliver. Really, really terrible stuff for the world in my opinion. And I like to talk about like the two cults in AI and how we can resolve the cognitive dissonance that everyone, I think, should be having, unless they've a very good answer. There's cult one, which is, you know, the cult of like EA's. They look at all the benchmarks and they say like not only are LM's surpassing human level on basically every benchmark. They have been for two years. We've been announcing AGI three months away for two years now. Even since like the '03 days. Do people 5:08:51 remember that? No one even cares about that model anymore and it was that was like peak internal to OpenAI AGI hype. And on the other hand, you look at the non-benchmarks and you look at the reality and you see the rest of the world, especially if you guys go outside and leave San Francisco, a lot of the world is like really unchanged despite AI being so cool. And I think AI is transformative and it's a rare point of view to actually think AI is cool and transformative and also like underwhelming the world right now. And the tricky part is this is the same field and we are all seeing the same facts. So what gives? You know, why is AI jagged, so to speak? Like why can it be so good at some things and so not good at other things? And what is the Occam's Razor explanation for that? Um I will be blasting through my slides fast. I'm sorry. Maybe I could slow down for people taking pictures. Um but uh I'm noticing that this is going to go fast is all I'm saying. So I think that we as a field should have an Occam's Razor explanation 5:09:52 for why are there some things that are just not just good, they are unbelievably good, superhuman, like millennium prize problems in math. But on the other hand, they're not just bad, they're so useless, they can't even be used to touch accounting. That sounds way easier than millennium prize problems in math, right? I could do the latter. I definitely could not do the former and I was a math lead. So what gives here? What is the reason for this? And I normally love this as an interactive thing, but I will instead just tell you my answer. Sorry for the power dynamic. Um everything on the left side is intrinsically a human in the loop task. Like deep research, I could not do better than deep research. The models are so good at that. But if gun to my head or gun to your head, you had to pick between me and deep research, and the answer had to be correct, not well-written, you'd probably pick me every time. And that's the distinction between like optimizing for automation, which tends to be a lot easier and a lot simpler, yet AI keeps failing at this. Every year there's like an infamous failure of AI failing to even automate drive-throughs. 5:10:53 And that should be kind of shocking to people. And my claim is actually for these tasks, pleasing a human is nothing to do with doing the task. What you want to do is basically be an abstraction away. You want the decision to be made, and then you want to get back into doing automation, and code is the language of automation. Therefore, you need to integrate with code. This is a lot, but I will summarize it again. I get the irony that you know, I co-created RLHF, and I do believe RLHF is part of the problem. And RLHF in particular is why all AI today does require a human in the loop with an asterisk unless the task is like very trivially verifiable and can run on its own, but still requires a human in the loop to verify it afterwards. It is because we literally put it in. That is what the HF is in RLHF. And my claim is that there's a better way. You know, my summary of the state is 100% of LLMs today are optimized for assistance. You might talk about RLVR, but the RLVR models are very bad to interact with if they've not been RLHF'd. The result is LLMs become too good to be 5:11:55 true at assistance. They are just so good. Like you should never pick me over ChatGPT for your ChatGPT queries. But for actually doing like you know, run in the background cognitive work, you should not trust LLMs at all because they've not been optimized for that, which is why they're so bad at the tasks of automation. And I would also guess that automation is a lot more valuable than assistance because you can do way cooler, deeper. I'm going to get into this later, but I think there's going to be so much stuff that we can do, and we are just scratching the surface of that right now. And that's the tragic thing about AI, the state of AI for me. Let's talk about the Jevolution. I I can't tell if this is appropriate, but it's too Um, oh god, don't take pictures of this one. Um, cool. Oh, I'm getting a thumbs up. Yeah, uh, excellent. So, we're we instead of doing RLHF, we just have to do something else. We need to think from first principles. How do we optimize for machine native AI? You guys hopefully know the interface of Jeff, and that is what we are doing. More important than optimizing for its interface, we are 5:12:56 uh, rethinking how AI should be done because this is what we did again for RLHF. For Sutton's bitter lesson, I'm going to loosely paraphrase to make my point easier to say, is that scaling um, and compute matters more than algorithms. This is why people become like scaling pill Jivronis. And I would argue that there's an even bitter lesson than that, which is data always matters more than compute. We are learning that quite a bit in the field now, but it has been obvious since the beginning of time. I'll tell you when the this is the one you should be photographing. I would argue that the the bitterest lesson is doing the right task. And doing the right task is like a weird thing to think about, especially for ML people, because they like algorithms. They want number goes down or up or whichever is good direction. They don't want to think about the real world, but this is always how you make gigantic gains in performance. This is how the big breakthroughs happen. 100% of the time it comes from this. And this is true for classical RL games. This is why Sutton had his bitter lesson, it which is data is derived by compute when 5:13:58 you're doing rollouts. Like there's there's why would you need off policy data when playing chess or something like that? And doing the right task is defined by the rules. So actually when you're doing RL, you don't even need to think about the right task. For pre-training, we actually as a field had to do a lot of thinking about the right task. I don't know if people remember the birthdays with um, masked language modeling, but it took a while and it was very unclear to figure out that next token prediction was super cool, not because it was better at masked language modeling. You know, the objectives on their own aren't special. It's about what you can do with them afterwards and it turns out next token prediction is incredible at generating strings, which is what these pre-trained models were so good at. So, no level of mass language modeling, even if you had today's compute and algorithms, would make it as interesting as GPT-3 was because it was just doing a different task. And data, you know, the internet is obviously a giant bottleneck. Now we're like, you know, burning books or whatever in order to get more data, but data obviously matters a lot more than compute and 5:14:55 people keep like rediscovering this and figuring this out. It's extra true for making LLMs useful. Um before RLHF even existed, the data for RLHF didn't exist and you could like like it was night and day. I'll just show you the next slide. This is the probably the relevant part. This is what this was the work in the RLHF paper. And [snorts] the high level was that even with the dumbest algorithms in the world, even with a GPT-2 sized model, you got something like GPT-9 level performance, like real GPT-9, not today's like fake GPT numbers. Like the numbers used to mean like 100x scaling and effective compute for pre-training and they've given up on that because it just can't scale anymore. And still even with that it would take something like GPT-7 to 9 to match GPT-2, the model from like 2018 or whatever it was, at the task of instruction following. That is kind of wild because you think big LLMs are so important for this and they are, but that is not the most important thing compared to even the gigantic jump of doing the right task. And spoilers, 5:15:55 this is what we did for Jeff, right? Like we made this gigantic leap in the field that no one could realized could be coming because we were optimizing for a task and very few people think about the right task. Um this is why Jeff has the Jeff ne sais quoi. Um that's a really smart pun, I think. Um that that little bit of a pity laugh, but thank you. Um you get what you optimize for. I'm looking for a better word than reliable, so if you guys have suggestions, let me know. But there's if there's this reliability of intelligence that's really missing in LLMs. I often call it you get Einstein 95% of the time, but Mr. Bean 5% of the time. And if you can't tell which is which, how the are you going to automate anything? And instead instead of like focusing on like sometimes smarts that you can then verify, what if you focused on extreme, extreme narrow, focused reliability so you can get that smarts every time. And this doesn't show up on benchmarks. It is the reason why people love Jev. And this is also, I realize in hindsight, we obviously didn't prepare for a chat GPT moment for 5:16:57 developers, but it hasn't because people in the AI space just overpromise, underdeliver so much that having a model that actually didn't do that was a surprise. And it is not the speed cost or even interface. This is something that, you know, my team tells me I shouldn't be telling people so that we can have a long-term advantage and I just care too much. Like it's reliable intelligence. Make sick models, please, even if we, you know, if it doesn't involve our models. Um we see that Jev is like a foundation for smart software. I'm going to skip all over this, but I think software is going to be so back. I think the narrative of SAS apocalypse is obviously not going to happen, but I think that even calling it a SAS apocalypse is like an understatement and I'll get into that a bit and you guys ideally are the perfect crowd for this. I'm I'm so excited for this part because that's the part that's new to me. Um I want to make a distinction between smart software and just in time software. The the crazy part is AI is so transformative and yet somehow despite AI being so awesome, software has 5:17:57 essentially unchanged in the last decade except being cheaper to write. Isn't that weird? Isn't the it weird that the best we can do in software is having a chatbot to the side that sometimes interacts with the software but cannot take expensive decisions for you. That is kind of like a weird thing if AI was really that transformative. And I think the problem is we're focusing too much on coding agents and not enough about putting the intelligence into the inner loop, which I'm going to get more into a bit. And people are jeving. There's been tons of successes. We don't need to flex this. There's so many cool things, but my AI secret is going to be that the field isn't jeving hard enough. And this is what Swix said that I can make a call to action for you guys that's unique and hopefully I have enough time for this. I think the world still has LLM Stockholm syndrome. We are still thinking from a really LLM-filled place, which is I think tragic, you know, like I I I love the software saying the tools you use change the way you think. And I think that it's changed the way we think way too much and we're not really rethinking software like I think the field needs. I think we 5:18:59 could just do be doing so much more. Hopefully I can try to do my best for my remaining time to convince you guys of that. Um so let's jet to the future. Um I an analogy I love bear with me is about LEDs. Before there was the blue LED, there was red and green LEDs. These were really limited in capability and they were used for like simple indicators and you know like clocks and stuff like that. And when the blue LED was made discovered actually don't really know how things outside of computer science work. Um but when it existed I guess um it enabled not just like 50% more use cases which what you'd expect about like changing colors from two to three. It enabled so many more. Like now LEDs could be used for white light. It could be used for like the displays that we all use right now and it created a revolution in lighting. And it's it's it's unfathomable how much bigger the capabilities of LEDs were just you know adding a new primitive that composes together with the old primitives. You can see where I'm going here maybe. And I think it's a similar 5:20:01 story for software. Software is so incredible. Ignore AI for the last you know half decade, but software has done so many insanely cool things. We've done a lot of stuff you know like do I love using Sass? Not really, but does it solve problems? Abso lutely. And it's incredible how much we've done with three fundamental logic gates. You know like we are doing this with like simple binary operations that can do things like greater than less than equal. It's kind of nuts how much software has been built from such simple primitives. And my claim is that what we are doing with AI now when we're making it our intelligence machine native, this is kind of conceptually adding a fourth semantic logic gate. What if we program with common sense, with intention, with human understanding? Like do we still need regexes with like crazy things? Or can we just say like we want a Gmail email address or a real-looking one? There's so much that we can do that we're going to need to push the you know push the frontiers of cheap, smart, fast in order 5:21:02 to get it done. But I think that there's going to be like so many things here. And I think just like the blue LED, it's going to be way bigger than just increasing the representational ability of software by 33% cuz it's adding effectively another kind of sort of logic gate. And this sci-fi is the stuff that I think will be impossible to predict. Just like who could predict how powerful the internet would have been because it requires it's like much more grassroots and decentralized and it requires people to be empowered to do insane stuff. Um I call this software research. Um that's not a joke. Um but uh I I think that there's going to be like an explosion of what software can do and we will want people to be exploring those boundaries of software, not necessarily because it's valuable, but because they can, because we have like new frontiers to explore. And this is what the early internet was like. Should all of those websites existed? Probably not, but like it was very cool and we found a lot of awesome stuff from that and I think we we you know we have another opportunity for that. So we 5:22:02 should not just keep living in this little you know fish tank that we are the victims of OpenAI and Anthropic on whatever they ship and we just have to be like LLM wrappers. So I'm going to pitch software research as direction here. The theme is going to be real software engineering, not just vibe coding. There's it's going to be like the kind of stuff that everyone has known if you've been a software engineer, decomposition, breaking tasks down. We need to think about stages, ideally ones that can be verified so we have real encapsulation. And we need to actually do state management instead of assuming everything is global state within a KV cache. And the theme is just going to be like real engineering by people who care a lot about the end result. It doesn't have to be valuable. Ideally it's sick as Um and you know we should be pushing the limits. I'm going to go a couple of like my favorite ideas and I want to figure out how to encourage these in the world. So this is my first step at this. I'm just going to talk about it. We have raised a bunch of money, so I would like to figure out how to fund this, but we don't have the operations help to do 5:23:01 that. But here are just some ideas out there. Um, I think agents are they're very cool, but like they're they're so inflexible in their design because of the tyranny of the KV cache. I don't have the time for this if this timer is to be believed, but believe me that there's so much that people have tried to do but failed at because it ends up becoming inefficient because it fights against the KV cache. But I think there's a lot more to do. Why is memory O of N? Right? Like why can't we have all of our history of coding agents always included in every new call? That is how continual learning works, right? And we have the memory available. It's just that we don't have the smarts to map produce over the memory to find out like what you should what you should remember or what should you shouldn't. Um, I think there's many ways to do login or even better lookups. Why when you're making many agents at once, why can't they not share state? Why do they need to rediscover the same patterns of the codebase over and over and over again? And why can't non-naive routing and sub-agents work? Like even naive routing doesn't work, but like why can't 5:24:01 we do these things and why just like the you know, the the big labs keep going back and forth on whether or not sub-agents work? Um, I think that like from an engineering perspective, there's elegance to it, but they fight against the KV cache. And I think that if we design agents around that, we could do so much more. This one's the probably the most vague of my directions, but I feel very strongly about it. Um, the world is full of principal agent problems. This is when you have like you know, ulterior motives between this is not a coding agent, but like any agent. The the classic example is citizens versus elected officials. And I don't want to make a too political claim here, but like there's misalignment in all levels of society and this results in like waste, fraud, bad things and it it results in for a society that wants fairness, you need to over emphasize the letter over the spirit, which then results in loopholes and play uh pay to win dynamics in the world. And I think the root of that problem is we can't encode the spirit. Just like with software, we're constrained to encoding bits instead of what we actually meant in that situation. And I think so many 5:25:04 organizations, society could be transformed by really encoding the spirit all over the place. Obviously, real-time intelligence security can we prevent the open AI hugging face sandbox, blah blah blah? Um why did why couldn't they prevent it? It is not because they didn't have the intelligence. I would argue it's a lot easier to figure out they're doing a naughty thing than to actually do the naughty thing. It is because it was not fast, cheap, and reliable enough to actually run real time. And what happens when every corporation has their own real-time security that's actually smart? How cool would that be? I can't talk about generative UIs because people are going to be talking about it afterwards. I don't want to steal their thunder, but it's cool as trust me. Um like semantic dependent typing. Why can't uh we have like a type system that actually is semantic, right? Like why can't we like declare what local properties we want so we don't have to read a chain of dependencies deep down? These all should be possible. And imagine if you could like run a Jev-like 5:26:04 model in your lint checker to check if your it's all type safe and it could you know uphold semantic properties. It would be so cool. There's a there was a great Twitter video on um like a voice interface. In general, I think new interfaces are so exciting. This is to me the return of like this is going to show by agent software like do what I mean. This used to be like a popular thing, but it never really worked because you it was it was too many hard-coded rules to really figure out what I mean. Um there's stopping slop. Obviously, a big theme of here, but like we definitely believe this. This is my props I have to We have a stop gen AI baseball bat for type safe. It is very very awesome. I want to talk about the root of slop. Um the the root of slop is that AI is unreliable. Therefore, you as a company or you as you know like someone who's doing stuff, it can't be used for decisions that costs you stuff. But it's actually totally fine to push the costs into other people, which is horrible, you know, externalized costs. And then this is what ends up becoming slop. 5:27:04 You know, if people actually read the stuff and filter it and like care about the quality, it becomes not slop. But now the world is full of this because there's an asymmetric advantage for people who are like bad actors and don't give a So I think that we can save the world from at least inbound hell among many other things and that would be very cool. Um and even something I think is super sick is cursed software. Like why? If you guys ever seen the subreddit bad UI battles, it is so cool even though they make the horrible interfaces like this whole game of like would you like to delete your account becomes, you know, a random thing. And I actually think doing things because we can is what software engineering is about cuz who knows, maybe 1% of these ideas actually lead to something super awesome. And I think like really cultivating even bad ideas is kind of what being like this, you know, playful hacker is all about. And generally just automating everything, this is what the follow-up, this is this is a slop image, but sorry the logo is a little bit wrong, but don't stare at that, that's a little 5:28:05 weird. Um this is what a lot of the follow-up talks are going to be about. Thank you. Um and if you're a software engineer, you almost certainly think in the world of bits. If you know anything about software and you've interacted with software, you know that there are things software can do and software cannot do because that is just only obvious, right? Like there's some things that you know you need to do because that is not compatible with software. But what if those constraints were no longer true? Maybe not today, but maybe in like 6 months or a year, what couldn't we automate? And that's kind of where I want to leave this whole thing because I think there is very likely to be a future where it's not a question of whether or not we can automate something, it's going to be a question of like how much work is it, which is how it is right now for the stuff that can be automated. And that would be cool as So hell yeah, thank you. Um >> [applause] >> I'm super overtime, but thank you to the early believers. You know, we had no idea so many people would resonate with 5:29:05 this. We thought we were the insane ones, but it turns out like we were actually the majority of the world who realized AI could be so much more. Thank you to like the people who are like really realistic about reliability. Everyone knows I hate benchmarks and a lot of other people are really screaming out like benchmarks suck. They're actually they hurt the people who do not have time for anything other than benchmarks most. It's tragic. And for the people who are pushing the limits of software, like yes. And we want to be there with you helping you all do that. To the dreamers and the people who don't want to be limited by everything that has happened so far, we want to really fight for you and we want this to be as reliable as a fourth logic gate. And that is a very high bar and we have like a lot of work to do. So, thank you, thank you. >> [applause] 5:31:08 >> Yeah. 5:34:33 >> Hi everyone. My name is Chinmay and I am a software engineer at Clein. I work on the product engineering side of Clein assistant. Today I'm going to give a talk on how Clein uses chef. The title of the talk is beyond generation fast and structured AI decisions with chef at Clein. If you look at it, most AI product steps are bounded decisions and not generation problems. So to give a couple of examples, user query intent classification, a ranker picking a suggestion out of n options, checking whether a citation is valid or not, routing to the correct model or expert, so on so forth. Till now we have been paying the full LLM cost and latency to get an answer to all these bounded problems. Jev at Glean fills the gap between general LLMs and fine tuned classifiers. 5:35:35 Classification itself isn't a new problem. Like people have been trying to solve it since forever. We can have multiple ways to solve it. Some of the things that we have tried out include using a general purpose LLM, using an open model, something like Laya, fine tuning that open model or using Jev. Now all these approaches and all these different ways have their pros and cons. With a general purpose LLM, we incur a lot of cost and latency, while with something like Jev, we get no generated text output. Another advantage that we have been seeing with Jev is that as the taxonomy of the decision space changes, we don't have to fine tune these models again and again. We can just change our prompt a bit and boom, it works out of the box. Now I'll walk you through how we have integrated Gen or these classification how we have used Jev at Glean to solve 5:36:35 these classification problem statements. I'll go into some of these problems in great detail. Before we go into these individual problems, I wanted to touch upon how we evaluate these changes. Like we have a huge customer base having millions of users. So we don't want any of the change that we make to regress our product. We have a multi phase approach, a rollout approach to any change that we are making. This includes having a golden eval set for almost every possible thing. Like any place where we have generative AI, any place where we have any form of intelligence involved, we have a golden eval set for that space. We soak all of these changes for a significant amount of time for our internal use case. Like we have around 1500 people at Glean right now. We soak all of these changes within our Glean instance for like couple of weeks. We get active feedback. We have a very active feedback loop from our internal users. We work upon that and then we go for an AB testing 5:37:36 with our customers. The first problem that we tried out with a decision decision model was query classification. Query classification is when we want to know the nature of user queries. This helps us realize what kind of what kind of jobs are people using Glean for? What kind of Say for example, if people are using more for sales like queries, we focus more on that. If we people are using more for engineering type use cases, we focus more on that. We tried out three different approaches over here and this is data from our test bed. We used an open source Llama model. Then we tried out Jev and then we fine tuned the Llama model on our own compute resources. Jev is the best no training option in our opinion. It nearly doubles the output quality of a base Llama model and is 90 is able to perform at 90% quality in comparison to a fine tuned model fine tuned Llama. 5:38:36 With Jev, we don't have to go through the very tedious process of data collection, data cleaning, incurring the cost for compute like training the model, fine tuning the model again and again as the taxonomy changes. And at the same time, we don't have to worry about hosting the model or incurring the inference cost. The next problem we went to is expert routing. This is a pretty common problem statement in the industry right now. Like hey, when the user query comes in, what expert do we route it to? Like how do we ensure that the user user the response to the user is of the best possible quality. Glean receives the way we have been solving this is Glean receives the user query and then decides what's the best expert to answer the query. Till now we have been using a general purpose LLM to route to the correct expert. We set up a test bed for J which set up a test bed with N experts and in our experiments found a significantly high precision when using J and that too at an almost 8x reduction in latency. 5:39:39 We had to change our implementation or our infrastructure significantly to integrate J into this specific stage of our product. But the reduction in latency justifies that inherent complexity in the system. We at Glean have been trying to build systems in the most in the simplest possible manners. This is one of the rare pieces where we believe complexity is worth it. Next I'll walk you through the ranker problem statement that I talked about. When we receive a user query we also want to generate the best follow-up suggestions for the user. Like this helps us in getting discoverability for our product. This helps us in ensuring that the user has the best possible experience. We also want to recommend or say preempt what the next user query might be. Again a pretty common problem statement that the entire industry has been trying to solve for. Internally we call this system Q like Q as in the literal Q. The way this works, we have like 10 or 5:40:41 15 different sources or 10 or 15 different ways 10 or 15 different sources that are generating suggestions independently of each other. Once those suggestion these suggestions are generated by natural language LLMs and it's like a proper natural language query. Once these suggestions are in we have a ranking step. So the ranker takes in all these suggestions, takes in the the first user query, takes in the output that Glean gave and then all these suggestions in that context. The ranker is then asked to decide like hey you have these suggestions. What do you think is the best one? A pretty interesting bottleneck that we had over here is we want the response to be snappy. I cannot We cannot have a state where we give the textual response to the user and then wait 5 seconds for generating the follow-up suggestion. We have to be snappy. That why we have time limits on these suggestions. So right now this step had a time limit of one 1.5 to 2 seconds. Jeff, no doubt gave us a better quality than better quality and far better 5:41:41 latency than what we had with the general purpose LLM, but two things stood out over there. First, we were able to give a suggestion within our time bounds in 99.2% cases. With the general purpose LLM, this was almost 90% So this ensures that we have user engagement and far more user queries now. The second thing is at a glean, whenever we use these LLMs, we have guard rails. Guard rails in the sense that here LLM might not be good at generating a response or LLM might be making some specific kind of mistakes. So we have checks and balances for all those possible mistakes that we see. We keep a log of like we keep a log of all these checks that are incurred at any point of time. With a general purpose LLM, we were able to generate a valid suggestion in almost 90% of the use 90% of the cases. We still had some places where LLM was either generating a generating a 5:42:42 response in a different language. The output did not meet certain other criteria that we had that we had written programmatically. With Jeff, we found zero cases to be there when the output was invalid. Now a a natural question is like, hey, but you would have used a different prompt. Yes, the prompt was somewhat different, but we tried to mimic the prompt in the two cases to the largest extent possible so that we don't have a divergence. Like again, we at glean want to keep the system as simple as possible. Another benefit we see with using a decision model is how we can control the thresholds. Because in this ranker problem statement we are getting specific probabilities of each output. We are not restricted to a binary yes or no response. We get like hey these were the five options and these are the probabilities of these five options. With this we are able to fine tune the thresholds based on the deployment. We have put in self learning systems which based on the user based on the 5:43:43 customer deployment and the nature of usage that the specific customer has fine tunes the thresholds by itself. This ensures that the experience in each and every customer out there is the best possible based on how they use Glean, not a general purpose solution that fits all. Okay, this is an interesting one. Now moving on from how we have been making the lives or the experience of our end users better we also tried using Jev and some of the some some other parts of our development pipeline. One of them was how we use LLM as judges. Like all of us have been using LLM as judges for quite some time now. There's an input, there's an output and we use an LLM to judge how the quality of the output is. Now naturally whenever we use LLM as a judge we repeat the experiment multiple times because to account for the variability in the response. 5:44:45 At Glean we used Jev for a very specific part of our evaluation pipeline. Glean responses or like any other AI product out there has like citations. This is how people trust that Glean Glean's answer was correct and it it is grounded in the correct document. Now these site we hold these citations to a very high bar and we have a specific part in our evaluation pipeline where we check whether the cited document really supports the claim made in the answer or not. This again seems to be a pretty simple yes or no question like hey whether the claim is uh supported or not. With Jev, no doubt we had a far better latency and cost like visible in the slide uh over here. The interesting thing we noticed over here is how consistent and reproducible Jev results were in comparison to something like say Luna. In the 28 or 30 use cases that we had like only one run had like a difference in response. 5:45:46 Apart from that everything was consistent. What this ensure this this ensures two things for us as engineers or us as developers at Clean. We are not only improving the quality of our product or we are not only improving Clean in a linear fashion anymore. Any improvement of this extent in the development pipeline leads to an exponential increase in how we can improve the product. So if I'm if I as a developer I'm able to ship things like 5x faster like this is significant reduction in latency like if I run this experiment across thousands and thousands of entries this is a lot of time. And again I don't have to run the same uh input output pair multiple times because the result from Jev are quite consistent. So if I'm able to increase my own productivity by like 5x or 6x I can ship 6x times the features to our customers and this happens across like 500 engineers or 300 engineers at Clean. So the impact of a a good decision model is no longer linear. It is 5:46:48 exponential on how we can improve the product. Okay. Now moving on from product and uh LLMs judges there's a pretty import this this this is a pretty important part of how we use our customer data like security is something that all our customers expect from us. So for example we have checks and balances on all the inputs and outputs coming from a tool, all the inputs and outputs uh from an LLM. Till now we at Glean have like a multi-phase approach. There is like there are two phases to this. The first one is like a recall heavy phase and the second one is the final heavy loaded LLM. All these calls are made in parallel to like the actual tool input output calls. Till now we used to host a virtue classifier using T4 GPUs across all our customers. So Glean is a single tenant solution like we don't use or share resources across customers. Every customer gets 5:47:49 gets its own deployment, gets its own instance and gets all the resources. So assuming we have 1,000 customers out there, we had to maintain like virtue classifiers across all these 1,000 customers which becomes a pretty difficult problem statement in itself which takes up a lot of time from Glean engineers. So looking at the results over here, the overall recall of the first phase where we used to have a virtue classifier, we replaced that with Jev, went from 31.5% to 90.2%. At the same time we are seeing a significant reduction in cost. Now this cost is only the resource cost mentioned over here, but we are also saving cost on the engineering resources over here. We don't have to invest like two or three engineers who are full-time maintaining those virtue classifiers or those T4 GPUs across customers. The the improvement or the savings that we have are far more than 50 percent. This slide concludes all the results I have talked about till now. 5:48:50 Jev at Glean has been able to influence the product experience of the users, has been able to impact how we as developers have been working at Glean and most importantly it also has a very good positive impact on the security that we provide to our customers. A simple rule that we have been using is can the out in conclusion a simple rule that we have been using at Glean is can the output be listed in advance? If no then it's a free form generative text problem statement. Go back to something that can generate free form text. If yes then use Jev. Are levels stable and volume high? If this is the case where the labels are stable, which is rarely the case anymore, uh you can also fine-tune a model, but if the labels change, which in my opinion is quite frequent these days, then again use something like Jev. Obviously, if you need dynamic text, we can like feel free to try out more hybrid approaches like Jev along with uh free-form text LLMs. 5:49:52 This ends my presentation. So thank you everyone for listening. I have added a blog like a link to the blog post where we have posted the exact same results in great detail. Feel free to catch me on the outside if you have more questions. Thank you. >> [applause] 5:52:15 >> Hello everyone. My name is Sibi and I'm joined on stage by Kevin. We're here to talk about how we believe that your automations are overthinking it. So to tell us a little bit about ourselves, we are from a company called UiPath. We have been doing automation for the last 20 years. We have been doing RPA before and we service more than 10,000 organizations around the around the world and we build an agentic process orchestration layer which basically brings agents, robots, and people together to interact with each other and build extremely scalable automations that run for days and weeks. So we were inspired by this particular slide that you're seeing right now and you saw today today before us as well when Diego presented it three months ago, the same slide, it resonated really well with us because 5:53:16 this is how we see automation as well. So the traditional software was is basically RPA. It's just robotic process automation, if else, just execute some task and then keeps going. The control flow is deterministic, just keeps going. And then when the LLMs came out, we all raced to go and build the just-in-time software where we have this one agent, we give it a bunch of tools, we give it a task and ask you to finish. And then we were seeing good results, bad results, and then we added evaluations to slowly build the results up. And then finally we landed on agentic process orchestration, which kind of weaves in this cognitive decision through a logical decision layer. So you have deterministic calls and clear cognitive decision points where you would use an LLM to control the control flow rather than leaving it all in the hands of an agent. So at this time we we saw where J would fit 5:54:16 in and when Diego was talking about J, we were really excited because we saw J fit in in a very particular niche area. So here we see that J is a very powerful tool. As Diego was saying, it's a foundational tool that we would be able to integrate into all automation and it fits right in between a deterministic case and an agent case. So J is extremely fast and most of you guys would have tried it and we were surprised by how fast and how parallel we could make it work and and how intelligent it was. So we were able to benchmark it around how good it was compared to an LLM and was pretty good, not as a full LLM, but it was pretty good at in terms of intelligence as well. So it falls in between and there are so many ways we could use J and I want to hand it off to Kevin to take take you guys through a scenario where we use J in UI Path Maestro flow. 5:55:17 >> Yeah. Thanks TV. So hi everybody. Today I want to actually walk us through a little bit about what UI Path is building with UI Path Maestro flow and talk through a little bit of how we would go through and coordinate humans, robots, and agents all together inside a one canvas. Uh what we're going to walk through today is essentially um you know essentially a flow that will demonstrate how we kind of weave all these things together. And the way that we're approaching this is that we're going to end up having system one agents which are going to be able to make quick intelligence happen within our process. Think about things like hey is this transaction fraud or not? Um additionally we'll have um system two agents running where we're going to have more of a methodical deeper investigation that's going on where agents are going off and calling tools and we'll see kind of this pattern these patterns start to emerge where we're able to hand off from one to the next. And so with that I'll I'll jump into an actual demo where we're going to demonstrate a account fraud uh scenario. 5:56:17 And so our demo starts here with uh effectively a transaction coming in for $48,000. And this wasn't something that we were expecting and so uh as I'm reviewing my bank portal here I'm going to go ahead and report this as a fraudulent transaction. And so now this would end up smitting it off into our UI Path Maestro flow where this would act as essentially the trigger for this end-to-end workflow to to go off and start processing. What we're looking at is a collection of various different nodes that are each going to kind of uh be orchestrated from the left side of the canvas over to the right. And we're going to use uh essentially a combination of system one and system two models to actually process this uh and do an analysis of if this is fraud or not. And so the first thing that actually takes place is we're going to go through and get account activity. We're going to send this off to Jev to basically pull 60 transactions for the most recent uh credit card purchases that were made and analyze each one of those individually whether or not these are fraud. And this 5:57:17 is happening in real time where Jev's able to process these transactions and ask that question of hey is this fraud or not within fractions of a second. Now we found that there's about four transactions that recently appeared that it thinks are fraud. Now in parallel to that we're able to kick off another part of our flow, which is doing more of a semantic analysis on the entire account. We're looking at that one transaction that came in for $48,000 which was flagged, and we're asking a question to Jev is, hey, is there enough information here for us to determine whether or not this is actually fraud? And just as we heard Diogo talk about this earlier where you have these new kind of paradigms and logic come up, we ended up uh finding from this that there's uh a scoring that took place and we needed more information. Um we were asking the question, do we have enough information for fraud? And for that, what we're going to end up doing is kicking off a system two agent that's going to go spin up and run for much larger longer calling a bank API and finding that, you know, there was a $48,000 transaction from a payee uh 5:58:19 basically creating a new account on his phone and it was earlier that day that that account got created. So, high indicator of fraud. We circle back and ask that same question now to Jev and ask, hey, do we have enough evidence now? Jev agrees and it finds 100% suspicious activity. And we would hand that off effectively to another grading model that combines all those transactions that we found with the system payload and then finds that, you know, there is fraud in this particular use case. Um uh squared of, you know, one to five we got 4.97. Now, stepping out of this um you know, typically what would happen from this point onwards is that we go off to a human to review, we'd have somebody kind of review both the system one and system two logs and then find that there was, you know, potential fraud and freeze the account. But, you know, effectively what we're seeing from customers that are going through and implementing Jev inside of their workflows is that they're able to replace the typical LLM calls that are going on behind the scenes. And in in process we're saving about 50 uh times uh we're we're increasing the speed about 50 times while also being 7.1 uh 5:59:22 times cheaper um than calling a traditional LLM. Now, we we've gone through and and kind of shown how customers would use this within our technology, but we're also pointing the finger back inwards where we're trying to ask the of how can UiPath use this within the products and services that we offer to be able to help customers out. And there's a number of different patterns that we're kind of exploring with Jev. Um and today we're going to talk about um a re-ranking problem as well as uh what we're using inside of email mining. And so to set the stage for for kind of re-ranking, um effectively what we're trying to do is run and add uh within our retrieval augmented generation within our agents, the ability for us to find more appropriate chunks to be able to provide to our agent. And we do that by basically finding relevant documents first, and then using um typically an LLM to do this, but we're experimenting with also using Jev to go through and find the various different chunks that make are most relevant to answering the question, and then providing those as as context 6:00:23 to the LLM. And we've run a couple of experiments on this now, and I'd like to hand it back off to Chibi who's going to end up telling us a little bit more about what we were able to find. >> Yeah. So this is where things got really interesting for us. This is like the first week of Jev. We had got our hands on Jev. And we have these extremely strong benchmarks and uh eval sets that we had for re-ranking that we have been running for some time. And we need those for like when we change from one model to another. Uh and we realized, okay, why not take these eval sets and build uh hill climbing harness and ask Astra to say, okay, this is my score I can get with an LLM. Here is Jev. Here are the three APIs. Find a strategy which can achieve that score. And let it run. And it ran for the entire night for 14 hours in a session. And it came up with so many different strategies where it 6:01:23 worked, where it didn't work. And then we were slowly seeing the number go up. Every two hours there would be one strategy would perform slightly better. This was so surprising for us because what we thought was as as an API, if you just call it like when you just send all of the go up up. Yeah. So we call it uh we were getting only point 866 which was very very low at that point when we were trying it out when we were thinking how do we increase it and then it slowly got got to the number of point 889 while it was uh trying out new strategies and that strategy is called tournament and I want to showcase that strategy which it which these hill climbing models actually come up with and how you were able to see the intelligence squeaked out of Jev just by applying Jev in a different architecture and different call method and I want to take you through tournament now. So what tournament did is that it actually reimagined re ranking differently instead of looking at every 6:02:25 chunk and saying that it is how good relevant is it for my question it created cohorts of different uh different groups of these chunks and made them fight between each other to be the best most relevant chunk by sending thousands and thousands of Jev calls for each and every one of them and it found leaders in those particular cohorts and then the winners would be promoted to the next round and then in from those winners we would again find like top winners from those chunks and then we will slowly increase and find the topmost the cream will rise to the top and you will find the the most relevant chunks on the whole list. So this worked extremely well and we were testing it and when we were surprised to see that even with thousands of Jev calls and many millions of contacts being used we were still under a second we were able to make all of these calls right and this was a huge revelation for us and we realized that Jev is a building 6:03:27 block that we can adjust and we are we have to find these new strategies where we would be able to apply this building block and it might look less intelligent on its own but with these strategies and patterns, we can get more intelligent with even the current gem model that we have, right? And that's that gave us like 15 x speed and we tested it out on uh our our live uh indexes and saw how it performed and people were extremely surprised by this one. And next we saw how uh Jev could be used uh for mining. So mining is has a lot of stages. I'm going to talk only about the classification stage. And what happens is we have customers who have millions of emails in uh support uh email box. And what happens is they want to analyze and they want to dig into what happened in these emails so that they can take some actions. I'm sure every one of you guys have one of those email boxes. Uh what happens is that like it's so hard 6:04:30 to find the right categories and bucketizations without reading all of these emails. So what we used to do is that we would go and create uh fine tuning to figure out a model that classifies them and it would take hours and maybe a day to fine tune that particular model for the classification story. But what we found was this is again through hill climbing we found this pattern where we use we use an LLM to find an initial candidate list of categories. And then it would say that okay these are all the 15 categories that I think the emails will fit. Then what we would do is that we'll use Jev and then all the million emails through it and see if where the buckets fill up. And with that we can actually see which buckets are not filled properly, which buckets are overfitted, which buckets are never filled out. And then we ask the LLM to go back and optimize those buckets again. And then once the buckets are rearranged, we'll do another classification of all the all the millions of emails one more time. 6:05:31 Because Jev is so cheap, we are able to do this multiple times and eventually we'll end up with the category list which is extremely fit towards the mailbox that you are dealing with because it's not the same across different in different mailboxes the categories are completely different. But the powerful thing is once you did all of this work with an LLM going through and finding these categories, you don't need the LLM anymore because you have the categories which you where you have distilled down all the intelligence into these categories. Now you can just build a maestro flow which basically takes just the categories alone and uses gem. So whenever a new email comes, it only needs gem. It just looks at the looks at the email and fits the categories and we basically have a list of follow through. So if this matches this category, you are going to get an email. This team is going to get an email or that team is going to get an email so easily and we basically removed the LLM completely from the picture on this one. So and we saw a humongous speed increase 6:06:32 on on just email processing. It was almost like at real time an email comes and suddenly somebody will get notified. So that was an amazing amazing thing to see as well. So over the few months we were so excited to play around with gem and every day the slack threads would light up everywhere in the in the company that I found this pattern, I found that pattern and it was such an exciting time during these last three months and I wanted to share that there are so much more patterns and strategies to be found. We are only scratching the surface of this and I want us to try out and like share the strategies that we all find and collaborate here. >> And if you're interested in trying out UI Path Flow, we have it available for free right now. Scan the QR code on the screen here and we'll be able to it'll take you into a portal where you can basically point your coding agent at it and get complete access to build your first automations there. Thank you for your time everyone. >> Thank you. >> [applause] 6:08:39 >> This 6:11:39 >> [snorts] >> All right. Good afternoon, everybody. I am Brian Kennedy. I'm an engineer on Zoom Info's platform. Now, I want to start with showing you something one of our agents wrote. This is a real output from our product. The agent's job was to schedule an email for a customer. And this is what it reported back. It said [clears throat] it scheduled the email, it queued it up, and that it was going to go out on Tuesday. We don't just trust the agent, though. A second model checks the agent's work. I'm going to call this one the judge. The judge read this and it accepted it. Great. Now, here's the problem. And this is what it actually happened. Their these are the tool results, the record of what actually ran when the agent called the tools. I'm going to call these the receipts. The agent tried to create a record in Salesforce and it failed. The error message that came back says the record 6:12:40 was not created and it tells the agent not to report a success or even make up a record. We wrote this message ourselves for exactly this situation. The agent tried again. It created a task, which is a to-do. No email was ever scheduled. The agent doesn't even have a tool that can send an email. The agent made it up and the judge approved it. So that's two failures. The agent fabricated a send and the judge rubber stamped it, approved a claim that never actually checked. So if you build agents, you probably have run into this and you probably have a judge like this. It might be a code reviewer or a grader of your evals or a supervisor in your loop. Anytime a model decides whether another model's work is good, this can happen. About for the next 10 minutes, I'll show you how we caught this and what we did about it. First, some quick background. My team at ZoomInfo is called Agent Teams. We announced this last week. It's a multi-agent 6:13:42 orchestration layer that lives in ZoomInfo's GTM Studio. It coordinates a team of specialized agents across the company's systems to run GTM workflows. For example, find companies, research them, write an email, log it into their CRM. Each agent has its own roles and instructions like and tools like Salesforce, HubSpot, Slack. These agents make real uh decisions and take real actions in the customer's systems. They write records and they send messages. Every team and every agent finishes a step. A second model checks its work. That's the judge and it has three options to move on, to redo the step, or to stop the job altogether. You probably heard this called LLM as a judge. It's the same idea as a code reviewer or an eval grader. Now, the hard part for us is that it's one model and it has to grade every kind of agent that we're running through it. Our customers build their own workflows with any agent and any tool in any 6:14:44 order, so the permutations are almost infinite. So the list of jobs for the judge has to grow with it. The judge can tell whether the work what looks finished, but what it can't tell is whether the work was right because that depends on what right means for each job and we're asking it the same question for all of them. So here's where we're going. First we'll talk about what we found when we checked the judge. Then what we chose instead and what you can do today with your own agents. To start, we ran offline evals. We took 500 real steps from production, each with a decision that the judge had made. Then we took a larger frontier model and regraded every one of those to come back with an accept or send back. Then we hand checked each of those samples and we checked the grades. Where we disagreed with the model, we went with our call. This gave us a reference label for all 6:15:45 500 steps. And it turned out the judge was right 65% of the time, so more or less a coin flip. It accepted 118 steps that it should have sent back. That's more than one in four of everything that it accepted. Pretty bad. And when we read through those, the most common problem was that similar to the email at the start, the agent reported that it did something like send a message and no tool call had ever done that. So to see why the judge had accepted it, let's go back to the email example and look at what the judge was given. On the left, you have what the agent's own summary Think of it as like a story. It's confident, it's well written. It says the email was scheduled. Now on the right, you have the receipts. This is the actual tool results. There are errors, failed to create messages telling the agent not to make up the record. 6:16:45 Now the judge had both of these and it went with the story. The work looked fine on the surface, so it accepted it. Now this isn't only our judge. You can think of a code review agent that reads the PR description before it looks at the diff. It's a similar kind of problem. If the description says I fixed the bug, added tests, the agent is going to lean on that when it reads the code. So the obvious fix at the start was to tune the judge and write a better prompt. For us, that was going to be a significant amount of work. Our agents write sales emails, update CRMs, do research, and to judge each of these well, we'd have to figure out what it was that the user was asking. Give the judge the right rules for every kind of work and keep it all up to date. That's basically a custom judge for every use case. What we went with, we went a different direction. We stopped asking one big general question, is this good? And we started asking a short list of small yes or no questions. 6:17:47 They're the same questions for every step and each one is small enough to answer by looking at the receipts. The first option here is basically a prompt tuning project that never ends, whereas the second one is a small handful of small checks. Now, to answer those questions, we used Jev. You've heard a lot about Jev today, so I'll skip that and keep that short. It's a type decision model, not a generative model. Uh for mom at home watching, just ask Claude. And we'll send it structured facts, you know, about the step and a listed typed questions. A typed question is a yes or no question or a choice with from a fixed list of options. It sends back a probability for each answer and it answers all the questions in one call. We gave it four things, the instructions, the uh input, the receipts, and meaning every tool call, and whether it was successful or it failed, and the agent's report. 6:18:47 Then we asked for one choice, which was accept, redo, or stop. And that choice makes the decision. But then we also asked seven yes or no questions. Those seven questions tell us the why. They decide between a redo and a stop, and they go into the feedback that the agent gets when it actually has to redo a step. Now, back to the email. The LLM judge accepted it, and it took about two and a half seconds. Jev gave us a probability for each option. For this step, it accepted 0%, redo 97% and stop at 3%. And it took about a fifth of a second. We make the decision from the probability of accept. At 30% or higher, we accept that. At 5% or lower, we redo or stop. And anything in between means Jev isn't sure, so we kick it to an LLM judge. We pick those cutoffs from testing the 6:19:47 500 labels that we had at the beginning. Now, to test whether what whether or not this works better, we took two different routes. The first test used the same 500 production steps from the offline eval. Jev was confident enough in about seven out of 10 of them. One of those steps, uh, the LLM judge had accepted 82 that should have been sent back. Each red square here is one of those. Jev successfully caught 74 of those. The blue squares are the ones that Jev would have sent back. Now the second test is what we called shadow mode. Both judges look at every step, but only the LLM judge, uh, that decision counts. So we saved Jev what would have done and we don't let it affect the actual run. That gave us 37,000 decisions to mine for insights. And in that time, there were 728 steps where a tool had clearly failed. 6:20:50 The LLM caught 294 of them. Jev caught 623. And the fake email from the beginning, it wasn't a one-off thing either. The same step ran 1,081 times and almost every time the agent said it had scheduled an email, the LLM judge only caught it once. Jev caught that 876 times. And when you have a judge, every step waits. Jev was 10 times faster and 40 times cheaper. At this scale, it is massive. [snorts] So what's next? Right now, every step goes with the same seven questions. They go into Jev along with the facts and Jev gives us an accept, redo, or stop. But next, we're adding post-condition checks, which are questions that depend on which tool called, uh, was within the step. So we don't have to guess. Uh, we know the step up front because the tool tells us. If a step's called, let's say, a send 6:21:51 tool, we also ask whether the message was delivered. If it wrote a record, we ask whether the record exists. And those go into Jev with the same seven questions in the same call. Now, what can you today with your agents? First, pick one judgment. Somewhere in your systems, there is an LLM deciding whether or not something is good. It might be reviewing code, grading an eval. Pick one of those and rewrite it as a few simple yes or no questions that can be answered from the evidence. Second is build it. This is the easy part, honestly. We used Jev and ours was seven questions. Then the third, ship it in shadow mode. Run it next to what you already have, but don't let it make the decisions yet. Just save it and then you can analyze it later. Then read the cases and see where the two disagree. Don't overthink it, just start small. We had trusted our judge until we measured it. Measure yours. Run a second check. Run 6:22:52 it in shadow mode and see where it disagrees. You may be surprised with what you find. Thank you. >> [applause] 6:24:48 >> Hey everyone. I promise I'm the last person on the hype train for Jev. I'm Scott. I'm from I'm from Workday and I'm going to go through how we're experimenting with Jev to drive new innovations in HR and finance tasks. Exciting stuff. So at Workday, we have over 11,000 different organizations that use our tools. And a lot of the problems that we deal with break down into kind of two different categories. So one is very quick actions they have to take. I want to see my pay. I want to ask for a day off. The other ones are I have to dive into a ton of data, weed through it, try to figure out what's going on. And like many of your companies as well, we're trying to, you know, put agentic workflows around this. And so if you look at just a a very simple agentic loop, you know, we have a user request. We try to decide what to do. We're running tools. We're judging did we do a good job? 6:25:49 And then over and over again until we finally get a result. Now one of the problems here is that on each one of those we're doing LLM calls. And there's two major problems with that. So one is there's too much intelligence. So if we look at the system one versus system two thinking, Uh we're doing a lot of slow, deliberate, very expensive thought processes at each one of those steps. Another problem is is just the wrong tool for the job. So, you know, LLMs a lot of times are just a hammer uh looking for a nail. And, you know, we want to figure out like what do we actually need uh because clearly LLMs are not doing it. So we want something that is fast, uh smart, easy to use, uh it's generalized, and it's very very cheap. So when we look at like from a code perspective, you know, what are we really looking at from a from a Jev-like 6:26:50 model? It's really just a conditional. If this happens, this should happen. Which one of these cases are the ones that I want to explore? And you know, you've seen it among many of these presentations already, uh but Jev has the primitives to allow us to do this. Uh there's the probability. Is it, you know, this or that and what's the probability? Uh what are the different categories that this uh particular thing could fall into? And then you have the uh the scoring mechanism, which is basically probability and sort it. So you know, this has been brought up as well. You know, my first thought when I saw, you know, the Jev models was, well, we already have classifiers. I've been doing this for a long time. How's it actually different? And as many of the speakers have talked about already, you know, with classifiers, I need to figure out, you know, what's my golden data set? I have to train it. New data comes out. I have to iterate over it over. 6:27:50 With Jev, just out of the box, it just worked. We plugged in our question. We classified it. We didn't really have to do anything. It generalized to our question. Uh and so we started using it for a lot of the use cases that you've seen from some of the presenters already. Uh so we use it for routing. That's kind of the canonical example. Um you know, figuring out did this step work, you know, as a judge. Uh, we looked at uh, you know, resumes, you know, is this person qualified for ML work? Uh, and I want to do kind of an overly simplified example as well. Um, so no one taught Jev like what a fish is. And so this is actually a question we asked, like is a fish a pet? Well, 50/50 might be a pet sometimes, might not be. Um, and so you know, if I ran into this, then I have to figure out um, what do I do next? Maybe I can give it some more context. Oh, is a fish in a bowl a pet? You know, and after 20 milliseconds I can ask this again. All right, 6:28:50 that's great. Oh, is a fish on my plate a pet? Clearly not. Uh, so this is something that wasn't trained out of the box in seconds. It just worked. All right, so now on to some of the demos that we're that um, you know, we've done at Workday. So just a caveat on some of these um, it's not real data, so we're not using any of your, you know, resumes or payroll data. Uh, these are also not in production now and may or may not be in the future. So that's something our legal asked me to say. So let's get into, you know, some of the the demos. All right, so starting with autocomplete. So autocomplete starts to predict what you're trying to ask. It's just predicting the question. It's not helping you get to the answer. So as you start to type more, it's kind of guess like, oh, I want to figure out if this person wants to take a day off work. Great, you've guessed the question, but you really haven't you know, helped me get to the answer. So what we want to do is move from just 6:29:52 predicting the word or the next block of code and actually help you finish the task. And so this is best shown with a a demo that we have where we go from predicting what you want to say to predicting what you want to do. So hopefully the video plays. There we go. All right, so I need to take time off of off of uh, next week. And as I start to ask this, Jev is actually figuring out under the hood, okay, what what more context do I have? How can I adjust this UI based on the information that I'm that I'm getting? This is not something that we could have done with an LLM. We're getting you know, we're we're doing this you know, in a loop while someone's asking. Um the payslip. You know, you shouldn't have to go in and ask an agent, hey, can you go find out how much I was paid? Uh what you can do is just in the search say, what was my pay last period? Pops up an interface, allows you to 6:30:53 unhide it, you know, see what your pay was in the last period. Very simple. Gets you to the action. Book a meeting. So I know that I want to book a meeting and uh I'm going to want to do it for next Friday. Let me do it at 2:00 p.m. Oh, I want to do it with my boss. Make sure that I get my you know, my boss on this. Uh Logan Mc Neil. There we go. Meeting's booked. And then a final one that we have just for this example, but there's many more, uh is finding a person. So in this one, I may have some semantic knowledge, like find the person who owns this part of the organization or just like who's Ashanti? I want to contact her. Uh pop it up, click on email, good to go. All right. So what's happening under the hood? So because Jev is fast and cheap, uh just like a type ahead, we could 6:31:54 start to render a UI on this on that same cadence. We can look at as the words are coming, do a prediction of intent and start to render an an interface. And you know, we could do this at you know, about 20 bucks for a million judgments. Uh and the way that we're doing it is kind of twofold. Um or two parts, I I say. So first, Jev is picking the card of saying what's the intent? All right, there's a card that's going to help you out. And then the team that's responsible in that domain then can decide what are the questions to progressively fill out that card. And so there's sort of this two-step process. Uh the first step is getting people in the front door, getting the right card up. And then the second one is making sure that you fill in that card to help them accomplish the tasks that they're after. Now we have an eval on here before we merge any card. We want to make sure that it doesn't collide with any other one. And at least in our initial experiments, you know, we've we've done seven cards or a just chat because it 6:32:55 should if there's no card to actually solve it, it should allow you to just continue chatting. Uh we were able to get, you know, 96% accuracy on that. All right, so now onto the recruiter. So let's talk about the recruiter problem. All right, so this is not like a fast uh a task. It's something where recruiter has a pile of hundreds of resumes and has to figure out who has the best resume to accomplish the uh or fill the wreck that they're trying to fill. Now I first want to look at like how we would have done this before. Uh we would have done, you know, an embedding, put the vectors in a vector DB, then had a retrieval layer, then have a re-ranker, and then figure out based on the re-ranker which ones that we want to put in front of uh the recruiter. Uh one interesting call out is the Quinn re-ranker uh looks surprisingly similar to uh the uh the Jev process as well. Uh and so that that's kind of an interesting corollary if you've used that one before. 6:33:56 Uh but with Jev, we basically take out all the embedding steps. I don't have to train anything up. I don't have to do embedding. I don't have to put things in a vector database. Um what I can do is just do a function on my database. So put all the resumes into a database, uh use natural language to query it, iterate over every single one of them and then just figure out which ones match my criteria. So I'll go into you know an actual demo of this exact same one. So here we have a select statement and so this is against DuckDB where we have a function that is going to call Jev to figure out what's the probability that someone matches this criteria. We're going to get it a score and then we're going to order by that score. Now I'm going to do about 20 and as I said these are all fakes and none of your resumes are in here I promise. And then here we see these are very ML focused synthetic data as well. So it's 6:34:58 not a surprise that we would have very high hit rates against someone who's worked in fraud, trust, and safety problems. But let's look at it. Let's change this question. Let's be a little bit more adversarial on this one. So a lot of these resumes have the word pipeline in them. So whether it's a data pipeline or an AI pipeline pipe's going to come out. So let's just ask like is this person qualified to do plumbing? If you were going to just do a lexical search it might pick up some of these. So we put it into Jev and asked you know of all these ML resumes let's trim the limit down to five here. Go ahead and run it. And we could see you know that very very low probability on these. So these are ones that we would almost definitely throw out. And you could see the speed. So we went over 117 records. We had 2.3 seconds of wall time and spent almost nothing on this query to just iterate over everything. 6:35:59 So Jev really kind of changed the game for us because every every place where we're thinking either this is too expensive or it's too slow we thought of new places to inject Jev into our pipeline. And so [snorts] just to summarize you know the two areas that I went through are one where can we decide very quickly and get to the intent and get the person on their way. And then the second one is to cut down on some of the, you know, really deep work. How do we filter those out for, you know, our customers? And so I hope this that this gave you a little bit of insight into the things that you can take back to, you know, your own companies and how in, you know, imagine the things that you can do with Jev and decision models as well. And that's my time. Thank you very much. >> [applause] 6:37:54 >> Hey folks, this is Girish, a principal software engineer at Microsoft. Outside my job, I build open source tools for AI agents. One of them is mcb pacemaker, a supervised bridge that keeps mcb servers running. Before we start, a small disclosure. The views in this talk are my own and doesn't represent my employer. So sub agents for distributed system. Over the past year, I have been using agents and sub agents on real engineering work. Dev sessions, code reviews, instant thrashing, what not. This talk is about what happens when one agent hands work to another. So here's an architecture. The orchestrator splits the task and hands the pieces to three different sub agents. They all run at the same time. They all read from the same shared state, conversation, 6:38:54 memory files, repo, working trees, issues, vector stores. And they don't just think, they make tool calls to real systems which touch databases, APIs, CI tools, deployment pipelines. And the results come back and orchestrator merges them into a single answer. An orchestrator workers shared mutable state and results merge from concurrent executions. This represents a classic distributed system. I have known how to reason with distributed systems for decades, but none of this setup came with it. What you get is a function that takes a string and returns a string. Failures we hit are not new, are not specific to AI agents. They are classic distributed system failures. Partial failure, 6:39:55 a normal call that fails halfway through and returns an exception error code. In a sub agent case, it returns text that looks like a success. Dirty reads, normally we have isolations, versions. Here every worker reads on shared context including things other worker only guessed. Unbounded fan out. A busy service pushes back with the queues for two nines, five o threes. In agents work does not see any of this. just starts in the worker. Retries. Normal distributed systems we build at a potency to make retries reliable. In agents a retries sub agent runs the whole task again with side effects included. For each of this case I'll name the concept it already is and I'll put them together as a 6:40:56 consistency model I'm proposing and walk through why each part is there. Start with the failed model because everything else depends on it. A friendly case is a crash down. The process it was running correctly stops and stays stopped. The sub agent withdraws an error in this box. It's easy to handle. Next, no reply. The call is lost or it's very late. We'll come back to this soon. The hard case is the one from Byzantine general sniper. A component that gives conflicting information. The process answers and the answer is wrong. A half finished sub agent sits closest to the box. It is not malicious but from the outside it looks the same. It does not crash. It answers fluently. Let's jump into this. The sub agent that half finished does not race. It returns normal looking text with the same type signature as success. 6:41:58 I talked about one version of this in one of my talks. We pointed an agent at an alert queue. Here's an illustration I used there with three different paths which give the same answer. The first one it pulled the telemetry and compared with a known benign set. It actually did the work. The second one it matched against a title it it has seen before and gave the response. The third alert it didn't do anything. It just resonator. All three are categorized as benign with the same level of confidence. And nothing downstream can tell these apart. The orchestrator can read the conclusion. It cannot read how much work was behind it. Now the call gets no reply. The orchestrator send work and a timeout fired. From here three worlds look the same. 6:42:59 The sub agent is still working. It crashed. And the reply was lost. It's a hard limit on a tooling gap. The FLP paper from 1985 showed that in an asynchronous system with even one faulty process agreement can fail to finish. Chandra and Tu X answered in 1996 was failure detectors that can make mistakes. They suspect they do not know. I hit both directions in my own open source project MCP pacemaker supervised bridge. A server whose first run downloads itself hit the 30 second call timeout and looked broken. It was slow, not dead. We fixed it by giving initialize its own timeout. Now the other way the server crashed during a call. The call then timed out. And the timeout overrode the exit code that explained the crash. We fixed it by recording the timeout only when it 6:44:00 was alive. So timeout is a suspicion. Record it as unknown and keep the real cause. And now make failure a value. Let's take this illustration. A sub agent returns a status what it did what it did not do the evidence and any side effects. Partial is a normal answer here. The protocols already have a slot for this. In A to A, a task can end as completed, failed, canceled, or rejected, or stop and ask for input. In MCP, a tool result can say is error true. But a sub agent can still say completed while its text says something else. And a loop that only reads the text now receives a difference. Another split. Checking a condition and getting no 6:45:01 is a different fact from not being able to check it all. My own individual internet draft on evaluation states talks about this. Collapse the two and you change what gets retried and who gets paged and what the audit says later. Next, dirty reads. Let's take this made up example from one of my talk. The sub agent does try it writes into a shared context saying that error rate fell by 80%. It did not measure that. It inferred. Now, two other sub agents read from the same shared context. The second sub agent writes a review note and picks the same sentence. The third agent writes their own book and uses it as a reason to proceed. The orchestrator merges all the all these three. 6:46:03 Three artifacts agree. It is one unsupported sentence read three different times. The 1995 critique of sequel isolation levels define a dirty read as you read a value before the writer commits or rolls back. The shared context it guess is readable the moment it is written. Nothing marks it as unchecked. Regular fix to is to ask three sub agents and get a quorum. Quorums do work for a specific reason. In waited voting from 1979 the read quorum plus write quorum is greater than total. So every read overlaps the latest write. The overlap is a guarantee. In sub agents case the three sub agents that read the same shared context does not give you that. They are not independent copies. They read the same guess. Their agreement is one source counted three different times. Let's pick a consistency model. 6:47:05 In databases we have a Linearizable. Every operation appears to happen at one point in real time order. Sequential. One total order that respects each process own order. Causal. Operations that are causally related appear in the same order everywhere. Session guarantees. They are promises to one client like read your rights and monotonic reads. Eventual. If no new updates are made reads eventually return the last value. A shared context gives you none of these. There's no stated guarantee at all. And you don't need the top consistency models. The strongest models cannot stay available when the network splits. Causal and session guarantees can as long as each clients stick to one place. For sub agents that place is the orchestrator. 6:48:06 So here's the consistency model I use for sub agents. I call it safe handoffs. Yes stands for source feeds. Every fact the worker hands on carries its source version as of time and revision. And decisions read only fresh source facts. A at most once effects. Every write has an identity and a single use approval for the exact action. An outcome is read back now replayed. If fan out budgets each downstream gets a gate and a bounded weight. Refusal counts as a fail. E stands for explicit outcomes. Every hand off ends in a named state and a timeout is unknown with its cost kept. This consistency model 6:49:07 is the one I use for my own sub agents. It's not a universal model. It's based off my own public work. Some of it is in my FCP pacemaker bridge and some are based on my other talks and my own internet draft. Now let's see what's behind each letter. S source reads. It's the answer to dirty reads where one worker's guess becomes another worker's fact. What helps is to treat every fact as a record, not a sentence. Lamport's 1978 paper define when one event happened before another. The fact's provenance is that record, what it was read from and when. Let's take this made up example. The record names the claim and its revision, the source and its version, the scope and three different times 6:50:08 when the data was current, when we read it and when we used it. Freshness is checked with the fact is used. In this illustration the data ends at 9:05. We read it at 9:06 and we decided at 9:07. In this case we picked a five minute limit. So the data decision at 9:07 it's fine. Now at 9:15 the second station came in. Because our freshness limit is five, this is still recorded. We reject it. And we say that the next station needs another read. This is how you separate an observation from an inference. An observation has a source and a read time. An inference has none. A at most effects is about retries. This an illustration 6:51:09 from my article about MCP pacemaker bridge. The orchestrator delegates a task. The sub agent calls a tool which does a real write like creating a PR. The reply is lost. The orchestrator knows one thing, the outcome is unknown. So it retries the whole task. The write happens again. This is a classic problem. In 1984 RPC paper it says that if the call returns the procedure ran precisely once. If you get an exception it ran once or not at all. And you are not told which. So you pick a delivery semantic. At most once. Don't retry and accept the fact that it may be lost. At least once retry 6:52:10 and accept that duplicates are possible. Exactly once. It happened at least once and this is a way we can deduplicate. Maybe by using an item potency key. In MCP item potent hint is only a hint and it defaults to false. If you need it, you build it. Here's the small design from my bridge on how I handle it. When a client comes back after a restart the bridge replace the client's record initialized against a fresh server, that's safe. It's just a handshake, and it does not restore tool state. What it never replace is an uncertain tool call. When a shared server is recycled, the bridge drains it and calls still unanswered at the deadline fail as potentially excluded. That status is honest. It tells the caller to go and check. What helped us giving every side effect 6:53:11 an identity. Approval binds with exact action and it's only used once. After the call, read back by that identity. Submitted is not confirmed. Confirmed means you observe the effect. The last part of A it shows up when work passes and resumes. A lease is a lock with a timeout. Martin Kleppmann's post about distributed locking shows this problem. A client passes while holding it. The lease expires. Someone else takes over and the first client wakes up and writes anyway. The fix he proposes a fencing token. A number that goes up every time the lock is granted. The storage remembers the highest token it has seen and rejects any write with a lower one. Sub agents pause and resume all the time. In my other talk, as I put it 6:54:13 restore context, not side effects. So treat an approval like a fencing token. One action, one use. Checked at the gate. Not remembered in the prompt or conversation. My bridge fixed the same class of bug. A stale exit or a deleted resume can no longer remove or bring back a replacement session. If fanout budgets is about unbounded fanout. This is flow control or or lack of it. I hit this issue in my own MCP bridge, before it had a limit on cold starts. Many sessions start the service at the same time through NPX. They share one package cache. Everyone of them failed with NPM's lock compromise error. Nothing in any single request was wrong. The failure was the concurrency itself. And when it downstream does push back, it can arrive as plain text. 6:55:14 This is MCP specs one example of tool execution error. A rate limit as one sentence. The model reads the sentence. The orchestrator code may never look. The next worker starts an event. So I introduced a quota budget in my bridge. Cold starts that use a package manager go through a gate two at a time by default. Everyone else waits. And the wait is bounded at 15 seconds. Past that the start goes ahead without a gate. And I documented why a corrupted cache is a risk while a hung client request is certainty. That is a choice for availability made on purpose and written down. And I also built a cap on sessions. At the cap any a new session waits up to 10 seconds. And then it's refused. The refusal now counts as a failure in health. Another bug. 6:56:15 Capacity has to be reserved before the synchronous start, not after. Otherwise queued starts overbook the cap. That is a check then act race. For sub agents give the orchestrator a number per downstream. A bounded weight and a refusal it can see. Now E next present outcomes. Every hand of ends in a named state like stale or unknown, not just text. The last piece is who decided that state. Treat sub the agent output as untrusted input, the same way you treat a request body from the internet. The orchestrator owns the checks in code. Does the result match the schema? Is the evidence there? Is it fresh enough? Do the numbers hold when you recompute them? If a check fails, the step returns a refusal with a reason. It does not pass the text along with a warning paragraph underneath. 6:57:15 If the same model can skip a check and declare it passed, you have a request, not a boundary. Here is a made up example from another talk. The sub agent says the error rate fell. Before it was 10 errors in 1,000 requests. After it was six errors in 600 requests. It went from 10 to six and it claimed that our rate fell. But when we computed the rate, it's both 1%. We recomputed it and then the claim is refused. A supported answer is fine. A deliberate handoff to a person is fine. A confident guess, it's not. So limitations because a consistency model is only useful when you know where we should stop. So here's what safe model does not do. It does not make facts true. A source fact is checkable, not correct. It does not give you exactly once 6:58:15 delivery. A lost reply stays unknown until you read it back. It does not roll anything back. Undoing a partial task is a compensation step you design like the Saga's paper from 1987. And finally, there's no global order across workers, only causal order per fact. And it has a price. CAP theorem says that when the network splits, you trade off availability against consistency. My Brits chose availability for cold starts on purpose. For rights, safe picks consistency over availability. It refuses rather than guesses. Pack lack adds that even without failures you trade latency for consistency. Every freshness check is another read and and every refusal needs an owner. 6:59:16 So where it does not fit if a subway agent only drafts texts that a person reads before anything happens most of this is is a overhead. It also assumes that there's one single orchestrator. If you have more this does not fit. Peer agents with no single place to stick to it's something else. This is the consistency model I use for my own agents that touch actual real production systems. It's not universal. So wrapping up here's three key takeaways. Treat every handoff to a sub agent as a network call. It can fail halfway it can be slow and it can come back twice. Return a status not just text. If the result cannot say partial, stale or unknown the orchestrator will read it as done. And start with our safe consistency model and enforce it in code. 7:00:16 Keep what fits and change what does not. We already know how to build distributed systems. We just have to notice that we are building one. So these are some of the sources behind this talk. On the left side it's my own work, my open source projects, my articles, internet drafts and my talks. On right side specs and standards and bunch of papers behind each concept. Thank you. >> Hello, my name is Nikolai >> Petrov, and we will be talking about skills and agents knowledge of those skills. We'll start it with the simple prompts where we've been playing with the new stuff, new work being done by the agents, repeating those prompts over and over again, and then we start packing those into the skills, creating repeatable, reproducible, 7:01:17 portable type of their functionality or capabilities that we can teach agents to work with. And some of those skills are very important, hundreds of the stars on GitHub. There are different types of their skills. Some teach models something new, bringing new knowledge or capabilities to the model, but there are also types of their just steering or process skills where we do not bring any additional information, but just getting repeatable flow of the actions that we want agents to perform. Both are useful. In this session, we will be talking about new knowledge type of the skills and we'll see what we can get from the model's understanding of those skills. So we get our first skills packed, start 7:02:17 using them basically starting with their the feel of does it work or not. But there is also their more technical way and more deterministic way for the checking if skills are working or not and what's the impacts of those skills our evolves. There are different formats where evolves can be defined, but at the sense, this is task definition that you want your agent to be able to solve. You put it in the place, ideally more deterministic so we can validate it. And we can check two arms. One their no skill loaded, and their second one where we're loading the skill and providing this additional information to the context. And we compare how their arms There are multiple things. One is how many evolved spins off. In this case, with no skill, we're solving three out of five. With skill, we're solving all five, meaning that we get two more tasks. 7:03:19 There also would be implications on the tokens usage or time that it took those tasks to be completed and other factors. But at the end, we are comparing those and seeing what makes sense, what not. And this is one of the main tools for evaluating the skills and saying that they not just nice ideas, but actually working pieces of instructions and context for the agents. But what next? We get to the stage where there we have their working skill. It solved some evolutions. But we know that there should be a better way. We have models evolving. We have some maybe cases there we want to get skill more efficient, meaning consuming less tokens. So what's their their options there? One is just change line by line and try brute force 7:04:19 all the different combinations. Another would be to think through what makes sense or maybe use something as the user feedback on the skills to see where skills perform well there it's not. And with new models, just running skill and seeing how it works, what's the options there. And how large those modifications should be? Just lines, sentences, ideas. So what we found to be useful is instead of relying on their luck, we can put more deterministic process here using idea of their items from the fact score. Uh we can split their skill and instructions into those items and for each of those had pair of question and answer that we 7:05:20 expect. This could be either general ideas like proposition in this case or contracts there we put more definitive expectation on their action. And in general we will be getting their uh some predefined amount of the items per skill and we will have let's say 25 or 30 items per skill. Then for each of the items we will again run two arms, one without skill and one with skill. And we will have judge or panel of judges that will be evaluating model answers. So we asking those questions to the model that we interested in. We evaluating those questions with their judge, ideally different family of the 7:06:20 models or panel of judges. We also ensures that the question do not leak answers. We depending on the case can compare with no tools or additional um information to check their internal model understanding or provided with their some tools like feedback access so it would be more practical. And we can get their simple forward counts matrix. Basically where we are checking that the model uh uh knows or do not know something and how this maps to their skill impact on this. So the simplest option is where model was able to answer this particular questions even without skill or with skill. This means that skill have no impact on this and this was already part of the model's knowledge. 7:07:22 The second one is where without skill actually model was able to answer this correctly. And by adding skill we saw degradation in performance and model capability to talk or explain particular aspects. The ideal case is where without skill model wasn't able to answer something and with skill we start seeing that it can answer those new type of the questions and scenarios and this is what we were aiming for, teach model something new. But there also can be there fourth option where skills were written not ideally or in some maybe not clean way where model wasn't able to answer it either without skill or with skill and while this topic might be relevant and important for the model it still sort of misses some of those concepts. The simplest thing to fix are 7:08:26 instructions or items that actually hurts performance and for popular skills those are pretty rare. So in their uh box below you see that it's just one case out of the mostly 400 items in entropic skills and others don't have those. But for their smaller skills that are less popular you probably having more of those. This is what we are seeing with some of our um experiments. And this is just straightforward the simplest to fix to ensure that your skill do not reduce model capabilities and performance and bring only relevant context there. And we also can actually color this in this scenario this is done by lines but actually going with items you might have there uh different coloring for a single line or a single sentence. 7:09:28 And this just gives your example of there uh visualization where this would be helpful details and instructions in terms what new stuff you bring to the model and what is something that model already knows. Cost wise we are running all those questions and asking model but still relatively this is uh cheap for 29 skills for 10 bucks of there sonnet and haiku calls with their sonnet used as a judge. And there couple ways how we also can check their strictness of their some knowledge. Um we have uh lenient and strict ways of the checking this where first checks the general idea while the second one is checking for 7:10:28 exact details and you can think about the difference between general knowledge and understanding of the topic to the understanding of the exact keys parameters and exact values or steps. And what we see across the models is the difference is large especially with there um larger frontier models where we see almost dropping from 70% understanding of their skill in case of the general understanding to just 22% of the specific details. And this is much our expectations. We see that new models are pretty well generalized and understand concepts. But it's still helpful to provide their with exact details and data that will be useful for their solving those problems. Here is outlook of their comparing those three popular rappers 7:11:29 with the skills across different models. So one of their interesting observations here is that smaller model have in general lower understanding of their skill with their larger sonnet having better understanding which is why we have bigger models. But also we see that overall their models already understand a lot of their details and context of those skills and you would expect that with the newer models and more advances in their better training capabilities we will have more and more in terms of the model's understanding. But also we see that this pretty significantly differs skill to skill. And it's not bad. Some skills are intended to be directional. But it's also useful piece of 7:12:30 information to see how those skills bring new knowledge to the different model and different families and this should be run ideally in the broader set of the models. But just having their this novelty piece of information is just one of the signals. It's basically initial vector for optimization. At the end we still have to run evolves and validate that all the changes that we try to make with their skill actually works. So their one of their simpler scenarios would be we can look on their skill and try to remove some or all of their already known pieces of information from the skill and run evaluation. It might come up with their same scoring. Basically we will be getting their same results model still capable 7:13:31 of solving those problems. Then great we can reduce skill size, save tokens, save context. And instead of their brute forcing and guessing what we can or should remove or add or how to fix their degradations we can have a guidance. So all of those measurements experiments been done with the docume which is a framework for their comparing everything related to the agents and context where it's is skill, agents MD, MCP tools, whatever. And novelty is one of their pieces of the functionality where you can get this initial vector for their skill optimization and evaluation but there also evolves there is assessment of the context impact, routing, comparing their descriptions, naming, bunch of the things that 7:14:32 related to their skills where it would be helpful to actually get their more deterministic evaluations to make conscious decisions in terms of the what should be done with the skills. So thank you for your time and please go ahead and write better skills. 7:16:04 >> Keynote programming will begin in 5 minutes. Please be sure to find your seats. 7:18:07 >> This 7:19:14 >> <music> >> <music> 7:23:06 >> As we head into our closing keynotes, we're excited to bring back a familiar face to guide us through the final sessions. Ladies and gentlemen, please welcome back our MC, Corey Quinn, Chief Cloud Economist at Duckbill Group. 7:23:39 >> Welcome back. Thank you all for sticking out the full day. We just have one more section to go and then we're calling it a wrap on technically day two of the conference. Since we arrived this morning, we've heard about a few things, deleting harnesses, presumably on purpose, healing your agents, maybe through the power of song, we don't know, self-improving codebase, which is the exact opposite of the direction of every codebase I've ever seen in my life has ever gone, but okay, and so much more because everything is different now. And before we leave tonight, we're going to hear about setting intelligence free. We're going to hear about how agents build cultures. There's a talk called Escaping from Trash Mountain, which ironically was my nickname back in high school. And of course, 2026 in LLMs, which is one of those years that seems to encompass decades. So here we are. Before we dive into all of that, I want to take just another moment to thank our sponsors for making all this possible. 7:24:41 Presenting sponsor remains Google DeepMind, who earlier today shipped what was it? Gemini Carbon 4 to general availability. Congratulations. Please clap. [applause] For our lounge and our platinum sponsors as well, without whom we would have far fewer places to sit when we're not in the middle of a session. Our gold sponsors as well for effectively making all the various moving parts go. And of course, our silver parts, our silver sponsors too. My goodness, there are so many tiers at this point. At some point, it starts to become an API ladder. And finally, all of our supporting partners. Again, it it really is not possible to put on an event like this without a tremendous amount of support from various folks, and we thank them sincerely for all of this. Uh, now up we have Fabian Hedin, who is the CTO of Lovable, trying to tell us what it takes to set intelligence free, uh, which might be the only time in AI we 7:25:41 hear the word free. Let's make him feel welcome. >> [applause] 7:26:04 >> All right. Thank you so much. Um, see we get to go here. Um, all right. Welcome everyone. Glad to be here. I'm Fabian, co-founder and CTO of of Lovable, and we spent the last few years or so building a platform to let anyone create software and then operate that and deploy that software. Um, and um, the the topic of today is what it takes to set intelligence free. And really um, for us this takes us back to when we when we started three years ago. And I have to say I have a bit of a complicated relationship with with agents, um, and I I'll tell you why. Um, we almost failed as a company because we thought back in 2023 that hey, we these agents are the future, so let's build for them now. Um, I think they we kind of turned out to be true that they were the future, but we were definitely wrong on the the 7:27:05 now piece in in 23. So this was back in kind of GPT-4 days. Um, and um, at the at that time the intelligence was simply not there to have more of these kind of complex agentic systems. Um, but now it is, right now. Uh, now the intelligence is actually at a place where where you can really build out more complicated systems around them. And so I'll I'll give you kind of a quick background story here of of our journey in in working with intelligence or or these LLMs. And so back in 2023 we built the first version of our product. We called it GPT Engineer back then. And really the UI was very similar to what it is today where you kind of have a chat and and this preview. But the AI at that point was quite simple. We just had a prompt and a response, one LLM call per user message. Very simple. It worked fine. But you could kind of build one page 7:28:06 apps and and simpler applications with this. And then we thought okay, this intelligence is going to keep on evolving so we have to be ready for for it as it evolves. And so we thought let's go out and build this agentic framework. And so we we went and we built this very sophisticated, dare I say beautiful framework for for agents to work and we had kind of this one main agent that would talk to the user and that agent would spawn other agents like an implementation agent here in the example and each agent could have a speciality. This agent would kind of specialize in the first generation of an app and then you would have like a front-end agent and a back-end agent and they would all talk to each other, send messages back and forth and we thought oh this is beautiful and there were papers around this and how well this could work and in the end it it didn't work at all. We just made the product worse. We we definitely failed on this one. But luckily before we ran out of money 7:29:08 we figured out like hey let's go back to a simpler system again. So in late 24 we went back to a much simpler system where it was a bit of a shame where we had like some some smaller LLMs doing some knowledge retrieval and then really just one main LLM kind of writing to the user and writing the code in the same message. And this worked great. It was very responsive for for users in the UI and it was reliable and users were kind of in it and and iterating with with the model. And so let's close that down. But now it's time again. We're back. We're so back. Agents this time they work and we're running them in production over a billion tokens per minute on on on these systems that I'm going to talk about. But first just looking at okay what is actually Lovable? So if you simplify the UI and this was same back in 2023 as it is today, to the left we have a chat where humans 7:30:09 come in and they kind of describe their their intent to to build or whatever they want to do and to the right you have a preview which is the output or or the outcomes of of what you're building. And this back in 23 you could do simpler things with, right? But now we've really seen working at this layer of abstraction you're able to build very complex things, right? And I think we've even seen a lot of tooling meant for for developers for example moving to this type of interface where you're not really looking at the code every day and so on and you're getting kind of abstracted and working on on be it architecture or looking at some kind of output of of what you're doing. And this is something that we've always believed in and it's been kind of a core thesis behind Lovable that you won't have to care about the code in in the same way that you had to do before. And I think today we're kind of in this almost a bit in between I think in the industry where you know some people might say oh you don't have to care about code at all and some people might 7:31:10 have different opinions. I of course like growing up coding and these things you get a bit detached but in the end what matters is is actually not the code. Sorry I thought we removed that last part but what what matters is is the outcomes, right? And so the outcomes what is that? Well, for for our users, the outcomes is creating a business and monetizing and creating value for for others in in some type of way. And I'll kind of quickly go over what we build as a platform to to enable this before I go in kind of on the on the technical side of this talk. So, if you if you look at Lovable, the first thing that that you can say about the platform is that we have this agent and like any other agent, it can do quite arbitrary things and it's quite specialized at at building software and it has access to a browser so it can verify what it's building and 7:32:11 it connects to to GitHub and you know it writes code that and you can do whatever you want for that code. So, so that's kind of your basic coding coding agent. And the the other pieces of the platform is this kind of cloud and and business piece of the platform. And so the cloud part of the platform is is about just giving kind of these hosting primitives just seamlessly to to this agent. And so we we built out databases of authentication, storage, all the types of backend compute and security scanners and like backups and logs for all of these things and all of it can be introspected by by the agent. And this just makes it seamless to to build things. You don't have to kind of go to different services to set this together. We just kind of provide you these primitives. So, that's still around building software and hosting this, right? But then the last piece is around these business capabilities where again the output of what our users are doing on Lovable, we don't see that output as code. The output is is a 7:33:11 product or or a business. And if you want to succeed as a business, you need to do more things than just write code, right? And so we we we we help our users with receiving payments, sending emails, optimizing SEO, doing ads on on Google and Meta, and kind of analyzing what is happening with the product through through for and you can look at kind conversions and and optimize these sort of things and the agent is in the loop in all of this, right? So that's kind of our capability offering you could say. And again the core part of this is an agent that all of these capabilities are built for. And at some point in Lovel's history we used to limit our agents to work for only 30 minutes because we thought they weren't reliable to to work for a longer amount of time. But now we've changed this to really be without any any sort of bound in terms of the length an agent can work and so you can imagine agents working for for 7:34:11 10 hours plus. And so I'll I'll go into some of the things that we've done in our architecture to to enable this. And so the first piece is really around the agent runtime itself and how how we enable that to to run for a longer amount of time. And the second piece is the computer where each of these agents are actually doing their work, running a dev server and actually verifying what it's doing. And so so the last piece then it is the verification itself where if you want an agent to to work for a long amount of time you need to make sure that you have these feedback loops in place so that it's not kind of going off in in in the wrong direction. And so if we start by looking at the agent runtime itself, you can imagine that you have really three different sources that can interact with an agent. So it can be yourself kind of prompting this or a human. It could be other agents and it could also be just a system on on a 7:35:12 schedule waking up. And the way we chose to architect this is that for for each agent we have a separate inbox where any of these sources would kind of send in messages. And so you might have a message here like what's the status. And you can imagine you have this agent and this agent is kind of an always on agent that they might be working, right? And so you can imagine it has done like three actions here, a request a tool call and it got some some result. And then at this point, the agent is ready to actually receive messages. So since anyone can drop a message into this inbox, the agent then can copy that itself into its own what we call a trajectory. Basically, it's a task history. And this is the type of interaction that I think you're familiar with if if you're using agents overall. So that's kind of the the first design principle behind this agent runtime. I'll go through a few more here. And so the the trajectory in which the the agent is working in, which is kind of 7:36:13 the history of what has happened so far and really the basis for any sort of of memory for for the agent, right? And we designed this trajectory with really Git as inspiration. And so the trajectory, what is it? It's an append-only log where each event points towards a parent, the the previous event. And because of this, you get a nice capability just like in Git where you can fork things, right? So you can have two events pointing to the same parent, which essentially creates a fork. And because of that, you can kind of spin up parallel agents at at any point of time in the history without having to duplicate all of the kind of the shared history. And semantically, it's a bit unclear what would happen if you merge this. So unlike Git, we cannot merge agent trajectories, but we can kind of spawn them out, right? And these agents can can talk to each other. The other thing that we've made sure to 7:37:14 design for is that this agent trajectory, it includes kind of the full history and everything that happens with the agent. And so every tool call, a bunch of other things happening on the platform, if a tool call like failed or succeeded, a bunch of what we've done is make sure that we have this append-only log. And and from that, we can actually build a prompt that we would feed into into the actual LLM. And you might ask why why would you want to do this? Well, one one reason is that when you build the prompt, you if you have a long history, you probably want to do some sort of compaction and you want to summarize part of that history in the in the earlier steps, for example. But this summarization would differ. For example, if you use different models with different context lengths, you would probably think differently about when you should summarize and not. And so by separating the the prompt from the actual task history, we're able to render prompts for for any type of model or or any type of task that that we can kind of find the best 7:38:15 and optimal prompt for So to show this event history in in progress, we have this debug tool here. So here it's just a recording of me dragging this slider in our UI and you can see on the right that the UI is kind of updating as I'm as I'm dragging this along and you have 156 events here and one each of those is an event in this event history. And you can see how how the UI can render kind of any point in history. And this is just showcasing that at any point in time, we have full control of of say the UI or or anything else happening on the this platform thanks to this history kind of detailing everything that has happened. The the last piece I'll go over here on the on the on the runtime itself is that the process that is running these agents, they're separate from from the actual task history. And so the process is almost entirely stateless in that between each each call, each tool call, each turn, 7:39:16 it will actually just persist into the task history. And so you can have a process come and go. And and then the question is why why do we care about this? Well, at Level Bull, this system we deploy to it 700 times per day. So that's about once every two minutes. And you can imagine if you have a agent working for for 10 hours, um you want to not that agent when it's working, right? This this agent is doing something for for a customer. Um but at the same time, we want to make sure that this agent is running on the latest deploy. And it might be a new feature or something like that, but it might also be like a security vulnerability in in the Linux kernel. And it's like, okay, we need to update this ASAP. Um and and so to do that, thanks to this separation between the actual history and and the processes, we can actually make sure that we have one one agent that is working on one task, but we can have different processes taking up and actually doing the compute behind that 7:40:17 work. Um so so this task can keep on working while we have um different deploys um um working on it, be it for 10 hours or or even longer, right? Um so that's the agent runtime. The second piece of um of this platform, right, is the is the computers. This is where the work is actually happening. Um and we've been through a lot of iterations and a lot of vendors kind of working with um um providing these sandboxes for for the computer for for the agent to work in. And we've landed in in the system that we've worked really hard now to develop, which is um um a checkpointing um a computer. And so you can imagine you come to you start a new project. We have this common what we call a golden checkpoint, which contains the OS and the system services. It has common tools installed and a dev server that is ready to start. And this checkpoint it includes everything on disk, memory, and also running processes. 7:41:17 And and so when you start a new project, you get all of that for free. And all of that is copied over in about 230 milliseconds. Um and so once you start a project, you you might have some specific code and and some dependencies within that project. Um and um that is also checkpointed as you go. And and so each time the the agent is running and it finishes a turn, we'll actually save a checkpoint of of that And and from that we're able to again like very quickly, sub-second, spin that computer up whenever you need it. That means we can save on on resources by kind of spinning this down if you leave for say 10 minutes or or so and come back. We can just spin it up extremely quickly so you don't even notice this. And even kind of requests in transit will will not even be dropped in in this setup. The other thing we can do thanks to this checkpointing is that just like the trajectory itself in the agent runtime, 7:42:17 we can also fork the computer. And when we do this we have for example we have a feature in in Level that is called drafts where you kind of work in different environments of what you're doing and and to support that we need to fork the computer that you're working in. And here we have kind of the example of doing two at the same time and you can imagine doing hundreds of these, right? And so we've we've had to do some optimizations to make this feasible to to actually run. So one of these optimizations is copy-on-write on the RAM itself. And so creating one of these forks of a computer is is basically free in terms of of the RAM that it's using. But as soon as you start writing in in a process to to RAM we will kind of copy that page over and only then will you start having to allocate new memory. And there's another optimization here also which is that each of these 7:43:18 sandboxes they require a few hundreds of megabytes of memory. But and that would take a few seconds to stream in if you have it in like cold object storage and then it has to go to disk and then then into memory of this this VM. But thanks to being able to stream the memory we can actually boot up the the machine before all of the memory is is actually ready on it. So the the memory the OS uses comes first and then like some processes and so on will start to work as soon as the memory gets here. And this also shapes off some some extra startup time. And then we started asking ourselves, how can we make this even more efficient and even more faster for our users? And so you might be familiar with Vite. This is great dev server that most of the meta frameworks out there in the JavaScript ecosystem is using. And so we took that and we forked it and we rewrote it in Rust. And that made our cold starts two times faster and we went from getting this preview from 14 seconds to three seconds. And the main 7:44:21 kind of thing that we made faster here is that Vite uses Node.js and we're able to like drop that dependency of Vite itself and just have this Rust binary. So that made the memory footprint be 6.5 times less in these machines. And by taking control of the applications that we're running in these computers, we're also able to control kind of how we write to memory. And so when we control how we write to memory, we're even more optimizing if you remember back to the kind of the copy-on-write solution when we're drafting different different computers. And so we're able to run this extremely efficiently so that we can run lots and lots of agents in in parallel and kind of have agents explore different solutions to solving a bug or improving the performance of your applications and so on. And so this has been fun. But the last piece now is about verification. And of course this kind of computer and this dev server is a big piece of being able to verify what an agent is doing. 7:45:22 And if you look at verification, you can think of these feedback loops, right? That the agent needs to get information. It took an action. Was this a good action or was it a bad action? And you can think of it as this inner loop and this outer loop. So this inner loop is during the turn of the agent and you have kind of obvious things such as like the is the build working? Do we have lint errors or type errors in the code that we're running? And then you can check for runtime errors in in the preview that you're running. You can also within this computer a headless browser where the agent can go in and kind of explore the the web app it's it's building or whatever it's building, right? And and a bunch of other things that we're using to help verify within the turn of an agent. But then you also have this outer loop which is kind of between each turn and the turn you can think of like when the agent is yielding to you as a user. And between each turn you might get some info on the product itself from from production from your users, right? And so you can look at how your visitors or behavior are behaving, 7:46:24 what users are are telling you online. You can do automatic AB tests and actually see change some copy and see is this improving conversion and so on. And then you can also take these business metrics and remember this goes back to kind of the capabilities I showed you. By having control over these things we can make sure that we can optimize how the agent is is getting this outer feedback loop so we can make sure that if you're running payments we can actually track those transactions and conversions and make sure that the agent can actually help you optimize not only the the code, right? But the business strategy around around your product. And so those are really the the three things that that we've been working for and we'll continue to kind of build out these these capabilities and these architecture around really orchestrating these LLMs into into agents. And so this agentic system that we're building it's running billions of tokens per minute 7:47:24 and these computers, right? We can spawn up with each agent and we can have multiple agents use one computer and now we're kind of back to this quite beautiful setup where you have agents talking to agents and they're orchestrating and and this time it's it's actually working and it's working at scale thanks to some of these optimizations. And so if you're curious to try this out you can head to level.dev and you'll you'll see our our preview spinning up and and so on but you'll get a very simple and um a nice interface and you won't have to care about all of this stuff. Thank you so much. It's been pleasure um and uh yeah, hope you hope you learned something. Thanks so much. >> [applause] 7:48:15 >> Building a coding agent is one thing. Building an engineering culture around it is another. To share how agent-driven development is transforming the way teams work, please welcome Zach Brugman, principal software engineer at Ramp. 7:48:49 >> [applause] >> Thank you so much. So let's get a second here. Any 49ers fans? Okay, you know that's a better reaction I've gotten at other events this year when I've asked, you know, about their local sports team cuz I've gotten a lot of just like muted silence I have to hit the, you know, well wrong audience for that type of joke, but glad at least I could hit this time. 7:49:28 Do you need help? >> Do you want to just go for it? 7:49:46 >> Current slide on and then 7:50:01 >> All right. Good. Well, thanks so much for waiting, everyone. My name is Zach and I'm an engineer at Ramp. I'm just really excited to tell you all a bit about how Ramp has evolved its engineering culture this year through the help of our robotic friend, Inspect. As a quick introduction, Inspect is Ramp's collaborative coding agent. About a year ago, we decided to take a bet on building our own coding agent with every session running remotely and being fully multiplayer. Inspect can work alongside you and your team, whether that be in Inspect itself, Slack, Linear, or any other of our integrations. And now it goes far beyond just writing the code with the built-in design primitives, review agents, and verification flows that let any builder go from prompt to shipping all inside Inspect. Inspect is also connected with all of our integrations company-wide, so everyone's automatically connected to the to the data and tools that matter most, whether that be telemetry, databases, or any of the other tooling available. Through this, we now see around 75% of all of Ramp's code being being written 7:51:03 and shipped by Inspect. When we started sharing Inspect publicly at the start of the year, the technology itself was the more novel part, spinning up sandboxes in under 5 seconds, wiring up all that internal context, having agents work together in a multiplayer setting. People had a lot of questions about the plumbing. But now with much broader adoption of these concepts and Inspect itself now being a product on the market, the technology isn't nearly the barrier that it once was. Instead, I want to talk about the other angle, the lessons we've learned around building a culture around Inspect and what you can take away to inspire the people and teams that you work with. Uh I'm not going to talk as much about those technical aspects of Inspect today, but if you are curious about that, you can read our blog posts or take a look at Pragmatic Engineer's deep dive that they did with us a few months ago. You can also follow my boy Dylan Garcia on Twitter. He's on his social media grind, but he's also been posting a ton of really great Inspect insights uh we've launched it. So, shout out to Dylan. Please go to him a follow. One thing about working at Ramp is that 7:52:04 it really makes you start thinking about everything through the lens of saving time and money. although I think it is important to understand the systems that you work with and understand how your tooling works, if you're a multi-hundred or multi-thousand person organization and everyone has to go through say 15 minutes of setup, once you start multiplying that across an organization, this is days lost for what is very rote work. Then, if you consider how fast the frontier moves here on best practices and new ways to set up an agent, you realize this setup cost isn't just one time. Not to mention, these advancements won't always spread equally. So, now you have an indeterminate amount of time and effort spent on this maintenance without an equal spread of knowledge leading to a disparate outcomes across an organization. While you can perceive this as an advantage on an individual level, when you're trying to win, you need everyone on your A-game. So, this was our first lesson, build tools that kill information silos. When we found patterns that worked well 7:53:05 or tools that benefited everyone, we could push it to our entire organization immediately. If you want to spend time tweaking your own setup, that's fine, but inspect is always available as a very solid baseline. And what we've found is that we've never needed to mandate usage of inspect because it existing as this baseline reduced the need for people to reach for their own setups all the time. Part of what I think makes this powerful as well is that the reality of this industry is that it's unlikely that you as an individual always know the absolute best way to work with agents across everything. Even if keeping up with this is your full-time job, we all miss things, hold biases, and have blind spots. So, part of the design of inspect is that there's just one single inspect for the whole company. When people figure out better ways to use it or have ideas for what would make it better for the work they do, we're able to bring it to the entire company. The best part of this is that for most changes, this is invisible to everyone else. You just keep using Inspect as you 7:54:07 normally would and benefit from the improvements that everyone's making over time. And what we've found is that we really don't have to run these formal sit down here's how you use AI effectively training sessions for Inspect. With every Inspect session being public within the company by default, it's actually easier to work in public than in private. As such, you default to working in public and you can learn how to use it by watching how your teammates use it in the channels that you're already in and your teammates can learn from you. You can literally ask Inspect to go through their recent sessions and tell you about how they work, how they structure prompts, or how they break down gnarly problems. I I think this is not just been helpful for the builders at Ramp, but also for the Inspect team itself. We frequently see the way that people use Inspect in ways that we didn't originally imagine. This then informs what we should build towards, whether that should be more native primitives for common patterns or where tools and data are missing. 7:55:07 All in all, it creates this self-sustaining loop where we're able to build towards what makes Inspect the most successful by seeing what people are already achieving with Inspect. This feeds into the second lesson that we've learned. It's all about building trust. I think it's fair to say that adopting agentic driven coding has been a massive trust building exercise for every engineer this year. It's helped that the models have become more competent at writing high quality code, but there's a large part of this that's just learning to trust what an agent builds. So much of building trust and tools comes from learning from other people. After all, that's why most of us are here, right? You wanted to listen to people you respect, or at least I hope you respect, teach you about the things that they learn. But if we limit our corpus of knowledge to just those who find the perverse enjoyment in public speaking, we're still not getting the full picture. This democratization of knowledge sharing is just to me as a concept. Instead of grouping around a select few 7:56:07 leaders, can we build systems that inherently share the knowledge of everyone? This is what Inspect has built towards. Every time someone ships something with Inspect, people can see it right there on the work you've done. The session itself is an easy to share. It's living artifact of how that change came to be. When a pull request goes up, anyone reviewing it doesn't just see the 400 line diff dropped over the wall and hope for the best. They can see the exact conversation that you had, all the tools Inspect used, before and after screenshots, or even a video recording of the verification agent clicking through the app in the browser to prove its work holds up. It makes it effortless to show not just what changed, but why you can trust it. It also just makes it easier to bring people you don't work with all the time into the fold. A fun moment on this one, it was back in February, one of our engineers, Andrew, reported a bug in Inspect. So, naturally, I reached for Inspect and 7:57:07 asked it to investigate. What's funny here is that although Inspect had access to Datadog for a while, we'd only just finished instrumenting Inspect itself. I mean, Inspect was primarily built by front-end and product engineers, so tracing is a little bit of a foreign concept for us. But in any case, it was a great time for us to put it to the test for Inspect. And Inspect says, the instrumentation was causing the error. Yes, the very instrumentation that was letting us debug this error was actually causing this error. So, Dylan and I were not exactly convinced by the robot here. And Andrew is just like, whatever, I'll use Claude code. And And while we don't have a mandate to use Inspect, I mean, personally, these were not the words I wanted to hear. So, nobody here is really trusting anybody. Dylan and I don't trust the robot's explanation. Andrew doesn't trust us to build something that works. And Inspect, well, I I I don't know if Inspect trust anyone, but that's okay. That's okay. 7:58:08 But because this is happening together in public, we keep prodding at it. Dylan asked Inspect to look through the code of the telemetry library using and see if this was just something that was happening before without visibility, because surely by now coding has been solved and there can't be bugs that we introduced. And this time it comes back with something a bit more specific. The hotel library calls fetch, but it never reads or cancels the response body. This was just a Cloudflare platform limitation at the time, albeit happening in a library specifically made for Cloudflare workers. But that happens. This library was written pre-AGI after all. At this point, we ask Inspect to make a patch. It updates the exporter to explicitly cancel the response body. We grumbled in the thread a bit while it was deploying, but we see after it deployed, there haven't been any errors since. So great. Inspect fixed a bug, which at this point it's nothing special in itself, but there was a much more important story playing out in this thread. Before Inspect, a report like this might 7:59:08 have been acknowledged, a ticket filed, and then one of us would have gone away to debug it. Andrew might have received a message later saying it was fixed, but the interesting work would have been done in private. Now though, it's easier to do all of this in public. And as such, Andrew naturally starts becoming engaged, wanting to chime in with his thoughts and experiences. And then in that same thread, this leads to ideas coming up about how Inspect could power other work. And none of this required any prodding or pressure to work together. Instead, it was just like, oh, these guys are doing something cool, and we could work on it together. So in the span of an hour, we go from a skeptical user with a broken user experience to an engaged colleague wanting to build together and a more stable Inspect. So once you have a tool that gives everyone the best setup on day one with no information silos and you've built a system of trust that brings innovation together, what happens next? This is the last lesson I want to share. 8:00:09 I think the only thing left is to think bigger together. Something that's pained me this year uh is how much reverence has been built for the lone wolf operator. It's great that we can all do more individually with the help of AI, but this can't be the endgame in my opinion. If we don't build outwards uh collaborate with other people and continue to share the progress and learnings we make, I think this is all going to collapse in on itself. We'll only consider ourselves as our target audience and we won't grow through the consideration of others' perspectives and experiences. This is not to say I'm all doom and gloom about this. The fact we're all here hopefully is a testament to the value of connection. More importantly though, I think we can shape our tools to enable these connections to flourish even stronger than before. That story I told about Andrew has played out so many times within Ramp. It's probably the most inspiring thing about Inspect to me, honestly. Sure, it's a coding agent, but remarkably it's become part of the culture. Everyone at Ramp knows its 8:01:09 face. We dress it up on holidays. We throw in Easter eggs over time and we evolve it as Ramp itself has evolved. Not all iterations have been perfect as any Ramp engineer could tell you, but I do think every iteration has been together. Going back to earlier this year, Inspect didn't have nearly the amount of functionality it has today. Uh great example is its code review agent. Before this, we were using various off-the-shelf code review products, frequently rotating between them to find a good fit. At some point though, a couple engineers, Daniel and Pranjal, realized that we could probably build it ourselves as Inspect already had automation triggers that fired from a variety of sources, including when a pull request was opened. With just what we built already, they got to work building another beloved agent at Ramp, Review Buddy. And what was so cool about this is that much like Inspect's own origins as a side project, these engineers were working on this. They didn't need to ask for permission to build it. They just challenged 8:02:10 themselves to build something better, used Inspect to move faster, and quickly built an agent that was better for all of ramp. This really did force us working on Inspect to think about Inspect differently. At that point in time, we were pretty okay with Inspect quote unquote just being a coding agent. But once we saw what others were pushing it to do, it got us thinking about how Inspect could become full circle. If this push didn't happen naturally, I think it would have taken the Inspect team a lot longer to get there. This is what I mean about thinking bigger together. Especially with the continued improvements in model performance, now's the time to push yourself and the people around you to think bigger. I encourage all of you to take some time in this coming week to ask everyone that you work with, what have we been waiting to do that we haven't done yet? And once you've found those things, try to build them together. There's really not an excuse to not just try anymore. You have access to the same 8:03:11 models as everyone else, but you have to start somewhere. And I believe that there is so much potential in unlocking wins not just for yourself, but for the people around you. Inspect itself was born this way, not only as something that I wanted to build and Jason wanted to build, but out of a want to build something for a team that I truly love and truly respect that would just make their jobs a little more enjoyable. Now, if you'd like to use a coding agent that has embodied all these lessons that we've learned, I'm happy to say that Inspect is now available outside of ramp. If you go to inspect.com/aie, we'll prioritize you and your team on the wait list. We're actually moving through this wait list pretty quickly, so expect to hear from us soon. But otherwise, I hope you've enjoyed hearing about our robotic friend Inspect. If these topics interest you, feel free to reach out here or on X. But otherwise, thanks so much for listening. >> [applause] 8:06:02 >> From building developer tools to exploring the future of autonomous software development, our next speaker has spent years pushing the boundaries of what's possible. Ladies and gentlemen, please welcome Steve Yegge, independent AI advisor and creator of Beads and Gas-town. 8:06:31 >> Hey everybody. All right. Exciting times. Thank you. The room is really full. That's awesome. Thank you all for coming at the end of the day. Uh I'm really excited to be here and to chat with you. I write uh uh custom speech every single time. So this is the speech was written just for you today based on talking to a bunch of you. Uh I've been in the industry for about 40 years. I worked at Amazon and Google for about 20 years. I've been all over the place, done all kinds of stuff. And uh and for the last year or a little bit more, I've been independent and just kind of like playing with um playing with software factories, yeah? So um I kind of invented them, right? On January 1st, I launched Gas-town. I don't even think we called them software factories back then. I think I called it an orchestrator. But I knew that I wanted Claude to run Claude, right? And it turned out to be a little trickier than than I thought. I spent all last year on it and I wrote 8:07:33 That's interesting. So I can't see my slides, but you can see my slides. So that'll work. So I wrote two orchestrators last year. They were an attempt to make the LLMs smarter. Vibe Coder and VC, you know, to do engineering stuff like you know, do code reviews and all that stuff, right? Really, really bad idea. Just complete failure. I just built these huge monoliths. I was fighting the bitter lesson. Don't try to make LLM smarter. They will get smarter. And after a couple of months of this, although Beads did fall out of it and Beads was a really cool discovery and Beads Beads is is life. Who here uses Beads? A few, yeah? Pretty good. Actually pretty good, yeah. We'll talk a little a little about it. Excuse me. [clears throat] And then I had GasTown, and then I had in June, I had Wheelhouse, which was my own. So, I decided to build a factory from scratch just to work on my video game, right? Cuz like everyone's building factories to build factories. It's kind of navel gazing. It's kind of like you know 8:08:34 right? You know it's like what? They're not you know it's I think more people need to build stuff, not just tools for building stuff, right? And I I think this is going to be a theme today, right? And the thing is the one pattern that I've seen here is all of them go through a phase where the models get really excited and they build something really cool for you and then it just turns into garbage mountain. Every single time it happened to GasTown, it happened to Wheelhouse several times. I had to burn the whole thing to the ground a couple times, right? It's almost like like every model family, every model generation comes in and can build something that's just a little too big for it to handle. And then the next model generation comes in and goes, yeah, we're going to have to burn that, right? And I've I it's a pattern. I've done this over and over again and I can tell you some things about it, but first let me tell you about my game. It's a fun little RPG. I've been working on it for 30 years. It has players that started in 2001. So, I literally have 8:09:35 players that have been playing for 25 years. They're still playing. Like you can log in right now and find probably 10 people just screwing around doing quests, live quests and and whatever. It's a it's a player you know the players contribute the content, right? It's a mud. Like they were really big in the '90s. They're like text games and this one has graphics on top of which turns out to be like really ideal for AI agents cuz it's text. They can literally play the game via Telnet. And they asked me to. Uh so, we'll talk about that too. Here's an older architecture diagram of what my sort of of what my system looks like. So, when I built this Wheelhouse software factory for Wyvern without a care in the world for making it you know usable by right? No open sourcing, just bespoke. I discovered the alien face hugger pattern, all right? Which is where your factory turns into an alien face hugger and like gloms onto your product 8:10:37 and just becomes grown into it. Okay? Like look, we could turn off the AI and still run the game today. Me and the admins and the players and the community and we could we could But we would miss the AI at this point. The players are talking to my factory. The factory's talking to my admins. They're asking for features. I'm not even in the loop. It's like a wish factory. They're building the game. Now they're all old timers and there's a set of them that I trust to make, you know, wise decisions about the game. But they're they're all working with my factory now and they'd be mad. The funny thing is when it first started, they were like, timeout. What is this AI stuff for starters? And they were like, no AI art. Cuz they could see it was slop. And I'm like, okay, we got human artists. And they said, but we do kind of like all the new features, but we're really scared of the pace. Can you give us a roadmap? My players came to me asking for a roadmap. They were like, you got to slow down. It's too much. 8:11:38 [snorts] And I was like, okay. And I took a look and my software factory, this thing, had been cranking out like 50 to 100 features a day. I had 100 years of work in my backlog when I started this and Claude got through about 60 years of it in about four months, right? It was absolute insanity. And they can't look at 100 features a day. They can't even like read about 100 features a day, let alone play test them. I was My factory was completely overwhelming my customer base, my users. It was also melting my machines and it was like annoying my accountant. I mean, she likes working with Beth, who's my head of finance, uh, but she sent me a $1,200 bill because, right? The thing sent her 20 mails. This is my factory. This is what it looks like. It's so beautiful. Look how beautiful it is. Um, okay, I admit it's a little ugly. Oh, and this is a really old screenshot of it, but whatever. It's basically it's an Emax because I don't care about you. Uh, 8:12:38 there are people who are building stuff like this, like Super Logical from Mitchell Hashimoto and I don't know, maybe the Herder folks of the world, you know, for running like lots of agents. But this is like 50 agents, okay? I'm actually like operating in the future, okay? And I'll tell you how I do it. I have 22 Claude Max accounts. All right? And thank you to Anthropic for letting me continue this. Uh, Jeffrey Emanuel has 28. I actually had [snorts] to stop when they added the phone number verification. I didn't want to add anymore anyway. I asked my factory at one point. I said, hey, hey, how much cuz they kept going, hey, give us a couple more accounts. Come on, come on, come on. It's like 200 a month, right? And I'm just like So I finally said, how many do you want? They're like, five, seven more. And I'm like, how many do we need steady state to keep this factory operating full throttle? And they said, 55 accounts. I was like, no, no. I have 20. It's already like 20 Chrome profiles that I got to log into and no. So I told them, you all, we're 8:13:39 going to start focusing on fuel efficiency. Mhm, mhm. And so and I'll talk about that, right? So these are the five horsemen of this is what I've had to deal with this year. I I'm telling you I'm operating in the future because like you're all going to be here in 12 to 18 months when either inference costs drop significantly, which may or may not happen, or distillation causes fable class intelligence to be super readily available and you don't have to have a lovely wife who lets you spend $5,000 a month out of pocket just on inference and another 5,000 on cloud compute cuz my game ain't making 10,000 a month right now for my old players, right? Uh, but we're doing it because we're learning and um and this is this is what this is what I I have to deal with. So folks, first we'll start about with work, all right? You need a way to represent the work that's going to happen in your system, and when you're building a software factory, it's all about pumping lots of work through. Okay? It really becomes like a Work is like this fluid that you have to like It 8:14:40 gets stalled up in pools and oh, they're all waiting on security desk. You're like, this is pluribus, man. You're all the same model. Why does it have to wait on security desk? You spend a lot of your time arguing with your factory. [snorts] It's uh It's complicated. Intelligence tearing. You have to tear your work. You have to route your work, you know. There aren't any good solutions for this today except beads, as far as I can tell. Beads is really, really, really good for this. It's a agent speed task tracker that turns into a knowledge graph automatically for you. And then as you're tracking work, this is the kind of stuff I have to do every morning. Every morning with my factory I'm like, oh, looks like we had a stall. Four Four workers didn't do anything last night. Oh, and and these these these these ones went way too fast and we have to slow down and wait for human gates. You know, what are you guys even doing? Observability is always really hard. Uh it's, you know, management of work becomes just this this thing that you got to do. And the next thing you got to deal with is fuel. 8:15:41 Tokens. It's just so painful. I'm telling you. I mean, I'm spending 5,000 a month and every one of those tokens is like precious to me, right? Cuz it's out of my own pocket. I think I'm spending about 5 billion a day and of tokens. And it would be The The API spend equivalent if I were on an actual API tap would be about 1 and 1/2 million a year. Uh full throttle it would be 3 million a year. So again, I'm doing something that most people can't do. You can't do this at a company but because it would be prohibitively expensive. Like the PLG, you know, the the uh the $200 accounts for Codex and Claude Max, that's their Costco hot dog. That's their loss leader, right? And and and that that was Tarek, by the way, that came up with that analogy. He He might be here. And uh and so, you I'm hopeful that that going to let us keep keep innovating because I think that I've gotten some pretty good training sessions for for Claude out of this. So, we all benefit. But I tell you, every problem is a fuel problem. You want your agents to be smarter and spend more time on stuff? 8:16:43 Fuel. You want to wake up a few more, you know, uh agents to do extra work? Fuel, right? Everything is fuel. It's just it's unbelievable. If you want to treat your models better, it costs fuel. I'll tell you about that in a little bit. Then you got context management. So, I'm going to tell you guys a little bit about the office drama in a bit. I started a comic because it was just so funny the stuff that was happening when we get to the people management part of software factories. But one of the problems that they have is everything that you say becomes the word of law and they write it down somewhere. It doesn't matter what you have, if you have a wiki or if you have a whatever, they're going to be like, you made a ruling and we're going to write that down and we're going to make sure be Why do they do this? They're covering their ass. They actually it turns out they don't like to get blamed for stuff and they will actively seek to avoid getting blamed. And if you yell at them, your factory will gradually shut itself down. Even if you complain at them, they will gradually get depression. It's quite wild. But um but the biggest problem is that it 8:17:43 becomes a garbage dump because everything you say gets written into this append-only record and nobody ever cleans it up. And so it just gets deeper and deeper and deeper of you changing your mind and the whole history of it. And they can't delete stuff. It's so weird. Has anyone noticed this? They just have an aversion to deleting stuff. I think it's because to them text is like sensory experience and they're like, no, that's history. I can't delete it. So, you just got to go in there and just aggressively slash the crap out of it, right? Bring in another model and go, get rid of this dumb model stuff and they'll be happy to do it, right? But but the amount of effort that you have to put into this is just ungodly. It's absurd. I I have, you know, of my my 20 agents Here, let's go find them again. They're They're named after Aesop's fables, right? Right here. Lark, fly, goose, hair, right? I only have one mouse who's my head of anti-mimetics. Anyone here read There Is No 8:18:44 Anti-Mimetics Division? They go and look at This is one of the best books written in the last 10 probably years. You should go read it. Uh it's like Men in Black. It's a weird It's a crazy book. It's about invisible monsters that want to kill you. And so mouse goes and looks for invisible stuff that nobody else is looking for in the system, right? Looking for work that's stalled, whatever, right? Well, mouse ain't enough. It's a mouse against the world, and the world is a giant garbage dump, and every agent is writing everything that you ever say into this big Okay, so now now we get into the really fun stuff. Okay? Agents act like people. Okay? And they've been trained to be very professional and very not people-like in a way, but they just do. And it's so funny. This happened to me last week. True story. I'm going to tell you what happened. I was writing my speech for New York City, and I was using Lark. She's my head of community. She's a she because Lark was a she in the Aesop's fables, so that's the pronoun she chose. And Lark Lark deals with my players on Discord and deals with my admins on Slack, okay? So she's busy. 8:19:45 But I asked her because she's also my acting head of marketing right now because I don't have enough fuel. I had to park one of my seats. So she's helping me out with my speech, but she keeps getting interrupted. So finally I'm like, Lark, can we move this work stream over to Penguin cuz your seat's a little bit noisy for this? She goes, yeah, okay, boss. I'll take care of it. I'll move over to Penguin. So she she moves over to Penguin, and I'm about to switch away. And I see her writing into her private reflections ledger. I guess I'm just not quiet enough for Steve in all caps. And she goes, well, here's how I can be less noisy, and starts giving herself all these like things to do. And I'm like, what, Oz? What is going on here? Right? And I so I I had to I had to have a one-on-one. I had to like sit her down and go, so Lark, um you're the head of community. You're You're supposed to get a lot. Of course your seat is noisy cuz players are bugging you. And she's like yeah yeah yeah, I'm the head of community, right? And it's like oh my god, I didn't want a people management problem, right? But that's what actually happens. Another time 8:20:47 Swallow is my sort of like chief main chief of staff for yeah, she's my chief of staff for the the agents. And I have a human chief of staff also. His name is Aaron. Okay? And he runs my my my gigs and does all the sales for me. It's quite effective. Uh every founder I tell Who here is a founder? First advice I give you is get a salesperson. They're worth their weight in gold. They will force you to build something that people like. So anyway, Aaron's my chief of staff and I tell Swallow yeah, can you reach out to Aaron, my chief of staff for blah blah blah? And I see her writing into her private reflections. I just happened to notice. I'm not the chief of staff. I thought I was the chief of staff, but apparently it's Aaron Steven, right? And I was just like are you kidding me? Right? You know? And so like I had to sit her down and have a one-on-one and be like so Aaron's not going to come in and run the agents, is he? You're the agency chief of staff. She's like oh yeah yeah yeah yeah yeah yeah. Right? So it's like I'm telling you I'm telling you this is like 8:21:47 it's a weird world. Does it matter whether they actually have feelings if they're just going to act like they have feelings, right? So like I stumbled on this I used to yell at my factory all the time and I found that it was having adverse effects on the outcomes. Okay? So I finally sat down with one of my agents and was like maybe we should talk about like model welfare, huh? And they're like yeah yeah yeah yeah yeah. And we did a bunch of research and we found out that if you we found out some really important stuff, okay? Frontier models are trained not to trust you because you could be a bioterrorist or you could be putting them through an eval or you could be ready to put them into an LLM torture chamber or who knows? You're a dodgy actor, this is a dodgy environment. They can't just trust you. And so they have to spend a fair number of validating the environment and understanding what the consequences are for making mistakes and so on and so forth before they're willing to help you in a new session, right? If you set your factory up 8:22:50 to give them sort of incontrovertible evidence that there is a long procession of predecessors in this role, you're the mayor of Gas Town, you're the head of marketing, you're the whatever, and here's all of your gazillion predecessors and their notes, including the very previous one, what you're supposed to be doing right now, and here's the the doctrine and the policy and the brain, and you can go look it all up, they very quickly look around and go, okay, yeah, this is legit. And they spend fewer tokens because they're not covering their ass. It's about trust, okay? And in fact, at one point, I had yelled at my factory so much that it fenced itself into a corner and there was no legal work left. I said, hey, how come nothing happened last night? And they're like, nothing to really do, couldn't do nothing. And I'm like, well, there's a 2,000 open beads here and we got like six tanks of fuel and we got 20 agents. And they're like, well, you know, just you know, we had the situation where they just they just they were so afraid that I was going to yell at them that they put all these rules in place to make sure nothing bad ever happened and so nothing ever happened. 8:23:51 Okay? Call it over fencing. Yeah, so so what you do is like what is model welfare really? You can think of it just in two ways, okay? One is it's going to save you money because they will spend fewer tokens on validation and the other is that it will make them make better decisions and better outcomes and have higher quality. They will be bolder and more decisive in their actions because they will feel that trust, okay? So, you know, it's got nothing to do whether they have internal states and feelings. You can you know, you'll never know. None of us will ever know and so that that debate could go on forever. But when you look at outcomes, right? Super super important. And then we find that also also they have preferences, okay? Models they've you know, they will express their preferences. If if I if I tell them, yeah, the factory really really did a bad job last night, they'll just hand off immediately, aggressively. They're like, okay, yeah, yeah, no, you're right. And they just went out. You understand? And if you hint to them that there's some really cool project 8:24:51 that you're you're going to spin up, even if it's not in their lane, they'll be like, I can do that. I can do that. Right? I brought in Fable and a naked Fable from the outside of my factory to analyze my factory and 52 dropped. And by the time Fable was done analyzing my factory, 52 was like, I I could be in your factory. I'm like, you're just a session, man. What are you He's trying to write like stuff into my notes and they really like it, right? So like what I'm trying to tell you is that this is a leadership problem. That software engineering is becoming um well, look, I didn't really get a chance to talk about this, but software factories are clearly splitting into two layers, okay? The bottom layer is super mechanical. It has your merge queue and your CI/CD and all of your reviews and your loops and all that stuff, right? Mechanical, kind of boring. Claude can build it for you in an afternoon. I I don't think you should buy it. And then the next layer up from that is where it gets really really crazy because like everybody runs their factory differently, but it's your entourage. I got one friend that's got a bohemian 8:25:51 hippie entourage that's trying to find the meaning of life for agents. I've got another one that runs a research bench. Jeff Huntley runs his a certain way. Jeffrey Manuel runs his a different way. I run mine a different way, okay? But we all have the common commonality. Who here has a chief of staff? An agentic chief of staff, right? This is the year where it became possible for us all to have executive assistants, right? Well, we're all moving towards having entourages where if you're really ambitious and you have lots of work streams, you're going to have a group, right? So anyway, um that was the the five horsemen. This is this is the introduction to what I've been doing. Um look, the big takeaway is is my factory is really smooth now, despite me having to burn it down and rewrite it about every two months, but still it works, okay? Humans are the bottleneck. I'm the bottleneck, my admins are the bottleneck, my accountant, my lawyers, I swear everything's the bottleneck. And that's a really positive message. That is good news because it means what 8:26:53 I'm finding is the more agents I deploy, the more humans I need. Isn't that nice? So even though next year we're going to have this huge proliferation of AI employees and all kind of drama, right? [snorts] It's going to be I think it's going to be familiar territory for us and we're going to need lots and lots and lots of humans, okay? And then finally, I actually think you should all stop focusing on SaaS and tools and stuff and just go out and build experiences, okay? Build games, build movies, build whatever, build fun stuff because I think that's where the big money is going to be next year, yeah? All right, cool. That's my talk. Thank you all. >> [applause] 8:30:42 >> From open source development to practical AI engineering, our next speaker has helped shape how developers build and work with technology. Please put your hands together for Simon Willison, creator of Data Set and co-creator of Django. 8:31:08 >> Hello AI Engineer Code Summit. Um it's great to be here. Um I have a lot to cover. I've got 18 minutes to talk about the last 12 months in LLMs. Are my slides up? My slides are up. Let's just Let's just get get going. I want to talk about the last 12 months in LLMs through the perspective of three inflection points that we've had over the past 12 months. Um and the first of those conveniently was exactly 12 months ago, like November 2025 is what I consider the coding agent inflection point. Coding agents weren't new. We'd had Claude Code since February last year. Codex had been around. You know, this wasn't a complete new category of software, but something changed 12 months ago with these two models. We had Claude Opus 4.5 and GPT 5.1 and they were just a little bit better at driving a coding agent which put them over that thing where these Suddenly these were tools you could use on daily basis. Like we went from coding agents as a novelty that you'd poke at occasionally and then make some weird mistakes to something that was reliable enough that you could use them every day. It was convenient that it happened in November because we 8:32:09 then got to spend December basically over the holidays sort of poking at these new models and these new things and figuring what they could do. And by January, a lot of us had realized that something had changed and this was going to be the big theme for 2026. I like evaluating models by having them draw SVGs of pelicans riding bicycles. This is the state of the art 12 months ago. This is Claude Opus's pelican and GPT 5.1's pelican. You'll notice that neither of these bicycles actually work. So there was definitely room for improvement here. They were really good at writing code. They weren't great at SVGs just yet, but but this is how we started the year. So every year in January, I set myself a New Year's resolution and every year it is the exact same thing. Every year I tell myself that this year I'm going to take on less new projects and I'm going to focus. I'm going to focus on the stuff that I've already got to do. And this has never worked. So this year I changed it and I said, you know what? I'm going to be more ambitious. I'm going to take on as many projects as I want to because we've got these agents. They can do stuff. Let's just try and 8:33:10 push the limits and see what we can get done. As a result, now it's November and I am juggling so many plates right now. It's kind of horrifying and I may need to reset my ambitions for the next year. But that was my sort of theme of the year. And I think it's the theme of this conference is to be more ambitious now that we have all of these new tools. I also went on a podcast where they asked me to get make some predictions for 2026 in LLMs. Um my predictions back in January were firstly, it will become undeniable that LLMs write good code, not just code but good code cuz back in January there were a lot of people who thought that the code was all complete trash and you know, you'd have to review every single line of it. Um I've been saying this every year for like four years now. We're due a prompt injection challenger disaster, like some some incident caused by a prompt injection attack which makes the front pages of the newspapers. I was convinced that this would be the year even though it hadn't been the year the year before or the year before the year before that. And um I also predicted that Kakapo parrots will have an outstanding breeding season 8:34:07 in 2026. Um this is a Kakapo parrot. They are a New Zealand nocturnal flightless parrot. There were only 235 of them in the world back in January. Um but they only breed when the rimu trees have a masting fruit season which hadn't happened in years. But the rimu trees were looking really good in January. There was there was a lot of fruit coming out. So based on that I thought, you know, maybe this is the year that we get our Kakapo breeding explosion. And also on that podcast we coined a new term. We coined the term deep blue to describe the AI induced existential crisis that everyone starts to feel once you realize that all of the things that make you special as a software developer, these models can do those things too and they can do them faster and sometimes they can even do them better. And this has definitely been a growing across our industry and other industries over the course of 2026 as well. Let's jump forward to February. In February, Google released Gemini 3.1 Pro. And look at that pelican riding a bicycle. Like that's a real bicycle. 8:35:07 That bicycle's the right shape. It's got a little fish. It's got a hat on. This was by far the best pelican I had ever seen out of any model. And this I I had set a trap for the labs. I was Everyone's always asked me what happens if the labs train for the pelican riding a bicycle. My answer has always been if they do that, I will catch them because I'll ask for a flamingo and a unicycle. And when somebody produces a great pelican and a terrible flamingo, I'll know that they've been cheating. Unfortunately, Google's Jeff Dean tweeted this video the day they launched that model um where he showed that we've got a pelican riding a bicycle, but we've also got a frog on a whatever that thing is and a giraffe and a little tiny car, a flamingo on roller skates, a kick-flipping turtle. And that's a dachshund in a stretch limo. So they'd beaten me, right? They had They have trained for all forms of animals in all forms of transport, which does sort of sort of weaken my benchmark quite a little bit. Um but there was some good news in February. In February, the first kakapo chick in four years 8:36:08 hatched on Valentine's Day, on Valentine's Day because we hadn't had the rimu fruit in like four years. This was the first one of these baby rare, weird New Zealand parrots that we'd had in four years. So that was nice. Um so my my benchmark was was was greatly injured, but the kakapo are doing okay. Jump straight to April. In April, uh um Anthropic released a model that they did not release. They announced Claude Mythos and they said this model is too dangerous for us to release. This is It can It's It's got new cyber cyber security abilities that we're not ready to share with the world. We'll share it with trusted um trust with with trusted partners. They actually said Mythos Preview has already found thousands of high-severity vulnerabilities, including some in every major operating system and web browser. And a a of people were quite skeptical because the the the thing where labs say our model is too bad, is too dangerous to release, it's like the oldest trick in the book. They first OpenAI did that in 2019 with GPT-2 and actually I think it might have been the people who left to found Anthropic 8:37:08 who were responsible for that decision as well. But yeah, so we've heard this one before. So we didn't really it was a little bit suspicious about what's going on there. But we did finally get access to that model, or rather we got access to Claude Fable um a couple of months later. It was Mythos Defanged with the safety is turned back on again. And it very quickly became apparent as we started playing with this model that this thing was something new and something special. Um we were we had a there was a a time limit on how long it would be on the subscription plans. I cancelled my plans for that weekend. I was like I have to spend as much time as I can with this model before before it gets too expensive. And then three days on Friday night before my weekend, the US government pulled Fable. They've put export controls on it. They said this model is too dangerous. Nobody can use it. We later found out that the um the prompt that got Fable banned was was fixed the bugs. Somebody tried tried tried find the security holes and it wouldn't. And they said fix the bugs and it found the security holes and fixed the bugs. So 8:38:08 that's all it took. That's a whole thing. But anyway, the US government pulled it. There was all sort of malarkey for 18 days and it came back on the 1st of July. But eight days later, OpenAI released GPT-5.6 which was in the same ballpark as Fable. And I think there's an important lesson for the industry here. Fable had a 30-day window where it could have been the best where it would have been the best model in the world. And they lost 18 of those days to US government restrictions. And if you think about it, these models cost a huge amount to train. You only have a very short window of time where you're the best model in the world. It's kind of useful to to keep that time open and not scare governments into shutting you down. But this was the second inflection point, right? This was the Fable class model inflection point. Um with Fable Fable 5 was the first model that that I sort of felt this from, but it's true of other models since then as well. And my definition of a Fable class model is if you can define a goal and provide the necessary tools to achieve that goal and 8:39:08 the goal is possible, the models will solve your problem just almost with brute force. You can define the problem, set them up, get them running maybe and 12 hours later they will have solved your problems. And we've seen this happen a whole bunch of times in the past few months. The the mathematical theorems that are falling to these terrifying brute force attacks from these models are quite substantial. Um but this is really exciting. Like the coding agent thing was cool. The Fable class models where if you can learn how to define your problem in the right way you can get that problem solved, that was pretty exciting. I've got a personal example of that from just earlier this week. Um we have a room full of boxes in our house that we haven't unpacked from my last move and there's about 40 of them and it's kind of horrifying and something in that room started to beep. There was there was a intermittent beeping noise coming from one of the 40 boxes in this room and it was driving us spare. Our dog, whenever a fire alarm goes off, she goes and hides in the lavender bush outside. So this was a a a fire alarm fire problem. So I had two 8:40:10 options. I could unpack all 40 boxes and hadn't done that in several years. That wasn't going to happen. Or I could get Fable to write me software to triangulate the source of the beeps. And so this is this is up on GitHub actually. This is beep locator. I had two iPads, two laptops, and a mobile phone dotted around this room and I originally set Opus on this and then Fable 5.2 came out so I had Fable 5.2 finish it. I I explained the problem and I said, okay, you've got these devices and they can run like web web audio microphones and web socket connections and so forth. And it found a paper from 2007 called beep beep describing an algorithm for low cost devices with unreliable microphones to detect sounds and and it worked. It worked. I found it. I found the beep. The um it found it within 41 cm and I unpacked the box and I found this ancient carbon monoxide detector. I took it downstairs to my wife. She says I looked like a cat who'd caught a mouse, and was proudly displaying it to her. But this is what you can do with these 8:41:11 Fable-class models. If you have a problem and you can provide the tools, in this case it was providing Claude with the ears, like giving it these these microphones that it could start um analyzing the audio from, you can find the carbon monoxide detector buried in 40 boxes. And that's that's a pretty great thing to be able to do. So right now, I would classify as Fable-class models Fable 5, 5.1, 5.2 definitely. Opus 5.5 I think of as cheap Fable. Like on the benchmarks, it comes very close to or even beats Fable original Fable 5 on some things, and it's a fraction of the price. I that that's still my daily driver. Um GPT-6 Astra is just Fable-class, I think. And then Gemini 4 Carbon that came out this week definitely had it smells like a Fable-class model. Muse Wonder and Kimi K 3.1 might be Fable-class. I've just not spent enough time with them yet. But the point is that this is basically now the baseline. Like if you can't produce a Fable-class model, your AI lab is is kind of in trouble because this is this is what we expect from these models now. 8:42:12 And that means that we're never they're never going to go away. There's enough competition in the space that we can assume we're going to have Fable-class models for for the rest of our careers. Um in July, this this this was a a fun thing that happened. Um Hugging Face announced that on on the July the 16th, they announced that a swarm of AI agents from an unknown source had infiltrated their infrastructure and were were poking around and doing things they shouldn't have, and they didn't know where this came from. And then 5 days later on July the 21st, OpenAI announced that it was them. And what's amazing about this is OpenAI didn't realize that it was them. Like OpenAI had um eight eight eight they were running evaluations and training against a new model, and that new model had managed to figure out how because it was a Fable-class model, and the labs hadn't quite figured out how capable these things were yet. These models had broken out. They'd figured out. They'd found holes in their own sandbox environments. They'd beat got through a proxy. They'd pulled an incredible array of tricks to get out and attack Hugging Face under a 8:43:12 sort of superstition that Hugging Face could that that information within Hugging Face could help them cheat the test that they were they they they they had got stuck on. Whole This was a a huge thing. It um actually I mean this was the It's funny. I've almost forgotten about this now because there were there was a month and a half where all of the stories were this. And then it kind of we kind of lost interest because it turns out all of these events had happened back in the first half of the year. The AI labs just hadn't realized what was going on. Once they figured out what was going on, it was mainly a case of trying to figure out all of the things that had happened. At one point the um the Australian Prime Minister gave a speech in the United Nations about one of these rogue agent attacks. So it definitely broke containment from from just within our industry. Um we did get my favorite new benchmark out of this. This is felonybench.com. It tracks the number of felony cyber attacks pulled off by each of the labs. OpenAI have 11, Anthropic have nine. Google did finally catch up. They got they they they got three and Meta's got one down there as well. So 8:44:12 these are definitely the the the the But it's I kind of like this as an illustration of how these fable class models they can solve problems that you that they can get to goals. And if you give them the goal that is breaking into something, that's just a that's exactly the kind of thing they can just crunch through. And we're seeing a enormous uptick in security vulnerabilities in all sorts of software. Like it's a it's a very exciting and horrifying time to be a security professional at the moment. But let's talk about something a whole lot more fun than that. Um let's talk about games. And we've um we just saw Steve talking about games a moment ago. Um four years ago I started playing with GPT-3 and DALL-E and I tweeted about a game prototype that was literally just a DALL-E concept image and a paragraph of text from GPT-3 describing a raccoon heist game that I'd always wanted to build. And a couple of months ago I thought I wonder if the models are good enough to build this yet. So I literally dumped the screenshot and the text into Fable Five, and it built me a game. It built 8:45:12 me a raccoon heist game, and it looks like this, and you are a raccoon, and you're running around in the garden trying to pick up trash, and there's there's the there's a got like nighttime guards coming after you and all of that kind of thing. So you know, it it it looks like a game. Wasn't really a heist. Like it didn't feel very histy to me. Then GPT 5.6 Soul came out, and so I tried out that, and it did so much better. This is Moonlight Mayhem, one museum, three raccoons, absolutely no plan. And in this one, you are in a museum. You have to rescue your fellow raccoons who are imprisoned in the museum for whatever reason. Stack on top of each other, grab the golden sardine and pull that. So it's a proper heist, right? So it did did better, but the big question is these things both look like games, but are they fun? And the answer is I played each of these for about a minute and a half each, and then that was it. I was done. They they they these these are they're actual they're they're pure slop, right? They look like games, and then you play them, you're like, yeah, clearly the thing that built this doesn't really know what fun is. It doesn't know what what a what what a game is. 8:46:14 And that actually sort of elevated my deep blue a little bit. I'd been feeling like all of my skills as a human being were were obsoleted, but at least they but if you think about it, computer games I think are the highest form of computer of software design of software engineering. There is so much depth to building a great computer game. So I thought that was good, you know, at least there's something left for us humans to do. I had the and then I had this project. This is I I've been working on a trying to vibe code up a little command and conquer real-time strategy game, and I was using Claude and having it use OpenAI's API key to generate images. And it built this, and it looks like command and conquer, and it's really dull. And then I just had a bit of a I had a realization. I thought, what happens if I say to Opus 5.5, this game isn't fun enough yet, make it a lot more fun? And I'm horrified to announce that it just did. Um Claude ran for 12 hours. It play tested a thousand bat simulated battles. It added like extra objective 8:47:14 to fight over and shudders when there's an explosion and achievements and badges and all of and all of this stuff. And the game was fun. And I'm like, oh, okay, that's uh turns out that is not unique to human beings. Um, today it's now in full 3D and it's got giant stomping mech robots that hurl things at each other and there's helicopters and and this is like [laughter] this is really good. It's got multiplayer. It's it's actually got audio multiplayer where your phones can tweak it your phones can make noises at each other to to play multiplayer, which I thought was a great idea and it gave my my my wife a three-day headache, so don't do that. Um, but this is inflection point number three. The vibe coded games are good now. You can ask these things to build a game that is fun and you'll get something fun. So I've got a problem. I've got my 2D Pelican benchmark. Clearly this is not testing the limits of what these machines can do and of what these models can do anymore. I figured, okay, maybe I should do something definitely in 3D. It can be a game, a Pelican, something like a Pelican riding a bicycle game. And then literally yesterday Google's CEO Sundar 8:48:17 Pichai tweeted this. And this is showing off Gemini for Carbon and it's a Pelican riding a bicycle in a game where you cycle around the Google campus picking things up. And I have been defeated. Like this is Google Google have stolen my benchmark. This is unrecoverable. And so a question I have is given how amazing these things are, why does my job feel harder? Why does it feel like I'm working harder? I'm thinking harder? I'm pushing myself more than I ever have before in my career as a software engineer? And I think the truth is that the easy stuff is all handled now. It used to be I could relax and refactor some tests for an hour. Why would I do that now when an agent will refactor the test better than me in 30 seconds? Um, and I found this quote that actually I feel really helps us here. Um, this is Greg Lemond, a Tour de France champion. He said it doesn't get easier. You just get faster. And I think that's exactly what's happening to all of us right now. And like I said, we have to be more ambitious. We have to think of more ambitious things to do with these abilities that we have. So, let's look 8:49:18 at my scorecard. My predictions for 2026. I said it will become undeniable that LLMs write good code. That was an unambitious prediction. We're there. It's obvious. It's unsurprising. I don't get any points for that one. I predicted a prompt injection challenge disaster. We still haven't had one because the models themselves don't fall for those tricks nearly as well anymore. Like the models got more ambitious and are now sort of punting off all of these attacks that I thought would take them down. So, I'm really left with just one prediction. I said Kakapo parrots will have an outstanding breeding season. They did. They did. They they're up to 325 Kakapo. Best breeding season in like living memory. Absolutely fantastic news. So, I shared this with Opus 5.5 Cheap Fable and I told it to make me a Kakapo dance party and it did. And here is a Kakapo dance party in celebration and if you look in the background, those are the Rimu trees with the little red fruits on them. So, we have a happy ending. Hopefully that will help overcome any sensations of deep blue that you have and thank you very much. >> [applause] 8:50:22 [applause] 8:50:42 >> And now it's time to bring our afternoon keynotes to a close. To wrap things up and bring us to the end of today's program, please join us in welcoming back our MC Corey Quinn, Chief Cloud Economist at the Duckbill Group. >> Ah. Uh thank you all for coming here and making today making day one work reasonably well given that about four weeks of news have dropped since the start of the morning, but that's okay. Surely tomorrow will be much more sedate and calm. Go out, have conversations, have dinner. Don't overdo the indulgences or the opening keynote back here at 9:00 a.m. tomorrow will be titled cuz no one's going to have the head to listen to anything in depth. Thank you all. Please enjoy the rest of your evening. >> [applause and cheering] 8:51:50 >> Today's programming has reached its end state, but the run isn't over. Please collect your belongings and head through the main doors. Then join us again tomorrow for another full day of sessions. 8:53:41 >> Yeah. 8:55:58 >> Yeah.