This has been so much fun. We learned so much already. But now let's talk about how you deploy, how you monitor it. And what is the value. What are the return of executing the crews in a production environment? I'm so excited because in this lesson we're going to have a special guest, an industry expert, working from a bigger enterprise. He's going to come in and answer a bunch of questions on how they are using crewAI, and how they are building multi-agent systems, and what is the value that that is unlocking for them. So let's look into that. So here I have Jacob to share with you all details about taking agentic workflows into real industry use cases. Jacob is a commercial GenAI CTO at PWC, and I'm blown away with what you folks have been doing. PWC is like leading the industry and doing so much with AI and AI agents. I'm looking forward to chatting with you today. Yeah, super excited to be here and really look forward to all the fun conversations around agents. Nice. So, Jacob, what were the main challenges that led PWC to explore AI agents as a solution? Like how did this problems shape your approach to adopting this technology? Yeah. You know, when we started our journey, in transformation two years ago, obviously there weren't a lot of agent frameworks just kind of laying around, and the technology was still pretty nascent. So at that time, we started with our own proprietary plugin development framework, and that kind of got us going down a path. But obviously, as the technology evolves, the complexity of the use cases evolves. We kind of need to take a step back and see, you know, what else we could be doing to kind of optimize the accuracy and kind of the overall user experience of our solutions. Got it. That makes a lot of sense. Well after that, what were the key factors that kind of convinced you that AI agents were the right approach for your needs? Yeah, I think, like I mentioned, it all comes down to accuracy and user experiences. I think, you know, when we started with our initial solutions, around kind of software development, lifecycle transformation and, you know, generating long, complex documents for functional specifications, technical specifications, these are really long running processes. And, you know, the feedback from our consultants was, you know, we need to provide more real-time feedback. And at that point, you know, that's where we're like, okay, this is this is like time for agents. We need to have kind of more real-time feedback and getting the feedback incorporated into the solutions and going through numerous rounds of validation, to get to the right outcomes. That makes a lot of sense. Yeah. I think like, it's honestly, there's so much options nowadays that you really want to think this through and like how you want to build this agent and make sure that you're bringing a good experience. As a follow-up, how did CrewAI specifically help in addressing this problems and what role did the platform play in your strategy? Yeah. Great question. You know, I think for us, you know, obviously not everyone, who's working on GenAI was born an agent expert. And so there's there's a lot of barriers to entry, I would say, in terms of using an agent framework to help us, you know, more complex use cases. I think the nice thing about CrewAI is very low barrier to entry and helping anyone kind of really jump in and get hands on starting creating agents. But also for the more sophisticated, kind of developers or data scientists, you also have the flexibility to kind of get into the lower-level APIs and really customize it the way that you need it. So I think that's been the power of AI is really the low barrier to entry, and also the flexibility to kind of customize as much as you need. Nice. Those are great points. And yeah, I can tell from the team and CrewAI that are really trying to make it things like simple for people to pick up on. So it's great to hear that that is coming through. And in your experience, what have been the most significant indirect benefits of deploying AI agents at PWC compared to kind of like the direct outcomes that you initially were aiming for? Yeah, I think, what's kind of come indirectly was transparency into the process. If you look at what we're trying to do and, broader kind of transformation across our businesses, it's a lot of different things that we're trying to analyze and monitor in terms of how we're driving efficiency gains and return on investment. You know, are we getting the outcomes, you know, from the investment dollars that we're putting in? So with CrewAI and the native integration with the agent monitoring tools, you really get that kind of direct visibility. And you can see, you know, how long it's taking the agents to complete the tasks. You know, what tools they're selecting as part of that process. And we can really analyze that a detailed level in the data. You know, how much time is taking agents versus our consultants to complete a process which helps us explain the ROI. That's so good. I think you mentioned the key word that ROI. I think in the end of the day, when you think about this, AI kind of revolution and trend. It's funny to see how this can have direct impact to the bottom line from sometimes day zero. So what early results or impacts have you seen from deploying these AI agents in your operations? Have these results matched or exceeded your expectations in some way? Yeah, a great question. I think one of the main use cases around code generation. If you look at a lot of the large systems that we implement for clients, there's a lot of proprietary development languages that we're dealing with, whether it's, Abap, Apex, goes to like you name it, like all these derivatives from Java. So when we, when we started kind of pre-agent workflow, I would say, you know, we were getting hit or miss results. You know, in some cases we were as low as 10% on the overall accuracy of the code generation. But as we started to involve agents, we got more real-time validation of code with agents. Also, agent execution of code in real runtime environments and analyzing log output and feedback to generate, better code and better outcomes. So in those cases, we jumped from like a 10% accuracy to a 70% plus accuracy on code generation with agents and leveraging CrewAI. Damn, that's a lot. 70% is a big number. I like that. And what have been the the main obstacles and challenges that PWC faced implementing these AI agents at scale? Because I know there are some challenges that come with that too. Yeah. I think, you know, obviously there's all the technology scale problems that you deal with and GenAI solutions and technology solutions overall. But I think as you look at agent specifically, it's you know, one of the benefits with CrewAI as you can you anybody can create agents and it's very easy to jump in. The harder part is how do I drive standards around the agents I'm creating. How do I get re-use out of the agents I'm creating across? You know, like for example, the code generation use cases, how do I create, a consistent foundation and pipeline for code generation, but then yet only kind of tweak the things that are needed to go from one proprietary language to another. So I think it's really been more about those development patterns, the standardization, and to really kind of optimize, you know, how are we creating these agents and getting the reuse out of agents across multiple use cases. Yeah, I think like in the end of the day, like as any technology boarder down to experimentation, getting our hands dirty a little bit. And, and how did the transition from kind of like testing these agents to actually running them in the production environment differ? Like, were there any unexpected hurdles or lessons that you learned during this transition, or any specific features that helped you with that? Yeah. You know, obviously, like you still, whether it's traditional AI or now GenAI, you still have some of the core infrastructure problems that you still have to deal with technically. Right. And then obviously, with GenAI, you have all the token limits, the rate throttling, all that stuff as you have agents working together, you need to monitor that stuff more closely. So it's kind of one thing, but I would say honestly, the harder part is the change management, the human workforce element. And once you get these technology solutions developed, how do you actually drive the change in adoption and get people used to working, you know, with agents in their day to day workflow and business process? So that's where we've been spending a lot of our time lately. And just really the change management aspects around this. That makes a lot of sense. Yeah, it's a different mentality and framework that people need to kind of like embrace, kind of like getting promoted to manage agents in a way. And looking back on the PWC's journey with AI agents, what do you see was the most like critical factor for success? Yeah, I think like I said earlier, all about accuracy and user experience. If we're not driving the right accuracy, we're not driving the right user experience, trust is lost pretty quickly. And if we lose that trust upfront in the process, it's a lot harder to get it back. So that's where we're really focusing on, you know, in terms of just using the agents to drive better accuracy, better experiences with our overall journey with genAI solutions. I 100% couldn't agree more. And based on your experience, what advice would offer to developers that other companies or other enterprises they're considering adopting AI agents and this technology? Yeah. Great question. I think with all things technology, start simple and then increase complexity as you go. So kind of the old adage of, crawl, walk, run, right. So make sure you kind of start off simple, you know, some of the kind of the basic prompt engineering use of RAG pipelines, things like that, and trying to see, you know, what gaps you need to solve from there. And then with those larger gaps and kind of more complexities, you know, that's where the agent frameworks and tools like CrewAI can definitely come in and help close those gaps and make sure you're driving that higher accuracy. And, better user experience. I love that. Jacob, thank you so much. Was great to have you here. And it's great to partner with PWC on all this. Absolutely. Thank you. Thank you. Thank you.