From Financial Data to AI-Ready Decisions: Power BI, Microsoft Fabric, Semantic Models & AI Agents with Rishi Sapra [MVP]

Welcome to another edition of the MC65FM podcast. Today we welcome Richie and we want to go behind dashboard and even beyond Microsoft fabric itself. Financial organizations already have enormous amount of data, they have earpiece system, budgets, forecast, actual cost centers, power BI reports, ex-surprise and yeah, increasingly using Microsoft fabric. Now we adding a co-pilot and AI agents on the top of all of this, but AI agents doesn't, yeah, automatically understand what revenue may margin, consume up a probability or even this quarter means to particular companies. So today I want to explore the layers between raw financial data and AI that actually understand the business and that brings us directly into the world of the Monteque models, context layers, ontologies, governments and present driven data. So, but I think we let's start with you, Richie. How would you get into the Microsoft data and analytics ecosystem? - Well, through the beloved Excel, power pair of power queries. And when power BI came out in 2015, I was a financial modeler, KPMG. So my background before I got into data world really was big four. So working to the big four Deloitte and KPMG and then some of the big banks in London, HSPC, Barclays, doing kind of process automation. I've got still a fairly technical role rather than pure accounting, but yeah, financial modeling was kind of that area where I really got into kind of the data side behind all of this and making decisions and how is data used to drive calculations? And yeah, power BI came out then and I saw a massive opportunity. I mean, I don't know if you do remember power BI in 2015. It was pretty basic to be polite, I mean, pretty craft. A little bit less polite. But even then, I think it just showed that comments of especially the semantic model stuff in the power query of making that accessible beyond kind of reporting being stuck in IT teams and data and BI teams and having to outsource it all to consultants because that's been the model for decades. And I think power BI and Tableau and Kleeck as well, obviously, kind of really tried to change that model around and introduce this whole idea of self-service BI. And I think it's great and I really like the idea that actually, as a non-technical business user who's not a developer, you can actually go and build these things yourself. You can go in rather than just use Excel, you can go and build solutions that can sit within your corporate environment and connect to your source systems, connect to all of these things. And this was before Power Quiz in Excel obviously as well, right? So, well, at least built into Excel. And it was an add-on in 2010. So it gave that platform to be able to take Excel users, I think, into a whole new world. So that's kind of where I got into it, through this self-service BI wave. I think the challenge has been that since 2015, Microsoft, do you want to take, I don't know, do you want to take a guess at the number of features that Microsoft have added to Power BI? It would include fabric in that as well because it's probably on the bottom of fabric. I mean, does it take a guess, right? Tens of thousands of features and capabilities. So it's not so easy to use any board. It's not so easy for those kind of Excel users to kind of pick up. And now you've not got the in copilot studio. It's said that you've got semantic model, that you've got all sorts of technologies and tools. And I think it just gets overwhelming very quickly. So really, I think now actually, we need to think about how we can delegate that solution design in architecture and planning and thinking about how to build these things and actually do the building of them, delegate that to agents so that we can then focus on what we're, we're good at humans, which is not trying to keep up with all this tech and know how to do everything, but actually understand the business and build that context. This is saying to to to to Gen A. I yeah, I've seen you often write about topics like operation models, governance and there's all that driven insights. What do is it mean in practice? And why is it becoming more important now with the, I say the generative AI is entering the analytics tech. Oh, it's way more important now. There's, there's this book, I really recommend your audience to read. Actually, it's called Fair Game. So I think the website's Fair Game book, AI by Rob Cully, who's who's actually one of the ex-mokes of pns in the Excel, Power BI team around this time, actually when Power BI was coming out. And in here, he's talking about the two different types of computing, like CPU, computing and AI computing. And he's really, comparing and contrasting actually, they're both completely different in terms of their strengths and weaknesses. So, you know, Gen AI is really clever, but it's also quite clumsy in the sense of, you wouldn't trust it to really come up with your numbers that you need to be reliable every month and grounded in your business processes. But that's exactly what you need to delegate to the other type of computing, which is still there, CPU, traditional computing, right? It's, it's deterministic. It's, you know, it's great at doing those calculations, of working, connecting to your sources, and understanding your data. That should be the grounding, you know, it's, I think you called it, reliable, but rigid, right? And how do you get the best of both worlds? How do you combine them? And that's the real essence, I think, of where we are right now. How do we combine, you know, deterministic, you know, structured data and logic that calculates our financial numbers, that calculates, you know, all of our kind of core KPIs and metrics in a business with a kind of single version of the truth? How do we combine that with the analysis capability, but also the cleverness of AI in terms of the flexibility? Because what ends up happening is you have, you know, processes that are run in a very rigid fashion. And so you want to try and automate things when you end up writing with VBA macros or RPA and how automate those. And you know, even Power BI reports. And before you know it, you've got hundreds of these solutions to try and manage. And you build technical depth, where us with AI, actually, it can reason over those things in flexible ways. So maybe you have far fewer artifacts, and actually they can flex to different scenarios and different customers and different types of data. You don't need to set up a separate spreadsheet process or set for Power BI reports or a set for a thing that every single type of variance that you have around the type of data and the type of process that it works with. So you combine, how do you combine the data and the flexibility of the AI? I think that's where we are now. And that's what's excited about it. So yeah, yeah. But let's start with the financial problem. So let's imagine I'm the CEO of a large company, I hope, but not for them. I get to give also their bankrupt. OK, I have SAP or another ERP system. I have Power BI. I have Excel everywhere. Maybe we are implementing Microsoft Fabric. And now someone walks into my office and says, great news. We have by an agent that can answer, glass to the body of financial support. Super power. So what's the first question you will ask? How are documented are your processes? I asked if that was Christian or at odds. And for my organization, it's not, is that knowledge exists in people's heads? And it's embedded within the logic in their Excel formulas or in their Power BI reports or in their source systems. Maybe some of that notchics in the ERP, some of that notchics in Excel, some of that notchics in Power BI, some of that notchics in the SharePoint files that ultimately come to it, or at least some of the context is in there. So everything is scattered all over the place. And what we've done up until now is kind of build and say there's point solutions to deal with every specific thing, right? We need to do this financial analysis. We need to produce this report each month and let's go and build a solution to that connected sub data, model it, structure it, put our logic in there and then have that as a Power BI report, have that in Excel spreadsheet. OK, then you do that for every single process, every single time. So now let's take a step back and say, look, if we were doing this right from scratch, let's forget we have any company to get re-hide, let's take a moment, right? We just have our systems, we have our technical infrastructure is, we have that all this and it, but how would we redesign this and the grounds up? How would we think about this? And it all starts with that business context. It all starts with that thinking, OK, what are we got? What processes do we need to run? What outputs do we need to do? Which decisions do we need to make? Once you start there, what decisions do you need to make every month, every week, every day? Once you start with the decisions you need to make and the business questions that you need answered, then you go from there into logic and into data and then you pass that to Gen AI because once it gets an understanding of the business insights and it's got an understanding of your data and this could just be some CSV files that you give Gen AI. It can go everywhere from there. So really I'd just, I'd want to understand how well documented are your processes, but actually my IG, the little bit to say how well documented are your business insights? Are your decisions are your personas? Like because what Gen AI really needs is to understand who the real people in the business are and what they're interested, what their pain points are, what keeps them awake at night. That's what Gen AI really needs. So I've given like a brain dump of that and I've just actually, I'm doing an event next week and I'm going to show an example at the end where we've got, it's for a retail company. It is one in the UK called Mums and Puppas and I met somebody who works there at a conference and I bought all my kids stuff from there, so I never company. I've just downloaded their financial statements from company's house and I asked Claude to generate a general ledger like I gave it a zero, a child of accounts. And so it generate me a whole general ledger for this and then it can be a general ledger that tied up to their accounts. So actually it's probably fairly realistic and then it gave me all the dimensions that they need staff, you know, what, they can use stores and real stores, suppliers, real suppliers that they have because it knows their stores, it has their suppliers with their website. So again, the order is the data, but now I have this data set. And what I did is passed it into Gen AI, like just the CSV files and I passed it like a page or two of kind of descriptions of the kind of things that someone would be interested into analyzes data, you know, the kind of, and it was quite broad, you know, it was data on, you know, inventory management and, you know, star feed rate as even or, you know, things like that, like just quite broad, but actually, you know, these are the kind of things that need to make, these are decisions, this person needs to make on a daily basis. And it went through a workflow that I've got kind of a genetic work though that basically built this gopet template, that summarized all the phase decisions and then it said, okay, what for each, for each area of analysis, who's the audience, what's their pain points, what's the business questions they need to answer? And then what's the kind of KPIs that logic that would help to answer those questions and what would you need to view and filter by it or the ones of them? And, you know, essentially what you do is to conceptually design a semantic model at a stage, right? You don't need to call it that. That's essentially what you do. And then you then generate, I take that input and then designs your semantic model of that designs your facts and dimensions as that and designs all the measures and gives you all of those to approve and say, yeah, this is good. Then it goes, goes, it builds it. It goes, it builds you a whole power of your semantic model using power of your MCB server. Okay, right? Now you, I've got, you know, 50, 60 measures all aligns to my business decisions, all align to my, you know, the pain points and all connected or working, right? This is, so this is way, way ahead of where we were with how VI before this, right? Where we were just kind of that queries could help you with personal productivity. Now we're taking business insights into data and it's building you false semantic models with measures and then it went to build a reports and then it went to build ontologies and platinum level lake houses because it was made well actually the semantic model can't answer all those questions that you had because it's aggregated data, it's, you know, tapio based, it doesn't tell you really why what's happening in your business. So then you know, it's just a gap analysis to say, well, what could we not answer this semantic model that we needed to know for our business decisions and then designed, you know, maybe that data's coming, maybe it's for me, our PCM, but maybe it's also from SharePoint to other places and it builds you that whole kind of platinum layer that's going to ask for the, the why they allow you to drill down and then it puts it all into an ontology and then the data agents, now the data agents connected to your semantic model to your, you know, to your lake houses, to your ontology, and then you put that just as one click into co-pilot studio and then you can ask it in teams. So someone could go into teams and ask any question or ask, run any process or anything because you've got workflows in co-pilot studio, you've got all of these things now. So that's the path that I think makes the most sense is, is public data agents into co-pilot studio. So coming back to your question, there's a log with the answer to the question, what will I ask the CFO is, yeah, how well documented have you got this, you know, context or can you document this context? You know, maybe we need to record some calls with his team or her team to go through their processes and then just pass those transcripts into Gen AI and get it to produce these subways of what people need. And it's not really more complicated than that, right? It really isn't. You give us a data, you give us some context. Hey, everything else could be built for you. Yeah, I think let's just think about something simple, like revenue or so. So there are different roles, I think, the CFO from the company, the regional edge of the sales, the marketing guy and so on. And they all ask what was our revenue last quarter? And normally they have their different systems and get their different answers. I think you, we can have the, yeah, the same KPI in different ways. Or short, we have one KPI that's the universal truth. Well, exactly. So that's where semantic models really come in because they are your single source of truth for those numbers, right? So you've got your governed enterprise level semantic models. Obviously, you've got the semantic models that people build off the side of the desk as well, that's great for a peerless, easy experimentation learning all the rest of that, right? But once it gets certified and you know, pull it up into the enterprise world, that's where then your semantic models are, that's what's the truth. And so when this is why you need that as a grandied for your agents, so you don't have this challenge where people are coming in with their own different versions of spreadsheets, cuts taking a different time. So therefore you're getting different numbers, some include these adjustments, some include these adjustments. You know, you'll always end up with different numbers and pick up. And yeah, there might be different numbers that you need to bring up to pendium on the context. So again, when someone's asking revenue, is it a sales manager? He's asking revenue and just should that include, you know, commissions or should it not include commissions like the ones that does depend. And that's okay. But what you need to do is tell it when it depends. And then you have skills that you encode into agents to tell it these kind of instructions to give it this kind of process to say, follow this methodology. If it's this person asking or if it's this context, go down this semantic model and maybe look at this and this is the kind of data that we're working on that. If it's this scenario, maybe go down this route and you're coding those instructions into a skill. So your data is coming from a ground source like fabric and semantic models, your instructions on how to work with those semantic models and how to interpret those semantic models and how to read the data is coming in skills. And that's like how we work as humans, right? That's, we have skills, we have abilities to do things. So I think we really need to start thinking about how we apply the human type roles and human type skills to agentech agents and treat them like digital employees, like performance management, give them, give them, give them clear guardrails, give them clear instructions, give them clear performance metrics in terms of what goes like, so give them access to data and tools. And then let them come up with, with the wide answers. Yeah, I like this, this context, context layer stuff, but I think where should this business context actually live, should it inside the power BI, inside the semantic model or in the meta data, like peer view, or whether in the on the, notology? Yeah, why is it mixed? It's a mix and it does, you know, it depends on what the context is needed. So as I say, semantic models are a brilliant source for AI because it has those calculations and, you know, the logic that really represents your core business. But it's not the early source. So when I, when I talk about the organizational brain, there's five layers of this organizational brain. So the first is data and that's your kind of source system, your schemers, you know, where your data is coming from and how it looks like, what it looks like in your, in your certified kind of hub workspaces if you like, maybe, yeah. And then you've got logic. This is your, again, your certified C core, Python, DAX, the logic that it needs to be able to transport, Aitor and work in a different ways. You then got tools. So tools are just any tool and it, you know, could be any MCP servers to connect into your systems. It could be a semantic model is a tool because it could be query, that's a tool. And the tool is a tool. So it could be tools by any tools that you've got. Then you've got skills and that's both utility skills or the skills that Microsoft make available on how to work with tools and platforms and things like that. And you know, copilot studio has got a whole bunch of skills and agents and plugins and all of that. But also your custom skills. So your business processes get turned into custom skills that live in this layer. And then you've got governance, which is things like, you know, your access, commissions, your policies, all the things that needs to do. Now with those kind of anything that you can think of can be fought across those five layers. So you take a business process and it has some data that it connects to it, has some tools that it needs to connect into those. It has some skills on how to work with those tools, how to interpret that data, how to calculate the right things or where to go. And plus it has, you know, governance, right, in terms of how it needs to be accessed. And that's logic, obviously. So I think it's a combination of all of those five layers that work together. And that's what provides the overall grounding. So this is something you could build is something like Foundry IQ. I think that if you have you come across it, have you worked with that tool? Yeah. That's what we fought for years ago. I did find that IQ exists four years ago. I think. Really? Oh, okay. Let me think. I'm not a lot of great. I'm not so. I mean, it's just a bug search index over over over your knowledge sources, right? So in a sense, it's been around for a lot of, I said a long time lag hasn't been around for that long, but yeah, it's been around for a while. But I think it could sexually it's there. And I think it's all just plumbing this stuff together. And again, Gen AI can help you do that. So I'm trying to look at how can we take existing processes like existing Excel spreadsheets and convert that into a custom skill in the organizational grain because it can take an Excel spreadsheet and it can look at the data, look at things that are static data and whether they're as inputs, you can look at your Excel formulas, which is the logic and it can look at your outputs, which is whether the results land. And it can then kind of work with you to understand that process that's happening and then convert that into fabric lake houses, convert that into semantic models, convert that into reports and ontologies and data agents and co-pilot studio agents and things like that in a way that makes sense. And you don't know how to know how to build all of these things individually yourself. You don't have to know all these tools inside out. You don't have to spend five years to become a data engineer and then an AI engineer who knows boundary and all of this because the agents can do all of that stuff for you as long as you give it the right context and as long as you get it to communicating the right way with you. And I asked the thing I'm struggling with the most is that if I'm honest at the moment, AI doesn't communicate very well. It likes to just jump ahead to do everything and tell you, hey, I've done it. All. And you know, when you try to get it to the development stuff, hey, should I run this the end of environment? Should I run this bash command? And you're like, I have no idea what you're talking about. I didn't know what the bash command is. You haven't told me that you're writing the bash command of what it's going to do. All you're doing is asking me for approval. It's like, come on. Yeah, really. Like, I know you want to show off that you clever, but we've gone past that stage. You know, I know I want you to communicate with me like a human, right? I want you to communicate and tell me what you're doing and tell me how I get what you need from me. That's where AI needs to come scale right back down. And that's not about intelligence. You know, ashtras not going to fix that. Vables not going to fix that. It doesn't matter how intelligent it gets. In fact, the more intelligent it gets, the worse it gets. The worse it communicates because it's now things it doesn't need to listen to you anymore. It can go off into everything itself. So that's the biggest risk, I see with AI. And that's the biggest thing I think I'd need to solve to be able to get this whole solution working. I think, yeah, for the people from Power BI, they know the data model, the semantic model, and now with fabric and AI, we have something new, I say, the onotology. How will you explain these three things? Are the data models a semantic model in the actuality to someone with new and, yeah. Okay, yeah. Let me give you a try. If it's not clear, let me know and I'll read you the information. So data model is basically just a structure of your tables. So if you look at transactional systems like at ERP, CRM, those tables are all scattered around. If you look at the say something like SAP's, hundreds of tables that have just got all different connections. And that data structure of tables works really well for running a transactional system. Because you've got no redundancy, everything's in only one place, there's no repetition anywhere. So everything's just connected and it's very good for reading the writing single records. It's an absolute nightmare if you try and use that data model in Power BI. Because suddenly, to do a simple DAX query, it has to do 50 joins in the back between all these tables and try and understand how they all relate and your DAX becomes stupidly complex because it has to then pick out all these tables at different places. So you need to denormalize and actually turn it from an OLTP or a transaction processor model, data model into a low lap, tabular model, which is just dimensions and facts. And there is with that repetition and redundancy. And that's OK. Because it's just a structure that Power BI could work within the Kineco get open and analyze. So that's data model. And you know, you've got different types of data model, but that's your physical data model. And you know, you do it through Kimble. So Kimble is the kind of methodology, what if you data model? Right. And then he's been said semantic model. And again, obviously data model is called a semantic model, but OLTP, the structure is for semantic model. That's the data structure you need or is ideally what's to do. And then you have your measures, your logic and very little security permissions and all these other things that you can add into a semantic model. So now this becomes a kind of asset. And I think this is the real interesting way to think about these things as assets, right? You've got a data model, which is foundational, kind of asset, you've got semantic model, which now contains logic and stuff, which is another type of mail assets. And then you've got, as I said before, right? You've got semantic models are great for those KPIs and measures, but AI needs more than that. AI needs to understand not just what the numbers were and what you were porting was over time, but why? And actually, it's not, it could tell you that your cache was lower this month, but in order to know why your cache is lower this month, it probably needs to go into your customer contracts and identify that, you know, the terms have changed for one of your big suppliers and, you know, actually, you've had to pay out earlier than you expected or when your customers and they, you know, they're not paying the same period or something. It's not going to know that from the Power BI report because Power BI doesn't have that level of granularity. You wouldn't put that net of granularity or data into your semantic model. So that's where then the idea of nontology comes in. nontology can connect to lake houses. It can use semantic models to start endpoint, but your relationships are also less rigid. It's not a single table to table, single relationship. It's verb based relationships. So it's saying, well, actually, you've got stores which sell products, which have employees, which, you know, are located in a region. But then all of these things can hold true at the same time. And it's up to AI to then use this kind of nerve based language to almost understand the business. So business processes and entities are actually really well represented in nontology, especially for AI, which needs context. And it could be a little bit restrictive to just use a semantic model. So semantic models great for your KPI logic data models, and see what your foundation of going from transactional to Kimball style tap into models. And then you got ontologies that can have a different lens in your business. And actually can be connected to what grandly the lake house data or different data, real time data even, right? But then go to a reason over your business in that way. Yeah, I think a little bit about the model, do is the model need to understand concept like product, customer region, cost center, revenue, map margin, and especially the relationships between them. And how? Exactly. So we've got the logic based relationships, if you like, which is what your semantic model has. It says, take this field, well, it's to this field and to do this calculation, you take the number from this field and divide it by the number from this field. Okay, great. That's the layer you need to be able to calculate business logic because you don't want it to be lucid. That's, you need that to be quite tight, right? Now when you go into AI and asking for context and asking why things happen, and all of that, you needed to be lucid. You actually do need it. You want to be able to have your KPIs and metrics and know, but then you start asking, well, why? Why is my revenue doubtment? Why is, you know, why is it down in this region or what's happened here? And that's where it needs to start to understand those entities in a slightly different way, in a less rigid way, in a way that can start to explore, you know, the different relationships that might exist and you know, should exist between anything. So, you know, in the semantic model, you would know, it'd be very hard to write a formula that does all of these different things and has all these relationships, but an ontology is better suited to that. So, I think this update is not a single short answer. I think it's a combination of these things, but these context also needs to live together. This is a status, skills, tools, governance. All of these things need to kind of coexist. And you kind of need to start building this before you really can take advantage of AI beyond personal productivity. And if you want to use AI beyond kind of, you know, sub-rises, email, or even, you know, generate, you know, especially generate, you know, this presentation. Even if you start to turn into skills, you still have the challenge of multiple skills. You know, two different people have built the same kind of skill, different, different context, different data, different ideas, you know, how do you manage that? You still get technical debt. So, it's not just about skills. It's then about the broader context there is the goal of the land as well. Yeah. I think a little bit about there was before the co-pilot joins the Power BI. There was also a function. You can ask questions in Power BI. Q&A, yeah. Q&A, yeah, the Q&A function. I think it's a bit difficult to say for co-pilot. What would you say from this function to a co-pilot function? What is the big picture? They're model in each other. Q&A feature actually had this thing called a linguistic schema, which if it's their existence, their existence in Tindall, Power BI, semantic model. You can see the linguistic schema. The linguistic schema is exactly what we're talking about with our technologies. It was based relationships. And things like, ah, in fact, layer on top of your semantic model that kind of tells things how entities relate or when someone asks through this field, how does that bind into data? Like, it was very fitting. It wasn't a great experience to build it and it was very hidden. Like, put it into this linguistic schema. There's no way to extract that out and manage it at a corporate level and things like that. So that's the problem that ontology solves. You couldn't put AI, I don't think, on top of the linguistic schema, Tindall. I don't know. I mean, you could try it if you've still got any other models of an old Power BI model. That has a Q&A. Then even just try and see if AI can leverage it. I know. I've never tried it. I imagine ontology is going to be a far better solution to something like that. So, we have also, or, yeah, where do you say Microsoft fabric fit into this architecture? What did you think? What's the benefit of using Microsoft fabric? Well, it's your core ground, all your data and logic. It's no small part of plays in that tiny ecosystem. Even a small company that has a couple of systems and a dozen or two employees. Even then, they would benefit, I think, massively from spending $300 a month or less on fabric, getting all of that data into a single place, into a single name house that's governed, managed it. You can run SQL queries against it. You could put out the AI against it. You can put data agents against the AI. Just having that data logic foundation is so key. And the fact that they've then made it so easy to literally say, "Okay, add it as an agent into Copilot Studio and use the GitHub Copilot hardest now and Copilot Studio." To be able to reason over it as well. And then expose that into teams, or something like that. The fact that they've made that integration, and don't get me well, Microsoft have made so many mess ups with integrations and stuff and different teams working at different silo products that have not really talked into each other. But I mean, the data agent talking to Copilot Studio, I mean, that's genius, whoever came up with that. Yeah, I think a little bit makes it, when I have fabric, and fabric also have a semantic model, makes it then also sense to have two semantic models, one living in Power BI and one living in fabric. No, Power BI is part of fabric. There is no difference. Power BI is just one of the workloads in fabric now. It's not too separable. Okay, okay. I think what another part of what I find really interesting in fabric is it's the one like stuff. So I think we have years before we have worked and then we have this, we have to make all, yeah, all data structure. And now we have also this option to work with unstructured data with the work like, what did you think about the benefits, what's the biggest benefit? That's what I'm saying, even for a small company, right? Like it makes sense for them to bring their data into a lake house. And you can synchronize, I didn't, you know, with one lake, you can even synchronize it with your machine, right? Like one drive. See, you download the one lake file explorer and you see the Windows Explorer and you can just drag and drop files into there or save files into there directly from Excel or anything else. And then those get synchronized to the file section of the lake house. And then you can have things that file off there to actually convert that into delta or actually if they're just playing CSVs and, you know, it's a bit picky on it, but, you know, if it were structural works, it could convert them automatically as shortcuts into delta tables. And then suddenly, once they're in delta tables, once they're in delta format, everything houses just open to it. It's then that, you know, AI is open to it, sequels open to it, you know, real time intelligence cake, you're also open to it, you know, every work related fabric and every tool that could clearly use a sequel that takes a sequel endpoint, suddenly has this data available to it. And that's that's the power, including Excel, by the way, obviously. And that's how one lake like. Yeah, you live living in, in, in, in, in, right? Yeah. I think that, that, that, there are, there are so many lakes, we have actually one lake, a lake house and so on. It's, it's, feel a little bit like Scotland, right? But, I'm great. Actually, I'm going for Scottish Summit. In a, in a, in a few weeks, I've had you been once for like, carbonate. Actually, don't be really interested trip, but yes, you're right. There's, I think the scene you're in in the landscape and the lakes are, are, are, are supposed to be amazing in Scotland. Yeah. I'm a whiskey drinker as well, which helps, you know, and kind of, and it books the time to do whiskey tour after that. Yeah, yeah. So, some really good distil, you know, far worse, far worse. Yeah. It's a bit, it's a bit, I've got two and a half year olds. It's a bit harder with that, really, top. Yeah. I don't think, I don't think she's allowed in these distilries. And I'm going to, we'll, we'll opt out, Scotland would be. Yeah, but, but what, what, what's the, what's the, the direct lake? Direct lake. So, I mean, direct lake is, um, essentially queer, so, so tabular models. So it's analysis services, tabular model, um, which is obviously what, how pivot is, um, and it's, um, you know, your, your semantic models have been in tabular models. And now, the analysis service engine, what that does is basically takes your data, um, compresses it massively, you know, like 10 times or so, um, you know, and indexes, indexes stuff indexes your data, essentially. So, you know, every row, if you've got, you know, a row with 10 million rows of data, but it's just one and zero flag. It's like one bite. It compresses it to one bite as a dictionary, right? So one or zero, right? And so it's commonly, commonly made like based, yep. So actually the amount of compression you can achieve and therefore your performance, you can achieve a semantic model is, um, based on that compression technology, um, vertepac, which, um, determines how, um, how much it could compress your data and therefore how quickly it can, uh, analyze it and form over it. So that's vertepac that's been there, you know, sits in that list of services and, and, obviously, been enriched and, you know, with DAX and everything else that sits on top of that. So when Databricks came out with Delta for not, um, you know, I think Microsoft flipped about and said, well, how got a second? This is another compression technology that's calling them as well. Um, could we actually just build the entire vertepac engine into park against park a files? And then we don't need to actually do all the transformations. You don't need to go and convert your data like with the tabular vertepac engine and analysis services. And now the services is its own layer, right? So you need to convert data from whatever spreadsheets, CRM systems, whatever that data is coming into, you need to convert, bring it refresh your semantic model and bring your data into there. You could you die it query, but it does, that's, that's horrible. Okay. But if you're not, if you do an import mode, you need to refresh your semantic model, which will take all the data, bring it compression and bring it into the semantic model. But actually, if your data coming into this layer is in Delta format already in park a, which is park a files with a Delta log, then you don't need to do that anymore. You don't need to go and transform your data because Kaubi could read directly against those park a files and actually still has the same similar level of column, like compression and therefore performance. So actually that was the key thing. The key thing, the key reason why you needed to get your data into a import mode, vertipack semantic model was performance. But if you can get that performance directly off park a files, then all you need to do is get your Delta files, that's all you need to do is get your data to that format. So that's what diet query is. It's basically replicated vertipack into park a files to get against park a files instead. Yeah. And we have the next lake, Delta Lake. How can it, I need Delta Lake Delta Delta Delta Delta Delta, Delta just a format of one or eight or more. Yeah. The call, let's go ahead. Yeah, I read on your profile and you often also have a topic, it's a little bit different. It's the persona driven insights. Is this a different to or what's the difference to self service BI? I mean, it's the same, it's the same thing. I think, or self service BI or self service AI, it's also as AI as this idea that I'm kind of trying to kind of build thought leadership around, which is actually like self service BI in it, as they didn't really deliver as well as should have done because of the complexity of these tools as grown. Actually, can we bring that right back down to actually having an individual and they need to know a bit about data, they need to look good data looks like a day's quality, but you know, they don't need to spend years or the same before to become data engineers and bioblists. Can we enable them to actually work with tools like power BI and fabric and co-pilot studio and Excel to actually build the artifacts they need and but have agents do that where they provide a context. So that's the idea of self service AI. Now you said you're asking how that relates to relates to what we just talking about. Yeah. So the idea then is actually, so the idea is that people can build with if agents have the right context of those developer agents, if you like, have the right context, then they can sit on top of that and grounding and could build you, somatic models, could build you reports, could build you ontologies, could build you data agents. What it needs to understand is the business context of a sonar driven insight. So say you ask about the sonar driven insights and that's exactly where it comes down to because what it really needs is that level of who is this for what business decisions are coming right back to where we started, right? What's the business decisions they need to make? What's the key questions they need to answer from data and the ones that knows that and it has a sample of data, everything else it can do for you. So, the sonar driven insights are your key starting point. So coming back to that question for the CFO, it really is actually, how well have you defined your sonar driven insights if we're going to ask the question in a nice six thing to it? Right, how well have you defined those presented driven insights? If you define those presented driven insights as well, especially in the context of data that you have available, then GDIA has what it is. And no, no, we have bring a eye inside the system. So, and I think, while for years, I think, Colin, just Pauma, I think, like, was a guru in the... Yes, I think, yeah, we tell you data, yeah, I love that. Was that a story telling? Yes. I think you have also, right, the book, storytelling with data, I think, yeah, yeah, yeah, okay, I write, great. But how did I change the way we do storytelling with data? Oh, it's changed it massively, but with exactly the same principles, which is brilliant, right? Because, so, you know, storytelling with data is also... So, I did a talk in secret, actually, storytelling with data, and I talked about the different types of reports, right? So, Colin, this is Bob and Affleck actually has an example where she talks about the different types of reporting, and it... She has the analogy of oysters and pearls. So, you could build a kind of exploratory report in dashboard, which is kind of for you, is your scratch pad, if you like. And that's where you're going hunting for pearls, you're hunting for insights, you're looking at what's important. You've then got the kind of... second example, which is kind of where you're building curated kind of insights, right? So, an exploratory journey, right? So, you're building kind of... It's like an essay style, kind of, right? So, you write in arguments, and an argument is basically a statement with context, because just to say, we had $2 million revenue last year, okay? Is that good or bad? So, what makes that useful is the context, how do you compare it to prior year? How do you compare it to budget? Or something like that? Or, you know, it's argument of context, and you're structuring it with this context to be able to put an end, and you've visualized it with this context, and that's really, really key. And then the third time, and that's a vast analogy, just to... You pointed people to oysters that contain pearls, right? So, that's the idea of storytelling now. You basically say, look, you know, I've done the hard work of figuring out kind of where things are interesting. You know, go look at this oyster, exactly. Tens this plug. Go look at this oyster, because that takes this plug. And, you know, you've got this breakdown, this visualization. And here's the context for you to evaluate whether that's good or bad. That's the key part of storytelling. And your third one is basically executive reporting where you're just giving them the numbers. And they trust that everything behind that is right. They're not beginning to go digging and do drill downs, and, you know, do all of this power be our exploratory work? They might ask Jenny, "Are there some questions now?" But, you know, they kind of trust that those numbers are fine, and that's basically the analogy of giving the pearls, technically. So, they're going into guaranteed oysters, or telling people, "Well, are you going to just give them the pearls?" So, I think that's the idea of kind of storytelling. And then I actually, at the end of that talk, I really talked about AI. And I said, "Well, AI is the best storyteller we've ever come across." Right? But, it gives you the same generic, plausible sounds and answers to everyone. Right? Now, actually, what you need is, is personalize intelligence. What you need is to know, you know, in addition, it would be great if AI actually learnt really about what you needed, what you want. And it tries to do that today by looking at all your chat history, and looking at everything you've uploaded, and looking at all of this, and storing it in a storage layer. But, actually, it's controlling how to use that context. So, this is where you have things like system instructions you can put in, and they help, but this is where your context is not just your organizational context, and it's not even just your team's context is your context. How do you work? What are you interested in? What's the business questions you want answered? So, this is where you get persona-driven insights, and then at the end you can tell the right stories for that, because, as we said at the beginning, where you really need to start is understanding those specific personas, those specific insights, and then AI can produce the right semantic models, the right reports, to tell the right stories for the right data. So, the changes, I think, before we have the dashboard, to tell me the story, the revenue decreased by 7%, and then I have to figure out why it's decreased, and I have to click through my, I don't know, throw down, and so on. And now, AI can say, okay, revenue decreased, 7% primarily, because the customer during increased the Germany, and then it's persona-driven. So, I say, the CEO, it says, because you have a Merco, and he is really bad to manage, and to me, it says, "Werco, you have to go and do this, this, this is the..." But this is what comes back to context, right? So, again, it's the context of the insights you need and what matters and how to interpret it. I give you an example, and I show this example again in SQL bits as well. So, say, "Costing Conversier", right? It's a ratio, example, and it has, so it's your operating cost divided by your revenue. Yeah. And, you know, we can calculate that quite easily from financial statements and draw balance. And we said, "Okay, so you've got a cost in Conversier." And now, how do we visualize that? How do you tell the story about cost in Conversier? So, you can show it broken down by division, and I showed you the bar charts, right? And I said, "Costing Conversier" was highest in A and B, and then division C and D had lower cost in Conversier's, and cost in Conversier's high as B bad, obviously. So, I said, "Okay, so what does the CFO do here?" Well, they go and call up their heads of division A and B and go, "Well, what the hell are you doing? Why is your cost in Conversier? Don't say much, right? Why are you spending so much when the benefit of the revenue here?" But I said, actually, let's now, and because the challenge overall is the cost in Conversier is high, right? This year. So then I said, "Actually, let's turn that bar chart into a scatter plot." So, now let's put cost in Conversier on Ralaxes and revenue on the other axes, because then you saw A and B actually, once they had a high cost in Conversier, they were actually relatively small units, divisions. So, actually, C and D had slightly lower cost in Conversier's, but higher revenues. Therefore, they're the ones that contributed probably more to the cost in Conversier being high overall, because revenue, because that's your denominator, right? So actually, maybe we don't need to call that A and B. Maybe we need to call that C and D, because they're the ones that, you know, a small change in their cost in Conversier is going to make a massive difference because of the revenue they contribute to it at all. So, actually, like, it's that context, which you need to know. Now, you need to know that context in order to know how to tell the right story, and to know that actually a bar chart by itself is a given enough context, maybe a scatter block with the revenue will work better. So that's your kind of visualization head of, how do I establish the context? Now, Gen A, I, yeah, this is where the intelligence comes in. I don't know. Maybe we need to give us a data and understand if it does do that kind of analysis for you, but at least given the right skills and instructions, it will. So, if you give it the kind of instructions to say, look, when you analyse these kind of ratios, you know, make sure you analyse the context of them and you can be quite genuine about this. You can say these kind of principles that we're talking about now, and you can bake some of those principles into your skills to interpret Power BI semantic models and faults. And then it will give you those kind of insights automatically, and it will tell you why it will tell you you'll drill into detail itself. You don't need to go and right click on a bar chart and drill down into detail to see what's going on and try and explore. It will do all of that for you. But by default, it will just probably say to you, oh, the cost of the conversation is this right? And yes, and then it will try and find some other reasons and those reasons might not be important and it might not be relevant. So you need to kind of guide it as to where personalised intelligence comes in. You need to guide it into what to look for. What's important. Like that example, we were just talking about actually think about what's important here. You're, you're denominated important. You're revenue important. I don't just want to focus on ones with the highest metrics. I want to focus on ones that are contributing most to my bottom line, which might actually just be the biggest divisions. So we've all become to the better static. We have, we have to do a little bit transition. So we'll be with some groups. So the people are, oh, we have, as I say, the co-pilot, AI, Tobic, is a little bit, yeah, last year. Now we're living in the agent. Agent, well, yeah, how would do is, where, what was the impact in Power BI and, and fabric when agents come? Well, it's, I think, I think Power BI and Fabric are a core grounding for agents, right? So it's a knowledge source with skills on how to interpret a data and logic, right? So that's, that's your core thing. So agents, you know, if, if we didn't have Power BI and fabric and we just had agents, or we didn't have a data platform and, you know, BI, I don't know, I think it's pretty, it's pretty risky already with how that and grand it is, it's even worse, right? So that's, that's one thing. But I think let's, let's come back a bit as well to what we mean by agents because actually there's different types of agents. So Copilot Studio, when up until fairly recently, I think maybe a few weeks or maybe a month or two, I'm not sure, Copilot Studio is essentially just Power Virtual Agents, right? So you built an agent in Copilot Studio, but all that agent was, was Topics, and the Topic Hand Knowledge Sources aligns to it. And then you, a chatbot. So you asked the chatbot an agent question, it tries to fit it into one of the topics and then use the knowledge sources, or if it can't answer with that, it just falls back to a full-back topic. Okay? That's all right. It was, it was the bad, but then people were saying, "Hang on a second, I'm not comparing this to Claude, I'm not Claude and Excel, where it's going through this chain of what we's in need and it's got, you know, it's calculated things that's producing reports, that's producing these things." So that is such a different level to a chatbot with Topics and Knowledge. So actually there are different types of agents. Now, Copilot Studio with a GitHub Copilot harness has now got this, and it's going to have this second type of agent standard. Right? Then you experience a Copilot Studio. I did a trade-in course on it last week with a Moxot product team in London. And I mean, I have looked at Copilot Studio for a few months, at least, but I was blown away by what the GitHub Copilot harness could do. It's got, you know, an agent sandbox. It's got, now you can add tools and skills and, you know, MCP connectors and all of this stuff into a single agent. And then you can combine them to get into the workloads and all of this. So I think when you say how agents have worked, I mean, that level of agents is really powerful. And yes, you have to be more careful with those types of agents because it's far more dangerous. Right? And this is why I think Microsoft actually probably did a safe thing, you know, a rich legal copilot where they kind of restricted copilot to be within these kind of topic, knowledge-based domains and, you know, it was very limited in terms of what you can do, but it was also less chance of hallucination, right? If you've got very defined topics and knowledge sources for those, your chance of hallucination are a lot lower. If you've got Claude and agents that, you know, all of this stuff now going off and thinking for themselves and going for a chain of thoughts and saying, hey, I can look at this or maybe it's to do with this or maybe it's to do with that. Or actually, here's analysis out there, you didn't know since then, but here you go. There's lots of fun. This is this. Now, suddenly, your risk level goes out in terms of your risk of hallucination goes up massively and your token cost goes up because it's charging you for all of this fault process as well, by the way, absolutely. But, you know, your everything's, everything's not becoming riskier and more expensive than agents. So, now that makes it even more important than it was before for your agents to be grounded in solid data foundation and solid logic foundation with semantic models of like houses and things like that. And so then it's not hallucinating. It's it's the and you need skills and instructions to tell it. Don't invent numbers. Don't go looking in places. I haven't told you to go and look. Don't go come up with bits of analysis that I haven't asked you to do. Like you need to be and the as the models get more and more intelligent, you need to be strict and strict and stricter is the instruction and tell it to use or take to foundation, your core logic foundation, your semantic models and like. And then you need to be a little bit more intelligent and you need to be a little bit more intelligent and you need to be a little bit more intelligent and you need to be a little bit more intelligent is where Ella Lemous, a judge comes in and you know critics and and again this stuff is built into copilot studio now. So, this is where you know, we're not having to reinvent the wheel and actually it's much more aligned to how we'd manage employees rather than this chatbot agent which we had before which was just topics. Yep, Now we come to the med path. I think everyone who would have like need these crew agents, man yeah, And zwar wants to yeah, nobody wants the governance meeting actually. The governance meeting. No, no, no, no. Yeah, how did we make governance great again? I do make cover this great again. Oh, maybe that needs to be a cap. Make governance great again. M-G-G-A. That should be your new tackle. How do they come from this great again? Go for this. So, you know what I was talking about is organizational brain. I've got six agents who are the core kind of team that sit on top of this brain as obviously you can define your own agents skills as well. And one of those is a governance manager. And I've got a little persona based avatar videos for each of these agents personas that talk about their tools, their skills, what they do. The first one with Gabrielle is the governance manager. She's like, I know governance has a reputation of where good ideas go to die, but let me let me reframe that for you, right? And it talks about governance in terms of kind of actually not being that a nabler, being that a nabler to be able to trust AI. So I think governance is great again. And I think it's been brought right to the top of the agenda for almost every company because as soon as these you started to use agents in AI, especially as become more intelligent, and it's no longer restricted to this kind of topic based chatbot, how do we make sure that we're getting reliable results? And that's governance in the nutshell. And governance is, you know, and also security, right? How do we make sure agents, not just employees, but now also agents, are accessing the right data using the right tools with the right skills, with the right, you know, access permissions with the right security policies or the right infrastructure. And actually, if you build all of that into a context layer, then suddenly whenever agents do and you force it to use this governance policies, like you tell it where to use it and you force it to use it with contracts, like again, like humans, right? We have employment contracts. And my employment contract doesn't say I need to go and use this skill in here, or it doesn't say, I need to go and make sure I apply this governance policy because the government contracts are not attended for that. But agent employment contracts are intended for that. So we write employment contracts for our agents that tell it, we need to use this governance policies. And then it will, that's how governance is automatically enforced. So for me, governance is security. How do we make governance great again? No, single word I'd use for that is automation, right? I think governance is be one of those things that's being too important to just write into 100 page documents that no one reads. Governance has been too important and security has been too important to have people manually apply road level security into their power BI to mantifoddles with. It's been too important to have, you know, to, to try to stick to these all these manual and things and too important to let it live in people's heads as the majority of its statusors. So how do we make governance great again? We put it into our organization or brain. We make it a foundation if our organization or brain and we put our agents on top of that and that in an automated way, right? Using some of that of Andrew Acky. So that's how we make governance great again. Okay, good. And will you say who should own own the governance or is it different from the semantic model to KPI definition or should we have one over? So I think the way I kind of look at it is that this organizational brain is obviously your source of truth, your data for your logic, for your tools, the skills, for your governance. Right. Now who owns that? Well, I think in terms of populating that brain, I think every business user who runs processes, which is all us, needs to convert their processes into a custom skill in that brain that has the right governance supply that has the right tools, skills, data logic. Right. Now I think the operating model that I see that really operating in is is people could take an exosperitiate and use my tool or other kind of processes to convert that into a custom skill that could be part of this brain and then managed by a skills manager in that brain. But I think initially that should probably be an unsertified level. Right. And maybe not part of the real brain that everyone uses, right? But maybe part of just a local kind of, okay, like whatever, right, that you run in yourself or one for your team. And then it should go through some kind of certification process like we do with some antiponels like we do with these other things, whether it's okay. Yeah. This makes sense. Let's make this why accessible to the wider organization. And that's where it then becomes part of the brain. And that's where you make sure that everything because governance is applied properly and all these things will apply properly. At the moment, this is still very hard to do. But I'm going to be, be realistic here, right? Even if you have found your IQ or something, it's not a magic bullet to this. And most of those problems are not technology problems. They're people who process problems. So how do we get the operating model right around this? And that's the bigger problem we've got. So, yeah, the operating morning is also an interesting topic. I think, yeah, we have for the sea level, we have the the bus word, the center of excellence. So, they might, but what does a good operation model for fabric and AI looks like from your perspective? So, center of excellence is absolutely key for that. And when I talk about center of excellence, I think it's one of those buzzwords, if you like, that people have different ideas about what it means. But for me, a center of excellence is just the enablements team, your core enablements team. They're the team that allow you to do self service. They're a team that allow you to do this. So, the fabric team are going to be responsible essentially for building all your hub workspaces. So, this is a hub and spoke operating model essentially, right? So, you've got your certified data sets, data and logic in certain workspaces that are all certified and those are all automated. Everything goes through a kind of central function for that. And it's probably, you know, maybe code first and that or even, or still AI, but you know, maybe, maybe more code first and dev-obsolated, that's your kind of certified layer, your hub layer. And then spokes, which is the kind of business user teams, kind of connect into the hubs and build their own versions of logic and data tools and asset gets wants to be used more widely and get certified back into the hub. Okay, for the process. So, that kind of operating model, I think, works very well for the BI, that I would speak, I would speak model. And kind of the sets of excellence is responsible for making sure there's a clopier owners for each hub workspace and a data set and for each spokes as well and making spokes that maybe help you then move them from there to there or making sure they could build stuff in the right way and use the hub data sets in the right way. So, I think that's the right kind of model of like for BI and for AI, it becomes even more important to kind of do some of this. So, I'm saying, I think with the organizational grain, now your assets are not just data assets or, you know, spreadsheets, your assets and our skills and your assets are now, you know, all of those things that look for part of your organizational brain, they're all assets, your governance policies, your assets, your tools, your assets. So, how do we combine these assets together into a catalog that agents consider top of and humans as well, right? Obviously, there's nothing stopping the human going, querying that organizational brain with van dreiky, right? It's Q and A things as well. So, I think that's the kind of model that works well. So, this is a people will build them in skills. I showed you in a context a couple of months ago, I showed how to build a custom skill for an Excel process in Excel using Cloud for Excel. So, just lightly implementation plan, great looks good, turn this into a skill, open up a new set of data, run the skill, create your cow, subgeneral sheets like a VBA macro wood, but you know, now it's driven from natural language. So, now, how do we manage those assets? That's the key and that's the off-ageable that's needed right then. Yeah, then, oh, yeah, we're running all the time. You have to go to the building. Oh, good. I'm a real story, but I have to wrap it up, I make it short. So, centralised or federated governance? A mixture, right? So, Zen, I think the hop and spoke is a mix of centralised and federated. One enterprise, I think, model or then you domain models. So, well, again, a mix, right? Like, I think you need the semantic models to represent your business logic. Now, where that, and maybe a different layers, right? So, you know, maybe at the silver layer, things are more centralised and more kind of enterprise level, and then maybe your kind of gold layers are some of the more business layers. And then again, maybe platinum layers, we need to start to think about it. So, maybe when standardising a bit more the gold, then maybe people could build their own things on platinum, and that's where agents can kind of then work and work across as well. And I think that's really interesting for the rest of the world. What's the best food in England? The best food in England. You know, the most, do you know, the most popular food in England? You'll take a guess at what it is. The most popular dish. Yeah, a foodie chip? No, haggers? Maybe a scones. I don't know. Chicken teacup, thunder? Huh? Chicken teacup, a bissala. Oh, okay, yeah, really British. Obviously the British style or the British version of it. So, I mean, I don't have asked the best food in England, but it's the most popular dish apparently. But yeah, maybe fish and chips if there's something coming to the record, they want to try something maybe fish and chips. Yeah, and you've come to London to give me a shout because I'm a, there's a nice relationship, please, maybe I'll give it to the same house in London for over 30 years. And probably, yeah, who shall I invite next and what, see what person should I ask? I think we definitely dive deeper into this Copilot studio agents and the copilot harness and stuff like that because I just, I saw saying I wasn't really aware of it until last week and it's just such a huge area and I think, how do we integrate this? How do we build the right context layers with the, with the copilot harness and how do we make it safe? Because again, great, you've got the capability that these agents are going to take you for a ride if you're not, if you haven't got the right guard 12, the right skills, the right tools. Yeah, then I say, which is thank you for for being here in my podcast or it's more of the new plan. Yeah, we have a really cool overview, I think, now from, yeah, from the start to the end. And I hope we have another session with G-Drive and the other topics. And yeah, then thank you for being here and, yeah, have a good meeting. No, you too, okay, thanks very much. Exist? [BLANK_AUDIO]