AI liability for MSPs: missing policies create audit risk — Ignacio Lopez, Work-Smart.ai

[00:00:01] When an AI system hands back a bad answer, what happens next? How's that audited? What questions need to be asked? Ignacio Lopez is a former SOX auditor, now AI developer and a former Kaseya product manager who understands this space. He wants to help out his clients with AI and we talk about what that means for them and for MSPs. This is the Business of Tech.

[00:00:31] Ignacio Lopez, you are the fractional head of AI at WorkSmart.ai. Welcome to the Business of Tech. Thank you for having me. I'm excited to talk to you. You build AI inside places like family offices, law firms, construction groups, and we know nearly all of them run working with an MSP. So when you walk in, what is the MSP already put in place for AI?

[00:00:59] Ignacio Lopez, I'm going to be honest, not much. So my work starts with an assessment to understand if they are ready to implement AI. And then I work in conjunction with the MSP to deploy, configure, and run the AI tool they decide to roll out. Okay. Now is the MSP usually in the room for the engagement? Do they find out later? Like, how does that work? So to be honest, they are not engaged from the beginning.

[00:01:28] And I feel that's something that it needs to change. The MSP should be involved from the beginning because we are talking about companies that are regulated, that they need to present evidence like any other company. So MSP should be involved from the beginning. But in my experience, I'm working with mainly wealth management firms, construction firms. They are not involved at the beginning. I have to involve them. I have to ask them to do, yeah, some things.

[00:01:57] But I think they shouldn't be involved in the definition plan, like the step zero. So how many of the firms had anything in writing about approving AI settings or configuration? Like how many policies? Like what's the landscape look like when you walk in? Ah, pretty terrible, I will say. So I will say like only 20% of my current engagement had an AI policy written. And not only written, because one thing is to have a document written,

[00:02:25] but then you need to communicate it to your company, to anyone who is using AI. And not so long ago, I did a webinar for family offices. There were 53 different family offices. And 80% stated, I did a poll, that they were using AI without a written AI policy. So that tells you how the market is doing in terms of AI and governance. It's pretty bad out there right now. It is.

[00:02:53] So one of the things, I also know you've got a background in some of the insurance side. You started your career on the auditor side of the table. And one of the things that's really interesting, like since January, insurers can attach generative AI exclusions, things like CG 4047 and renewal. Now you've got experience around this. When a claim or an audit lands, like what evidence does the auditor or insurer like accept? And how does it compare with the logs most MSPs are keeping right now?

[00:03:22] So just to give you a bit about my background, I was an editor, a SOX editor for KPMG. And they, I work as an information risk manager in JP Morgan. So we are talking about one of the biggest bank in the world. So it was highly regulated. So I learned how, you know, what an editor wants to see and how to produce the right evidence. So nowadays it's not just about having a log. Like that's just a small piece.

[00:03:49] What an editor wants to see is the evidence that there's a process in place that was discussed and approved, and that you have the right controls in place, and that you can produce evidence that shows that you have the process, that you have the control. And it was done and it's dated because what I see on a lot of my clients, they have some logs, but they don't have it dated. And an editor wants to see what happened in March, what happened in April.

[00:04:18] So you can produce evidence or logs from March in September. They need to be able to trace back from the logs, the controls you have in place to enforce what you approve, like your policy. They want to see the full cycle of the process. Well, that makes sense. Now, the other thing I want to talk a little bit about is, so in SOX terms, right, you test whether a control was designed right. And whether it actually operated. Yeah.

[00:04:46] Well, let's apply that to an AI agent. Like can an MSP today reconstruct who approved the agent's access and what it actually did on a given day? So, that's a really good question. And yes, it depends on the tool. Well, obviously, I do a lot of cloud enterprise rollouts and you as an MSP, you can get the logs, the access, everything that an agent does.

[00:05:16] But it's like, it's like a pretty complex log. So you have to be mindful of how you like digest it. But now the tools, they all have the right information for the MSP to present to their clients or even an editor. But what you were saying, they want to prove that the controls are in place and are working well. That's why I'm always very, how can I say it, pushy on having an AI policy.

[00:05:45] It's not just having the document signed. It should be discussed. Like all my clients have like, since it's web management, like boards, like investment committees. So they need to have a presentation where they discuss the policy. Then they need to have a minute of the meeting where they discuss it. And then they need to have the policy sign. And then they have, they need the MSP to enforce the controls.

[00:06:09] And when I mean enforce the controls in terms of AI, we are talking about ensuring the right controls are in the brain of the AI. For example, Claude, whenever you send a problem, we'll be right back after this message. So that's where you need to enforce all the control. Let's say a company decided that you can say, you can send bank statements to AI. So that's something that the MSP needs to configure.

[00:06:37] It needs to configure on the brain that if any employee sends a bank statement, it needs to say like, no, you can send it. And it will be loved. Who sent that document? When? And it shows that the control work. So I feel the MSP needs to be involved because all the settings are done by the MSP. It's like any other tool. Like you, you control the access, who has access. You need to invite the users. You need to check the logs.

[00:07:06] So it's kind of the same. We'll be right back after this message. Here's what I'm hearing from MSPs on backup. They want control. Control over storage. Control over costs. Control over what happens when something breaks. Comet Backup gives you that. Bring your own storage. White label it for your clients and keep margins where they belong. With you. It's why Comet Backup keeps showing up when MSPs ask each other what actually works.

[00:07:36] See for yourself at cometbackup.com. And we're back. Gotcha. Now, what's like the minimum record that you think would satisfy an auditor? Like what has to be there? So, an AI policy. Evidence that it was discussed on the top level of the company. Assigned policy. Then, it's not just about the policy. Like now the AI tools can connect to other tools.

[00:08:07] If you have connections, because let's say Claude, you connect your Gmail, your Microsoft 365. Now it has access to everything that you have access to. So, that needs to be documented. So, the access that the AI tool has. Incident. What happens if something wrong happens? You need to have an incident management process. Who is responsible? How is the rollback?

[00:08:31] I feel like the minimum is an AI policy, a definition of the integration connectors it could have, and an incident management policy. Okay. Now, one of the things you've also said that in your clients, nobody has decided who approves the AI settings. So, I kind of want to know like who should? The client? The MSP? Who built it? Like and where does that liability sit when nobody's decided?

[00:08:56] So, to be honest, they hired me to decide to work with them and do the policy. But I feel this is the right opportunity for the MSP. Like the owners, they won't know. Like again, that's why they are hiring an MSP. They don't know about IT. They don't know about access control. So, I feel the MSP should recommend controls settings, data privacy. Everyone is mindful about data privacy.

[00:09:23] Can a new model use my data to train on? No or yes. Depends. But I feel like the MSP should suggest that the settings of the tool. Like again, I think the owner and this is a liability. It's not the MSP who decide. It's the MSP who propose, the client decide and then you enforce and execute.

[00:09:47] But I feel the MSP should help the client to decide what is best for them. Gotcha. Now, I want to talk about some of your own successes. I looked into, you know, one of the case studies you talk about is a fourth generation real estate developer, right? And so, your system regenerates their quarterly investor reports and it checks every figure against two years of prior signed ones. So, walk me through that system and walk me through what happens when the numbers don't match.

[00:10:15] Every quarter, they send a report to their investor telling how their properties are doing. It takes them around, it took them around like 15 days to 45 days after the quarter was finished to send the report. So, a long time. So, I built an engine that uses two paths to validate and something that everyone should be mindful. AI is not suitable for financial statement. As everyone knows, AI hallucinates. It has error. It lies.

[00:10:45] So, when you ask like, hey, create this financial report, it can flip numbers and you will never know. So, that's why I create engines, deterministic, mainly in Python. So, what it does, it checks all the numbers, like it does all the numbers in a deterministic way. So, nothing will be wrong. And then it checks against the past.

[00:11:08] So, to create the engine, I took the last eight quarters from this company and I compared it against what my engine did. So, if it didn't match, it stopped. It didn't create the report. And if it was okay, all the numbers were right, it will proceed and did the report. So, the evidence is that the control is working is when it stopped. Like, how do you know a control is working?

[00:11:36] Well, it stops generating the report because there was something wrong. So, for example, my engine found nine like ref errors. Do you know when you have like a reference error on an Excel? They had nine. On that file that they already sent to an investor, it was signed, reviewed by humans. So, that's what happened when you're working with like huge financial statement and a human is doing, there's always like human error. It's like something common, like typos.

[00:12:06] So, on my engines, what I do is I calculate everything like they have like a model that we define and then I compare a gal's previous quarter to make sure it works. And if it doesn't, if it doesn't match, it stops. Well, one of the things that really interests me is when you kind of pitched this to me, right, it was the idea of a second independent calculation that blocks the output on a mismatch.

[00:12:31] You know, your case study talks specifically about that and it says unprovable numbers get flagged and a person signs off. So, talk to me, if it's a flag, what stops a busy reviewer from just signing anyway? So, no, because it wouldn't work. Like they need like, let's say it flagged a number or something is missing. Typically, they are missing some report or like, yeah, it's some missing data.

[00:13:00] They need to fill it out. It won't let them continue unless they fix it. And again, that's how you prove the control is working. Okay, makes sense. Now, one of the things I'm curious, on a recent episode, I talked about the idea of a false guidance acceptance rate. Like how often a person accepts a confident wrong answer. Have you ever measured that for people signing? We all experienced some problems with our AIs where it tells us lie.

[00:13:29] So, that's why creating controls, not only relying on human judgment, but now with technology, you can be like, for example, on the MSP, having like when you're using an RMM and you check if a script works, like it tells you, yeah, it ran, but did it do what it had to do? Well, how do you know? Or a backup. Yeah, the backup was done. But unless you do a rollout and you restore it, you don't know if it works.

[00:13:56] So, now with engines like our AI, you can check if the control work and try it out, run it again, and see if it was done properly. Now, one of the things you pitch your clients is that they own everything you build. So, the AI stays when you leave and you also build on clawed teams.

[00:14:18] So, I'm curious, when an auditor asks what the system did and why, how much of that answer depends on a vendor's log and a vendor's terms? That's a really good question because now, exactly, most of my client wants everything on their premises. And that's why the MSP work, it's so valuable. Like, I build engines that run in a local machine, in a laptop, in Python, and I need the MSP to deploy it.

[00:14:47] Like, my client, they don't know how to run on a terminal, a Python script. So, I need the MSP help. And that produces all information locally and is storing the client's infrastructure. So, you don't rely on a third-party app to have all the logs. So, everything, like for example, Cloud, they have an API where you can, it's called the compliance MPI,

[00:15:10] and you can extract all the logs, what your team prompt, what files they sent, what answer they got. So, I feel like the MSP needs to take all the logs from third-party platforms and put it on the client's premises. So, you need to have access to all the logs anytime. You can't rely on a third-party. We'll be right back after this message. This episode is supported by Halo.

[00:15:40] There's a moment many MSPs eventually reach. The PSA they started with worked well early on, but as the business grows, workflows get harder to manage, automation becomes complicated, and the systems start shaping how the company operates. Halo PSA is designed for service providers who want more control over how their operations run, from ticketing and service delivery to billing and workflow automation.

[00:16:05] That's one reason Halo PSA often comes up when MSPs start evaluating their next PSA platform. You can learn more at usehalo.com. And, we're back. So, if the model changes underneath them, do you have to do all the back tests to have them run again? So, when there's a new model, yes, things change. That's a reality. Yeah. Yeah.

[00:16:33] I also want to get your take on something. I've argued on my show that the one document an MSP actually controls is the client agreement. That named client acceptance of what the AI is permitted to do and who owns the result when it does something else. Now, you sit on the client side. Would your client sign a document like that? Yes, they will sign it. They want to feel secure that they are doing the right things. But IT is not their expertise.

[00:17:03] So, again, the MSP can assess, suggest when the client approves, enforce, control, and then, for example, the policies need to be reviewed every quarter, like any other, like, private access platform. So, that's something that the MSP should, like, present to the client. Hey, you need to do the review. Is this right? Did anyone leave? What happens when someone leaves the company? Is the off-boarding done correctly?

[00:17:33] Secondly, I feel like with AI, there's a huge opportunity for MSPs to assist, enforce, control. And especially, for example, on my clients, I have to tell the MSP what to configure, what to do. And that's something that they should already know. They should be familiar with Cloud, Gemini, ChatCPT, Grok. There are so many models.

[00:18:00] They need to understand the different subscriptions, the different configurations, so they can do it without anyone having to tell them what to do. I feel they need to be more mature on the AI tools, their plans, the configurations, the risk. And they can package and offer a new service. And I feel like my clients are willing to pay, for sure. They are already paying on an MSP. So, why not adding the AI side?

[00:18:30] That is really important. So, you feel like the right exact person asked this question. If you're doing all this AI build work inside clients that already pay an MSP, why isn't the MSP doing it? What do you think is holding them back? I feel like the MSP are not, I will say, someone proficient in AI. And that's not their job. They are more on the infrastructure side. What I do is I train the team because once you implement AI, you need to change the way you work.

[00:18:58] It's not the same you as an individual working with your Cloud or ChatCPT that an organization working together with Cloud or ChatCPT where they share the memory. So, I feel there are certain areas the MSP should do, but there are some others in terms of the implementation that they need to rely on someone who is AI proficient or understands how to work with AI as an organization.

[00:19:26] But the work of an MSP is fundamental, especially like saving the logs in the right folders with the right timestamp. That's not something that our clients are going to do. It's not something that I am going to do. I don't have access to the tool. I always have to ask the MSP, hey, can you do this? Can you do that? And I send them like an operator guide. I feel they need to know it. I need to tell them, hey, we are installing this. This is the privacy. Do it. Make it happen. All right.

[00:19:55] So, let's give them some practical advice. An MSP has 50 clients and they haven't put anything together around AI. What is the place you think that when they listen to this interview, what is the first place they should start working on?

[00:20:36] You can connect it to Slack, your teams. So, they have all information to propose an AI policy. Once it's approved, you enforce it. And then you give them the peace of mind that they are secure. The controls are there. They are enforced. And there's evidence for if there's ever a problem and any answer comes, they will have all the evidence and everything in the right place. So, it's just a matter of proposing.

[00:21:01] I feel on my side that I sell AI, everyone is open to talk about AI, the risk, the limitations. I feel like in every conversation, even with your friends, your family, everyone talks about AI. So, this is the moment for an MSP to approach their client and say, hey, have you ever thought about implementing AI as an organization? Do you want me to help you with the policy? Not only the policy, the controls, the enforcement. I'm going to do all these settings. Don't worry.

[00:21:31] I have you covered. I'm going to save all the evidence. And it depends on each of the industry. Like there's different regulations that have different requirements, but the MSP knows they are already a customer. So, I wouldn't go and pitch it to someone new. I will use someone in their customer base who they think they are talking about implementing AI and work with them. Yeah. Help them.

[00:21:57] Well, Ignacio, if people are interested in continuing the conversation, finding you, learning more about your work, what's the best way for them to get in touch? Yeah. My email is my first name, Ignacio, I-G-N-A-C-I-O, at work-smart.ai. Or you can go to my website, work-smart.ai. Or you can find me on LinkedIn, Ignacio Lopez. And happy to help. I work with a lot of MSPs. All my clients have an MSP.

[00:22:26] And I work for Kaseya. So, I know you are so happy to be connected with MSPs. Well, Ignacio, I really appreciate you joining me today on the Business of Tech. No, thank you for having me. Produced by Picture This Video. Part of the MSP Radio Network.