To Grok or Not? Using AI for Risk Management & Governance Decisions

In this episode of Risk! Engineers Talk Governance, due diligence engineers Richard Robinson and Gaye Francis how AI in Risk Management?

Richard begins with a deep-dive into how large language models work, and where they fall short. He explains why AI systems are sophisticated inference engines rather than true reasoning machines, and why that distinction matters enormously for high-stakes decision-making and risk management.

The conversation covers the parallels between AI and Monte Carlo simulation (great for likely scenarios, unreliable for rare critical events), the growing wave of fabricated legal citations produced by AI tools, and why the common law system itself mirrors how large language models operate.

Gaye and Richard then bring the discussion back to governance and what does responsible AI use look like for boards and organisations? Who carries liability when a decision is based on AI output? And how do you ensure the sources AI cites are actually real?

They conclude by agreeing that AI is a powerful tool for gathering information faster than ever before, but it demands that essential second layer of human thought, verification, and documented decision-making. 

They reiterate that thinking, and SFAIRP, is hard.

If you’d like us to cover a specific topic or have any feedback we’d love to hear from you. Email admin@r2a.com.au.

For further information on Richard and Gaye’s consulting work with R2A, head to https://www.r2a.com.au, where you’ll also find their booklets (store) and a sign-up for their quarterly newsletter to keep informed of their latest news and events. 

Gaye is also founder of Australian women’s safety workwear company Apto PPE https://www.aptoppe.com.au.

Transcript

Megan (Producer) (00:01):

Welcome to Risk! Engineers Talk Governance. In this episode, due diligence engineers, Richard Robinson and Gaye Francis discuss AI in Risk Management.

(00:12):

We hope you enjoy the chat. If you do, please support our work by giving us a rating and subscribing on your favourite podcast platform. And if you'd like more information on R2A, our newsletter and resources, or have any feedback or topic ideas, please head to the website www.r2a.com.au.

Gaye Francis (00:33):

Good morning, Richard. Welcome to another podcast.

Richard Robinson (00:36):

Hi Gaye. I wonder if it's going to go like last time?

Gaye Francis (00:39):

Let's hope it goes a little bit more smoothly.

Richard Robinson (00:42):

I'm actually going to take a bit of a book on whether you're going to Finland this year at Christmas.

Gaye Francis (00:49):

We'll have to wait and see. Depends how well I can plan. But today's podcast, we're going to talk about AI because we get asked lots and lots of questions on this.

Richard Robinson (00:59):

In the Risk Management space.

Gaye Francis (01:01):

In the risk management space, how it's used, etc. So we're going to title this one to Grok or Not. And Richard's been doing some research, so he's going to give you a bit of a brain dump first and a summary of things. And then we're going to talk about how it's used in the risk management space.

Richard Robinson (01:19):

Well, this originally popped up because a couple of weeks ago I spent the weekend trying to get some of our old software going, in particular, the simulation in supercard for Singapore. Now, that simulation basically put down all the ship tracks and then it basically fired different ships at different speeds doing different things to see if you threw enough ships at it for long enough for it to tell you where the potential collision points were. But it was a bit of crude sort of simulation. And it was obviously under 32 bits. So as Apple moved on, it was Apple software, we couldn't use it.

(01:51):

But anyway, because we're now onto M4 computer chips and these things are so powerful, you can actually get an emulator and run the old interface and the old software and it's faster than it was before. But every time I sort of started doing this, strange things happen.

(02:04):

So I just kept typing into (AI), if this thing happens, what's the problem? What happens next? And obviously what was I thought the Apple AI was coming back with answers. But I commented to you, this thing was coming back with really good answers. I mean, I was asking very complicated questions like I'm running an M4 and I'm trying to use a UTM simulator with System 10 and Supercard and this thing isn't working and it's giving me this message. What's the answer? And it came back three answers. One of these, probably if this is the problem, this is the problem, do this, do that. And it all worked.

(02:35):

It wasn't until I was reading later, they suddenly realised that Apple had done a deal with Google and it adopted Google's main AI, Gemini, to run the core answers from Apple. And that had happened in January this year (2026). And it commenced. Apple are paying a billion dollars a year, I think, to Google for the privilege. Although Google is currently paying Apple $20 billion a year for being able to put Google on Apple instruments.

Gaye Francis (02:58):

I have noticed that there's a lot more Google reminders and sign up for Google and sign in.

Richard Robinson (03:03):

It's getting very annoying.

Gaye Francis (03:04):

For Google and Apple machines.

Richard Robinson (03:05):

It's getting very annoying.

Gaye Francis (03:06):

Yes.

Richard Robinson (03:06):

But that meant I suddenly had to have a look at what was the AI actually doing and where was it coming and what was it doing? Now, I mean, you listen to these podcasts. Anyway, I found this podcast by a fellow called Gary Marcus. He's being interviewed by financial people. And he'd been apparently in the AI business for the last 30 years or something like that. And what he was pointing out is in the last 10 years, all the AIs are basically focused on large language models. Now effectively, what you've got to do to think about that is it's basically scraping the internet. And as the new AI has come up, they scrape more and more. But the logical inference engines from his point of view, they're not actually thinking.

(03:43):

And the example he gave, which I thought was kind of entertaining was when Tesla was trying to get their software going up in the early days, it's an inference engine in the sense that what you do is you throw a whole lot of experiences at this thing. So it looks for cars, bikes, pedestrians, and things like that, and it gives them all the scenarios in which these things could happen. So when it's looking at something, it picks the scenario that's most like that, and then that tells it what to do. It's not thinking through what should be done of itself. And the example it gave was some Tesla guy was at an airport somewhere, and he said, "Well, I'll show you how it works." And he called the Tesla to go somewhere to come to him. And the first thing he did was take out an aircraft because it'd never been told about that you shouldn't run into aircraft and it wasn't in the database so it didn't recognise what it was. And that means it's just an inference engine.

(04:32):

And then he went on to explain that human cognition is a bit like that because the example he sort of gave ... Well, he gave it, but it was my inference from it. But when you drive to work in a car, you drive there and you wouldn't even remember when you stopped at a red light, you just get there. And it's only every now and then when something strange happened. You sort of go, "What the hell is going on? And do I have to think about this? " And he said, there's an extra bit where you're actually thinking about something as opposed to just sort of doing what's in background and that you know. And he said, basically these large language models from his point of view is the background thinking. It's not the high level thinking, the actual reasoning path.

Gaye Francis (05:08):

So it's giving you information that then you're supposed to think about to interpret into your context.

Richard Robinson (05:14):

It's the new and novel experience. If it's a repetitive routine experience, a large language model will get it right because you're scraping the internet for all the circumstances. So if it says the right way or the best way, if you take the top 10 people that have written English grammar or something like that, this is how they'd express what you'd want to say. So that's what it does.

(05:33):

He then sort of points out, I think that in 2024, I think the US courts had 300 cases where people misquoted cases because the lawyers use an AI to bring the case law together, and I think last year it was 600 cases so this is going exponentially. And so there was one particular case, and he was talking about how these bits of information get disconnected, but the AI had sort of said, well, you're obviously looking for this and you want this, so I'll actually construct an artificial case and put it in the reference. <laughs> So he was having quite a good time pointing it out and basically saying is that most of the time the AI is going to get it right just because if that's the way the world's done.

(06:12):

Now, in terms of that simulation, for example, we had a pretty dumb simulation. It was just sort of basically delaying the start, but every object was just every ship object was just moving around at a more less constant speed and so forth. Whereas with the power of these things, we could actually program the AI to actually be each ship and to actually slow this ship down or put helms on a sleep or put the guy watching out the front or the radar.

Gaye Francis (06:37):

Different weather conditions.

Richard Robinson (06:39):

You can manipulate it. But obviously that wouldn't, unless you ran the simulation for several hundred years, you wouldn't find all the bad things that could happen, but you could do that theoretically and obviously the more powerful these things get. And that's what you then went on to say because Apple's just done this deal with Google.

Gaye Francis (06:55):

Does that come down to similar to your Monte Carlo simulations that it's really good for predicting and helping you with the most likely scenarios?

Richard Robinson (07:04):

Correct. Very much so.

Gaye Francis (07:05):

And the things that happen all the time, but what it's going to miss is those credible, critical long tail events potentially, because they're rare and they don't happen very often.

Richard Robinson (07:14):

And they require thought.

Gaye Francis (07:15):

And they require thought.

Richard Robinson (07:17):

Well, the other thing which then struck me, which I hadn't actually thought of, but when I was just reflecting on our various experiences with the legal system, the adversary of legal common law system, actually, to one extent, acts like a large LLM, a large language model, because what happens is all the lawyers and barristers work out all the case law that's relevant to the particular situation they're looking at. And so if that's the case and this case law comes up for this particular circumstance, this is what the outcome should be. But then the judge or the jury, more likely the judge mostly in a common law case, actually acts as the higher level thinking saying, "Well, is that right?" The lawyers have put it all together, they put the case together. This is the facts. This is what the case law suggests is what the large language model saying should be the outcome, right?

(08:04):

Now, that actually could be done because that's what the lawyers are doing when they use a large language model to sort of dream up the case law. I mean, what the lawyers ... There's no reason why you shouldn't ask the AI to search all the case law and tell you which case is relevant. There's nothing wrong with that. The point about it is don't rely on it.

Gaye Francis (08:21):

But what you're saying is it's a tool. And I think Megan, our producer used that as an example today. When she's doing the podcasts and putting the transcripts together, you can use AI to get your basis, but you've got to put in human effort to make sure that the terminology is correct, that what you and I say, the AI has picked up, but also that it's verified what it refers to and it hasn't just put in something that sounds like what we thought we said.

Richard Robinson (08:47):

Yeah. And you've got the same problem marking those postgrad units of Swimburne that we do. I mean, one of academia's main problems in life is, particularly if you're dealing with postgrad students, was this original thought or how much has the AI had a benefit in here?

Gaye Francis (09:03):

And I think part of the stuff with the due diligence process that we're going back to, it's that documentation and the governance processes around how AI is going to be used.

Richard Robinson (09:14):

Well, it's interesting because you might remember, we sponsored the professional and public policy officer from Cambridge and he was pointing out, one of the examples he gave was the impact of computer aided design. And he said that the real advantage of computer aided design, there used to be draftsman who drew things up, but a lot of them lost their jobs, but didn't sort of just disappear. They turned into CAD operators. And I said the real advantage of the CAD operator and the whole system was that is that the engineers could test more ideas faster than having everything done manually. And if that's what AI does, because I can see that you could test a whole lot of ideas faster using AI, but that means you're acting as the second order brain and using the AI to help you get there. But I don't think people have been thinking about it like that.

Gaye Francis (09:58):

And I think that's when we've been and talked to boards because we get asked particularly by boards, what's the use of AI and how are the liabilities associated with if we take something that AI says and do that or implement it, who takes responsibility for that decision? And I think it does come back to that human interface, isn't it? So you're really using AI as a tool to get the information to then make an informed decision.

Richard Robinson (10:27):

So far as you can.

Gaye Francis (10:28):

So far as you can. And you're then asking, all right, how do I document that decision? I think part of the importance of AI will be when you use AI is to document the decision why you've gone with that and where you've got your sources from.

Richard Robinson (10:44):

And verifying the source is real and not just something that AI dreamed up as as an inference machine.

Gaye Francis (10:50):

And as older people have been industry and haven't had the technology available to them, we automatically do that. But I've got a daughter who started high school this year and she had to do some research. And when I asked her where she got that information from, the first thing that pops up on your screen is the AI summary ... And you had to explain to this 12-year-old that, well, you actually have to go and look at where the source came from and then does that source seem real and does it match up with other sources.

Richard Robinson (11:19):

And how many kids do you think are getting that advice?

Gaye Francis (11:22):

I'm not sure. But she got a bit of a lecture on that. And then I got an eye roll to say that was a lot more work than she expected to have to do for her homework.

Richard Robinson (11:31):

Quite. You better reward her with a holiday to Finland. That's what she wants. <laughs>

Gaye Francis (11:36):

But I think that's where the trap with AI is going to come in. We've talked about it as a tool, whereas organisations are looking to use it as the answer. And I don't think AI is there at that point to use it as the answer.

Richard Robinson (11:56):

No, I think that's probably right.

Gaye Francis (11:59):

So from a due diligence perspective, we'd say absolutely continue to use AI, use it as a tool, get the information that you can out of it, but it still requires, as you said, that second level of thought.

Richard Robinson (12:10):

It still requires thinking, which is hard.

Gaye Francis (12:13):

Which is hard. And I think that's all of this stuff. It's really tricky to do. Thinking is hard and demonstrating SFAIRP is hard.

Richard Robinson (12:24):

Yeah. And document in a way, which is pithy, but useful is hard. Actually, it's not that hard, but it does require clear thought.

Gaye Francis (12:31):

Clear thought. Clear and transparent thought. Yeah.

Richard Robinson (12:34):

Yep.

Gaye Francis (12:34):

All right. So we hope you found AI interesting and we look forward to you joining us next time. Thanks, Richard.

Richard Robinson (12:40):

Thanks, Gaye.