Project Due Diligence - Using a Military Intelligence Approach & Critical Success Outcomes

In this episode of Risk! Engineers Talk Governance, due diligence engineers Richard Robinson and Gaye Francis discuss project governance due diligence. 

They highlight their unique approach that has incorporated a military intelligence approach and the importance of focusing on critical success outcomes and vulnerabilities rather than threats. The also emphasise the need for early involvement of key stakeholders. They also discuss the airport rail link project in Victoria and the Commonwealth Games in Victoria, and touch on the limitations of Monte Carlo simulations in identifying high consequence, low likelihood events. 

For further information on Richard & Gaye's work, including their Project Due Dilligence booklet, head to www.r2a.com.au

Transcript

Megan (Producer) (00:03):

Welcome to the latest episode of Risk! Engineers Talk Governance. In this episode, due diligence engineers Richard Robinson and Gaye Francis talk about project governance due diligence. They discuss how they have applied a military approach to their work and also how they focus on critical success outcomes.

(00:26):

Enjoy the episode. Please give us a rating if you enjoy our work, and also ensure that you subscribe on your favorite platform.

Gaye Francis (00:36):

Hi Richard. Welcome to another podcast session.

Richard Robinson (00:38):

We're here again, Gaye.

Gaye Francis (00:40):

We are. Today we're going to talk about project due diligence, which is, I guess, close to our heart. And I attended a recent conference and presented a paper at Risk 2023, which was in Brisbane in September, 2023. And the theme of the conference was Risk Engineering for Resilient 2030. I think one of the key takeaways for me from the conference was that it was a really interesting conference and there were lots of papers given and everything. But there was a real focus, especially from the keynote speakers, around top-down criticality and vulnerability. And the people that were speaking as those keynotes were probably people that were putting projects together, so bigger projects. For example, the cladding replacement in Melbourne, that was one of the guys that spoke. But when you went and talked to people and the participants in the conference, one of the things that was still happening was those rare showstoppers were still bringing projects undone, those things that were critical and rare.

(01:51):

And my presentation was all around project governance and the little booklet that we've got out, "How to demonstrate due diligence - A practical guide for directors". And it really is that top down, focusing on the credible critical things, project showstoppers.

Richard Robinson (02:08):

High consequence, low likelihood.

Gaye Francis (02:11):

Yeah. And we've used this process for very many years on projects that have ranged from - am I going to get this right!? - $500 million to $2.5 billion?

Richard Robinson (02:27):

I think we started about $10 million up to $2.5- $3 billion.

Gaye Francis (02:30):

There you go.

Richard Robinson (02:30):

Australian dollars. I think Marina Coastal Expressway was about $2.5 billion US, as I recall.

(02:38):

And I guess if we just talk through today about what makes our process that little bit different from the traditional processes that we've seen, and we've said in many of our podcasts that there's a whole lot of tools and techniques out there, and they provide you with different insights into the risk issues that you do and some of the safety stuff that we - we've focused on safety in some of these podcasts - but the tools and techniques really do apply to projects as well, or can apply to projects as well.

(03:06):

Yep. Mostly.

Gaye Francis (03:10):

I guess, Richard, do you want to outline the process?

Richard Robinson (03:13):

Well, so far as we're concerned, it's really a military intelligence process, and it was originally developed to deal with people going on missions. So if you get it wrong, you're all going to die. So it's actually worth getting it right. What you don't do is take on a course of action where there's a credible critical showstopper that can come and bite you, even if it's rare, you just don't do it. We just adapted that.

(03:32):

Now, part of the reason why I adapted, I got it off Bob Browning who used to work with us. Unfortunately he is no longer with us. And he used to be the head of protected services for Asia. He said he got it off MI6 because he was working there for three years with the English for a while. And when he came back to Australia, I was working with him, he just explained how it works and what it was, and we just kind of adapted it.

(03:55):

But basically it sort of says, well, the simplest way to describe is to compare it to way most people do project risk management now. And that is what they do is they get people who have suffered pain in similar projects together in a room and they try to express their pain, usually by adapting a risk register from the previous nearest project they've been able to find. And you can normally tell that's the case because somebody has failed to do a global search in the spreadsheet and the name of the price project still buried in there. And then they all pour the pain they've previously suffered on this similar project and augment this risk register.

Gaye Francis (04:31):

So we're talking about hundreds of items in a risk register, which is just unmanageable.

Richard Robinson (04:36):

Yeah. And they do this usually around the tender stage. And as the project goes through, every now and then they drag this thing out and try to make sure they haven't overlooked something. And by the time they get to practical completion, they say, well, this is the end of the risk management process of practical completion. And then they hand it over to the poor sod who's got to live with the result who then has to spend a fair amount of time fixing all up and making sure that it really does achieve the outcomes that everybody intended.

(05:00):

The other way to do it, which is the way we've always done it, and this is what the military intelligence people do, is you actually walk forward in time to the end of the project when it's in and working and it's kicking all the goals you really want it to kick. We call them critical success outcomes. And you get the senior decision makers to explain what those essential critical successful outcomes are. And then having established what this happy place is when it's in and functioning, not just practical completion, it's actually achieving all the things it's supposed to do when it's actually working. This is a happy place, by the way. This is where promotions are possible. Excellence awards abound. And even bonuses can sometimes be represented. But this future happy place. And then you step backwards from that happy place and say, what are the threats to those outcomes? Those threats create vulnerabilities to those critical success outcomes. They are the vulnerabilities. And if you eliminate all those vulnerabilities, then these will be the high consequence, low likelihood ones as a rule, then you'll always wind up at this happy place on time, on budget, to spec, and it's clean and simple.

Gaye Francis (06:06):

I think you said a couple of important things there, and I just want to touch on those. And that was really about that you're focusing on the vulnerabilities. We've seen a lot of projects focus on threats that don't necessarily affect a critical success outcome.

Richard Robinson (06:21):

Correct. And that was one of Bob's special points (that) he kept hammering.

Gaye Francis (06:24):

So the key is to focus on the vulnerabilities rather than the threats per se, because if it doesn't affect one of your critical success outcomes, you don't want to spend resources and time on that particular threat. So you're focusing on the vulnerabilities and they are the ones, the threats that affect your critical success outcomes.

(06:43):

The second thing is about stakeholders and making sure the right stakeholders. Now we've seen large projects and there are a number of interested parties that are very vocal.

Richard Robinson (06:55):

And have opinions

Gaye Francis (06:56):

And have opinions that are often very loud, but they are not necessarily the key stakeholders for the project. So it's really important to make sure that you've identified the key stakeholders.

Richard Robinson (07:09):

And what you mean by that are those who live or die by the success or otherwise of this project.

Gaye Francis (07:14):

And I think sometimes that's the difficulty around this project governance and project due diligence stuff is that there's a number of stages in a project. And unless you can get all of the key stakeholders involved in the project very, very early on, then there's often a mismatch of what the critical success outcomes are for the project.

Richard Robinson (07:35):

It's like the tram project we did the, or Gaye did the project due diligence for the procurement of the Class E trams, the first trams for the Class E trams in Melbourne. And the real problem we had was getting Yara Trams on board as I recall.

Gaye Francis (07:51):

At the design stage.

Richard Robinson (07:52):

At the design stage because until you've actually chosen who the tender was and all that sort of stuff. But obviously the people who have to sign off with the safety cases is the tram operator, and that case, it's Yara Trams. So whilst there's a tendency for them not to be involved at the start, if they're the people who have to live with the final result and actually get the credited safety case up, if they're not involved in the initial design stages, you get something which is quite peculiar.

Gaye Francis (08:18):

So it's almost critical that they're involved. And when we've seen this with other projects as well, that you really do need to get the ultimate operators and maintainers on board very early on because it's a lot easier to make changes at a design phase or at a design stage than it is to be delivered with a piece of infrastructure or vehicle and then it has to be modified once it's had been built. A lot cheaper as well.

Richard Robinson (08:48):

Well also your project will blow out. If you're talking about a five-year project and you only discover at the end of year four that you've got a basic design flaw, you won't fix that. You are over time and over budget no matter what happens.

Gaye Francis (09:00):

And you are looking for work-arounds to deal with that particular issue.

Richard Robinson (09:03):

And we've seen that happen a fair bit. We shouldn't mention any rail projects, which we're familiar, for example.

Gaye Francis (09:09):

One of the interesting other things that I guess we've been, and there's been a couple of high profile projects in the news lately, and one of them is the airport rail link for Victoria out to Melbourne Airport and the project due diligence around that. And we understand that the federal government's just committed to providing half the funding for that particular project and for it to go ahead. And now there's some discussions with the airport around what that project looks like. Apparently they want all underground, and the offer on the table is above ground solution. But working towards that, you need all of the stakeholders involved. And look, don't get us wrong, we've worked on projects and they have gone relatively smoothly. But it doesn't mean that doing this process doesn't mean that issues won't arise. <Correct.> What it does typically mean is if you correctly identify your critical success outcomes in your threats and your key vulnerabilities, credible critical vulnerabilities, those project showstoppers, typically you don't get surprises.

Richard Robinson (10:19):

That's right. You know there's a problem and you know how hard it is and you can make a decision about it, but I don't think we've ever had a surprise with any major project we've dealt with.

Gaye Francis (10:27):

No. We've had to refine some controls before and tighten up those when they did manifest itself. And often that's in terms of quality assurance systems, but it's relatively straightforward. As Richard said: Make sure you got your key stakeholders in the room. Make sure critical success outcomes are clearly articulated. That's where we've also seen projects come undone, where the ultimate owner has been unable to clearly articulate what they're trying...

Richard Robinson (10:58):

You're better off not proceeding with the project if the ultimate owner can't articulate what it is they're actually trying to achieve.

Gaye Francis (11:04):

And trying to second guess that is just impossible. You setting yourself up for a fall.

(11:09):

The other question we've been asked is what are our thoughts around the Commonwealth Games in Victoria saying that they're not going to proceed with the Commonwealth Games? Our understanding was that there was some promised funding from the federal government and that was withdrawn.

Richard Robinson (11:24):

About half, I believe.

Gaye Francis (11:26):

So from a governance viewpoint, if half your funding gets withdrawn, you can't proceed with your project.

Richard Robinson (11:34):

You won't achieve your critical success outcomes. It's just one of those things. It's just simply don't have the resources to make it so.

Gaye Francis (11:40):

And we've seen a number of projects have to be withdrawn and not proceed because there are these potential showstoppers that can't be addressed.

Richard Robinson (11:48):

But the other point is just to make a remark about Monte Carlo simulations, because we see a lot of this and people keep saying, Monte Carlo simulations has solved all your problems. No. We're not against Monte Carlo simulation because it'll tell you the most likely outcomes and that's worth knowing, we're not any doubt about that at all. But what Monte Carlo simulation simply can't do is pick the high consequence low likelihood event, because generally a rare combination of circumstances has to get together. The way I normally explain it, if you've got three probability distributions, you've got to add up from Monte Carlo simulation, the only way to do that is to do a number of trials to get the final probability distribution, and you might only do 15 or 20,000 trials if you're lucky. What happens to the low likelihood long tails of each of those distributions? Well, the answer is they just go missing. So if you had a one in a thousand chance for each of the three probability distributions in order to find the one in a billion chance where those three of those combined to cause a disaster, you'd have to do, well, at least a billion trials or to be statistically significant 10 billion trials. And nobody does that. So that means the long tail low likelihood events that can conspire together are just ignored and thrown away. Our process doesn't do that.

Gaye Francis (12:57):

I think that was my takeaway from the conference was that there was a real want to focus on top-down criticality and vulnerability. And that was from the owner's viewpoint to deliver, because there's a lot of big projects going on in Australia at the moment.

Richard Robinson (13:13):

And they're still going over budget badly.

Gaye Francis (13:16):

But the tools and techniques that are being currently used by organisations to identify the key risk issues for projects is still very much bottom up and lower level, the more likely lower consequence ones. So identify your potential project showstoppers first, and then the other stuff is done secondary. But you do need to do both processes.

Richard Robinson (13:41):

That's correct. And you'll find that all the senior decision makers, once you hit a potential showstopper, they will not let the project proceed. They will not let the tender be let until you have shown why that will not be the case, why it can't happen.

Gaye Francis (13:55):

So this is sort of a passion subject for me in particular, but I hope you enjoyed the podcast today and that you can join us next time.