FX Campion

[00:00:0] That was not an easy thing to get in the first couple of months of the pandemic. There was concern about the need to gown and glove properly, to have that equipment widely available in small offices as well as in hospitals, and that just didn’t exist for many medical offices, for example.

And so when Monday morning came many medical groups didn’t see patients for some number of days or a week, but often within a week they needed to be in operation and transitioned heavily to telephone and video. And they were using the technology that was in their pocket at the time.

Courtney Clark

[00:36:9] That was Francis X Campion talking about the early days of the pandemic. In March of 2020 millions of Americans went home on a Friday and didn’t return to the office Monday morning. What followed was months of figuring out society’s new normal.

But what happened to critical industries like healthcare?

Francis X also known as FX works at MITRE researching the benefits of AI in the medical field while also working in the field himself as a physician. For FX, the pandemic required his team to adapt quickly and illustrated the importance resiliency plays in an organization. Overnight doctor’s office had to transform the way they provided care.

Hi, and welcome to MITRE’s Tech Futures podcast. I’m your host, Courtney Clark. And I’m a cyber business strategist here at MITRE. At MITRE, we offer unique vantage points and objective insights that we share in the public interest. And in this podcast series, we showcase emerging technologies that will affect the government and our nation in the future.

Today, we’re going to talk about the TRUSTS Framework, a framework focused on resiliency developed by Dr. Kelly Neville, that bridges the gap between new technologies and the critical industries that would benefit from them. We’re going to hear real life examples in the healthcare and defense industry, embracing new technology and learn more about how TRUSTS makes that process easier and safer for everyone involved.

Before we begin, I want to say a huge thank you to Dr. Kris Rosfjord, the Tech Futures Innovation Area Leader in MITRE’s independent research and development program that supported this effort. This episode would not have happened without her support. Now without further ado, I bring you MITRE’s Tech Futures podcast, episode number 17.

[INTRO MUSIC]

Transitioning a new technology into a work system is always risky. Even as the technology benefits a work system’s performance, it can negatively impact the system’s ability to be adaptive and resilient to high demands, surprise events and diverse conditions. In fact, almost any change to increase a system’s efficiency is guaranteed to decrease its adaptivity and resilience.

And what if you work in fields such as aviation, emergency response, or military operations using high consequence work systems? Or, what if you’re facing a pandemic?

 

 

FX Campion

[03:00:4] It was not a time where they could go out and do vendor selection for a telemedicine platform, or it wasn’t time where they could even go to their existing electronic health record and say, what are the four or five things I need to do to make telemedicine happen through our existing EHR, all of that may have been things they were planning and were taking a year or two to think through, but they had literally days to do something.

So it was a bumpy ride. There was a lot of uncertainty and I would say, when push comes to shove, clinicians do what’s best for their patient and there was a good willingness all around to try to get patients good care.

Courtney Clark

[03:40:5] As we now know the pandemic transformed the way we receive care in a way no one was expecting. The medical industry had to quickly adapt by offering services across a new medium. As a result, healthcare became a widely more available to those in rural areas and anyone who is struggling to make it into a physical office.

But it is highly unusual for a critical industry to make such drastic changes so quickly. Usually, an organization or industry has more time to prepare. Although time doesn’t guarantee success if organizations like the Department of Defense, don’t think about resilient technology transformation early on in the program.

Scott Lee

[04:20:1] The DoD is trying to improve based upon mission need, and contextual requirements of the commander or the decision maker situational awareness across war fighting domains. So think air, maritime, land, space, cyberspace. That awareness requires automation of data flows, automation of workflows and the application of analytics in a resilient manner so that the commander can have awareness and understanding and visualization of assets, dynamic operations from a current and future state, and the potential effect that the application of certain assets may have on mission objectives.

Courtney Clark

[05:00:7] That was Scott Lee who led MITRE’s National Security Engineering Center in building resilience and decision-making for the DoD. He worked with the T R U S T S or TRUSTS framework team to update and strengthen resiliency while introducing new technology into DOD high consequence work systems. But how? How do you introduce a new technology, which includes retraining people, ensuring security, and successful adoption, all while ensuring no grave mistakes are made in industry such just healthcare or military.

That’s why we talked to Kelly Neville. The principal investigator and lead for the TRUSTS Framework.

TRUSTS, which stands for Transform with Resilience during Upgrades to Socio-Technical Systems is a framework designed to help organizations conduct resilience-aware development and transition. In other words, it ensures that all of the updates and investments made to secure and advance our critical industries actually get used by working with the people and inside of the systems that we’ll be using them.

But what does it mean for a tool to have resiliency? Why didn’t any guidance in the first place? As Kelly goes on to describe, things don’t always go to plan, particularly in high stress situations. In those cases, it’s important to be able to deviate from the plan without breaking where you’re using.

Kelly Neville

[06:21:5] They’re the ambulance folks, they’re the emergency responders, the crisis response team. They’re boots on the ground seeing “this is what we’re dealing with and we have all these practices and policies in our organization, and I’m gonna apply them to the best of my ability.

Or if I don’t do this, if you’re in combat or what have you in that kind of situation, that’ll be serious consequences. I’ve gotta deviate from standard operating procedures to save somebody, this kind of thing.”

The frontline operator always needs to be able to make that quick decision.

Courtney Clark

[06:53:0] Resiliency isn’t about making sure things go right. But ensuring the system doesn’t break when things go wrong and you deviate from the plan. That explains a little bit about why critical industries are less willing to try new things. If a new system or tool doesn’t stand up to pressure, the results could be deadly.

Organizations need to understand how a new technology may impact adaptivity and resilience during the engineering process. By doing so they could help ensure that their technology transition will be successful.

But how exactly do you ensure resiliency? The TRUSTS framework has five working system resiliency factors. Demand and Deviation Awareness, Progressive Responding, Response Coordination. Maneuver Capability now called Maneuver Flexibility, and Local Guided Control.

Kelly Neville

[07:43:1] So the Demand and Deviation Awareness is the idea that you’re aware and you have a means to share that with everybody else and everything else; other technologies that may care or need to know about that change.

Progressive Responding is more about here are some of the more effective ways to respond to events, challenges, deviations, new stuff in an environment that is complex, that’s changing where there’s uncertainty. It’s about anticipating what may be coming down the pike for you. It’s about initiating those early responses and picking “what will work?”, but not the perfect, “what will work?” and then looking for feedback, getting feedback so you can adjust.

The next one was response coordination and especially if you’ve got a deviation a challenging situation, a surge event if you’re a hospital or what have you, you can’t just individually respond to it. It’s gotta be a coordinated response, and it has to be coordinated for resilience.

And then we have Maneuver Capacity, now Flexibility, and that is to be adaptive and resilient, if you can only do something through a series of, these four steps in this order and something in those steps breaks or is unavailable to you today, you’re up the creek without a paddle. So that’s rigidity.

Guided Local Control is the idea that frontline operators, and this assumes that you’ve got educated, well-trained frontline operators, you want those frontline operators to be able to respond to what they’re seeing. They are the pointy end of the spear. And they’re the ones most directly facing the challenges that the work system is dealing with.

Courtney Clark

[09:28:6] Along with the big five resiliency factors, sub factors and resiliency requirements are also contained in the TRUSTS framework to ensure the pointy end of the spear actually lands where it needs to, even if there are detours along the way. With all of this under consideration, it’s easier to see why resiliency is a factor that needs to be considered in critical systems. Particularly when developing new technologies for them.

Okay, that sounds great. But how exactly you’re supposed to do that?

Steve Dorton

[09:56:3] In an hour, we were able to generate 15 user stories or requirements. We also were able to identify at least five or six different stakeholders. So, a lot of this was framed as what’s the physician doing and needing, what’s the patient doing and needing, but we identified through this tabletop the roles of the family of the patients, of the chief medical information officer, the clinical decision support team.

Courtney Clark

[10:18:3] That was Steve Dorton, who predominantly works on the operationalization of the framework – the scientist who puts it into motion.

By working in tandem with those who would be using the technology, engineers and developers are able to look ahead at the intricacies that come with specific systems, as well as identify stakeholders that may not have been considered. Steve has seen firsthand the benefits from a tabletop exercise held with a healthcare developer last year.

Steve Dorton

[10:46:1] So we got teamed up with a team developing a decision support system. for telemedicine. So basically, they had a really cool technology, synthetic populations, they had different clinical factors, they had social determinants of health factors, which my understanding, very limited in the healthcare world. And basically, this would help assess whether a whole population or an individual patient is an appropriate fit for obstetrics or OB telemedicine.

So, we worked with them, and we do these three, we call sense making sessions where we learn about their technology, why it exists, what they’re trying to accomplish.

We learn about the work. So who uses it? How do they use it? What steps happen upstream from it, downstream from it? And then, from there with that understanding, we map their use cases to the TRUSTS framework and come back and then the actual tabletop itself is only an hour. The 90 minutes includes the intro, rules and engagement, but, we provide these injects that test resilience so we wrote a scenario where, the head of the hospital asked you to evaluate this and provide evidence. And then we had a fast forward a year later and your individual patients that you see were how we did the story that we used to give the inject. So, we would do things like, they’re really frazzled, they didn’t seem to want to come into the office, they mentioned that they had to catch like three buses to get there.

And the data they’re submitting through the app is very uniform and right in line with medical guidance. So, we introduce ambiguity. You’re seeing different things versus what they’re telling you. How would you use this system to make sense of that?

So, if you’re building stakeholder maps or user journey, those types of user-centered design artifacts. The RAD-X feeds directly into that. Of the 15 user stories that came out of it, we had some resilience experts code them to see whether they met certain criteria or not.

And all but one or two of them actually supported work system resilience. So that to me was a huge measure of success because, we did this, really lightweight, rapid thing. We met with them three times. We did an hour-long tabletop, we generated 15 user stories, the vast majority of which weren’t just functionality or usability, but they actually were things that if implemented with support work system resilience.

Courtney Clark

[13:00:7] Let’s bring that back a second. Through a total of four and a half hours with the developers of this app, the TRUSTS Framework team was able to increase resilience and therefore increased the likelihood of the new tech being adopted by those who need it. Amazing.

It’s obvious the TRUSTS Framework has its place in healthcare. Now let’s talk about some other use cases in building resilience in decision-making for the DOD.

Scott Lee

[13:25:2] Part of my job as the Joint All-Domain Command and Control Cross-Cutting Priority Lead was to help cohere a vision, a strategy, and a technical approach for the US Department of Defense to advance JADC2, Joint All-Domain Command and Control, the next generation architecture that allows us to have awareness, make decisions, and act at the speed and scale required against peer adversaries. The number one peer being the People’s Republic of China, Russia being a close second. So, as we look at JADC2 and how do I get to speed and scale decision making across an operational environment?

And so, if I look at sense making and decision making and the need to have resiliency in that design, not only from a network, from a data and from a C2 perspective. Human machine teaming frameworks for the future in terms of what might next generation situational or dynamic awareness look like.

The joint staff J6 leading that effort asked MITRE to engage and to help them with this document called ADEPT, Artificial Intelligence Development Environment, so that we can inform how industry builds algorithms and how industry can inform human machine teaming solutions for the future.

So, we brought Kelly and the team in to brief their research on TRUSTS, and then we’re looking at prototyping those opportunities going forward here in FY23.

Courtney Clark

[14:46:3] As we’ve heard in this podcast, the opportunities for the TRUSTS Framework are endless. The Framework can be used to usher in new technologies to countless industries and ensures transition is safe and successful.

FX Campion

[14:58:9] I like what we’re doing with the TRUSTS framework because it’s a message primarily to the development community. We needed resilience when this pandemic started because everyone’s current operations were shaken, right?

So, it was such a societal experience that we went through, every organization did. So just trying to think about how do we prepare ourselves for all the change that’s coming whether it’s abrupt or not abrupt in the future.

And this TRUSTS Framework really is a lot more than just software development. It’s about change behavior in complex systems. And I think once we start thinking about it then the development community and those that are gonna be using the technologies can work together and talk the same language.

Courtney Clark

[15:42:5] Thanks for tuning into this episode of MITRE’s tech futures podcast. I wrote, produced and edited the show with the help of my cohost Eliza Mace, Dr. Kris Rosfjord, Dr. Heath Farris, and Beverly Wood. Our guests for this episode, were Dr. Kelly Neville, Steve Dorton, Scott Lee, and FX Campion.

The music in this episode is brought to you by Ooey, Truvio and Airae.

Copyright 2023, the MITRE Corporation. Approved for public release. Distribution unlimited. Public release case number 23-1181. MITRE: Solving problems for a safer world.