"I really want to have a strong position to operate from and to think from, and a strong advisor who’s committed to that, then I think that’s why you want to work with IF."

[.quote-author]Head of Innovation[.quote-author]

Problem

Our client had developed an AI system that could detect disease earlier in a care pathway, recommending whether people need to be sent for a cervical cancer screening. This had the potential of reducing costs of providing healthcare, as well as delivering better health outcomes for members. 

The team knew the AI system performed reliably for the specific context and demographics of one state of the US, but needed to adapt the technology so that they could prove it was safe to use across other states.

Part of the challenge was earning public, clinical and member trust in the use of AI in healthcare. However, there could also be reputational and clinical harm if the power that comes with predictive models were misused, if predictions are made without meaningful consent, or if testing impacts are not rigorously considered.

Our approach

Earning trust with AI is innovation work. There are no silver bullets. The systems that AI is applied to are often complex, and the impacts of changes difficult to predict. But we felt that starting early would enable our client to explore more ambitious uses of AI, and ultimately deliver better services for people.

Emerging technology and platforms

We ran a series of download workshops with the client, and used our experience of building machine learning systems to ask questions that explored where the hard problems were. From this we were able to draw an AI lifecycle, and identify who should be accountable for each stage.

The term bias in AI sparks a lot of fear for companies but not many people understand what it means. We started describing bias as whether there is a balance of representation in training data. This could be in terms of social demographics, or how long someone had been a member. We shared how imbalances in any of these factors will have an impact on how accurate a model will be on different groups. They need to be understood and designed for. It depends on what impact the bias will have, and whether that’s relevant to the people our client wanted to apply the model to.

This technical work resulted in a series of audit questions that spanned from setting up the model development environment to storing and acting on model predictions.

Full stack design and development

We drew user journeys to draw out unhappy paths through the service, where trust is more likely to be relied on, to find points of risk. In particular we were building an understanding of the unintended consequences of model predictions, and the impact on downstream services.

This helped give the team a shared understanding and alignment of the opportunity space and prioritisation of risks. This resulted in a short term plan, with practical next steps, and a risk register for the technical team to run a prototype audit against.

Hype free digital strategy

Throughout the project we were drawing our insights and expertise into a digital strategy for an AI audit process. The strategy spanned company policy, audit structures and communication strategies for different user groups. It was situated in our clients’ product maturity plans, in the context of changing regulation and impacts on how and when they develop their AI audit programme.

Outcomes

Our work we helped the team lay the foundations for responsible innovation, which had lasting impact within the business. We were able to change their cost model, which had two significant benefits: the team was able to reduce medical and operating expenses by $50M just in year one, and it became easy to do the right thing. 

Ultimately, the team were able to deploy the AI systems within care pathways to provide more proactive care. They also created a Medical Council, enabling the organisation to undertake more innovation with AI well ahead of competitors.

As a result of being able to deploy the AI system, our client not only benefited from cost savings but also twice the level of engagement from members too.

‍