Veera Häkkinen: When AI supports patient self-management, who stays in control?

When AI supports patient self-management, who stays in control? 

For people with long-term conditions, care continues between appointments. They monitor symptoms, manage medication and decide when to seek help. 

Supported self-management can make these daily decisions easier. NHS England, for example, already recognises it as a core part of personalised care. 

The need is growing. Non-communicable diseases account for 80% of the disease burden across EU countries, according to the European Commission. 

While AI could help providers offer timely guidance and identify cases needing attention, it raises an important question: who stays in control when its output could affect patient care? 

A question grounded in clinical experience 

For Veera Häkkinen, this topic brings together her experience in healthcare and technology. Now a Solutions Consultant at Digital Workforce, she previously worked in elderly care, hospital wards and for nearly seven years as an operating theatre nurse. 

Drawing on this background, her Tampere University thesis explores professional responsibilities and human oversight in AI-supported self-management. 

Making human oversight concrete 

The thesis followed a constructive study approach in two parts. 

First, Häkkinen reviewed existing research to identify the roles, tasks and responsibilities professionals may have in algorithmic healthcare processes. She then compared these findings with relevant requirements in the EU AI Act. 

Finally, she created four AI-supported self-management scenarios. These covered: 

• Feedback based on patient questionnaires 

• Processing follow-up information 

• Recommendations for self-care materials 

• Responses to patient messages 

Häkkinen used the scenarios to explore two levels of human oversight. 

In a human-in-the-loop process, a healthcare professional reviews each AI-generated output before it is used or shared with the patient. They decide whether to approve, amend or reject it. 

In a human-on-the-loop model, the system completes agreed tasks independently within defined limits. A healthcare professional monitors its performance and steps in when a case is flagged or something requires closer review. 

This difference affects how work is organised. It also determines when professional judgement enters the automated pathway. 

Häkkinen presented the scenarios and related responsibilities to three nurses and one doctor working with self-management. Their discussions explored whether the proposed tasks would fit their roles and where concerns might arise.

What AI oversight requires from healthcare professionals 

Human oversight covers more than approving a result at the end of a process. 

Incomplete information may already have shaped the AI generated output. The system could miss an exception or direct the case down an unsuitable route. 

Healthcare professionals may therefore need to: 

• Understand the system’s purpose, capabilities and limitations 

• Assess the quality of its inputs and outputs 

• Compare recommendations with the patient’s wider situation 

• Recognise when an output should be questioned 

• Correct, override or stop the process when necessary 

• Report problems and protect professional integrity 

Critical assessment was one of the strongest themes in Häkkinen’s findings. 

This skill is already central to healthcare. Clinicians assess information from patients, colleagues, records and diagnostic systems every day. AI introduces another source that requires professional judgement.

Responsbilities in the context of EU AI Act

The responsibilities assigned to healthcare workers depend on how the AI system is classified and used. Each service therefore requires its own assessment. 

For high-risk systems, the EU AI Act requires the organisation using the system to assign human oversight to people with the necessary competence, training, authority and support. The organisation must also monitor its operation, manage risks and act when concerns arise. Further obligations may apply, including a fundamental rights impact assessment.

Treatment-related cases demand close review  

In the research, participants generally preferred a human-in-the-loop model for treatment-related cases.

Their concerns came from real care situations. A patient may have several conditions or find it difficult to describe their symptoms accurately. Living arrangements, financial circumstances and health literacy can also affect suitable guidance. 

AI may not recognise this wider context. A response can sound convincing while missing something important. 

Participants also raised questions about responsibility. They wanted to know who would remain accountable and whether the system could consider each person’s circumstances fully enough. 

These concerns did not lead to rejection of AI. The professionals were interested in the scenarios and suggested ways to make them safer and more trustworthy. 

They were more open to human-on-the-loop oversight for lower-risk activities. General lifestyle information and administrative communication were examples where greater independence could be appropriate. 

The level of control should therefore reflect the task, its risks and the people using the service.

Involving professionals from the beginning 

Häkkinen’s research highlights the value of involving healthcare teams in design and implementation. 

Professionals understand the exceptions that are difficult to see in a process diagram. They know where patients may struggle and which information carries clinical importance. 

Their involvement can help organisations define: 

• What the AI system is allowed to do 
• Which outputs require professional review 
• When a case must be escalated 
• Who owns each decision 
• How users can question, correct or stop the process 

This work also supports adoption. A solution may meet legal requirements and still fail to gain the confidence of its users. 

Acceptance matters because the intended benefits depend on people using the service as designed. Roles, training and escalation routes must make sense to those responsible for daily operation. 

Häkkinen also found that the level of risk can depend on how healthcare professionals use the system and act on its outputs. Organisations should therefore define these responsibilities clearly and reflect them in service design, implementation and training.

How to make implementation manageable and safe 

For Digital Workforce, Häkkinen’s findings reinforce the importance of designing services around the care pathway and the people responsible for it. 

We work with clinical teams and process owners to define the challenge, map key decisions and identify exceptions. Together, we agree where technology can execute the workflow independently and where professional review or other human input is necessary. 

The service can then be introduced in manageable stages, with users involved in testing and training shaped around their roles. A nurse reviewing guidance needs different support from a process owner monitoring performance and escalations. 

This approach reduces disruption and gives healthcare professionals a clear role in shaping a service they can use confidently in their daily work.

Professional expertise remains essential 

Häkkinen is careful about the limits of the research. The interview part of the study only involved four participants, so its findings cannot represent every healthcare professional or use case. AI technology and the regulatory environment are also developing quickly. 

The thesis still offers a useful way to think about human oversight. It turns a broad requirement into practical responsibilities that healthcare teams can discuss, test, and improve. 

Häkkinen’s conclusion is grounded in her experience as both a nurse and a technology specialist. Professional expertise will remain valuable throughout healthcare processes. AI may change where and when that knowledge is needed. 

The participants brought both concerns and ideas to the discussion. That combination provides a responsible starting point for designing AI-supported care. 

With clear roles, suitable safeguards and healthcare professionals involved throughout, AI can support timely self-management while keeping decisions that affect patients under human control.

Watch Veera’s inteview about her research on our YouTube -channel.