Manufacturing technology What do regulators and standards bodies say?
The US FDA’s guidance is specifi c to software used in medical-device production and quality systems, while the US National Institute of Standards and Technology documents address AI across sectors. Together, they provide useful principles for managing software assurance and AI risk.
Take a risk-based approach The US FDA’s guidance on computer software assurance provides recommendations for computers and automated data-processing systems used in medical-device production or a quality management system. It recommends a risk- based approach to establishing and maintaining confi dence that software is fi t for its intended use, with assurance activities proportionate to the risks associated with its use.
Keep monitoring after deployment NIST’s voluntary AI Risk Management Framework promotes managing AI risks throughout the AI system life cycle. The framework includes post-deployment monitoring as part of AI risk management. NIST’s report, Challenges to the Monitoring of Deployed AI Systems, examines practical challenges in monitoring AI systems after deployment.
Match controls to the use case Together, the documents support an approach in which organisations defi ne an AI or software system’s intended use, assess the risks associated with that use, generate appropriate evidence that it performs as expected and maintain proportionate assurance and monitoring activities throughout its life cycle.
Source: US Food and Drug Administration’s ‘Computer Software Assurance for Production and Quality Management System Software’; National Institute of Standards and Technology’s ‘Artifi cial Intelligence Risk Management Framework’ and ‘Challenges to the Monitoring of Deployed AI Systems’.
Siemens Healthineers. She adds that humans with the right expertise should always be in the driver’s seat to ensure AI models are running safely and effectively. To get the most out of AI, Sainath explains that Siemens Healthineers uses custom models tailored to specific use cases. She says this fit-for-purpose approach can help reduce the risk of inappropriate or erroneous outputs, including hallucinations in generative-AI systems.
A hallucination occurs when a generative-AI model reports incorrect, misleading or fabricated information to a user. In medical device manufacturing, such outputs could affect decision-making or the quality of associated documentation. Manufacturers should therefore assess and control the risk of erroneous or fabricated outputs through appropriate testing, human review and other safeguards.
Sainath adds that, while building the infrastructure for these AI systems to operate, the company has placed a keen focus on developing the architecture guiding them in-house.
Juanita Schoen, senior engagement manager at consultancy Columbus, says accountability, documented procedures and auditable risk-monitoring controls are important for introducing AI into regulated operations.
Schoen also believes that appropriate human oversight should remain in place for relevant regulated use cases. Intervention may be required during potential model breakdowns to help ensure devices are manufactured safely and to the required standard. In the eyes of Erez Kaminski, founder and CEO of medtech manufacturing life-cycle management software company Ketryx, a key consideration for operators is ensuring they set up their systems to be ready for AI before it is implemented. This, he says, requires creating efficient controls as a by-product of the work already being done by the manufacturer.
www.medicaldevice-developments.com
A growing need for AI validation As medical device manufacturers adapt to the growing use of AI, Kaminski says a broader challenge is emerging around capacity for model validation. Kaminski says some companies are incorporating large numbers of AI agents into their workflows and questions how they will manage the need to continuously validate their activity as models update rapidly and use cases change over time. “Today, the way people think about validation compliance is through documenting everything and testing it once; with AI, you can’t govern a dynamic model through this type of static documentation,” Kaminski notes.
Kaminski says some companies rely on manual validation cycles lasting more than 90 days, while some AI models are updated every few weeks. He cautions that introducing AI into the workflow could result in a large increase in work linked to validating the models used across an organisation. To overcome this challenge, Kaminski calls for the embedding of automated control points within systems, which he says should be designed to collect evidence and document system activity reliably. This, he says, can create a continuous digital trail to support monitoring and controlled corrective action over time.
Addressing the risks of model shift In addition to AI’s potential to add to a company’s workload through increased validation requirements, Schoen also points to the technology’s “probabilistic” nature, which she says can present hurdles for medical device manufacturers.
The most notable example, Schoen says, is model drift, used here to describe a change in model performance or behaviour as deployment conditions or input data change. If a model’s
39
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28 |
Page 29 |
Page 30 |
Page 31 |
Page 32 |
Page 33 |
Page 34 |
Page 35 |
Page 36 |
Page 37 |
Page 38 |
Page 39 |
Page 40 |
Page 41 |
Page 42 |
Page 43 |
Page 44 |
Page 45 |
Page 46 |
Page 47 |
Page 48 |
Page 49 |
Page 50 |
Page 51 |
Page 52 |
Page 53 |
Page 54 |
Page 55 |
Page 56 |
Page 57 |
Page 58 |
Page 59 |
Page 60 |
Page 61 |
Page 62 |
Page 63 |
Page 64 |
Page 65 |
Page 66 |
Page 67 |
Page 68 |
Page 69 |
Page 70 |
Page 71 |
Page 72 |
Page 73 |
Page 74 |
Page 75 |
Page 76 |
Page 77 |
Page 78 |
Page 79 |
Page 80 |
Page 81 |
Page 82 |
Page 83 |
Page 84 |
Page 85 |
Page 86 |
Page 87 |
Page 88 |
Page 89 |
Page 90 |
Page 91 |
Page 92 |
Page 93 |
Page 94 |
Page 95 |
Page 96 |
Page 97 |
Page 98 |
Page 99 |
Page 100 |
Page 101 |
Page 102
