Romano Setzu 1, Andy Olivares 2, Jordi Mill 2, Carlos Albors 2, Oscar Camara 2, Kristian Valen-Sendstad 3, Maria Teresa Mora 4, Beatriz Trenor 4, Maxime Sermesant 5 and Michèle Barbier 5
1 MICROPORT CRM, Clamart, France, 2 Universitat Pompeu Fabra (UPF), Barcelona, Spain, 3 Simula Research Laboratory, Oslo, Norway, 4 Universitat Politècnica de València (UPV), Valencia, Spain and 5 Inria Center of the University Cote d’Azur, Sophia Antipolis, France
Computational models in healthcare
Computational models (also referred to as ‘in silico’ models) are digital representations of a physical object or a phenomenon. Computational models can be used in healthcare for a variety of reasons, including:
- running in silico clinical trials to study the safety and efficacy of new medical devices and drugs
- developing digital twins of patients to tailor specific treatments or therapies in the context of personalised medicine
- accelerating the development of new products by reducing the number of prototyping-testing iterations.
As digital technologies are more widely available and powerful, using computational models in healthcare becomes a clear and straightforward decision. Indeed, computational models have the potential to meet opposite constraints: on the one side, the quest for improving the safety and efficacy of new medical devices and drugs; on the other side, the urge to reduce time-to-market and developing costs.
Fostering adoption of computational models
Accepting computational models as streamlined elements to support the certification of new medical devices and drugs constitutes a paradigm shift for regulatory bodies and certification agencies.
Until recently, most of the evidence supporting the safety and efficacy of new medical devices, drugs, and therapies consisted of real-world data. This includes in vitro experiments, in vivo preclinical (animal), and clinical studies.
Nowadays, more and more computational models are used at some stage in the development of new medical devices and drugs. Some computational models may even constitute the product itself (in the case of ‘software as medical device’) if they can play a role in the clinical pathway.
Gaining credibility on the computational model and its predictions is paramount for its adoption by all stakeholders (Lesage et al., 2023). Model credibility is achieved through verification and validation (V&V) activities (ASME V&V 40, 2018; Viceconti et al., 2021). V&V of computational models is a complex task for several reasons, including, for instance, having access to or creating the necessary data for proper validation and accounting for the large number of parameters potentially involved in a given model.
International guidelines for verification and validation of computational models
Landscape
Only recently, international efforts led by regulators, academia, and industry started to progressively address the impelling need for harmonised guidelines and standards concerning in silico methods.
Late in 2023, the FDA issued a final guidance for assessing the credibility of computational models used in medical device submissions (FDA, 2023). This guidance was issued after a two-year period during which academics and industry were called for feedback.
Similar community efforts led to a white paper defining “Good Simulation Practice” guidelines for developing computational models to assess the safety and efficacy of medical devices and drugs (Viceconti, 2023).
ASME V&V40 standard is undoubtedly the most established document in this field, creating a consistent framework for assessing the credibility of computational models in healthcare (ASME V&V 40, 2018).
That said, the healthcare community is still lacking thorough, worked-out V&V examples of complex computational models. Examples reported in VV40 are definitely a good start. However, they are not exhaustive, as they only approach selected aspects of the required V&V.
Key notions in model V&V
A model is, by definition, a simplification of the real-life phenomenon we want to replicate, and it is the result of a trade-off among several factors, including the:
- quality and quantity of information we dispose to develop the model
- finality for which the model has been developed, and the context in which it should be valid
- risk inherent in accepting that the model may result in a wrong prediction.
The credibility of a computational model measures the trust we can put in the prediction capability of such a model for a specific context of use (COU), i.e. the perimeter in which the model is operated.
ASMEV&V40 proposes that the credibility of a computational model is inferred through verification and validation activities. Each activity is associated with a well-defined credibility factor. The computational model is evaluated for each activity, and the obtained score is weighted against the model risk, which is the risk incurred by the patient in case the model fails to predict the question of interest it addresses.
Verification and validation are key terms in the development of computational models and are unambiguously defined within the V&V40 standard.
Given a mathematical model that aims to replicate a real-life phenomenon, model verification comprises all activities intended to demonstrate that the numerical implementation of the model is robust and accurate. Activities include software quality assurance, evaluation of numerical and discretisation errors, and procedures to minimise use errors.
Model validation comprises all activities meant to show how well the computational model represents reality. Since the quantities of interest needed to validate the model behaviour in its context of use may not be readily accessible, we often rely on comparators to obtain real-world data instead.
Consequently, there may be a gap between the space in which the computational model is validated and the space in which the model (once validated) is used to predict real-life phenomena (context of use). The magnitude of such a gap constitutes the applicability of the validation activities. It needs to be addressed and weighted (once again) against the model risk.
The SimCardioTest project
The SimCardioTest project, funded by the European Commission (2020–2024), focuses on developing a cloud-based platform for the virtual testing of cardiac medical devices and drugs (Barbier et al., 2021). Three different use cases (UCs) are addressed within the project, covering several domains in cardiovascular medicine.
- UC1 [active implants]
Pacing devices used to deliver electrical stimuli to the heart and treat cardiac arrhythmias. - UC2 [passive implants]
Left atrial appendage occluders (LAAO) used to reduce the risk of stroke. - UC3 [drugs]
Evaluation of new molecules to ensure their safety against malignant arrhythmias and assess their efficacy.
Computational models developed within each use case need to go through extensive V&V activities before being implemented in the cloud-based platform. V&V activities ensure that all models have gained a satisfactory degree of credibility before being included in in silico trials and used to make predictions about the quantities they are meant to model.
SimCardioTest aims to be a precursor in establishing a clear path to streamline V&V of computational models, offering a consistent template applicable across various fields in cardiovascular medicine, one which may positively serve to guide future work of both industry and regulators.
SimCardioTest also has the ambition to seek dialogue with regulatory bodies and certification agencies, collecting constructive feedback on their perspective and expectations in this emerging field.
Focus on V&V activities
We now present a few examples to illustrate typical V&V activities within SimCardioTest addressing both medical devices (UC2) and drugs (UC3) in cardiovascular medicine.
Use Case 2 (medical devices)
Verification example
Figure 1 shows space and time convergence analysis performed on the computational model. Based on the results, the best time-space combination was then selected for model validation.
Validation example
Figure 2 shows an example of validation activity for LAAO applications. In this example, mitral valve velocity (MV) was used as the quantity of interest for the comparison.
Use Case 3 (drugs)
Verification example
Due to the model complexity, the electrophysiological signals do not have a unique true solution. To verify the computational model, we ensured that action potential and calcium transient traces replicated real electrophysiological recordings and that the derived biomarkers were within physiological ranges.
In Figure 3, the biomarkers were computed for increasing coarser time and space resolutions starting from a set of sufficiently small values (0.025 ms and 0.1 mm, respectively). The small relative convergence error associated with changes in discretisation values shows stability in the selected temporal and spatial ranges.
Validation example
For validating the model’s capability to predict the effect of drugs, data from the scientific literature was selected as the comparator. Selected data included experiment results at the cellular and organ level. Changes in action potential duration (APD90) was the index used to quantify the prolongation of the action potential, and the QT interval was the biomarker used to evaluate changes in ECG duration.
Conclusion
Computational models are meant to become an essential tool in the healthcare sector for developing new medical devices and drugs. Consequently, gaining credibility on the computational model and its predictions is paramount for its adoption by all stakeholders. Credibility is achieved through V&V activities.
SimCardioTest is committed to exploring in-depth V&V aspects of computational models addressing several safety and efficacy aspects in cardiovascular medicine through three different use cases (active implants, passive implants, and drugs). Results will be shared with the larger community to provide systematic and harmonised examples of how to gain credibility on computational models.
View: SimCardioTest YouTube channel.
References
Ansys (2023) Ansys ® Fluent. Available at: https://www.ansys.com/products/fluids/ansys-fluent.
ASME V&V 40 (2018) Assessing Credibility of Computational Modeling through Verification and Validation: Application to Medical Devices. New York: The American Society of Mechanical Engineers.
Barbier, B., Sermesant, M., Camara, O., Coudière, Y. and Trenor, B. (2021) ‘What computational sciences can do for your heart’, Project RepositoryJournal, 10, pp. 76–79. Available at: https://www.europeandissemination. eu/project-repository-journal-volume-10-july-2021/14505 (Accessed: 15 November 2023).
FDA (2023) Assessing the Credibility of Computational Modeling and Simulation in Medical Device Submissions. Available at: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/ assessing-credibility-computational-modeling-and-simulation-medical-device-submissions (Accessed: 1 December 2023).
Finsberg, H.N.T., van Herck, I.G.M., Daversin-Catty, C., Arevalo, H. and Wall, S. (2023) ‘simcardems: A FEniCS-based cardiac electro-mechanics solver’, The Journal of Open Source Software, 8(81). doi: 10.21105/joss.04753.
Gluais, P., Michìle, B., Jacques, C., Monique, A. (2003) ‘Comparative Effects of Clarithromycin on Action Potential and Ionic Currents from Rabbit Isolated Atrial and Ventricular Myocytes’, Journal of Cardiovascular Pharmacology, 41(4), pp. 506–517. doi: 10.1097/00005344-200304000-00002.
Himmel, H.M., Bussek, A., Hoffman, M., Beckmann, R., Lohmann, H., Schmidt, M. and Wettwer, E. (2012) ‘Field and action potential recordings in heart slices: Correlation with established in vitro and in vivo models’, British Journal of Pharmacology, 166(1), pp. 276–296. doi: 10.1111/j.1476-5381.2011.01775.x.
Lesage, R., De Michele, R., Contin, M., Camara, O. and Geris, L. (2023) ‘Digital innovation in cardiovascular medicine: a multi-stakeholder business’, Project Repository Journal, 16, pp. 50–53. Available at: https:// www.europeandissemination.eu/project-repository-journal-volume-16-february-2023/19892 (Accessed: 15 November 2023).
Mendez, K., Kennedy, D.G., Wang, D.D., O’Neill, B. and Roche, E. (2022) ‘Left Atrial Appendage Occlusion: Current Stroke Prevention Strategies and a Shift Toward Data-Driven, Patient-Specific Approaches’, Journal of the Society for Cardiovascular Angiography & Interventions, 1(5), 100405. doi: 10.1016/j.jscai.2022.100405.
Mortensen, M. and Valen-Sendstad, K. (2015) ‘Oasis: A high-level/high-performance open source Navier– Stokes solver’, Computer Physics Communications, 188, pp. 177–188. doi: 10.1016/j.cpc.2014.10.026.
Viceconti M., Pappalarado, F., Rodriguez, B., Horner, M., Bischoff, J. and Tshinanu, F.M. (2021), ‘In silico trials: Verification, validation and uncertainty quantification of predictive models used in the regulatory evaluation of biomedical products’, Methods, 185, pp. 120–127. doi: 10.1016/j.ymeth.2020.01.011.
Viceconti, M. (2023) Toward Good Simulation Practice: Best Practices for the Use of Computational Modelling & Simulation in the Regulatory Process of Biomedical Products’, In Silico World – Good Simulation Practice (GSP) Task Force, Version: R6 – prefinal, open to public comments, Unpublished. s://insilico.world/sito/wp-content/uploads/2023/06/Position-Paper-GSP-R6.pdf. (Accessed: 1 December 2023).
Project summary
Cardiovascular diseases affect 15 million people in Europe and digital solutions are now seen as very useful tools in the search for new drugs and medical devices. SimCardioTest is a 4-year project funded by the European Commission that aims to develop credible computer modelling and simulation approaches on a cloud-based platform for testing cardiac drugs and devices in silico.
Project partners
SimCardioTest brings together leading experts in the field of cardiac simulation, drug effect, medical devices and regulatory process. It includes large companies (Microport – CRM), SMEs (ExactCure and InSilicoTrials), research organisations (Inria and Simula), universities (University of Bordeaux, University Pompeu Fabra, Polytechnic University of Valencia) and international non-profit organisation (The Virtual Physiological Human Institute).
Project lead profile
SimCardioTest is led by Inria, the French national research institute for the digital sciences, world-class research and technological innovation organisation, developing and supporting scientific and entrepreneurial projects that create value in France and Europe. Dr Maxime Sermesant, head of Computational Cardiology at Inria Epione and Chair of AI & biophysics at 3IA Côte d’Azur, ensures the scientific coordination.
Project contact
Michèle Barbier
Inria Sophia Antipolis-Méditerranée
2004 Route des Lucioles,
06902 Valbonne France
+336 3307 9899
www.linkedin.com/in/simcardiotest-eu-project-ehealth-eu-207a08204/
Funding
This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 101078351.
Figure legends
Figure 1: SimCardioTest UC2 – discretisation error assessment using commercial software (Ansys, 2023) or open source software (Oasis FEniCS (Mortensen and Valen-Sendstad, 2015)).
Figure 2: SimCardioTest UC2 – Validation of left atria (LA) fluid simulations using the in vitro test in Massachusetts Institute Technology (MIT) in Roche Lab (Mendez et al., 2022).
Figure 3: SimCardioTest UC3 – Convergence tests of cellular electrophysiological properties on a slab of size 3x7x20 mm with the software Simcardems (Finsberg et al., 2023). APD90: action potential duration at 90 per cent of repolarisation; SystCa: systolic intracellular calcium concentration.
Figure 4: SimCardioTest UC3 – Validation of the prolonging effects of drugs A) at the cellular level (action potential) and B) at the ventricular level (ECG). Examples with A) Clarithromycin (experimental data from (Gluais et al., 2003)) and B) Sotalol (experimental data from (Himmel et al., 2012)).
