Marko Topalovic, Nilakash Das, Pierre-Régis Burgel, Marc Daenen, Eric Derom, Christel Haenebalcke, Rob Janssen, Huib A.M. Kerstjens, Giuseppe Liistro, Renaud Louis, Vincent Ninane, Christophe Pison, Marc Schlesser, Piet Vercauter, Claus F. Vogelmeier, Emiel Wouters, Jokke Wynants, Wim Janssens on behalf of the Pulmonary Function Study Investigators
European Respiratory Journal 2019 53: 1801660; DOI: 10.1183/13993003.01660-2018
The interpretation of pulmonary function tests (PFTs) to diagnose respiratory diseases is built on expert opinion that relies on the recognition of patterns and the clinical context for detection of specific diseases. In this study, we aimed to explore the accuracy and interrater variability of pulmonologists when interpreting PFTs compared with artificial intelligence (AI)-based software that was developed and validated in more than 1500 historical patient cases.
120 pulmonologists from 16 European hospitals evaluated 50 cases with PFT and clinical information, resulting in 6000 independent interpretations. The AI software examined the same data. American Thoracic Society/European Respiratory Society guidelines were used as the gold standard for PFT pattern interpretation. The gold standard for diagnosis was derived from clinical history, PFT and all additional tests.
The pattern recognition of PFTs by pulmonologists (senior 73%, junior 27%) matched the guidelines in 74.4±5.9% of the cases (range 56–88%). The interrater variability of κ=0.67 pointed to a common agreement. Pulmonologists made correct diagnoses in 44.6±8.7% of the cases (range 24–62%) with a large interrater variability (κ=0.35). The AI-based software perfectly matched the PFT pattern interpretations (100%) and assigned a correct diagnosis in 82% of all cases (p<0.0001 for both measures).
The interpretation of PFTs by pulmonologists leads to marked variations and errors. AI-based software provides more accurate interpretations and may serve as a powerful decision support tool to improve clinical practice.
There is poor accuracy and substantial disagreement between pulmonologists when interpreting complex pulmonary function data. Automating interpretation with artificial intelligence provides a powerful decision support tool in clinical practice.
This article has supplementary material available from erj.ersjournals.com
This study is registered at ClinicalTrials.gov with identifier number NCT03264417.
Author contributions: All authors critically revised the manuscript and approved the final version. All authors organised evaluation sessions in hospitals, examined patient files and interpreted results. M. Topalovic performed the data acquisition, analysis, interpretation as well as contributed to the study design and wrote the manuscript. N. Das contributed to data acquisition. W. Janssens takes responsibility for the content of the manuscript, contributed to the study design, and assisted in the data analysis, interpretation and writing of the manuscript.
The Pulmonary Function Study Investigators: R. De Pauw, C. Depuydt, C. Haenebalcke, S. Muyldermans, V. Ringoet, D. Stevens (AZ Sint-Jan Hospital, Bruges, Belgium); S. Bayat, J. Benet, E. Catho, J. Claustre, A. Fedi, M.A. Ferjani, R. Guzun, M. Isnard, S. Nicolas, T. Pierret, C. Pison, S. Rouches, B. Wuyam (CHU Grenoble Alpes, Grenoble, France); J.L. Corhay, J. Guiot, K. Ghysen, L. Renaud, A. Sibille (University Hospital, Liege, Belgium); H. De La Barriere, C. Charpentier, S. Corhut, K.A. Hamdan, M. Schlesser, G. Wirtz (Centre Hospitalier de Luxembourg, Luxembourg, Luxembourg); E. Alabadan, G. Birsen, P.R. Burgel, A. Chohra, C. Hamard, B. Lemarié, M.N. Lothe, C. Martin, A.C. Sainte-Marie, L. Sebane (Cochin Hospital, Paris, France); Y. Berk, B. de Brouwer, R. Janssen, J. Kerkhoff, A. Spaanderman, M. Stegers, A. Termeer, I. van Grimbergen, A. van Veen, L. van Ruitenbeek, L. Vermeer, R. Zaal, M. Zijlker (Canisius Wilhelmina Hospital, Nijmegen, The Netherlands); J. Aumann, K. Cuppens, D. Degraeve, K. Demuynck, B. Dieriks, K. Pat, L. Spaas, R. Van Puijenbroek, K. Weytjens, J. Wynants (Jessa Hospital, Hasselt, Belgium); V. Adam, B.J. Berendes, E. Hardeman, P. Jordens, E. Munghen, K. Tournoy, P. Vercauter (Onze-Lieve-Vrouw Hospital, Aalst, Belgium); T. Alame, M. Bruyneel, M. Gabrovska, I. Muylle, V. Ninane, D. Rozen, P. Rummens. S. Van Den Broecke (Saint-Pierre Hospital, Brussels, Belgium); A. Froidure, S. Gohy, G. Liistro, T. Pieters, C. Pilette, F. Pirson (Université Catholique de Louvain, Brussels, Belgium); H. Kerstjens, M. Van den Berge, N. Ten Hacken, M. Duiverman, D. Koster (University Medical Center Groningen, Groningen, The Netherlands); B. Vosse, L. Conemans, M. Maus, M. Bischoff, M. Rutten, D. Agterhuis, R. Sprooten (Maastricht University Medical Center, Maastricht, The Netherlands); B. Beutel, A. Jerrentrup, A. Klemmer, C. Viniol, C. Vogelmeier (University Medical Center, Marburg, Germany); H. Bode, C. Dooms, D. Gullentops, W. Janssens, K. Nackaerts, D. Rutens, E. Wauters, W. Wuyts (University Hospital Leuven, Leuven, Belgium); E. Derom, S. Dobbelaere, S. Loof, G. Serry, B. Putman, L. Van Acker, Y. Vandeweygaerde (Ghent University Hospital, Ghent, Belgium); M. Criel, M. Daenen, R. Gubbelmans, S. Klerkx, E. Michiels, M. Thomeer, A. Vanhauwaert (Hospital Oost-Limburg, Genk, Belgium).
Conflict of interest: M. Topalovic has nothing to disclose.
Conflict of interest: N. Das has nothing to disclose.
Conflict of interest: P-R. Burgel reports personal fees from AstraZeneca, Boehringer Ingelheim, Chiesi, Novartis, Teva and Vertex, outside the submitted work.
Conflict of interest: M. Daenen has nothing to disclose.
Conflict of interest: E. Derom has nothing to disclose.
Conflict of interest: C. Haenebalcke reports personal fees from Novartis, Chiesi, GSK and AstraZeneca, outside the submitted work.
Conflict of interest: R. Janssen has nothing to disclose.
Conflict of interest: H.A.M. Kerstjens has nothing to disclose.
Conflict of interest: G. Liistro has nothing to disclose.
Conflict of interest: R. Louis reports grants and personal fees from GSK and Novartis, personal fees from AstraZeneca, and grants from Chiesi, outside the submitted work.
Conflict of interest: V. Ninane has nothing to disclose.
Conflict of interest: C. Pison has nothing to disclose.
Conflict of interest: M. Schlesser has nothing to disclose.
Conflict of interest: P. Vercauter has nothing to disclose.
Conflict of interest: C.F. Vogelmeier reports personal fees from Almirall, Cipla, Berlin-Chemie/Menarini, CSL Behring and Teva, grants and personal fees from AstraZeneca, Boehringer Ingelheim, Chiesi, GSK, Grifols, Mundipharma, Novartis and Takeda, grants from German Federal Ministry of Education and Research (BMBF) Competence Network Asthma and COPD (ASCONET), Bayer Schering Pharma AG, MSD and Pfizer, outside the submitted work.
Conflict of interest: E. Wouters reports personal fees for board membership from Nycomed and Boehringer, grants from AstraZeneca and GSK, and personal fees for lectures from AstraZeneca, GSK, Novartis and Chiesi, outside the submitted work.
Conflict of interest: J. Wynants has nothing to disclose.
Conflict of interest: W. Janssens has nothing to disclose.
Support statement: This work was supported by the Vlaams Agentschap Innoveren & Ondernemen (VLAIO, government body, 2016–2018). The funder had no role in study design and conduct of the study; collection, management, analysis and interpretation of the data; preparation, review or approval of the manuscript; and decision to submit the manuscript for publication. Funding information for this article has been deposited with the Crossref Funder Registry.