The development of artificial intelligence (AI) has revolutionised the medical system, empowering healthcare professionals to analyse complex nonlinear big data and identify hidden patterns, facilitating well-informed decisions. Over the last decade, there has been a notable trend of research in AI, machine learning (ML), and their associated algorithms in health and medical systems. These approaches have transformed the healthcare system, enhancing efficiency, accuracy, personalised treatment, and decision-making. Recognising the importance and growing trend of research in the topic area, this paper presents a bibliometric analysis of AI in health and medical systems. The paper utilises the Web of Science (WoS) Core Collection database, considering documents published in the topic area for the last four decades. A total of 64,063 papers were identified from 1983 to 2022. The paper evaluates the bibliometric data from various perspectives, such as annual papers published, annual citations, highly cited papers, and most productive institutions, and countries. The paper visualises the relationship among various scientific actors by presenting bibliographic coupling and co-occurrences of the author\'s keywords. The analysis indicates that the field began its significant growth in the late 1970s and early 1980s, with significant growth since 2019. The most influential institutions are in the USA and China. The study also reveals that the scientific community\'s top keywords include \'ML\', \'Deep Learning\', and \'Artificial Intelligence\'.

The black-box nature of machine learning models hinders the deployment of some high-accuracy medical diagnosis algorithms. It is risky to put one\'s life in the hands of models that medical researchers do not fully understand or trust. However, through model interpretation, black-box models can promptly reveal significant biomarkers that medical practitioners may have overlooked due to the surge of infected patients in the COVID-19 pandemic. This research leverages a database of 92 patients with confirmed SARS-CoV-2 laboratory tests between 18th January 2020 and 5th March 2020, in Zhuhai, China, to identify biomarkers indicative of infection severity prediction. Through the interpretation of four machine learning models, decision tree, random forests, gradient boosted trees, and neural networks using permutation feature importance, partial dependence plot, individual conditional expectation, accumulated local effects, local interpretable model-agnostic explanations, and Shapley additive explanation, we identify an increase in N-terminal pro-brain natriuretic peptide, C-reaction protein, and lactic dehydrogenase, a decrease in lymphocyte is associated with severe infection and an increased risk of death, which is consistent with recent medical research on COVID-19 and other research using dedicated models. We further validate our methods on a large open dataset with 5644 confirmed patients from the Hospital Israelita Albert Einstein, at São Paulo, Brazil from Kaggle, and unveil leukocytes, eosinophils, and platelets as three indicative biomarkers for COVID-19.