1. Introduction

Hepatocellular carcinoma (HCC) is the sixth most common cancer and the fourth leading cause of cancer-related death worldwide [1]. Despite the eradication of Hepatitis C Virus (HCV) and the available vaccination for Hepatitis B Virus (HBV), the incidence of HCC is predicted to increase in the coming years, due to population growth, aging and the expansion of Nonalcoholic Fatty Liver Disease/ Nonalcoholic Steatohepatitis (NAFLD/NASH) in Western countries [2]. In HCC, the prediction of prognosis is complex since not only tumor burden, but also residual liver function and general clinical conditions (e.g., performance status) must be take into account. Furthermore, ongoing treatment represents one of the most important prognostic parameters [3]. Despite noteworthy improvements in screening and treatment, the recurrence and/or progression rate and the cancer-related mortality rate remains high [42]. HCC has a high recurrence rate after both surgical resection and orthotopic liver transplantation. In particular, after a liver transplantation patient show a five-year survival rate of 65-81% despite using specific criteria (Milan, USFC, Kyoto) aimed at selecting patients thought to have better long-term outcomes [4,5]. Conventional predictive models to assess the prognosis of HCC include survival analysis, logistic regression, and Cox models [6,7]. These statistical models are usually based on multivariate predictors, such as demographic, clinical, radiological, pathological, and genetic parameters. However, considered the extremely complex prognostic prediction of HCC patients, linear systems might be too simplistic [8]. Artificial intelligence (AI) is a very promising tool that may be useful in overcoming the limitations of traditional statistic and that can improve the prediction of survival and recurrence after treatment [43]. AI refers to the use of computers and related technologies to simulate the intelligent behavior and critical thinking of humans [9]. Although AI has been applied in medicine for the past 30 years, only recent studies demonstrated the ability of this tool to achieve accurate diagnosis and tumor classification, as well as the evaluation of survival and recurrence [44]. Among AI technologies, machine learning (ML) relies on the computer’s ability to “learn” and improve from past examples without being programmed [45]. Deep learning (DL) is a subset of ML and a computer software that mimics the network of neurons in a brain, where the learning phase occurs through a neural network. Since ML models can include a lot of variables, it is advantageous and can become a promising tool compared to the traditional statistical models [45]. Consequently, with the introduction of AI several computer models have been developed for the prediction of tumor characteristics or of recurrence risk, using ultrasound-based radiomic signatures [10], contrast-enhanced computed tomography (CT) [11-13] or magnetic resonance imaging (MRI) [14]. Other authors have also developed predictive models in patients undergoing radiofrequency ablation (RFA) based on Support Vector Machine (SVM) [15]. Algorithms have been created using the combination of CT images and clinical data to predict response to transarterial chemoembolization (TACE) [16], DL models using CT or ultrasound images [17,18], ML techniques by combining MRI with clinical data [19], or SVM techniques to genetic analysis [20]. To date, algorithms have been developed to predict the response and survival after TACE, stereotaxic body radiotherapy (SBRT), RFA and liver resection. In addition, AI techniques were evaluated to providing information on the predictive power of biomarkers [21-34]. The aim of this study was to create a predictive model for elaboration the recurrence of HCC after treatment, and to identify the most important clinical features associated with disease recurrence, regardless of the type of treatment, using Supervised Machine Learning techniques.

2. Materials and Methods

2.1 Cohort description

Data were retrieved from the Italian Liver Cancer (ITA.LI.CA.) database, a national registry that prospectively collect the data of patients with HCC managed in 20 Italian centers. Currently, the database includes 171 patients diagnosed with HCC between January 1998 and December 2018. Informed consent was obtained as usual for medical, surgical, and radiological treatments, but not specifically for patient data to be used in this retrospective study. Details about patient data collected for this study are described in S1.

2.2 Machine Learning Models

For the training of ML models and subsequent accuracy evaluation, the dataset was randomly divided in a training/validation set (70% of cases) and a test set (30% of cases). 5-fold cross-validation with 20 repetitions was applied. Considered that the Receiver Operating Characteristic (ROC) curve is more robust than classical accuracy, especially in the case of unbalanced outcomes [35], it was used as metric in the training process. In this study, both non-parametric models and parametric models have been evaluated. The nonparametric models trained were the following: Random Forest, Support Vector Machine and K-Nearest Neighbours. As parametric model, we used the logistic regression model with the elastic net algorithm [36]. A disparity in the frequencies of the observed classes can have a significant negative impact on the fit of the model. To solve this problem, we used the "up-sampling" method [37], which randomly samples (with replacement) the minority class (absence of relapse) to be of the same size as the majority class (presence of relapse) (Figure 1).

Therefore, the metrics used in this study are the following:

1. Balanced precision: the quality of a positive prediction made by the model. Precision refers to the number of true positives divided by the total number of positive predictions (i.e., the number of true positives plus the number of false positives).
2. Balanced recall: the ability of a model to find all the relevant cases within a data set. Recall refers to the number of true positives divided by the number of true positives plus the number of false negatives. In binary classification, recall is called sensitivity or true positive rate.
3. Balanced accuracy: the percentage of correct predictions for the test data.
4. F1 balanced metric: The F1 score is the 2*((precision*recall)/(precision+recall)). It is also called the F Score or the F Measure. The F1 score conveys the balance between precision and recall.

Having used the ROC metric in training the model, the optimal cut-off was appropriately investigated, and the performance metrics have been calculated using this cut-off (and not the default one equal to 0.5) [38].

The statistical analyses have been performed using the caret package in R, version 4.1.0 [39].

3. Results

3.1 Characteristics of the population used for Model Building

For the present study, the 171 patients managed at the Bolzano Regional Hospital from 1998 and 2019 were analyzed. After the exclusion of 5 patients without complete data, the final dataset was composed of 166 patients. Medical examinations were conducted between 1998 and 2020. The number of medical examinations for each patient varies from a minimum of 1 to a maximum of 10. The median follow-up was 520.5 (range, 23 days – 11.8 years). About 2/3 of the patients (65.7%) showed at least one recurrence during the study observation and therefore the outcome variable is unbalanced. Demographic and clinical characteristics of patients included in this study are described in table 1.

Quantitative variables were expressed as mean (SD), qualitative variables as number (%) unless otherwise specified.

Table 1: Demographic and clinical characteristics of the patients.

3.2 Selection of the Most Suitable Model Building

In general, in the ML models, it is possible to define a specific function to choose the optimal tuning parameters. By default, the algorithm chooses the tuning parameter associated with the “best” performance (in our case the largest). Moreover, it is possible to use the "one standard error" rule of Breiman et al. [4], who suggest that the tuning parameter associated with the best performance may overfit. They advise that the simplest model within one standard error of the empirically optimal model is the better choice. All the models adopted in this study have been trained both with the “best” rule and with the “one standard error” rule explained above.

The k-NN is an algorithm used in pattern recognition for the classification of objects based on the characteristics of the objects close to the one considered. It is the simplest algorithm among those used in machine learning [36]. In our study, the model selected with the “best” rule and with the “one’” rule coincide and the tuning parameter selected is equal to 1. If the tuning parameter k is equal to 1 then the object is assigned to the class of its neighbour. In this specific case, it is called "the nearest neighbour algorithm". Table 2 shows that the k-NN is inaccurate for our data (0.5 < AUC ≤ 0.7). It was expectable as it is known from the literature that this model produces not accurate results. This model was considered as a benchmark for this study.

Table 2: Machine Learning models results

According to the Random Forest (RF) model trained with the “best” rule, there seems to be overfitting. The mtry equal to 11 seems too high, loosing quite a bit in terms of model accuracy. According to this, the model trained with the “one standard error” rule seems to be faster even if something is lost in terms of test set accuracy (Table 2). In general, the model trained with the “best” rule seems a moderately accurate model in the test set (0.7

3.3 Performance of the Recurrence P of the Most Suitable Model Building

According to the obtained results, the Random Forest model seems to be the most suitable for our data (AUC value: 0.712). The variable importance analysis was assessed using the average decrease in the nodes’ impurity measured by the Gini index during the construction of the random forest model [36]. Age at diagnosis, (Model for End-Stage Liver Disease) (MELD), anamnestic obesity, type of diagnosis, BMI, and BCLC were the six most important variables for recurrence prediction found in this analysis since they are associated with the highest Gini importance. To better understand the effects of the most important variables over classification results, the partial dependence plots [36] for the top 5-predictors have been explored (Figure 2). These plots give a graphical depiction of the marginal effect of a variable on the class probability. Greater y-values indicate that observation for a specific variable is associated with a higher probability for classifying new instances as recurrence. For age at diagnosis the Partial Dependence Plot (PDP) shows an increasing probability from about 50 to 70 years and then a decreasing probability. Conversely, lower MELD score is associated with a higher probability for a relapse (Figure 2). Moreover, surveillance or a casual diagnosis of the HCC is associated with a higher relapse probability compared to symptomatic diagnosis, as well as the absence of previous obesity (Figure 2). A low BMI is associated with a higher recurrence probability. Finally, a higher recurrence probability was also demonstrated for BCLC stage B patients. The learning curve is a plot that relates the accuracy of the learning system (ROC in our case) and the number of training examples used for learning [5]. The learning curve is useful for many purposes including comparing different algorithms, choosing model parameters during design, adjusting optimization to improve convergence, and determining the amount of data used for training. The learning curve (Figure 3) shows that as the training set increases, the quality of the prediction improves (ROC). Therefore, it may make sense to increase the number of cases to improve the performance of the model and make it more generalizable on new data.

4. Discussion

Artificial intelligence and machine learning have the potential to revolutionize the management of cancer patients by predicting their clinical course and outcome. In the past two years, AI methods have been published for the prediction of overall survival and treatment outcomes in HCC. To date, however, there are few clinical prognostic factors capable of predicting HCC recurrence. For this reason, we have performed this study with the aim to establish a new ML model for predicting outcome of HCC patients. ML models are capable to include a large amount of clinical, histological, radiological as well as molecular variables compared to traditional statistical models. The most important challenge in using ML is the careful selection and validation of algorithms to create a predictive model In this study, considering a large number of variables (socio-demographic, diagnosis of HCC, patient's medical history, comorbidities, residual liver function, HCC staging at diagnosis, treatments), including patients` outcome, we developed a supervised ML model to predict HCC recurrence. Moreover, six variables that can predict the recurrence of HCC, regardless of the type of treatment, have been identified. Among the three non-parametric models (Random Forest (RF), Support Vector Machine (SVM) and K-Nearest Neighbors (KNN)) and the parametric one (ENET) adopted, the Random Forest algorithm appears to be the most suitable for predicting the risk of HCC in our data. The six most important variables predicting HCC recurrence according to this model are age at diagnosis, MELD, the absence of previous obesity, type of diagnosis, BMI, and BCLC stage. According to our model instead, the type of treatment does not appear among the most predictive parameters. In our study, the likelihood of HCC recurrence increases from age 50 to 70 and then decreases. Cho E. et al. [40] described distinct clinical features among elderly HCC patients that could explain this finding. Elderly patients with HCC had fewer HCC nodules than younger patients and less liver fibrosis [40]. Furthermore, cancers in elderly patients tend to be more encapsulated, well-differentiated, and associated with less vascular invasion [40]. However, the size of HCC nodules in elderly patients was larger than that found in younger patients most likely due to the absence of regular HCC surveillance [40]. According to the model used, patients who did not have previous obesity and were underweight have a higher probability of HCC recurrence. Although this finding may be unexpected, 88% of the population in our study developed HCC on cirrhosis, and sarcopenia is a well-known characteristic of cirrhotic patients. Chang KV et al. [41] demonstrated that muscle loss is associated with increased all-cause mortality and tumor recurrence in HCC patients. Furthermore, we found that lower MELD values are associated with a higher likelihood of recurrence while both surveillance and random diagnosis of HCC are associated with a higher likelihood of recurrence. These results could be determined by the small sample size and the characteristics of our population. Survival in patients with chronic liver disease and successfully treated HCC is mainly affected by early hepatic decompensation (cit.). Longterm preservation of liver function may allow for the emergence of HCC and therefore of initiating a specific treatment (cit.). Finally, the learning curve shows that as the training set increases, accuracy improves. This could be promising for increasing sample size and obtaining more generalizable models in an independent data set. The present study had several limitations. The analysis of the results may have been limited by the sample size. Second, the few histological data may not emerge as a prognostic factor. In conclusion, AI and the models obtained with ML, analyzing a large number of variables, could provide predictive models that help predict the risk of recurrence. This study represents a "proof of concept" (also given the large number) and it opens the way to other studies with a larger number of patients, which also allows for differentiation by type of treatment.

Figure 1: Model Building of Machine Learning

Figure 2: Partial dependence plots

Figure 3: The learning curve of the Random Forest model

References

1. International Agency for Research on Cancer. Globocan 2018. IARC (2020).
2. Sagnelli E, Macera M, Russo A, et al. Epidemiological and etiological variations in hepatocellular carcinoma. Infection 48 (2020): 7-17.
3. Llovet JM, Brú C, Bruix J. Prognosis of hepatocellular carcinoma: the BCLC staging classification. Semin Liver Dis 19 (1999): 329-338.
4. Breiman, Friedman, Olshen, et al. (1984) Classification and Regression Trees. Wadsworth.
5. Meek C, Thiesson B, Heckerman D. The Learning-Curve Sampling Method Applied to Model-Based Clustering. Journal of Machine Learning Research 2 (2002): 397.
6. Vitale A, Lai Q, Farinati F, et al. Utility of Tumor Burden Score to Stratify Prognosis of Patients with Hepatocellular Cancer: Results of 4759 Cases from ITA.LI.CA Study Group. J Gastrointest Surg 22 (2018): 859-871.
7. Lai Q, Vitale A, Halazun K, et al. Identification of an Upper Limit of Tumor Burden for Downstaging in Candidates with Hepatocellular Cancer Waiting for Liver Transplantation: A West-East Collaborative Effort. Cancers (Basel) 12 (2020): 452.
8. European Association for the Study of the Liver. European Association for the Study of the Liver. EASL Clinical Practice Guidelines: Management of hepatocellular carcinoma. J Hepatol 69 (2018): 182-236.
9. Amisha, Malik P, Pathania M, et al. Overview of artificial intelligence in medicine. J Family Med Prim Care 8 (2019): 2328-2331.
10. Dong Y, Zhou L, Xia W, et al. Preoperative Prediction of Microvascular Invasion in Hepatocellular Carcinoma: Initial Application of a Radiomic Algorithm Based on Grayscale Ultrasound Images. Front Oncol 10 (2020): 353.
11. Ma X, Wei J, Gu D, et al. Preoperative radiomics nomogram for microvascular invasion prediction in hepatocellular carcinoma using contrast-enhanced CT. Eur Radiol 29 (2019): 3595-3605.
12. Xu X, Zhang HL, Liu QP, et al. Radiomic analysis of contrast-enhanced CT predicts microvascular invasion and outcome in hepatocellular carcinoma. J Hepatol 70 (2019): 1133-1144.
13. Ji GW, Zhu FP, Xu Q, et al. Machine-learning analysis of contrast-enhanced CT radiomics predicts recurrence of hepatocellular carcinoma after resection: A multi-institutional study. EBioMedicine 50 (2019): 156-165.
14. Zhou W, Zhang L, Wang K, et al. Malignancy characterization of hepatocellular carcinomas based on texture analysis of contrastenhanced MR images. J Magn Reson Imaging 45 (2017): 1476-1484.
15. Liang JD, Ping XO, Tseng YJ, et al. Recurrence predictive models for patients with hepatocellular carcinoma after radiofrequency ablation using support vector machines with feature selection methods. Comput Methods Programs Biomed 117 (2014): 425-434.
16. Morshid A, Elsayes KM, Khalaf AM, et al. A machine learning model to predict hepatocellular carcinoma response to transcatheter arterial chemoembolization. Radiol Artif Intell (2019): e180021.
17. Peng J, Kang S, Ning Z, et al. Residual convolutional neural network for predicting response of transarterial chemoembolization in hepatocellular carcinoma from CT imaging. Eur Radiol 30 (2020): 413-424.
18. Liu D, Liu F, Xie X, et al. Accurate prediction of responses to transarterial chemoembolization for patients with hepatocellular carcinoma by using artificial intelligence in contrastenhanced ultrasound. Eur Radiol 30 (2020): 2365-2376.
19. Abajian A, Murali N, Savic LJ, et al. Predicting Treatment Response to Image-Guided Therapies Using Machine Learning: An Example for Trans-Arterial Treatment of Hepatocellular Carcinoma. J Vis Exp (2018): 58382.
20. Ziv E, Yarmohammadi H, Boas FE, et al. Gene Signature Associated with Upregulation of the Wnt/β-Catenin Signaling Pathway Predicts Tumor Response to Transarterial Embolization. J Vasc Interv Radiol 28 (2017): 349-355.
21. Mähringer-Kunz A, Wagner F, Hahn F, et al. Predicting survival after transarterial chemoembolization for hepatocellular carcinoma using a neural network: A Pilot Study. Liver Int 40 (2020): 694-703.
22. Abajian A, Murali N, Savic LJ, et al. Predicting Treatment Response to Intra-arterial Therapies for Hepatocellular Carcinoma with the Use of Supervised Machine Learning- An Artificial Intelligence Concept. J Vasc Interv Radiol 29 (2018): 850-857.
23. Morshid A, Elsayes KM, Khalaf AM, et al. A machine learning model to predict hepatocellular carcinoma response to transcatheter arterial chemoembolization. Radiol Artif Intell 1 (2019): e180021.
24. Liu D, Liu F, Xie X, et al. Accurate prediction of responses to transarterial chemoembolization for patients with hepatocellular carcinoma by using artificial intelligence in contrastenhanced ultrasound. Eur Radiol 30 (2020): 2365-2376.
25. Ibragimov B, Toesca D, Chang D, et al. Development of deep neural network for individualized hepatobiliary toxicity prediction after liver SBRT. Med Phys 45 (2018): 4763-4774.
26. Wu CF, Wu YJ, Liang PC, et al. Disease-free survival assessment by artificial neural networks for hepatocellular carcinoma patients after radiofrequency ablation. J Formos Med Assoc 116 (2017): 765-773.
27. Saillard C, Schmauch B, Laifa O, et al. Predicting Survival After Hepatocellular Carcinoma Resection Using Deep Learning on Histological Slides. Hepatology 72 (2020): 2000-2013.
28. Zhou W, Chen H, Han W, et al. Prediction of hepatocellular carcinoma patient survival using machine learning classification rules. J Clin Oncol 37 (2019): e15649.
29. Ji GW, Zhu FP, Xu Q, et al. Machine-learning analysis of contrast-enhanced CT radiomics predicts recurrence of hepatocellular carcinoma after resection: A multi-institutional study. EBioMedicine 50 (2019): 156-165.
30. Wang W, Chen Q, Iwamoto Y, et al. Deep Learning- Based Radiomics Models for Early Recurrence Prediction of Hepatocellular Carcinoma with Multi-phase CT Images and Clinical Data. Annu Int Conf IEEE Eng Med Biol Soc 11 (2019): 4881-4884.
31. Xu D, Sheng JQ, Hu PJ, et al. Predicting hepatocellular carcinoma recurrences: A data-driven multiclass classification method incorporating latent variables. J Biomed Inform 96 (2019): 103237.
32. Feng ST, Jia Y, Liao B, et al. Preoperative prediction of microvascular invasion in hepatocellular cancer: a radiomics model using Gd-EOB-DTPA-enhanced MRI. Eur Radiol 29 (2019): 4648-4659.
33. Tsilimigras DI, Mehta R, Moris D, et al. Utilizing Machine Learning for Pre- and Postoperative Assessment of Patients Undergoing Resection for BCLC-0, A and B Hepatocellular Carcinoma: Implications for Resection Beyond the BCLC Guidelines. Ann Surg Oncol 27 (2020): 866-874.
34. Guo D, Gu D, Wang H, et al. Radiomics analysis enables recurrence prediction for hepatocellular carcinoma after liver transplantation. Eur J Radiol 117 (2019): 33-40.
35. Branco P, Torgo L, Ribeiro RP. A Survey of Predictive Modelling under Imbalanced Distributions. ArXiv (2015).
36. Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning. Data Mining, Inference, and Prediction. Second edition, Springer Series in Statistics (2008).
37. Max Kuhn. caret: Classification and Regression Training. R package version 6.0- 86 (2020).
38. Habibzadeh F, Habibzadeh P, Yadollahie M. On determining the most appropriate test cut-off value: the case of tests with continuous results. Biochem Med (Zagreb) 26 (2016): 297-307.
39. R Core Team. A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria (2021).
40. Cho E, Cho HA, Jun CH, et al. A Review of Hepatocellular Carcinoma in Elderly Patients Focused on Management and Outcomes. In Vivo (2019): 1411-1420.
41. Chang KV, Chen JD, Wu WT, et al. Association between Loss of Skeletal Muscle Mass and Mortality and Tumor Recurrence in Hepatocellular Carcinoma: A Systematic Review and Meta-Analysis. Liver Cancer 7 (2018): 90- 103.
42. Llovet JM, Zucman-Rossi J, Pikarsky E, et al. Hepatocellular carcinoma. Nat Rev Dis Primers 14 (2016): 16018.
43. Rajula HSR, Verlato G, Manchia M, et al. Comparison of Conventional Statistical Methods with Machine Learning in Medicine: Diagnosis, Drug Development, and Treatment. Medicina (Kaunas) 56 (2020): 455.
44. Huang S, Yang J, Fong S, et al. Artificial intelligence in cancer diagnosis and prognosis: Opportunities and challenges. Cancer Lett 471 (2020): 61-71.
45. Cui S, Tseng HH, Pakela J, Ten Haken RK, El Naqa I. Introduction to machine and deep learning for medical physicists. Med Phys 47 (2020): e127-e147.