Implementation of Novel Methods, Markers or Sample types in Molecular Pathology – Quality Assurance Aspects
Article Information
Cleo Keppens, Jan De Smedt, Elisabeth MC Dequeker*
Department of Public Health and Primary Care, Biomedical Quality Assurance Research Unit, University of Leuven, 3000 Leuven, Belgium
*Corresponding author: Prof. Dr. Elisabeth MC Dequeker, Department of Public Health and Primary Care, Biomedical Quality Assurance Research Unit, University of Leuven, Kapucijnenvoer 35d, Box 7001, 3000 Leuven, Belgium
Received: 31 May 2020; Accepted: 09 June 2020; Published: 23 July 2020
Citation: Cleo Keppens, Jan De Smedt, Elisabeth MC Dequeker. Implementation of Novel Methods, Markers or Sample types in Molecular Pathology – Quality Assurance Aspects. Archives of Clinical and Biomedical Research 4 (2020): 302-324.
Share at FacebookAbstract
Molecular pathology is continuously evolving and laboratories are challenged to implement tests accurately prior to administration of targeted therapies. External quality assessment (EQA) programs revealed method-specific problems for laboratories who switched methods, and a good adherence to guidelines during method validation/verification in the USA. This study evaluated current guideline adherence in Europe and experienced hurdles during test, marker or sample implementation.
EQA participants from the European Society of Pathology were invited to complete an electronic survey if they: (i) recently changed their assay, (ii) implemented PD-L1 analysis, (iii) or introduced analysis of circulating tumor DNA.
In total, survey data from 54 laboratories was analyzed. The majority of laboratories implemented a written procedure for validation (68.5%) or verification (59.3%), in 53.7% and 44.4% cases based on standards or available literature. For 20.4% of respondents, a specific guideline was not available yet for their test strategy. Method revalidations and staff training, as well as EQA participation were frequently performed. Most reported hurdles included controlling of pre-analytical variables (87.0%), finding appropriate controls for rare mutations or antigens, and for varying positivity ranges/frequency (56.5%). In the post-analytical phase, interpretation of complex bio-informatics was a main concern (70.0%), while staff limitations, increased costs and workload (53.1%-57.1%) were barriers affecting the entire test process.
Documentation of procedures and guideline adherence was higher but not limited to accredited institutes. The data stressed the importance of further quality efforts to aid laboratories with controlling pre-analytical variables, the selection of appropriate controls, and test interpretation of complex data.
Keywords
Validation; Verification; ISO 15189; Molecular pathology; Biomarker; External quality assessment
Validation articles, Verification articles, ISO 15189 articles, Molecular pathology articles, Biomarker articles, External quality assessment articles
Article Details
Abbreviations
ALK, ALK receptor tyrosine kinase; CAP, College of American Pathologists; CDx, companion diagnostic; CE-IVD, European Conformity In Vitro Diagnostic; ctDNA, circulating tumor DNA; DNA, deoxyribonucleic acid; EGFR, Epidermal growth factor receptor; EMA, European Medicines Agency; EQA, external quality assessment; ESP, European Society of Pathology; FDA, Food and Drug Administration; FFPE, formalin-fixed paraffin embedded; FISH, fluorescence in situ hybridization; ICI, immune checkpoint inhibitor; IHC, immunohistochemistry; IQN Path, International Quality Network for Pathology; ISO, International Organization for Standardization; KRAS, KRAS proto-oncogene GTPase; LDT, laboratory-developed test; mCRC, metastatic colorectal cancer; NGS, next-generation sequencing; NRAS, NRAS proto-oncogene GTPase; NSCLC, non-small cell lung cancer; PD-L1, programmed death ligand 1; RNA, ribonucleic acid; ROS1, ROS proto-oncogene 1 receptor tyrosine kinase; TKI, tyrosine kinase inhibitor; TTP, total test process; USA, United States of America.
Introduction
In molecular pathology, testing of several predictive biomarkers is required by the Food and Drug Administration (FDA) and European Medicines Agency (EMA) for decision making prior to the administration of targeted therapies [1, 2]. Examples include the analysis of ALK receptor tyrosine kinase (ALK) and ROS proto-oncogene 1 receptor tyrosine kinase (ROS1) rearrangements by fluorescence in situ hybridization (FISH) or expression by immunohistochemistry (IHC), as well as mutation analysis of the epidermal growth factor receptor (EGFR) gene to predict the response likelihood of non-small cell lung cancer (NSCLC) patients to specific tyrosine-kinase inhibitors (TKIs) [3]. In metastatic colorectal cancer (mCRC), mutation analysis of exons 2 to 4 of the KRAS proto-oncogene GTPase (KRAS) and NRAS proto-oncogene GTPase (NRAS) should be performed to assess patients’ eligibility to anti-EGFR therapy [4]. Recent advances in molecular oncology have identified other molecular targets of which several hold prognostic value. Approved therapies are not yet available for all of these markers and recommendations for testing may differ between countries or individual laboratories depending on reimbursement status or assay availability [5].
Besides TKIs, immune checkpoint inhibitors (ICIs) along with the detection of programmed death ligand 1 (PD-L1) expression as a predictive biomarker were recently approved, as well as different detection platforms related to these ICIs with varying cut-offs of positivity [6]. Also, while analysis of formalin-fixed paraffin-embedded (FFPE) tissue remains the gold standard, testing of circulating tumor DNA (ctDNA) in plasma was approved by the FDA and EMA to detect EGFR mutations in clinical settings when tissue is limited, or to identify inactivating EGFR mutations in NSCLC patients with progression or secondary clinical resistance to TKIs [7, 8]. An international study in 2016 revealed that 90% (151/167) of laboratories have implemented ctDNA testing for EGFR, KRAS and NRAS, of which 37% for diagnostic purposes [9].
Correct test results and clear interpretation are critical given the implications for treatment [10] and should thus undergo complete validation or verification for several performance characteristics before implementing a clinical test in practice. Laboratories testing these markers can use a commercially available test, which can be CE-IVD labeled or FDA-approved as a companion diagnostic (CDx) test [11, 12]. Such tests should undergo verification of the performance characteristics previously validated by the manufacturer. In contrast, laboratory-developed tests (LDTs), or modified commercially available tests should be more extensively validated [13]. In addition, the laboratory should use quality control materials (cell line DNA, or internal and external reference control materials) for validation or verification as negative and positive controls, that react to the examining system in a manner as close as possible to patient samples, especially at or near clinical decision values [13]. Accordingly, molecular tests should be validated for each of the specimen types likely to be encountered, and testing should be performed and reported only on validated specimen types [14]. The validation or verification procedures should be well documented and adequate training should be foreseen to enable smooth introduction and validation of new tests [13, 15, 16].
Several guidelines and standards have been published on the validation and verification of diagnostic assays [17-25], which can be part of the requirements within the framework of a laboratories’ accreditation according to the International Organization for Standardization (ISO) 15189 [13, 26, 27]. Participation to external quality assessment (EQA) programs can further aid laboratories in validating their test methods, by monitoring and comparing their performances to international peers and receiving individual feedback [10]. EQA schemes organized by the European Society of Pathology (ESP) have revealed increased error rates upon the implementation of novel markers, both for NSCLC and mCRC [28-31]. Error rates also varied between the technique type used, and were higher for IHC compared to FISH [28], but lower for next-generation sequencing (NGS) based techniques compared to other mutation detection assays [28, 29, 31]. More errors were also reported for methods that do not include all required variants for testing by the FDA and EMA, stressing the importance of careful method validation [30]. A further follow-up on underlying error causes revealed that 20.4% of problems were method-related problems, arising when laboratories switched to novel or more complex assays such as NGS [32]. However, it remains unclear if specific issues during validation or verification are at the basis. A study by the College of American Pathologists (CAP) including 1085 EQA participants both in the USA and in Europe revealed that 96% of laboratories (n=1085) had written validation procedures (of which 43% for LDTs) for IHC and specifications for the minimum number of cases tested [33]. These numbers as well as guideline adoption increased significantly in a follow-up study after dissemination of an evidence-based guideline in 2014 [20, 34]. The aim of this study was to provide an overview of current guideline adherence in European laboratories and to assess if laboratories experience specific hurdles during the implementation of novel tests, markers or sample types in the total test process (TTP) and related to the availability of control samples.
Material & Methods
The ESP organizes yearly EQA schemes for testing of common biomarkers in NSCLC and mCRC, in collaboration with the Biomedical Quality Assurance Research Unit of KU Leuven as an ISO 17043-accredited EQA provider [35]. The schemes are open to all laboratories worldwide, and their detailed set-up has been previously described [28, 30]. Participating laboratories had to enter their method of analysis for testing the offered biomarkers, along with their analysis results for the provided samples, in an electronic datasheet. Laboratories who participated to the 2017 ESP NSCLC and mCRC schemes were invited to complete the survey if they changed their test protocol or assay between the 2016 and 2017 schemes, or between one of the three distribution rounds in the 2017 NSCLC EQA scheme, for any of the offered markers. In 2017 testing of PD-L1 expression by IHC was added as a novel marker to the schemes, and laboratories participating in the PD-L1 IHC subscheme thereafter (in 2017 and 2018) were additionally invited. All laboratories had the opportunity to access their previously entered data on the EQA website after login to their participant’s area.
Besides the schemes on FFPE material, a first pilot EQA scheme was organized in 2017 to assess the ability of laboratories to detect common EGFR and KRAS mutations in ctDNA from plasma samples. This scheme was run by four international EQA providers, including ESP, under the umbrella organization International Quality Network for Pathology (IQN Path). A subgroup of 32 participants was selected based on a prior survey on the implementation of ctDNA analysis in routine practice. Results of the pilot EQA scheme and implementation survey have previously been reported [9, 36]. All 32 participants to the pilot scheme (i), laboratories who replied to the survey on the ctDNA implementation (ii) [9], and laboratories who registered to a follow-up ctDNA EQA scheme run in 2018 by ESP (iii), were invited to complete the survey. Additionally, data from nine other institutes was added who were invited to participate as part of a master thesis project. These institutes consisted of eight laboratories from Italy and one from Denmark, who were not ISO 15189 accredited, and took part in either the ESP EQA schemes or Italian national EQA schemes [37].
The selected laboratories were invited via e-mail to complete an electronic survey created in FormDesk (Innovero Software Solutions B.V., the Netherlands) in March 2018, as part of additional quality research outside the scope of the ESP or IQN Path schemes. Laboratories received a first reminder after two weeks and a second reminder after one month. Invitees were provided with an accompanying letter explaining the survey goal, estimated timeframe, and laboratory inclusion criteria. The nine additional institutes were invited by telephone in December 2018 to complete an identical survey drafted in SurveyMonkey (SurveyMonkey, CA, USA). Terms describing the laboratory setting were previously conceptualized [38]. Data on the institute setting and accreditation was validated afterwards on the websites of the laboratories and their relevant accreditation/certification bodies. Data is represented on a descriptive level as reported by the survey respondents, and missing data is indicated in case of occurrence. This survey did not aim to evaluate different elements included in the validation reports (such as sensitivity, specificity, accuracy, or precision) but focuses on general practices and problems during the implementation of novel methods, markers (PD-L1) or sample types (ctDNA).
Results
Survey Respondents
In total, the survey was sent to 361 unique laboratories, of which 120 and 39 laboratories because they switched their analysis method in the 2017 ESP NSCLC and mCRC EQA schemes, respectively. Seventy-six and 81 laboratories were invited as they took part in the 2017 and 2018 PD-L1 IHC subscheme of the ESP NSCLC EQA scheme. Sixty-five laboratories enrolled for the 2018 IQN Path ctDNA scheme, and 175 completed the previous survey on ctDNA implementation. Of those 361 institutes, 45 unique laboratories responded. Their data was supplemented with survey responses from nine additional institutes monitored during a master thesis internship, resulting in 54 laboratories from 22 different countries for further analysis (Supplemental Table 1).
Of the 54 respondents, 40 introduced ctDNA analysis, 23 laboratories recently introduced PD-L1 IHC as a predictive biomarker, and 22 laboratories recently changed their test protocol. Twenty-three laboratories reported a combination of two or more elements to be relevant for their current situation. The majority of respondents were affiliated to a university (18/54) or general hospital (16/54), or were private laboratories (10/54) and specialized anti-cancer centers (10/54). Twenty of 54 laboratories had a specific laboratory accreditation conform ISO 15189 [13]. The other 34 were not accredited conform the ISO 15189 standard, but had a general hospital accreditation, ISO 9001 certification [39] or ISO 15189 accreditation was underway (Table 1).
Number of respondents |
% of respondents (N=54) |
|
Laboratory setting |
||
University hospital or laboratory |
18 |
33,3 |
(Community) hospital |
16 |
29,6 |
Private hospital or laboratory |
10 |
18,5 |
Anti-cancer Center |
10 |
18,5 |
Accreditation status |
||
ISO 15189 laboratory accreditation |
20 |
37,0 |
No ISO 15189 accreditation |
34 |
63,0 |
ISO 15189 underway [13] |
3 |
5,6 |
ISO 9001 certification [39] |
16 |
29,6 |
Hospital accreditation |
2 |
3,7 |
None of the above |
13 |
24,1 |
Survey inclusion criteria* |
||
Implementation of ctDNA analysis |
40 |
74,1 |
CE-IVD commercial kit [11] |
13 |
24,1 |
Commercial NGS |
9 |
16,7 |
LDT NGS |
1 |
1,9 |
LDT ddPCR |
6 |
11,1 |
Missing data |
11 |
20,4 |
Introduction of PD-L1 marker |
23 |
42,6 |
CE-IVD commercial kit [11] |
10 |
18,5 |
LDT kit |
8 |
14,8 |
Missing data |
5 |
9,3 |
Recent change in test protocol* |
22 |
40,7 |
Mutation analysis: switch in NGS platform |
3 |
5,6 |
Mutation analysis: switch to another commercial kit |
2 |
3,7 |
Mutation analysis: switch to another non-commercial sequencing method |
1 |
1,9 |
Mutation analysis: switch from a commercial kit to NGS |
3 |
5,6 |
Mutation analysis: New DNA extraction method |
1 |
1,9 |
FISH: change of probe/protocol |
3 |
5,6 |
IHC: change of antibody/protocol |
3 |
5,6 |
Missing data |
13 |
24,1 |
*Multiple answer options were possible which is why percentages add up to more than 100%. Abbreviations: CE-IVD, European Conformity In Vitro Diagnostic [11]; ctDNA, circulating tumor DNA; ddPCR, droplet digital PCR; DNA, deoxyribonucleic acid; FISH, fluorescence in situ hybridization; IHC, immunohistochemistry; ISO, International Organization for Standardization; LDT, laboratory-developed test; NGS, next-generation sequencing; PD-L1, programmed death ligand 1. |
Table 1: Characteristics of survey respondents
Laboratories’ Strategies for Validation and Verification
The majority of laboratories has a written procedure implemented in the laboratory that specifies the procedure for validation (68.5%) and verification (59.3%) (Table 2). For the procedure on validation, 29/54 laboratories incorporated recommendations specified by their accreditation or certification standard or additional recommendations available in literature. For verification, 24 laboratories adhered to one or more standards and guidelines to base their written procedure on. The applied standards and publications reported by the survey respondents are presented in Supplemental Table 2. For validation and verification, 8/29 and 4/24 laboratories reported to only adopt several elements, for instance on the number of samples to be tested, on pre-analytical factors, or on the reportable range and quality metrics for NGS (data not shown). Laboratories frequently reported participation to EQA schemes (74.1%), comparison of the new method compared to a previously validated method (64.8%), and determination of intra- and inter-run variability (each 53.7%) as main aspects to be included in the procedure (Table 2). Other elements included inter-personal tuning, determination of the limit of detection and linear regression analyses (data not shown). Almost all respondents (51/54) indicated to train their personnel to perform the validation or verification, of which 39 also retrained their personnel when changes were implemented.
Number of respondents validation/verification |
% of respondents (N=54) |
|
Do you have a written procedure that outlines the validation/verification procedure for new assays, markers, or sample types? |
||
Yes |
37/32 |
68,5/59,3 |
No |
9/9 |
16,7/16,7 |
I don't know |
1/1 |
1,9/1,9 |
Not applicable because not performed |
5/10 |
9,3/18,5 |
Missing data |
2/2 |
3,7/3,7 |
Do you adhere to a specific guideline or standard for validation/verification? |
||
No |
12/16 |
22,2/29,6 |
No, but we plan to adopt one |
2/3 |
3,7/5,6 |
No, because not available yet |
11/11 |
20,4/20,4 |
Yes, our accreditation standard |
7/8 |
13,0/14,8 |
Yes, accreditation standard in combination with recommendations in literature |
4/3 |
7,4/5,6 |
Yes, recommendations in literature |
18/13 |
33,3/24,1 |
If yes: we adopt all recommendations |
21/20 |
38,9/37,0 |
If yes: we used the guideline as a basis but included the laboratory's own interpretation for several items |
8/4 |
14,8/7,4 |
Which of the following actions did your perform during the validation or verification?* |
||
Participated to external quality assessment scheme |
40 |
74,1 |
Compared to a previously validated test |
35 |
64,8 |
Tested the sample on multiple days to assess inter-run precision |
29 |
53,7 |
Tested the sample in the same experiment to assess intra-run precision |
29 |
53,7 |
Compared to the morphology and expected results |
20 |
37,0 |
Running parallel tests on the same case at another laboratory |
16 |
29,6 |
Other |
3 |
5,6 |
Missing data |
9 |
16,7 |
How was the personnel trained to perform these actions?* |
||
Internal: learning from colleagues with gradually more independence |
37 |
68,5 |
Internal: performing validations |
34 |
63,0 |
Internal: participate to laboratory meetings |
24 |
44,4 |
External: workshops, courses |
22 |
40,7 |
Depends on the type of action |
11 |
20,4 |
No specific training is required |
3 |
5,6 |
Manufacturer-based training |
2 |
3,7 |
Does the written procedure requires a new validation or verification? |
||
No |
9 |
16,7 |
Yes |
45 |
83,3 |
A change in instrumentation |
35 |
64,8 |
Change of reagent / antibody / primer |
33 |
61,1 |
Small changes in protocol, e.g. change in incubation time |
24 |
44,4 |
Introduction of a new reagent lot |
16 |
29,6 |
Change in fixative (not applicable for liquid biopsy) |
13 |
24,1 |
Change in environmental conditions |
8 |
14,8 |
Only for certain assays |
9 |
16,7 |
After a fixed amount of time |
2 |
3,7 |
Only for certain markers |
1 |
1,9 |
In case of problems |
1 |
1,9 |
When a new assay, sample type, strategy, or marker is implemented in your laboratory, is the personnel retrained? |
||
Yes |
39 |
72,2 |
Only for new methods |
4 |
7,4 |
No |
4 |
7,4 |
Missing data |
7 |
13,0 |
*Multiple options were possible which is why percentages add up to more than 100%. |
Table 2: Survey answers regarding practices for validation/verification
Sample Selection and Reference Materials
In total, 33/54 respondents mentioned to specify a minimum number of positive and negative samples to validate or verify their assay in general (Table 3). Of those, 27 laboratories provided the exact number. On average, the 27 laboratories included 12 positive cases (min. 1, max. 50, median 10), and 10 negative samples (min. 1, max. 50, median 8). The distribution of the amount of cases is shown in Figure 1. Cases were mainly selected based on the requirements by the assay manufacturer, or on the minimum number specified in relevant literature (each 32/54 respondents, 59.3%). Another 53.7% of respondents always includes borderline cases as part of the sample validation set (Table 3). The majority of samples originate from the laboratories’ own biobank (77.8%) or from tissue exchange with other laboratories (53.7%), and to a lesser extent from commercially purchased reference materials (46.3%). The suppliers reported included Horizon Discovery (22 laboratories), Life Technologies (4 laboratories), SeraCare (3 laboratories), Histocyte and American Type Culture Collection (ATCC, each 2 laboratories), Coriell institute, Invivoscribe, Qnostics and Ventana (all reported by 1 laboratory) (data not shown). Twenty-five laboratories thus included commercial reference materials. Ten of them indicated to experience unmet needs concerning the availability or costs of FFPE tissues, specific fusions, and mutations.
Number of respondents |
% of respondents (N=54) |
|
Does your written procedure specify a minimum number of cases to be included in the validation or verification? |
||
No |
21 |
38,9 |
Yes |
22 |
40,7 |
Yes, for validation but not for verification |
7 |
13,0 |
Yes, number depends on the assay |
2 |
3,7 |
Yes, depending on the prevalence in cancer population |
2 |
3,7 |
How are cases selected as appropriate positive and negative cases for the purpose of validation or verification?* |
||
Based on the assay manufacturer’s recommendation |
32 |
59,3 |
Based on literature review |
32 |
59,3 |
Based on cases available from another laboratory with an established procedure |
29 |
53,7 |
In-house cases available and tested by another validated assay |
4 |
7,4 |
Differs depending if it is the procedure for validation or verification |
3 |
5,6 |
Are borderline positive cases (weakly or focally positive staining, mutation frequency around limit of detection) included in the sample set? |
||
Yes, always |
29 |
53,7 |
Depends on the marker/variant or technique |
4 |
7,4 |
Depending on tissue availability |
3 |
5,6 |
No |
18 |
33,3 |
What kind of material is included?* |
||
Samples from our own biobank |
42 |
77,8 |
Tissue exchanged with other laboratories |
29 |
53,7 |
Commercially available reference material |
25 |
46,3 |
Differs depending if it’s the procedure for validation or verification |
4 |
7,4 |
Which references materials are you interested in?* |
||
FFPE – catalogue available |
33 |
61,1 |
FFPE – custom created |
25 |
46,3 |
ctDNA – catalogue available |
28 |
51,9 |
ctDNA – custom created |
19 |
35,2 |
Genomic DNA - catalogue available |
22 |
40,7 |
Genomic DNA - custom created |
13 |
24,1 |
RNA - catalogue available |
15 |
27,8 |
RNA - custom created |
10 |
18,5 |
In case you purchase commercial reference materials, are there any unmet needs concerning the available material? |
||
No |
13 |
24,1 |
Yes, unspecified |
1 |
1,9 |
Not all mutations (i.e. rare variants) are available |
1 |
1,9 |
An own composition of control material (custom control material) is needed |
1 |
1,9 |
FFPE materials with fusions for RNA-based assays |
1 |
1,9 |
FFPE material more suitable for DNA extraction |
1 |
1,9 |
Less expensive reference materials |
1 |
1,9 |
Provision of more FFPE material |
2 |
3,7 |
Missing data |
4 |
7,4 |
*Multiple answer options were possible which is why percentages add up to more than 100%. Abbreviations: ctDNA, circulating tumor DNA; DNA, deoxyribonucleic acid; FFPE, formalin-fixed paraffin embedded; RNA, ribonucleic acid. |
Table 3: Case selection for validation or verification
Hurdles During Test Implementation
Overall, 42.6% respondents indicated to encounter hurdles in the pre-analytical and analytical steps of the TTP, 37.0% in the post-analytical step, and 90.7% reported general hurdles that might affect the entire TTP (Table 4). In more detail, pre-analytical elements to be most difficult in the experience of the respondents were controlling several pre-analytical variables (87.0%) and finding an appropriate number of cases for rare mutations or antigens (56.5%). On analytical level, the selection of adequate positive and negative controls as well as of the most appropriate assay for testing the new markers or sample types were the main sources of experienced problems (56.5% and 52.2%, respectively). Analysis of bio-informatics, for instance for NGS, was the main cause in the post-analytical phase. Finally, on a general level many institutes still report staff limitations, additional workload and increased costs to be at the basis of encountered hurdles during test implementation.
Number of respondents |
% of respondents (N=54) |
|
Pre-analytical* |
23 |
42,6 |
Controlling pre-analytical variables (fixation, sample storage, centrifugation, …) |
20 |
87,0 |
Finding an appropriate number of cases for rare mutations/antigens |
13 |
56,5 |
Identification of sample quality |
6 |
26,1 |
Definition of the medical rationale (diagnostic, therapeutic, prognostic) |
5 |
21,7 |
Determination of the appropriate sample type (fresh frozen, FFPE, blood,..) |
3 |
13,0 |
Identification of the patient population |
2 |
8,7 |
Determination of the expected clinical/technical performance |
2 |
8,7 |
Finding the necessary information before the change (e.g. possible methodologies) |
2 |
8,7 |
Analytical* |
23 |
42,6 |
Finding appropriate positive and negative control cases |
13 |
56,5 |
Selection of the most appropriate method for testing of the new marker/sample type |
12 |
52,2 |
Incorrect outcome with unknown cause – commercial black box method |
11 |
47,8 |
Validation of different ranges of positivity of cases (% positive cells for IHC, % signals for FISH) |
11 |
47,8 |
Validation of different variant types (indels, insertions, deletions, complex variants) |
10 |
43,5 |
Confirmation of expected sensitivity/specificity/coverage |
9 |
39,1 |
Post-analytical* |
20 |
37,0 |
Analysis of bioinformatics (e.g. NGS) |
14 |
70,0 |
Correlation with clinical context |
9 |
45,0 |
Getting familiar with the interpretation of the test outcome |
7 |
35,0 |
Implementing an acceptable turnaround time |
6 |
30,0 |
Elements that need to be included in the report |
5 |
25,0 |
Monitor changes in test conditions |
2 |
10,0 |
Total test process* |
49 |
90,7 |
Staff limitations |
28 |
57,1 |
Additional costs/expenses |
27 |
55,1 |
Increasing workload |
26 |
53,1 |
Reimbursement issues |
23 |
46,9 |
Time constraints |
15 |
30,6 |
General implementation in the quality management system (creation of standard operating procedures) |
14 |
28,6 |
Organizational/institutional/system barriers that prevent me from making a change |
13 |
26,5 |
Lack of resources (equipment) |
12 |
24,5 |
Keeping thorough documentation |
9 |
18,4 |
Training of/communication to laboratory personnel |
4 |
8,2 |
Inter-professional and intra-professional team work |
1 |
2,0 |
*Multiple answer options were possible which is why percentages add up to more than 100%. The test phases were defined to the participants as: Pre-analytical phase: sample reception, sample identification, sample preparation: deparaffinization, hematoxylin and eosin staining, micro/macrodissection, pathologist review (selection of neoplastic cell region and scraping of the tissue). Analytical phase: DNA extraction, library preparation, mutation analysis, gene rearrangement/protein expression analysis according to pre-determined protocol. Post-analytical phase: readout of the analytical results, scoring of the samples according to predetermined criteria, clinical interpretation, communication of results, entering of the results in the laboratory information system/EQA datasheet, drafting of the written reports. Abbreviations: DNA, deoxyribonucleic acid; EQA, external quality assessment; FFPE, formalin-fixed paraffin embedded; FISH, fluorescence in situ hybridization; IHC, immunohistochemistry; NGS, next-generation sequencing. |
Table 4: Hurdles reported during the implementation of novel markers, sample types or assays in the different test phases
Discussion
Molecular pathology guides targeted treatments tailored to the cancer’s specific molecular features. The number of predictive molecular targets is rising vastly, and laboratories are challenged to implement and maintain accurate test procedures and to offer reliable results within an acceptable timeframe [10]. When implementing new tests, laboratories should decide on (i) the best test strategy to use for the intended purpose, (ii) the availability of a suitable biological starting material (pre-analytical phase requirements) and the choice of internal quality control materials, (iii) and participation to international EQA programs [40].
Laboratories’ Strategies for Validation and Verification
Our findings confirm the previous study performed by CAP [33, 34] as the majority of survey respondents implemented a written procedure for validation or verification practices (Table 2). The fact that 95.9% of CAP laboratories implemented a procedure compared to 59.3%-68.5% in this study might be explained by several factors. Firstly, laboratory accreditation status was not considered in the CAP survey or analysis of the results. Even though CAP accreditation is mandatory in the US, 19% of previous study results were obtained from laboratories not enrolled in the CAP EQA program, and 11% were laboratories outside the USA [33, 34]. In this study, 62.5% and 53.1% of non-accredited laboratories had a documented procedure for validation and verification respectively, compared to 85.0% and 70.0% of accredited laboratories. This seems logical, given that documenting the procedures for implementing new tests is a requirement by ISO 15189 [13]. Laboratory accreditation has previously been advised [10], and was correlated to higher EQA performances when implementing novel markers in the laboratory [38]. The increased implementation of written procedures in this study might be a possible explanation for the better performance by accredited institutes, although more data is needed to make solid assumptions. It must be noted that accreditation for molecular pathology tests is not mandatory in all countries [41]. For instance, ISO 15189 accreditation is mandatory in Belgium and France, while Italian laboratories adhere to ISO 9001 certification, as exemplified in Supplemental Table 1.
Secondly, the previous study only questioned IHC methods, while laboratories in this study were also selected based on the implementation of ctDNA analysis, or a recent change (within 12 months) of their test methods for FISH or mutation analysis. According to ISO 15189, preferred procedures for validation or verification are those specified in the instructions for use or those published in peer-reviewed journals, consensus standards or guidelines, or national or regional regulations [13]. While the majority of respondents indeed based their procedure on their applicable standard or international publications, 20.4% of respondents indicated that guidelines were not yet available (Table 3), in 9/11 cases for ctDNA analysis. It has previously been reported that the extent of evidence-based guideline adoption is improved if the evidence is strong and the guidelines are clear and supported or disseminated by professional societies [42]. For ctDNA analysis, there remains a need for standardization in pre-analytical sample processing and applied assays. Well-designed guidelines for validation were thus lacking at the time of survey distribution. Over time, progress has been made as several recommendations on pre-analytical conditions were published or are in the development phase, including by European standardization organizations [43-45]. In addition for IHC especially PD-L1 validation has been reported to be difficult, even though several guidelines are available, as many assays exist with varying positivity cut-offs for different therapies [46]. A lack of guidelines has previously also been reported to be at the basis of increased interpretation errors for IHC tests for ROS1 and PD-L1 at the time of implementation [28, 32].
When changes are made to a validated examination procedure, the influence of such changes should be documented and a new validation should be carried out when appropriate [13, 25]. The most common reasons in Table 2 confirm the previously reported results [33]. However, only 67.5% of non-accredited laboratories revalidated their assays compared to 95.0% of accredited institutes. This might further explain previous observations where accredited laboratories had a better performance when implementing new tests [38] and encountered fewer reagent problems possibly due to entry validation of new reagent lots [32].
Laboratories reported to frequently participate to EQA schemes to validate their methods, and train their personnel to perform the actions associated with validation or verification. Frequent participation to EQA as well as training in the form of workshops have previously been reported to exert a beneficial effect on test performance [28, 29, 31, 47].
Sample Selection and Reference Materials
In addition to guideline adoption, laboratories are also including a number of cases as specified in the manufacturer’s instructions for their commercial test method or in relevant literature. It must be noted that at the time of survey distribution, IVD classification was regulated by the European IVD Directive (98/79/EC) [11]. In May 2017, a novel European IVD Regulation (2017/746) entered into force and will apply from May 26th, 2022 [48]. The change from a directive to a regulation will have a major impact on the IVD industry as well as on institutes using LDTs (meaning all in-house developed tests, and off-label modifications to or combinations of CE-IVD approved tests). Namely, many devices falling under the scope of the IVD regulation will require new or re-certification.
The number of cases was not always documented in the written procedure by all respondents (Table 3). While the CAP guideline recommends confirming predictive marker performance with 40 validation cases [20], the numbers used by the laboratories in practice may depend on the technique (FISH, IHC or mutation analysis), the marker (predictive or non-predictive) and whether validation or verification was performed. This resulted in a variable number of cases reported (Figure 1). The majority of respondents indeed included borderline cases, which is especially important for the several reasons. First, previous EQA results demonstrated increased errors for cases with multiple and low frequency variants in NSCLC [29]. Such cases often occur in routine molecular oncology, due to the occurrence of resistance mechanisms upon progression to TKIs. Second, recent analysis in mCRC stressed that not all laboratories have well implemented rare RAS variants in exons 3 and 4, resulting in higher error rates [30, 31]. Third, the analysis of underlying causes of incorrect tests revealed that laboratories mentioned insufficient method sensitivities or omission of rare variants in their kit to be at basis of the false negative results [32]. Fourth, one of the main hurdles reported in the pre-analytical phase was finding an appropriate number of cases for rare mutations/antigens in this study (Table 4). Previously it was shown that this hurdle was correlated to laboratories testing higher sample volumes [34]. Once a test validation is completed this is not the end of performance evaluation, but the accumulation of data over time is an important additional component of the initial validation to continually improve the assessment of test accuracy and quality. Such ongoing or dynamic validation should include results of EQA, internal quality control, and nonconformities related to the test [25]. In this way, EQA could guide laboratories by further validating their test methods for challenging cases with rare variants that are not always encountered in a routine setting on a day-to-day basis.
Hurdles During Test Implementation
While few laboratories mentioned unmet needs for commercial reference materials, one of the main issues reported in the analytical phase was the finding of appropriate negative and positive control samples, and validation of different ranges of positivity of cases (Table 4). The search of appropriate positive and negative controls for IHC to evaluate the sensitivity and specificity of the test is often a challenge to laboratories. Control material has previously been reported to be highly variable and often suboptimal in EQA schemes specifically designed to assess IHC staining quality [49]. In this case, participation to EQA schemes that provide feedback on both the staining quality as well as interpretation of the final test outcome could further enhance performance.
One other hurdle that was frequently reported is the selection of the appropriate test method. Besides the inclusion of all required markers and variants, other factors such as costs, turnaround time, and complexity should be considered. For instance, even though several commercial kits for mutation analysis have the advantage that they include a positive and negative control, they only report the test outcome and users have limited insight into troubleshooting without contacting the manufacturer (Table 4). Over time however, EQA schemes have reported on an increased use of NGS [28, 29, 31]. In the meantime, NGS-specific EQAs have been organized and guidelines for NGS implementation were established [22-24, 50, 51]. The implementation of multi-parallel techniques brings along additional challenges. As such, one of the main issues reported in the survey was the analysis of complex bio-informatics data. This is a well-described issue. Also, it is expected that laboratories might outsource this part, resulting in additional concerns on how to ascertain the quality of the TTP [10]. Laboratories are also faced with difficulties on how to report variants of unknown significance for the various markers tested, for which some institutes have already established a molecular tumor board to guide therapy decision making.
The selected method of analysis should also be carefully validated for the pre-analytical conditions, which is another barrier reported during test implementation. Besides ctDNA analysis, pre-analytical conditions in FFPE are also essential to the test outcome, as the fixation type and duration, decalcifying reagents, instrumentation for tissue processing, and estimation of the minimum number and area of neoplastic cells in relation to the assay sensitivity, have all been reported to affect molecular pathology tests [19]. Indeed, laboratories who reported to have problems in the pre-analytical phase have previously been shown to encounter technical failures in future tests, possibly due to an inadequate selection of tumor cells for further analysis [32]. A framework for EQA schemes designed to assess the pre-analytical phase has been established [52], however remains difficult to date due to the intrinsic heterogeneity of tissue materials, ethical constraints and limited availability of sending blood samples, and the distribution of prepared samples that omit parts of the earliest test phases.
Finally, hurdles such as increased workload and costs or accompanying staff limitations might further contribute to determine a laboratory’s test selection, and confirm previous findings where higher test volumes resulted in increased personnel errors [32]. Interestingly, laboratories in the USA were more likely than non-US laboratories to cite resource limitations as challenges [34]. Access to appropriate testing techniques for predictive biomarkers differs, as their reimbursement varies by the local health care systems in Europe [53]. The vast complexity of the many available markers and tests in molecular pathology compared to other medical diagnostic fields further adds to these inequalities. In the Netherlands, it has been suggested that a regionally coordinated approach may assist in organizing complex molecular testing, but first involves the development of capacity, logistics, and information sharing in centers, which is more likely to become adopted when outstanding issues on the reasoning, costs, cost–effectiveness, and reimbursement are adequately addressed [54].
Conclusions
Molecular pathology is a continuously evolving field, and many markers and techniques have become available over time. This study presents benchmark data on procedures and practices of validation or verification for European laboratories who recently adopted new methods, markers (PD-L1) or sample types (ctDNA). The study revealed that documentation of validation/verification procedures and guideline adherence was higher but not limited to accredited institutes. The reported hurdles stressed the importance of further quality efforts to aid laboratories with test interpretation of complex data, controlling pre-analytical variables, and the selection of appropriate control materials.
Acknowledgements
We would like to thank the ESP and IQN Path for the joint organization of the NSCLC, mCRC and ctDNA EQA schemes, and the laboratories for completing the survey. We thank Nicola Normanno and Francesca Fenizia for the joint master thesis project at the National Cancer Institute, Fondazione G. Pascale-CROM in Mercogliano, Italy.
Author’s contribution
CK and EMCD conceived and designed the study set-up. CK was responsible for survey creation and result collection. JDS drafted the survey, contacted the institutes and collected the data for the nine additional institutes. CK analyzed and interpreted the data and was responsible for writing of the manuscript. All authors critically revised the text for important intellectual content.
Disclosures
The authors have no relevant conflicts of interest to declare.
Funding
Not applicable.
References
- Food and Drug Administration (2020): Available from https://www.fda.gov (2020).
- European Medicines Agency (2020): Available from https://www.ema.europa.eu (2020).
- Lindeman NI, Cagle PT, Aisner DL, et al. Updated Molecular Testing Guideline for the Selection of Lung Cancer Patients for Treatment With Targeted Tyrosine Kinase Inhibitors: Guideline From the College of American Pathologists, the International Association for the Study of Lung Cancer, and the Association for Molecular Pathology. J Mol Diagn 20 (2018): 129-159.
- Sepulveda AR, Hamilton SR, Allegra CJ, et al. Molecular Biomarkers for the Evaluation of Colorectal Cancer: Guideline From the American Society for Clinical Pathology, College of American Pathologists, Association for Molecular Pathology, and the American Society of Clinical Oncology. J Clin Oncol 35 (2017): 1453-1486.
- Pennell NA, Arcila ME, Gandara DR, et al. Biomarker Testing for Patients With Advanced Non-Small Cell Lung Cancer: Real-World Issues and Tough Choices. Am Soc Clin Oncol Educ Book 39 (2019): 531–542.
- Lantuejoul S, Damotte D, Hofman V, et al. Programmed death ligand 1 immunohistochemistry in non-small cell lung carcinoma. J Thorac Dis 11 (2019): S89–S101.
- Rolfo C, Mack PC, Scagliotti GV, et al. Liquid Biopsy for Advanced Non-Small Cell Lung Cancer (NSCLC): A Statement Paper from the IASLC. J Thorac Oncol 13 (2018): 1248–1268.
- Jenkins S, Yang JC, Ramalingam SS, et al. Plasma ctDNA Analysis for Detection of the EGFR T790M Mutation in Patients with Advanced Non-Small Cell Lung Cancer. J Thorac Oncol 12 (2017): 1061–1070.
- Deans ZC, Williams H, Dequeker EMC, et al. Review of the implementation of plasma ctDNA testing on behalf of IQN Path ASBL: a perspective from an EQA providers' survey. Virchows Arch 471 (2017): 809–813.
- Tembuyser L, Dequeker EM. Endorsing good quality assurance practices in molecular pathology: risks and recommendations for diagnostic laboratories and external quality assessment providers. Virchows Arch 468 (2016): 31-41
- European Parliament. Directive 98/79/EC of the European Parliament and of the Council of 27 October 1998 on in vitro diagnostic medical devices. (1998): Available from http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:1998:331:0001:0037:EN:PDF (2020).
- Food and Drug Administration. List of Cleared or Approved Companion Diagnostic Devices (In Vitro and Imaging Tools) (2020): Available from https://www.fda.gov/media/119249/download (2020).
- International Organization for Standardization. ISO 15189:2012 Medical laboratories - Particular requirements for quality and competence (2012): ISO, Geneva.
- Thunnissen E, Bubendorf L, Dietel M, et al. EML4-ALK testing in non-small cell carcinomas of the lung: a review with recommendations. Virchows Arch 461 (2012): 245–257.
- Halling KC, Schrijver I, Persons DL. Test verification and validation for molecular diagnostic assays. Arch Pathol Lab Med 136 (2012): 11–13.
- Cree IA, Deans ZC, Ligtenberg MJ, et al. Guidance for laboratories performing molecular pathology for cancer patients. J Clin Pathol 67 (2014): 923-931.
- Elliott K, McQuaid S, Salto-Tellez M, et al. Immunohistochemistry should undergo robust validation equivalent to that of molecular diagnostics. J Clin Pathol 68 (2015): 766–770.
- Maxwell P, Salto-Tellez M. Validation of immunocytochemistry as a morphomolecular technique. Cancer Cytopathol 124 (2016): 540–545.
- Torlakovic EE, Cheung CC, D'Arrigo C, et al. Evolution of Quality Assurance for Clinical Immunohistochemistry in the Era of Precision Medicine. Part 3: Technical Validation of Immunohistochemistry (IHC) Assays in Clinical IHC Laboratories. Appl Immunohistochem Mol Morphol 25 (2017): 151–159.
- Fitzgibbons PL, Bradley LA, Fatheree LA, et al. Principles of analytic validation of immunohistochemical assays: Guideline from the CAP Pathology and Laboratory Quality Center. Arch Pathol Lab Med 138 (2014): 1432–1443.
- Jennings L, Van Deerlin VM, Gulley ML, the College of American Pathologists Molecular Pathology Resource Committee. Recommended principles and practices for validating clinical molecular pathology tests. Arch Pathol Lab Med 133 (2009): 743–755.
- Jennings LJ, Arcila ME, Corless C, et al. Guidelines for Validation of Next-Generation Sequencing-Based Oncology Panels: A Joint Consensus Recommendation of the Association for Molecular Pathology and College of American Pathologists. J Mol Diagn 19 (2017): 341-365.
- Roy S, Coldren C, Karunamurthy A, et al. Standards and Guidelines for Validating Next-Generation Sequencing Bioinformatics Pipelines: A Joint Recommendation of the Association for Molecular Pathology and the College of American Pathologists. J Mol Diagn 20 (2018): 4-27.
- Deans ZC, Costa JL, Cree I, et al. Integration of next-generation sequencing in clinical diagnostic molecular pathology laboratories for analysis of solid tumours; an expert opinion on behalf of IQN Path ASBL. Virchows Arch 470 (2017): 5-20.
- Mattocks CJ, Morris MA, Matthijs G, et al. A standardized framework for the validation and verification of clinical molecular genetic tests. Eur J Hum Genet 18 (2010): 1276-1288.
- Code of Federal Regulation 21CFR809.3, in vitro diagnostic products for human use. Title 21 Volume 8 (2019): Available from https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?FR=809.3 (2020).
- Centers for Disease Control and Prevention. Clinical Laboratory Improvement Amendments (CLIA). (2020): Available from https://www.cdc.gov/clia/index.html (2020).
- Keppens C, Tack V, ‘t Hart N, et al. A stitch in time saves nine: external quality assessment rounds demonstrate improved quality of biomarker analysis in lung cancer. Oncotarget 9 (2018): 20524-20538
- Keppens C, Dequeker EMC, Rouleau E, et al. Sensitive detection methods are key to identify secondary EGFR c.2369C>T p.(Thr790Met) in non-small cell lung cancer tissue samples. BMC Cancer 20 (2020): 366-376.
- Tack V, Ligtenberg MJ, Tembuyser L, et al. External quality assessment unravels interlaboratory differences in quality of RAS testing for anti-EGFR therapy in colorectal cancer. Oncologist 20 (2015): 257-262
- Dufraing K, Keppens C, Tack V, et al. Evolution of RAS testing over time: factors influencing mutation rates in metastatic colorectal cancer patients. Colorect cancer 9 (2020).
- Keppens C, Schuuring E, Dequeker EMC. Causes behind error rates for predictive biomarker testing: the utility of sending post-EQA surveys. Submitted (2020).
- Stuart LN, Volmar KE, Nowak JA, et al. Analytic Validation of Immunohistochemistry Assays: New Benchmark Data From a Survey of 1085 Laboratories. Arch Pathol Lab Med 141 (2017): 1255–1261.
- Fitzgibbons PL, Goldsmith JD, Souers RJ, et al. Analytic Validation of Immunohistochemical Assays: A Comparison of Laboratory Practices Before and After Introduction of an Evidence-Based Guideline. Arch Pathol Lab Med 141 (2017): 1247–1254.
- International Organization for Standardization. ISO 17043:2010 Conformity assessment - General requirements for proficiency testing. (2010): ISO, Geneva.
- Keppens C, Dequeker EMC, Patton SJ, et al. International pilot external quality assessment scheme for analysis and reporting of circulating tumour DNA. BMC Cancer 18 (2018).
- Associazione Italiana di Oncologia Medica. (2020): Available from: https://www.aiom.it (2020).
- Tack V, Schuuring E, Keppens C, et al. Accreditation, setting and experience as indicators to assure quality in oncology biomarker testing laboratories. Br J Cancer 119 (2018): 605-614
- International Organization for Standardization. ISO 9001:2015 Quality management systems - Requirements. (2015): ISO, Geneva.
- Mancini I, Pinzani P, Simi L, et al. Implementation of a companion diagnostic in the clinical laboratory: the BRAF example in melanoma. Clin Chim Acta 439 (2015): 128-136
- Boursier G, Vukasovic I, Brguljan PM, et al. Accreditation process in European countries - an EFLM survey. Clin Chem Lab Med 54 (2016): 545–551.
- Sheldon TA, Cullum N, Dawson D, et al. What’s the evidence that NICE guidance has been implemented? results from a national evaluation using time series analysis, audit of patients’ notes, and interviews. BMJ 329 (2004).
- Meddeb R, Pisareva E, Thierry AR. Guidelines for the Preanalytical Conditions for Analyzing Circulating Cell-Free DNA. Clin Chem 65(2019): 623–633.
- International organization for standardization. ISO 20186-3. Molecular in vitro diagnostic examinations – specifications for pre-examination processes for venous whole blood - Part 3: Isolated circulating cell free DNA from plasma. (2019): ISO, Geneva.
- Heitzer E, Haque IS, Roberts CES, et al. Current and future perspectives of liquid biopsies in genomics-driven oncology. Nat Rev Genet 20 (2019): 71–88.
- Tsao MS, Kerr K, Dacic S, et al. IASLC atlas of PD-L1 immunohistochemistry testing in lung cancer. International Association for the Study of Lung Cancer (2017): Editorial Rx Press.
- Regulation (EU) 2017/746 of the European Parliament and of the Council of 5 April 2017 on in vitro diagnostic medical devices and repealing Directive 98/79/EC and Commission Decision 2010/227/EU. (2017): Available from https://eur-lex.europa.eu/eli/reg/2017/746/oj (2020).
- Nordic immunohistochemical Quality Control (NordiQC). (2020): Available from: https://www.nordiqc.org (2020).