Validating Rapid Microbiological Methods

Method validation is the process used to confirm that an analytical procedure employed for a specific test is reliable, reproducible and suitable for its intended purpose. All analytical methods need to be validated prior to their introduction into routine use, and this is especially true for novel technology platforms, such as rapid microbiological methods (RMMs).

Because many RMM technologies consist of a combination of instrumentation, software, consumables and reagents, in addition to specific detection, quantitative or identification methodologies, it is important to develop a comprehensive and holistic approach to the validation process to ensure that the entire RMM system is suitable for its intended use. The following sections provide an overview of how to design a meaningful validation program in order to effectively demonstrate that the new RMM is suitable for its intended use and is equivalent to, or better than, the existing method you intend to replace.

The majority of the guidance provided on this page has been excerpted from Dr. Michael J. Miller’s Training Course on RMM validation and implementation.

Initial Activities

Prior to purchasing and validating a RMM, there are a number of due diligence activities that should be undertaken. These can include identifying the scientific and technical benefits the RMM possesses as compared with the existing method, regulatory impact, financial advantages (e.g., return on investment), and the capabilities and role of the RMM supplier in terms of providing support during the initial assessment, validation and after the system has been placed in service for routine use. Each of these considerations are discussed in greater detail below.

From a scientific perspective, it is important to understand what technical capabilities are required, including, but not limited to, method sensitivity and specificity (e.g., detection levels and for what types of microorganisms), sample throughput, sample type, automation, data handling and archiving, report management, if the system needs to meet 21 CFR Part 11 expectations, and the required degree of operator training.

Next, proof-of-concept or feasibility testing can be performed to determine if incompatibilities exist between the RMM and the intended product or test sample(s). These types of studies can also be performed in the event the RMM supplier has little or no data on testing similar product or test materials. This can be accomplished using a rental or loaner instrument, or by sending samples directly to the RMM instrument supplier for evaluation. The data obtained from these initial studies will help with the decision to purchase the RMM and proceed with formal validation activities.

The due diligence process also involves a review of existing regulatory commitments and whether implementing the RMM will result in significant changes that will require a formal submission to relevant regulatory bodies. Additionally, a financial assessment of the costs (and cost savings) associated with the purchase, validation and implementation of the RMM should be performed. Information on both of these topics can be reviewed by visiting the Regulatory and ROI (return on investment) pages.

Finally, the selection of a rapid method supplier is just as important as the technology itself, and it is important to have a thorough understanding of the supplier's technical capabilities and their ability to support each phase of the validation process as well as continuing assistance once the RMM is placed into service. When deciding on a RMM and a RMM supplier, some points to consider may include the following:

  • Does the supplier have a robust quality, change control and manufacturing system in place?
  • Do they have appropriate documentation with regard to the design and manufacture of their instrumentation?
  • Has the supplier been audited by other companies or regulatory agencies?
  • Are they financially secure?
  • Are they the sole provider of the RMM consumables, reagents, supplies or replacement parts?
  • Do they provide training programs for the end-user?
  • Do they provide on-site technical services, calibration and preventive maintenance programs? Can they respond to technical issues in a timely manner?
  • Does the supplier provide validation protocols or similar documentation?
  • Have they published results of their own testing or have they submitted a Drug Master File of similar document to a regulatory agency?
  • How does the supplier manage software updates and notification to the end-user?

Suppliers should be assessed to determine if they can meet your requirements. This can be accomplished through a formal audit or supplier questionnaire.

In summary, the initial assessment of a RMM should include a comprehensive scientific, regulatory and business due diligence review, in addition to matching the appropriate technology with the desired microbiology application. It is not uncommon for companies that have purchased a RMM system to spend considerable time, resources, and expense in validating the instrumentation and method, only to find at a later date that the technology is incompatible with the process and/or product being evaluated, or that the sensitivity and/or specificity of the system is not what was originally anticipated. Therefore, careful planning and fact finding during the due diligence phase is critical to a successful RMM validation and implementation program.

The Validation Strategy

In order to design a holistic approach to RMM validation, it is necessary to develop a comprehensive strategy that includes qualifying the RMM instrumentation, software and the analytical method. The validation plan can be comprised of a number of process steps, which are outlined below and discussed in greater detail in the subsequent sections.

  • Risk Assessment
  • Validation Planning
  • User Requirements Specification (URS)
  • Design Qualification (DQ)
  • Functional Design Specification (FDS)
  • Requirements Traceability Matrix (RTM)
  • Technology Training and Standard Operating Procedures (SOPs)
  • System Integration
  • Installation Qualification (IQ)
  • Operational Qualification (OQ) and Computer System Validation
  • Performance Qualification (PQ)
    • Method Validation
    • Method Suitability
  • Ongoing Maintenance and Periodic Reviews

Risk Assessment

Quality risk management (QRM) is an important part of science-based decision making. The ICH Q9 guideline, Quality Risk Management, defines QRM as a systematic process for the assessment, control, communication and review of risk to the quality of drug product across the product lifecycle. Similarly, the FDA Final Report for Pharmaceutical cGMPs for the 21st Century - A Risk-Based Approach, states that using a scientific framework to find ways of mitigating risk while facilitating continuous improvement and innovation in pharmaceutical manufacturing is a key public health objective, and that a new risk-based pharmaceutical quality assessment system will encourage the implementation of new technologies, including RMMs, to facilitate continuous manufacturing improvements via implementation of an effective quality system.

A risk assessment should be performed prior to the start of any RMM validation activity. Identified risks will vary depending on the RMM technology and the RMM supplier, the method the RMM is intended to replace, the product or sample(s) for evaluation, whether the new measurements are qualitative or quantitative and if the resulting data are significantly different from the existing method, method variability, method robustness, pharmacopeial equivalence, regulatory acceptance, and other attributes.

First, the user should identify the hazards (i.e., what might go wrong when implementing the RMM), the likelihood of occurrence, severity of harm and the ability to detect the hazard. Next, the user will analyze the risk against predefined criteria, and determine how the risks will be addressed. Tools such as Failure Modes and Effects Analysis (FMEA) or Hazard Analysis and Critical Control Points (HACCP) may be utilized in assessing the potential risks when implementing the RMM.

The Validation Plan

A validation plan should be followed which will provide the roadmap for all of the activities that will be required to demonstrate that the RMM is validated and suitable for its intended use. The plan should include the overall project deliverables, the organizations or individuals that are responsible for each phase of test execution, review and approval requirements, and the documentation required to satisfy the expectations of the validation strategy. At the end of each phase of the validation plan, a summary of the results, whether the acceptance criteria have been met, and any deviations from the test plan should be documented and approved prior to initiating the following phase, unless it is acceptable to run phases in parallel.

User Requirements Specification (URS)

When choosing a RMM, the end-user must first establish the basic expectations that the system must meet. For example, the system may have to detect and enumerate bacteria, fungi and spores, have a sensitivity level of a single viable cell, process at least 80 samples within an 8-hour shift, and show (at least) equivalent results to the current method. From here, the user can develop specific requirements for the entire RMM system, including the instrumentation, software and the analytical method, all of which will demonstrate that the system performs as expected. The document that describes the functions and characteristics that the RMM system must be capable of performing is known as the URS. The requirements specified in the URS will also form the basis for all of the validation testing requirements, test scripts/protocols and acceptance criteria. Examples of what may be included in the URS are as follows:

  • The purpose of the RMM system
  • The scope or application of how the RMM will be used
  • Desired technical performance and functionality characteristics
  • Excluded characteristics
  • Hardware and software requirements
  • Computer system communication, data management and report needs
  • Safety requirements
  • Engineering and physical requirements
  • Preventive maintenance and calibration expectations
  • Operator qualification and training
  • Economic and regulatory considerations
  • Supplier requirements

Design Qualification (DQ)

Design Qualification (DQ) is documented verification that the proposed design of the equipment or system is suitable for the intended purpose. Because most RMMs are commercial off-the-shelf systems (COTS), DQ is accomplished by verifying that the supplier's design specifications meet the design requirements as specified in the URS. This activity can be completed prior to purchasing the RMM system or can be incorporated into the formal validation plan.

Functional Design Specification (FDS)

The FDS is the document that describes all of the functions and requirements for the RMM system and what will be tested to ensure that the system performs as specified in the URS. The FDS can be quite extensive, covering system functionality, configuration, input/outputs, environment, utilities, architecture, interfaces, data and security. Additionally, the FDS will point to specific test scripts/protocols where each requirement will be evaluated and verified against pre-established acceptance criteria. These test scripts/protocols are normally contained within the Installation, Operational and Performance Qualification documents. Example sections and requirements within the FDS may include, but are not limited to the following:

  • Purpose, scope and description of the RMM
  • Documentation
    • User manuals
    • Guidelines
    • Standards
    • SOPs
  • Physical specifications
    • Size
    • Electrical power
    • Voltage frequency
    • Operating temperature
    • Environmental requirements
    • Utility requirements
  • Computer system specifications
    • Processor, hard drive, RAM and video graphics
    • Network address and connections
    • Operating software
    • Printer ports
    • Software and algorithms
    • Databases
  • Security specifications
    • User ID and password
    • Access to data
    • Record retention
    • Audit trail
    • Administrative control
    • Data view and print reports
    • Data transfer to a dedicated server
    • 21 CFR Part 11
  • Functional specifications
    • Accuracy
    • Precision
    • Specificity
    • Limit of detection
    • Limit of quantification
    • Linearity
    • Range
    • Ruggedness
    • Robustness
    • Equivalency
  • Functions that will not be used (or tested)
  • System customization
  • Alarm configuration and error handling

Requirements Traceability Matrix (RTM)

The RTM is a document that provides traceability that all the requirements listed in the FDS have been verified and/or tested. Think of the RTM as a checklist of the validation process. The document identifies the test script or protocol where a function or requirement will be tested, such as the Installation, Operational and Performance Qualification (IQ, OQ and PQ, respectively) protocols. The RTM also specifies which SOPs and other documentation that needs to be in place in order to satisfy the criteria for meeting a specific function or requirement. The RTM is a living document during the execution of the validation test scripts/protocols.

Technology Training and SOPs

Training with the RMM supplier and the proper qualification of analysts are required for the effective execution of the testing protocols and are critical to the success of the overall validation plan. Training may be scheduled during initial proof-of-concept or feasibility testing, either in-house or at the supplier's facility. Many suppliers will also offer training as part of the initial installation and commissioning of RMM instrumentation.

SOPs that facilitate the proper execution of the RMM instrumentation, as well as those that are required to be in place as specified in the URS and FDS should be written and approved prior to the execution of the validation plan. Examples of SOPs that may be required in order to execute the RMM validation plan may include:

  • System operation
  • Training
  • Calibration
  • System and software security
  • Data management, backup and recovery
  • Corrective and Preventive Actions (CAPA)
  • Preventive maintenance
  • Business contingency plan
  • Change control

System Integration

System integration brings together all of the component subsystems into a single, operating system and ensures that all of the components function appropriately. An example may include setting up the RMM to communicate with a Laboratory Information Management System (LIMS). Many RMM systems are not required to be connected to an external server or IT platform; however, if this is required by a firm’s IT organization, system integration testing may be necessary.

FAT and SAT

In some instances, it may be appropriate (or required by the end-user’s company) to conduct Factory Acceptance Testing (FAT) or Site Acceptance Testing (SAT) prior to accepting the RMM instrumentation and initiating the IQ. A FAT is performed at the RMM supplier's facility to ensure that the system meets certain design criteria prior to the system being shipped to the end-user's site. FAT is appropriate when the end-user cannot test certain requirements at their own facility, when custom made systems have been manufactured, or when the safety of the end-user may be at risk. A SAT may be performed when the system arrives at the end-user's facility to ensure that the system operates properly after shipping. In some cases, the RMM supplier may conduct initial calibration and commissioning activities, as well as their own IQ procedures.

Installation Qualification (IQ)

The IQ establishes that the equipment is received as designed and specified, that it is properly and safely installed with the correct utilities in the selected environment, and that the environment is suitable for the operation and use of the equipment. Simply put, the IQ verifies that the equipment was received and meets the design specifications for the equipment that was ordered. The IQ can be carried out by the RMM supplier or by the end-user. Examples of what may be verified or tested during the IQ include:

  • Equipment delivery verification
  • Equipment descriptions and registration
  • Equipment installation
  • Environmental conditions
  • Preventive maintenance
  • SOPs for the operation, maintenance and calibration of the equipment
  • Establishment of an equipment log book
  • Safety checks
  • Required utilities
  • Power and wiring
  • Computer system capabilities
  • Secure server installation and communication
  • Computer access
  • Firmware and software installation and access
  • Data backup and recovery

Operational Qualification (OQ)

The OQ provides documented verification that the equipment, as installed in the selected environment, performs effectively and reproducibly as intended throughout the anticipated or representative operational ranges, defined limits and tolerances. The OQ is also the focal point for the majority of the computer system, software and security validation activities.

Computer system validation encompasses both hardware and software functionality and security, and demonstrates that these components of the RMM system operate accurately and reliably. Depending on the complexity of the RMM technology and the end-user's company policies, CSV can be quite extensive. Examples of what may be verified or tested can include the following:

  • Administrator control and operator access
  • User ID and password set up
  • User and system lockout
  • Data archiving and access
  • Audit trails
  • Report generation
  • Data transfer and server communication
  • Electronic signatures, 21 CFR Part 11
  • Data backup and recovery
  • Database management and integrity
  • Interference (radio frequency, electromagnetic, wireless)

Performance Qualification (PQ)

The PQ confirms that the instrumentation, as installed, consistently performs in accordance with predetermined criteria and thereby yields correct and appropriate results. Two very significant phase of the validation plan are performed during the PQ: validation of the microbiological method and method suitability testing.

Validation of the microbiological method first involves the use of an appropriate selection of standardized microbial suspensions in a suitable diluent and/or other target material in order to demonstrate that specific validation criteria can be met. Separately, the RMM is demonstrated to be equivalent to the existing method using actual test samples or product. Method suitability demonstrates that the presence of product, test material or the sample matrix does not significantly interfere with the performance of the RMM. Examples of interference may include background noise, false positives and/or false negatives.

When validating the method using standardized microorganisms, guidance on test procedures and acceptance criteria have primarily come from three documents:

  • PDA Technical Report No. 33, Evaluation, Validation and Implementation of Alternative and Rapid Microbiological Methods
  • United States Pharmacopoeia Informational Chapter <1223>, Validation of Alternative Microbiological Methods
  • European Pharmacopoeia Chapter 5.1.6, Alternative Methods for Control of Microbiological Quality

A brief review of the validation criteria from each guidance document is provided below. Please note that there may be differences in how the validation criteria and acceptance criteria are employed in each document and it is up to each end-user to determine what guidance works best for their products, processes and regulatory needs.

Validation Criteria for Quantitative Tests

PDA TR33
USP <1223>
Ph. Eur. 5.1.6
Accuracy
Precision
Specificity
Limit of Detection May be needed in some cases
Limit of Quantification
Linearity
Range
Ruggedness Assessed as intermediate precision
Robustness
Equivalence or Comparative Testing

Validation Criteria for Qualitative Tests

PDA TR33
USP <1223>
Ph. Eur. 5.1.6
Accuracy May be used instead of LOD
Specificity
Limit of Detection
Ruggedness
Robustness
Equivalence or Comparative Testing

Validation Criteria for Microbial Identification Tests

PDA TR33
USP <1223>
Ph. Eur. 5.1.6
Accuracy
Precision
Specificity
Robustness

The following definitions and recommendations are aligned with PDA TR33. Similarities may exist with the USP and Ph. Eur. chapters.

Accuracy is the closeness of the actual test results obtained by the new method to the actual test results obtained by the existing method (e.g., plate count). Accuracy is demonstrated across the practical range of the test, and is usually expressed as the percentage of recovery of microorganisms. Accuracy is usually assessed for quantitative methods, although microbial identification systems requires Accuracy testing as well (see below).

Precision is the degree of agreement among individual test results when the procedure is applied repeatedly to multiple samplings of the same suspension of microorganisms and using different suspensions across the range of the test. Precision is associated with the use of the method within the same laboratory over a short period of time using the same analyst with the same equipment (also referred as repeatability, within-run variability or intra-assay precision). Precision is usually expressed as the standard deviation or coefficient of variation of a series of measurements.

Specificity is the ability to detect a range of microorganisms which demonstrates that the method is fit for its intended purpose. Specificity is usually assessed during testing of the relevant validation criteria (e.g., Accuracy) for both quantitative and qualitative methods. Inclusivity and exclusivity testing may also be included for this methods that are designed to specifically detect a target organism and exclude all others. Organisms may be sourced from culture collections (e.g., ATCC), environmental or facility isolates, in-process or sterility failure isolates, slow-growing, fastidious or anaerobic strains, and/or clinically relevant cultures.

Limit of Detection (LOD) is lowest concentration of microorganisms in a test sample that can be detected, but not necessarily quantified, under the stated experimental conditions. The test determines the presence or absence of microorganisms in the original sample (e.g., sterility test).

Limit of Quantification (LOQ) is the lowest number of microorganisms in a test sample that can be enumerated with acceptable accuracy and precision under the stated experimental conditions.

Linearity is the ability to elicit results that are proportional to the concentration of microorganisms present in the sample within a given range, where accuracy and precision are demonstrated .

Range is the interval between the upper and lower levels of microorganisms that have been demonstrated to be determined with Accuracy, Precision and Linearity.

Ruggedness is the degree of intermediate precision or reproducibility of test results obtained by assessing the same samples under a variety of normal test conditions, such as different analysts, different instruments, different lots of reagents or on different days. Intermediate precision is performed within the same laboratory, and reproducibility is performed between laboratories. Ruggedness is best suited to be determined by the supplier of the test method who has easy access to multiple instruments and batches of components; however, similar studies may be conducted by the end user.

Robustness is measure of a method’s capacity to remain unaffected by small but deliberate variations in method parameters and provides an indication of its reliability during normal usage. Robustness is best suited to be determined by the supplier of the test method who has easy access to multiple instruments and batches of components; however, similar studies may be conducted by the end user.

Equivalence or Comparative Testing is the level of agreement in accuracy, precision, specificity, limit of detection, limit of quantification, linearity and/or range between the existing and the new method. This is initially demonstrated using standardized microbiology cultures (see above) and separately using actual product and other sample matrices that will be routinely tested using the new method once it is validated and implemented. However, prior to testing with actual product or test samples, these materials should be assessed for their potential to cause background noise, interference, false positive or false negative results (see Method Suitability below). Test samples should be identified that are expected to contain microorganisms in order to demonstrate that the new method will detect microorganisms similarly as the existing method. Furthermore, low levels of microorganisms may need to be inoculated into test samples in order to conduct the evaluation for equivalency (e.g., when using a sterile product to demonstrate the LOD for a rapid sterility test). The new method should be run in parallel with the existing method for a specified period of time or number of product batches or test samples (the end-user should determine the most appropriate strategy for the duration and extent of these studies). NOTE that the revised USP <1223> implies that product is not tested during equivalence studies; however, there is at least one statement in the chapter that infers at least one product type should be assessed during equivalence testing. Also, Ph. Eur. now states that equivalence is demonstrated by performing the validation parameters using standardized organism preparations and alternatively, and in some cases in addition to performing testing using a panel of microorganisms, equivalence should be demonstrated via parallel testing of a predefined number of test samples or for a period of time. This is justified based on a risk assessment.

NOTE TO READERS:
Detailed validation procedures, specific acceptance criteria and recommended statistical models are NOT provided on this webpage.
Readers should consult with a relevant guidance document (PDA TR33, USP and/or Ph. Eur.) for this information.

Method Suitability

Each test material should be evaluated for the potential to produce interfering or abnormal results, such as false positives (a positive result when no viable microorganisms are present) or false negatives (a negative result when microorganisms are present). This may also include evaluating whether cellular debris, dead microorganisms or mammalian cell cultures have any impact on the ability of the new method to operate as it is intended to. If false positives or negatives cannot be resolved, then the test sample may be incompatible with the new method.

Validation of Microbial Identification Systems

New or rapid microbial ID systems are usually tested for Accuracy and Precision (Repeatability). The end-user should establish suitable acceptance criteria for each, taking into account the ID method’s specific capabilities. For example, the ID system may not be able to identify an isolate because the organism is not included in the database, the system parameters are not sufficiently comprehensive to identify the organism, the isolate may be nonreactive in the system, or the species may not have been taxonomically described.

Ongoing Maintenance and Periodic Reviews

Following equipment verification, software validation and method validation as described above, procedures should be established to maintain the RMM system in a validated state. This wold also include a periodic review of system performance and compliance with cGMPs.

Implementation and Secondary Site Qualification

Once the validation plan has been executed and approved, the RMM may be implemented for routine use. In the case where the RMM will be implemented at a location other than where it was originally validated, it is usually not necessary to repeat the same VMP at the secondary site. In this case, a copy of the original validation package can be provided, and a reduced testing plan developed for the installation of the identical RMM system at the secondary site.

A standard equipment IQ and OQ can be performed that includes basic functionality and computer system testing. The original validation package can be used as a reference/template for what is expected during the IQ and OQ for a particular system. Furthermore, the original hardware/software security configuration testing may not need to be repeated unless there will be systems in use at a secondary site that were not evaluated in the initial test plan. An example might be the use of a different data handling or archiving platform (e.g., LIMS, LAN server, external drive) than what was originally qualified. Each new installation, or technology transfer, must be separately evaluated to determine the extent of additional IQ and OQ testing that may need to be performed.

Because an exhaustive microbiological testing plan will be completed during the PQ at the initial qualification facility, it is not necessary for the secondary facility to repeat this testing in its entirety when a like-for-like instrument is installed. The original validation package can be used to support this strategy; however, there should be limited microbiological challenges performed to demonstrate that the system is performing as intended. For example, a few reference organisms, identical to what was used during the original qualification, may be used to confirm basic functionality and demonstrate that key qualification requirements are met (e.g., Accuracy and Precision).

A review of locally recovered microbial isolates should indicate if the reference strains used during the original PQ are representative of the isolates recovered at that site. If it is determined that they are not, then the local qualification plan should include a list of microbial isolates to be evaluated and what qualification requirements they will be tested against.

Next, if the originating qualification did not include the actual product and/or process material that the site will be evaluating routinely, then these materials must be evaluated during the local PQ where the impact of sample material on the test method is assessed (i.e., during Method Suitability studies). The site will then determine the nature of the test plan to provide meaningful data about the ability of the RMM system to operate as it is intended. This may include some required number of batches, replicates, location of sampling points, length of study, etc. Additionally, sufficient data will be required to evaluate whether the data from the RMM is statistically greater than the method being replaced, and whether the data warrants a modification to baseline acceptance levels or product specifications (see additional information on changing acceptance criteria).

Assistance in Developing Your RMM Validation Plan

The development of a meaningful and comprehensive validation strategy can be a significant undertaking. For assistance in designing your own RMM validation plan, our Consulting Team is available to support your company's needs, in addition to providing in-house training sessions on all aspects of RMM validation and implementation.