The Master Test Plan (MTP) defines the general approach to testing at the architectural level within a device’s software. This document applies to the product’s software testing as a complete platform. It addresses testing of the product’s software and its components to understand the extent of testing necessary to ensure the software performs as required.

The MTP is intended to identify the various methods of testing planned for development and verification. The MTP considers the product’s system as convenient functional blocks in the context of the system architecture as defined in the Architectural Design Specification to describe the overall verification strategy and test methods. The MTP is subject to review and updates as necessary throughout the software development lifecycle.

Testing Best Practices

Test Objectives
As a best practice, create test objectives to utilize during the project’s testing phase. Write specific objectives in terms of consistency, efficiency, reliability, security, and testability (to name a few).

Test Schedule
Test planning for the product’s system software and software components should follow the development lifecycle process for software verification: document the requirements, document the design, and document the testing. Tests must trace back to the product’s design elements and functional requirements. Determine a preliminary test schedule.

Testing Approach

• Unit Testing: Unit testing is the lowest, most detailed level of testing. Unit tests target specific functionality in individual modules. The design of the system should rely heavily on abstractions and dependency injection to facilitate using mock objects and stubs in unit tests. This strategy allows unit tests to replace the actual functionality of objects upon which the object under test depends. Unit tests are written as modules are developed. This practice promotes the development of loosely coupled, highly testable, and highly reusable software modules. Proper unit testing greatly reduces the occurrence of bugs, but it does not guarantee a bug free system. However, developers tasked with making changes to the software can quickly expose any new bugs that are introduced to the software. Passing the entire set of unit tests is considered a minimum requirement for regression testing the system once it has been released.
• Code Coverage Testing: Code coverage testing is done to ensure every line of code is tested and proved to execute in a manner consistent with desired operations. This method verifies that there is no dead or unreachable code within the function under test. Code coverage testing is performed during software development by the software engineer writing the function.
• White Box Testing: White box testing is performed in a desktop development environment for platform-agnostic application code. It is also possible to perform white-box testing directly on the target platform, tool availability permitting.
• Black Box Testing: Black box testing is performed to ensure the software system operates as intended irrespective of internal system manipulation or behavior. Automated and manual tests are executed upon the system ensuring the system meets all the external testing as a unit. In addition, black box testing verifies that the software responds as anticipated given any stress conditions applied to the system.

Testing Requirements and Methods
Testing at each level is built on the previous level of testing so verification becomes a component of the system. Each level of testing is verification of the previous level of testing.

• Software Test Methods
  • Development Testing Methods: Development testing is an on-going process conducted by the programmer to ensure a quality product.
    • Development testing utilizes a combination of Unit testing (for critical sections only), white box testing, and black box testing as described above. Development test results are presented in weekly status reports and are not necessarily defined in a formal test plan.
    • Smoke Testing is a rapid, preliminary testing technique which examines all the basic components of a software system to ensure that they work properly prior to entering full testing procedures. Smoke testing may be conducted by the testing team immediately after a software build is made.
  • Sprint Release Testing Methods: Sprint testing is an extension of the Development Test Methods.
    • Incremental integration testing is a bottom-up approach for testing and applies continuous testing of an application as new functionality is added. Application functionality and modules provide interfaces to associated subsystems allowing for separate testing.
    • Functional testing focuses on the outputs of the system satisfying the requirements of the system. Black-box type testing is geared to functional requirements testing of an application. Functional testing is defined by a formal test plan for the product’s software.
  • System Verification Testing Methods: System verification testing utilizes Integration and functional testing described above.
    • System testing involves verifying the entire system against the functional requirements. Black-box type testing that is based on overall requirements specifications covers all combined parts of a system. The complete interaction between the hardware and software is evaluated. Software operation is verified for system crash recovery.
    • Load/Stress testing checks system behavior under load and under conditions outside specified limits against the functional requirements.
    • Install/uninstall testing – The software installer is fully verified. A formal test plan is written to verify the software installer against the installer requirements document.
    • Security testing – Access to software configuration and log elements via the tablet-PC interface is assessed. Database, configuration, and history files are examined for encryption and protection against copying, moving, deleting, and changing.

Software Requirements Testing

The software functional requirements for the software system are specified at the instrument level.  Requirements testing implies testing as a “black box” without knowledge of internal workings, as independent components capable of operating on their own, and as internal verification of key components.

At the system architecture level, all external and internal interfaces within the software system are verified. Separate levels of testing are required to ensure quality at all levels.

Test Considerations
Test considerations are for software only.  Hardware testing is not considered within the scope of this document.  This document does consider the following within the scope of architectural testing.

• Testing for verification of product requirements
• Testing with consideration of compliance standards

Recommended Types of Testing
The following tests are recommended highly recommended for the software system:

• Development Test: Unit Testing for Code Coverage
• Development Test: Functional Testing
• Release Test: Functional Requirements Testing
• Release Test: Stress/Load Testing
• Release Test: Install Testing
• Release Test: System Failure Management

Effort of Testing
Detailed test plans allow test engineers to execute tests in the same sequence and manner and repeat tests, if necessary, in the event of failures. Failures are fully documented to provide a method of recreation. The set of test plans for the software system can be executed multiple times generating the same results regardless of the operator executing the test.

Considerations

• Automated Test System: An automated test system is an environment with actions or events scripted on both ends of the system. Inputs are time tagged and logged. Once an entire script is completed, the log files are analyzed and matched up, in time sequence, verifying all requests, events, actions, and responses were correctly handled. This system is entirely automated and can rapidly manage testing of a system in minimal time and cost with the greatest amount of coverage. Automated systems provide testing capabilities with the greatest flexibility, coverage, and efficiency in the long run.
• Dependency Injection: Dependency Injection allows high-level software modules to assign service dependencies to lower level modules. This is typically achieved by passing concrete implementations of Abstract Classes or Interfaces into object constructors or properties. The benefits of Dependency Injection are multi-faceted; but among the most important is a flexible, loosely coupled system that is easier to test.
• Manufacturing: Consideration is given to the manufacturing process to help define the software supported tools required to determine if the system is operating as desired and the hardware components on the board are fully operational.
• Field Servicing: Consideration is given to field installation and servicing of the software. Instructions, diagnostics, statistics, or tools are required for field service support.
• Software Updates: Consideration is given to security and feedback during the download and installation process. In addition, failure of the download and the subsequent consequences must be considered and addressed.

Need assistance engineering your product? We can help!