Test Errors and Severities
It's important to understand that a Test step can encounter failures for various reasons. These failures can arise from a wide range of factors, including but not limited to software bugs, system glitches, or data inconsistencies. Relicx not only identifies the failures but also categorizes them based on their impact and significance. This assessment is crucial because it not only helps in understanding the nature of the issue but also plays a pivotal role in determining the overall status of the test run.
There are four distinct severity levels that Relicx uses to classify step errors: Critical, Significant, Minor, and Low. Each of these levels represents a different degree of impact on the application or system being tested:
- Critical: This severity level is reserved for issues that have a severe and immediate impact on the test run.
- Significant: Errors categorized as significant are still impactful but may not necessarily result in complete test failure. If a test execution encounters two significant errors the test run is terminated
- Minor: Minor errors are those that have a relatively low impact on the overall functionality of the application. These issues may cause inconveniences or minor glitches but are unlikely to disrupt essential processes. Tests with Minor severity issues do not result in outright failure but are marked as validation failed.
- Low: This is the lowest severity level, indicating errors with the least impact. Low-severity errors are often cosmetic in nature, such as minor UI glitches or non-critical data display issues. They are usually tolerable and do not significantly hinder the user experience.
By categorizing errors into these severity levels, Relicx provides valuable insights into the criticality of issues encountered during testing. This information helps testing teams prioritize their efforts, focusing on critical and significant issues that require immediate attention while allowing minor and low-severity issues to be addressed in due course.
In the table below, we list the default severities of the errors that are often encountered during a test run
Default Error Severity
API invocation mismatch
Discrepancy or misalignment between the way an API is called or invoked
When API execution returns an error code
App freeze seen
Application freeze is reported when the main thread doesn’t respond for more than 3000 ms
Checks whether the application URL navigated to during the test execution matches the expected URL
Programmatic issues affecting functionality of the app
Non-critical issues or potential concerns in your application
Custom Application Error
Catches user defined or unexpected application states e.g. Something went wrong error
Dead Link error
Indicates a broken link or URL that is no longer functional
The target defined for a step no longer exists
Relicx test executor failed due to a System issue
Locator Match Failure
Test executor failed to find a matching target as defined in the test
The relicx test executor could not log into the application
Error or Unable to navigate to the specified URL
Navigation timeout on static URL
The navigation timed out on a static URL during test execution
Navigation timeout pending requests
The navigation timeout on an external URL
User defined assertion
A user defined assertion failed
Visual validation error
Visual error detected when comparing against a baseline
The severity of some of these errors can be adjusted based on the application’s needs. Please navigate to Application Settings → Configuration to adjust the severities if you want to change them. You can also contact the Relicx team if you want to adjust the severity of one of the error types that is not listed here.