Test Errors and Severities

it's important to understand that a test step can encounter failures for various reasons these failures can arise from a wide range of factors, including but not limited to software bugs, system glitches, or data inconsistencies relicx not only identifies the failures but also categorizes them based on their impact and significance this assessment is crucial because it not only helps in understanding the nature of the issue but also plays a pivotal role in determining the overall status of the test run there are four distinct severity levels that relicx uses to classify step errors critical, significant, minor, and low each of these levels represents a different degree of impact on the application or system being tested critical this severity level is reserved for issues that have a severe and immediate impact on the test run significant errors categorized as significant are still impactful but may not necessarily result in complete test failure if a test execution encounters two significant errors the test run is terminated minor minor errors are those that have a relatively low impact on the overall functionality of the application these issues may cause inconveniences or minor glitches but are unlikely to disrupt essential processes tests with minor severity issues do not result in outright failure but are marked as validation failed low this is the lowest severity level, indicating errors with the least impact low severity errors are often cosmetic in nature, such as minor ui glitches or non critical data display issues they are usually tolerable and do not significantly hinder the user experience by categorizing errors into these severity levels, relicx provides valuable insights into the criticality of issues encountered during testing this information helps testing teams prioritize their efforts, focusing on critical and significant issues that require immediate attention while allowing minor and low severity issues to be addressed in due course in the table below, we list the default severities of the errors that are often encountered during a test run error type description default error severity api invocation mismatch discrepancy or misalignment between the way an api is called or invoked low api status when api execution returns an error code low app freeze seen application freeze is reported when the main thread doesn’t respond for more than 3000 ms minor assert url checks whether the application url navigated to during the test execution matches the expected url significant console error programmatic issues affecting functionality of the app low console warning non critical issues or potential concerns in your application low custom application error catches user defined or unexpected application states e g something went wrong error minor dead link error indicates a broken link or url that is no longer functional minor disconnected target the target defined for a step no longer exists critical framework error relicx test executor failed due to a system issue critical locator match failure test executor failed to find a matching target as defined in the test significant login failure the relicx test executor could not log into the application critical navigation error error or unable to navigate to the specified url critical navigation timeout on static url the navigation timed out on a static url during test execution minor navigation timeout pending requests the navigation timeout on an external url minor user defined assertion a user defined assertion failed significant visual validation error visual error detected when comparing against a baseline low the severity of some of these errors can be adjusted based on the application’s needs please navigate to application settings → configuration to adjust the severities if you want to change them you can also contact the relicx team if you want to adjust the severity of one of the error types that is not listed here