Error Classify Report
Introduction
Error test report is used to get information on asserts caught in suites's executed tests. Most error test reports concept is just rendering list view of accumulated asserts with any artifacts attached (images, logs, videos e.t.c) .
Whatever format it has, after processing You mind about list of unique asserts as result. Then You inspect errors rootcases further and can conclude that few some is coming from one rootcase.
If executed suites size has few tests and same for quantity of asserts, then you might do sorting out job quickly. But what it comes for when You need to handle suites with few hundred scenarios and few hundred asserts reported among em?
This case is more typical for large automation suites.
Classify errors report model
Classify report is another paradigm of errors report.
Primary idea is to let program do all routine sorting/analyse job for You
and provide effective view for smart browsing of asserts information.
Assert is defined by text split in 2 parts with sign ",,".
Static text is part before ",," to be main assert description.
Dynamic text is part after ",," to be detailed attributes assert description .
If there is no sign ",,", then all text is static.
Examples:
1) "this is error message"
2) "Wrong web site title ,, expected 'Welcome' but actual 'Hello'"
static text : Wrong web site title
dynamic text : expected 'Welcome' but actual 'Hello'

It is possible to add reference to bug_id. Bug pattern _([A-Z]+-\\d{1,7})
Link to your bug tracking system is added.
It event can retrieved bug status from your bug tracking system (will be included in later versions).

Bug pattern examples: _BUGID-123_, _JTESTER-555_, _ABC-888888_
Assert, containing bug: "Export to db failed _BUGID-123_"

Asserts are grouped in one error class:
1) if they contain reference to same bug id.
2) if their static part of text are equal

error classes are not intersecting.
click "view details " to open details layer.
Classify report advantages
Time economy spent on error report processing

problems:
  • Spent a lot of time in order to surf all scenario reports one by one and preview asserts.
solution:
  • classification of errors and bugs is done
  • no need to analyse errors with reproduced bugs
  • no need to analyse errors with already set reason. New ones doesn't have it.
estimations:
  • instead of 1-2 hours auto-tester's time per day on standard report, it is 5-20 minutes required.
  • 1-1.5 hour economy, 70-90% time spent on analyse. estimations:
  • instead of 1-2 hours auto-tester's time per day on standard report, it is 5-20 minutes required.
  • 1-1.5 hour economy, 70-90% time spent on analyse.
Not missing errors cause lack of attention

problems:
  • Human factor using standard error report. Quality of logs analyse (volume of new bug findings and required fixes for tests is
    back linear to volume of the report and quantity of asserts in it, cause drowning in big data and limits of the inspection)
solution:
  • classify report provides full information about all errors, focusing attention on its unique error classes.
estimations:
  • Among ~50-200 suites asserts potentially their unique instances quantity is 5-10.
  • That is You will be focused on. Was it reported once or hundred - no matters.
Report interface usability

problems:
  • Collecting information by routing actions.
  • Need to remember overall sorted test results
solution:
  • classify report provides full information about results of
    suite execution on one page in resume format with effective usability interface
Made on
Tilda