History of tool

Starting point

Some years ago I was working on test automation of web application. Selenium WebDriver was selected as remote control. Java as coding language. JUnit as test framework (execution, reporting). First suite contained about 10 cases and was executed with ant task configuring build. Some time later suite grew up to 50 scenarious.

What was recognized as annoying and boring actions:
1) Not comfortable with creating custom test set suites, required a lot of keyboard work and remember long test classes full names, possibility of errors to create ANT's build.xml file
2) Browser was initiated at each test start, then closed. Generating a lot of default profile logs (10 Mb per start).
3) Assuming avg test run as 5 minutes, suite take 1-4 hours to complete. Pretty long delay for quick feedback.
4) Rerun of failed tests recognized as "run error" required again a lot of boring actions.
5) Need to wait till suite run complete to get test report.

Development of execution tool was started under wish to resolve process pitfalls and optimize boring work.
On each next stage of usability evolved, new ideas were added as remains actual same for today.

First steps: root concept applied - orchestrate execution from web

First concept of Bali created
1) web app was developed
2) In config.xml user pointed path to java projects bin folder, app scanned folder for tests, built test tree
3) User can go on web and browse the projects tests tree
4) User can pick test set from tree and click run to execute
5) Execution engine prepared ANT build.xml file for selected items and started it in separate process like user did before in command line
6) Link to result was generated, but user required to ping and check its availability (after suite completed).
7) user can create/edit suites

New enhancements:
1) New conception applied
2) Browse tests tree online
3) Pick up and start tests just within few mouse actions
4) Create/modify/run suites

stage 1: multi-threading, concurrency, classloading tests project code

1) execution engine was remastered
a) execution in external process by ANT removed
b) code is classloaded and executed within same JVM
c) concurrency added

New enhancements:
1) execution is accelerated, speeded up ratio up to thread workers count

Pitfalls:
1) Concurrent creating and destroying browsers was slowing down performance

stage 2: idea of external resource pools, synchronized sharing 'em among executors jobs

1) Resource pool (class, factory, storage) conception was integrated to tool
2) Browser became implemented as resource
3) Test has required resource annotation which gives info about what it requires and Bali engine locks it before
start and releases at the end.
4) Thus pool stores already initiated resources, no need to recreate
5) Engine limits max concurrent jobs depending of required resources availability on pool
6) special web view created to monitor and manage resource pools content

New enhancements:
1) Problem of concurrent creating and destroying browsers was solved
2) User able to develop his personal resources and create pools of em

stage 3: classify and other reports integrated

1) JUnit standard report removed
2) Classify report feature added
3) States flow report added (on same engine)

New enhancements:
1) use next gen smart test report
2) get errors results online, dont wait till finish

stage 4: live update of suite execution&control view and rerun from test report

New enhancements:
1) add AJAX to online quick render of suite progress with many details and controlling
2) Rerun operations directly from classify report added

stage 5: load testing

1) It was short step to adopt current execution engine to work for load testing purpose
just added how many more threads than default 1 should be doing scenario and all.
2) Metrics report to store and render digital series added

New enhancements:
1) emulate load of users behaviour on web

stage 6: integrate with any javacode

New enhancement:

JUnit runner was hardcoded and other frameworks were not supported to onboard on Bali.
Thus plugin approach was developed so user can elaborate custom plugin and using it onboard his
runnable items and runners to be orchestarted from web.

BALI