Test  Exercise 2008

Deadline: 5/5 2008 12.00

Two standard options are available. If you have different wishes please discuss this with the instructor - it must be approved.


The objective is to evaluate the usability and effectiveness of two approaches to automated real-time testing: Manually assisted test purpose based test generation and full automatic randomized online testing. Each group is given a Uppaal specification of the light controller, and an (assumed!) correct implementation in java, including two example mutants. The experiment proceeds in two phases: Preparation phase and evaluation phase. 

In the preparation phase, each group will independently (manually or Uppaal assisted) generate a set of test cases and a set of mutated implementations based on the given specification and implementation. Each group runs their test suite on their own mutants, and possibly extend it as desired. The mutants must be non RT-IOCO conforming to the specification, i.e. killable by some test case.  The errors may of course both involve logical errors as well as timing. Use your imagination and create plausible mutants that could result from real coding errors.

In the evaluation phase, each group swaps mutants (java byte-code) and runs 

  1. their own test suite against the groups own and other groups mutants, collecting statistics with the results (which own/other mutants were killed?). 

  2. Uppaal-TRON (randomized online testing) to the collected mutants (e.g 10 runs per mutant)., again collecting statistics about the percentage of runs killing the mutants, and preferably also the duration of the test runs.


Each group is to hand in the following deliverables to be evaluated. 

The deliverables must be signed by each student who, by his signature, declares active and equal participation in preparation of the deliverables


  1. Java sources for Light Controller is distributed with TRON  Requires > java 1.5:  /pack/j2sdk-
  2. TRON. Please download and install according to the Tron  webpage. Then read the instructions in the readme.txt file (also found in the tron package) to start tron.
  3. If you use CS linux server hosts (simpsons) beware that each group uses distinct sets of tcp ports!


Instructions (execution of offline generated tests):

We recommend executing your preset input sequences using TRON by modeling the test input sequence as a timed automaton and by replacing the environment with this automaton.  An example is provided in LightContr4.xml (Template: LightCov and Envy Closure. See system section).

The difference bwteeen LightContr.xml and LightContr4.xml is that LightContr.xml contains a model of the adaptor, and more options for configuring tolerances important for real-time execution, but is also more complex. For the simulated clock framework the simpler model and a tolerance of 1 as in the LightContr4.xml suffice. Beware of the tolerances allowed by the test specification if you cannot kill a time-based mutant. 

Start tron as described below. You can also try different guide options for randomization (-P option, see tron --help):

The command line options mean the following: Start tron using  "SocketAdaptor" adaptor to communicate with IUT,  using test specification "lightContr4.xml", testing a java lightcontroller-implementation running on host ip number (can also be either "localhost", a dns-name, or an ipnumber), -P options hints how uppaal-TRON should select inputs at random times within environment behavior : The options given after -- are handed to the adaptor component rather than tron itself.  -v indicates the verbosity level, ie the amount of printed information on the console. 

Instructions (Real-time problems):

Here are some tips that may reduce real-time scheduling problems (alternatively use simulated time as in the above instructions)

  1. use lightly loaded machines
  2. use local isolated, single user computers, and and perferable 2 cpu machine (or two machines interconnected via cross cable)
  3. use lightly loaded multi-cpu machine
  4. use Linux Kernel 2.6 (marge, not homer)
  5. Disable "Nagle's" algorithm by setting the socket option: setTcpNoDelay(true); This should already be done in all the applications and adaptors you will work with, including the "TestLightController" application.
  6. Do not X-tunnel the lightcontroller gui. If you cannot run the lightcontroller locally, try to remove GUI from Java Lightcontroller implementation by uncomment the code that create the graphics objects: run the light controller with -N option.