View on GitHub

RadarGun

Benchmarking framework for data grids and distributed caches

Benchmark configuration

Context

One of the most important purposes of RadarGun is to support benchmarking of distributed caches/data grids. Generally speaking, a benchmark on a distributed cache is performed as follows:

Replicated put

Architecture

In order to be able to benchmark distributed caches, RadarGun is using a client/server architecture, in which the RadarGun control node (Master) coordinates multiple slaves. Each slave runs as an independent process, that handles one of the nodes of the benchmarked cluster. The Master has the following responsibilities:

For a complete reference of the architecture behind RadarGun please refer to the Design documentation

Configuration

Here is where you can find the complete example (see conf/benchmark-dist.xml). This section will take each individual element and detail it. An important aspect is the fact that the configuration will only be defined on the master, and it basically defines the way in which the master coordinates the slaves.

Root element

     <benchmark xmlns="urn:radargun:benchmark:2.1">

During compilation RadarGun generates schema (XSD) with documented properties for all stages in the distributed benchmark. With this you can be sure that the properties in schema are always in sync with source code. Schema files can be located in schema directory of RadarGun distribution.

Master configuration

     <master bindAddress="${master.address:127.0.0.1}" port="${master.port:2103}"/>

Master will open its sever socket at that address (host/port) and wait for connections from slaves. "${master.address:127.0.0.1}" syntax can be read: if there is a system property named "master.address" then use that one; otherwise default to "127.0.0.1". This way of specifying values can be used for all XML attributes.

Cluster configuration

     <clusters>
        <scale from="2" to="3" inc="1">
           <cluster />
        </scale>
     </clusters>

This section contains definition of clusters, which specify number nodes that the benchmark will run on. In this example, the benchmark will initially run on 2 nodes (from - initial size), and then 3 nodes (to - final size). After each run, the cluster size is incremented with "inc", until the "to" is reached. If scaling is not required we omit scale element and use cluster element directly.

Product configuration

     <configurations>
        <config name="Infinispan 5.2 - distributed">
           <setup plugin="infinispan52">
              <embedded xmlns="urn:radargun:plugins:infinispan52:2.2" file="dist-sync.xml"/>
           </setup>
        </config>
        <config name="Infinispan 6.0 - distributed">
           <setup plugin="infinispan60">
              <embedded xmlns="urn:radargun:plugins:infinispan60:2.2" file="dist-sync.xml"/>
           </setup>
        </config>
     </configurations>

In this section we specify configurations we want to benchmark - our example uses two products: infinispan52 and infinispan60. The scenario will be run against every (config, cluster size) combo: infinispan52 configured with "dist-sync.xml" file will be run on a cluster of 2, then 3 nodes. Once this is finished, RadarGun will run the next infinispan60 configuration (dist-sync.xml) on 2 and 3 nodes. There's no restriction on the number of configured products or configurations.

For some scenarios you may also want to pass different configuration to different slaves. This can be achieved by configuring groups.

     <config name="Infinispan 5.2 - distributed">
        <setup plugin="infinispan52" group="group-lon">
           <embedded xmlns="urn:radargun:plugins:infinispan52:2.2" file="dist-sync-lon.xml"/>
        </setup>
        <setup plugin="infinispan52" group="group-nyc">
           <embedded xmlns="urn:radargun:plugins:infinispan52:2.2" file="dist-sync-nyc.xml"/>
        </setup>
        <setup plugin="infinispan52" group="group-sfo">
           <embedded xmlns="urn:radargun:plugins:infinispan52:2.2" file="dist-sync-sfo.xml"/>
        </setup>
     </config>

This requires groups to be explicitly specified in clusters definition

     <clusters>
        <cluster>
           <group name="group-lon" size="4"/>
           <group name="group-nyc" size="4"/>
           <group name="group-sfo" size="4"/>
        </cluster>
     </clusters>

Each stage (see the next section) can be restricted to run only on selected set of groups. For this purpose use groups attribute of any stage element.

Scenario

     <scenario>
        <service-start />
        <jvm-monitor-start />

        <load-data num-entries="5000"/>
        <basic-operations-test test-name="warmup" num-requests="10000" num-threads-per-node="5">
           <key-selector>
              <concurrent-keys total-entries="5000" />
           </key-selector>
        </basic-operations-test>

        <clear-cache />

        <load-data num-entries="10000"/>
           <repeat from="10" to="30" inc="10">
           <basic-operations-test test-name="stress-test" amend-test="true"
                                  duration="30s" num-threads-per-node="${repeat.counter}">
              <key-selector>
                 <concurrent-keys total-entries="10000"/>
              </key-selector>
              <statistics>
                 <default>
                    <operation-stats>
                       <default />
                       <histogram />
                    </operation-stats>
                 </default>
              </statistics>
           </basic-operations-test>
        </repeat>

        <jvm-monitor-stop />
     </scenario>

In this section we define sequence of stages:

Report generation

     <reports>
        <reporter type="csv" />
        <reporter type="html" />
        <reporter type="serialized" />
   </reports>

In this last section report generation is configured. Use csv reporter if you want to process the results outside of RadarGun. html reporter generates graphical output and includes JVM monitoring output, histograms, configuration properties and many more. It is a good practice to define serialized reporter in your reports section, as this enables you to rerun all the reporters without the need to run the benchmark again (e.g. if something goes wrong during reporting). Recently perfrepo reporter has been added, which enables you to store performance results into an external repository for further analysis. Please see README file located in reporter-perfrepo folder for configuration example.

Running the benchmark

The sequence in which RadarGun should be started is the following:

RadarGun comes with a number of scripts that help starting the master and the nodes. They are present in the distribution: