View on GitHub

RadarGun

Benchmarking framework for data grids and distributed caches

General benchmark configuration

Context

One of the most important purposes of RadarGun is to support benchmarking of distributed caches/data grids. Generally speaking, a benchmark on a distributed cache is performed as follows:

Graph

Architecture

In order to be able to benchmark distributed caches RadarGun is using a main/worker architecture in which the RadarGun control node (Main) coordinates multiple cluster nodes (Workers). Each worker runs as an independent process, that handles one of the nodes of the benchmarked cluster. The Main has the following responsibilities:

For a complete reference of the architecture behind RadarGun please refer to the Design documentation

Configuration

Here is where you can find the complete example (see conf/benchmark-dist.xml). This section will take each individual element and detail it. An important aspect is the fact that the configuration is only defined on the main, and it defines the way in which the main coordinates the workers.

Root element

    <benchmark xmlns="urn:radargun:benchmark:3.0">

During compilation RadarGun generates schema (XSD) with documented properties for all stages in the distributed benchmark. With this you can be sure that the properties in schema are always in sync with source code. Schema files can be located in schema directory of RadarGun distribution.

Main element

    <main bindAddress="${main.address:127.0.0.1}" port="${main.port:2103}"/>

Main will open its sever socket at that address (host/port) and wait for connections from workers. “${main.address:127.0.0.1}” syntax can be read: if there is a system property named “main.address” then use that one; otherwise default to “127.0.0.1”. This way of specifying values can be used for all XML attributes.

Clusters element

    <clusters>
        <scale from="2" to="3" inc="1">
            <cluster />
        </scale>
    </clusters>

This section contains definition of clusters, which specify number nodes that the benchmark will run on. In this example, the benchmark will initially run on 2 nodes (from - initial size), and then 3 nodes (to - final size). After each run, the cluster size is incremented with “inc”, until the “to” is reached. If scaling is not required we omit scale element and use cluster element directly.

Configurations element

    <configurations>
        <config name="Infinispan 5.2 - distributed">
            <setup plugin="infinispan52">
                <embedded xmlns="urn:radargun:plugins:infinispan52:2.2" file="dist-sync.xml"/>
            </setup>
        </config>
        <config name="Infinispan 6.0 - distributed">
            <setup plugin="infinispan60">
                <embedded xmlns="urn:radargun:plugins:infinispan60:2.2" file="dist-sync.xml"/>
            </setup>
        </config>
    </configurations>

In this section we specify configurations we want to benchmark - our example uses two products: infinispan52 and infinispan60. The scenario will be run against every (config, cluster size) combo: infinispan52 configured with “dist-sync.xml” file will be run on a cluster of 2, then 3 nodes. Once this is finished, RadarGun will run the next infinispan60 configuration (dist-sync.xml) on 2 and 3 nodes. There’s no restriction on the number of configured products or configurations.

There is however memory limit on the amount of data gathered during benchmarking, some demanding benchmarks might require adjusting JVM memory parameters for Main node

Scenario element

    <scenario>
        <service-start />
        <jvm-monitor-start />

        <load-data num-entries="5000"/>
        <basic-operations-test test-name="warmup" num-requests="10000" num-threads-per-node="5">
            <key-selector>
                <concurrent-keys total-entries="5000" />
            </key-selector>
        </basic-operations-test>

        <clear-cache />

        <load-data num-entries="10000"/>
        <repeat from="10" to"=30" inc="10">
            <basic-operations-test test-name="stress-test" amend-test="true"
                duration="30s" num-threads-per-node="${repeat.counter}">
                <key-selector>
                    <concurrent-keys total-entries="10000"/>
                </key-selector>
                <statistics>
                    <default>
                        <operation-stats>
                            <default />
                            <histogram />
                        </operation-stats>
                    </default>
                </statistics>
            </basic-operations-test>
        </repeat>

        <jvm-monitor-stop />
    </scenario>

In this section we define sequence of stages:

Reports element

    <reports>
        <reporter type="csv" />
        <reporter type="html" />
        <reporter type="serialized" />
    </reports>

In this last section report generation is configured. Use csv reporter if you want to process the results outside of RadarGun. html reporter generates graphical output and includes JVM monitoring output, histograms, configuration properties and many more. It is a good practice to define serialized reporter in your reports section, as this enables you to rerun all the reporters without the need to run the benchmark again (e.g. if something goes wrong during reporting). Recently perfrepo reporter has been added, which enables you to store performance results into an external repository for further analysis. Please see README file located in reporter-perfrepo folder for configuration example.

Running the benchmark

The sequence in which RadarGun should be started is the following:

RadarGun comes with a number of scripts that help starting the main and the nodes. They are present in the distribution: