Automate Performance Testing With STAF and JMETER - Part 1 / 4

So, your manager asks you to performance-test an application deployed on several machines (e.g., Web servers, app servers, database server, and maybe even third-party application or data). You write scripts to drive load, you prepare each machine before each test, you log into each machine to monitor resource usages, you have tons of terminal sessions open on your workstation at any given time, you gather numbers, maybe capturing some screen shots, you paste results into a spreadsheet, format it so your manager can understand, you present the results, and your manager says "great job! can you run the same test for 10, 100, 200,...1000 users and get the same reports? And, can you also run the same tests on versions v1.8, v2.0 and v2.1?" "uhmmm...ok?"

Testing is very neccessary and tedious work. Lots of repetitive steps and error-prone. If you can save 1 hour each day by automating the test as much as you can (and it is too easy with STAF), you are saving half a week in one month. And your job will be more interesting too! So let's take a look at STAF and a step-by-step guide on how to use it to automate your performance testing.

STAF is an Open Source Testing System from IBM. It is short for Software Testing Automation Framework (STAF). Some key features:

   > light weight (less than 2% CPU for high workloads)
   > open source automation framework
   > remote agent to control tests on multiple machines
   > supports most operating systems: unix, windows, as/400, mvs
   > provides core services through a wide arrary of languages: java, perl, tcl, python, C/C++
   > command line interface available (easiest way to get started)
   > peer-to-peer environment
   > read more here

The following diagram shows a sample test environment consisted of a number of servers, running Unix and Windows - for which different admin and scripting skills are needed - STAF can make them "talk" the same language. STAF daemon runs as a process called STAFProc, as shown. STAFProc takes requests from any other peer to execute a command, copy a file to or from a peer to another, queries status on a running process, etc.

STAF is designed to do any kinda of software test (e.g., system integration test, functional test, performance test). The example here is tailored to performance test because it shows some example of how to gather server resource uages through STAF. Once you see the point, you see the power of STAF.

Step 1. Install STAF on all hosts

STAF comes with a installer. Download it, install it, and auto-start STAFProc when server starts. Your hosts should be configured "trust" each other. If all the hosts are behind a firewall, the easiest way to make all hosts trust each other (with STAF, you can configure different trust levels, for example, trust level 3 cannot execute remote command but trust level 5 can) to the highest level: 5. Here is how it is done in the STAF configuration file called "STAF.cfg" file:

...
# Enable TCP/IP connections
interface tcp library STAFTCP

# Set default local trust
trust machine local://local level 5

TRUST DEFAULT LEVEL 5
...

If you set the above in STAFProc on the "Web/App Server 2" host, you are basically telling "Web/App Server 2" to trust any hosts that sends STAF command/requests and to grant them access (i.e., "Load Generator 1" can run command "more /proc/cpuinfo" on "Web/App Server 2"). Of course you can specify different levels for different hosts.

Once STAF is installed and configured, you can issue the ping command to verify that a peer is alive and reachable via STAF services:

C:\>hostname
load1

C:\>staf webserver1 ping ping
Response
--------
PONG