Please read this license agreement carefully. Your use of the software or any related documentation (called the "Software") indicates your acceptance of the following terms and conditions. If you do not agree to these terms and conditions, you may not install or use the Software.
The Software is currently made available without charge in the development stage in order to allow you to evaluate it in advance of commercial release. Your feedback and suggestions are encouraged.
The Software is owned by Steve Markoff, and is copyrighted and licensed, not sold. Steve Markoff grants you a non-exclusive, non-transferable license to use it for any lawful end use. Implied licenses are negated. You may copy the Software for backup only. You may not sublicense the Software. You may not distribute (for free or for sale) the Software without written permission from Steve Markoff. You may not decompile, alter Java byte-code or binary code, or in any manner reverse engineer the Software.
Steve Markoff licenses the Software to you on an "as is" basis, without warranty of any kind. Steve Markoff hereby expressly disclaims all warranties or conditions, either express or implied, including, but not limited to, the implied warranties or conditions of merchantability and fitness for a particular purpose. You are solely responsible for determining the appropriateness of using this Software and assume all risks associated with the use of this Software, including but not limited to the risks of program errors, damage to or loss of data, programs or equipment, and unavailability or interruption of operations. Some jurisdictions do not allow for the exclusion or limitation of implied warranties, so the above limitations or exclusions may not apply to you. Steve Markoff will not be liable for any direct damages or for any special, incidental, or indirect damages or for any economic consequential damages (including lost profits or savings), even if Steve Markoff has been advised of the possibility of such damages. Steve Markoff will not be liable for the loss of, or damage to, your records or data, or any damages claimed by you based on a third party claim. Some jurisdictions do not allow for the exclusion or limitation of incidental or consequential damages, so the above limitations or exclusions may not apply to you.
You hereby grant to Steve Markoff an irrevocable license under all intellectual property rights (including copyright) to use, copy, distribute, sublicense, display, perform and prepare derivative works based upon any feedback, including materials, fixes, error corrections, enhancements, suggestions and the like that you provide to Steve Markoff.
To provide a load testing tool for web applications (and other applications) that contains many of the features of "big-name" tools, for roughly one-tenth the price.
The feature emphasis is on the collection of detailed and accurate data and the flexibility of how the load test is performed. For displaying extensive graphs of the data, the tool will provide some visualization features, but will attempt to save data in a format that can be loaded into popular spreadsheet programs for a wider range of possible presentation options.
The tool is designed to run on multiple platforms. In theory, the tool will run on any platform that supports Java (trademark of Oracle), however only the following platforms will be tested and officially supported:
(This is a partial list.)
Ability to create highly flexible testing scripts, including:
Large number of runtime options:
Detailed data reporting, including:
Currently, there is no installer program, however installation is quite simple. (A basic understanding of the Java environment is assumed.)
Scripts are in XML format. Below are two example scripts. Read the discussion that follows to understand what they mean.
-------- Script 1 -------- <?xml version="1.0" ?> <!-- (comment) This is a very simple script. --> <collection baseURL="http://myhost"> <request url="/home.html" label="Home page" /> <request url="/contact.html" /> <!-- Override the baseURL in this next request --> <request url="http://host2/index.html" /> <!-- Override the baseURL again, because this time, even though the host is the same, this page is secure (https) and uses a non-standard port (8443). --> <request url="https://myhost:8443/login.jsp" /> </collection> -------- Script 2 -------- <?xml version="1.0" ?> <!-- A more complicated script --> <collection baseURL="http://myhost"> <!-- Make the request a POST instead of the default GET. Set the frequency (see notes below). Set some content parameters that this page requires (username, password). Check for a particular string in the response page that indicates that something went wrong. --> <request method="post" frequency="3" url="https://myhost/login.jsp"> <param name="username" value="jsmith" /> <param name="password" value="passW=&ord" encode="true" /> <fail>Login failed</fail> </request> <!-- Alternatively, provide userid and password in an Authorization header. Check for a string to indicate success, instead of one indicating failure. --> <request url="/home.jsp" label="Home page with Auth header"> <auth username="username" password="password" /> <success>Welcome</success> </request> <!-- Provide different values for content parameters each time this request is made. --> <request url="/page1.jsp"> <param name="name" value="#steve,fred,bill,joe#" /> <param name="id" value="#400-900:50#" /> <param name="number2" value="#1000-1:1#" /> </request> <!-- Do a request with some special headers. --> <request frequency="8" url="/page2.jsp"> <extraHeader>User-Agent: LoadTest</extraHeader> <extraHeader>Special: text</extraHeader> </request> </collection>
Each request has the following data elements:
url: If a baseURL is specified on the collection, the url for each request may be relative to that base url. An absolute url (beginning with http:// or https://) will override the baseURL.
For query string or content parameters, suppose you want to have a string or number that is different for each request. You can use the following format:
<param name="name" value="#steve,fred,bill#" /> or <param name="id" value="#27,95,100#" />
For a large series of numbers that are evenly spaced out, you may also use the following format:
<param name="id" value="#10-20:1#" />
The notation indicates to start with 10, and then for each request add 1, until 20 is reached (inclusive), then go back to 10. The following syntax indicates a decreasing pattern:
#20-10:1#
Note that -1 is not specified as the third number. All that is necessary to indicate decreasing order is to make the start number greater then the second one.
Here is an example of the behavior if you use multiple special notations in the same request.
<param name="name" value="#steve,fred#" /> <param name="id" value="#1-4:1#" />
The content parameters will be as follows:
name=steve&id=1
name=fred&id=2
name=steve&id=3
name=fred&id=4
(repeat)
Examples for success or fail string:
<success>Welcome</success> <fail>Login failed</fail>
In the first example, the request will be considered a success only if "Welcome" is contained in the response. In the second example, the request will be considered a success only if "Login failed" is NOT contained in the response. The string is treated case sensitively. Multiple success and/or fail declarations can be made and there is an implicit AND between these conditions - all must be met for the request to be considerd successful. For example you could have two success conditions and two fail conditions such that the response must contain "string1" and "string2" and must NOT contain "string3" and "string4". Note that these conditions are ignored if the response type is not "text/html" or "text/plan" or something beginning with "text/" - for example a resposne of type "image/jpeg" would not be checked for these conditions.
extraHeader elements should contain whatever special headers you need/desire but you must not specify the Connection header because this is taken care of for you by the program automatically, depending on the flags you choose for keep-alive.
From the directory that you have chosen as your working directory (the one with your license.xml file in it and the scripts that you have created) type the following:
java stress.url.URLStress
This is actually not all you need to type, because several program arguments are required. However, typing just this much will cause a list of these arguments to be printed out to remind you what they are. Here is the list of these arguments with full explanation (the program does not give you the full explanation -- it will tell you to look here if you are confused.). After reading about all the options you can decide which ones are best for each test you want to perform.
General format of the command to start the program:
java stress.url.URLStress <xml file> -c <#clients> <-r #repeats | -s #seconds> [more options]
-c <#clients> indicates how many simultaneous clients to use for the load test. Each client can be thought of as virtual user.
Either -r or -s must be specified (but not both).
-r causes a fixed number of repeats to be done
-s causes repeats to be done until a fixed number of seconds has
passed -- each client will not stop until the end of a repeat,
so the number of seconds may be extended somewhat.
Generally, you want all the clients you have specified to finish at
roughly the same time. Another way of saying this is that you want
all the clients simultaneously active for the whole test period.
In order to achive this goal, best results will be obtained if you
use the -r for short tests (less than a few minutes) and
use the -s for longer tests.
With a short test period, it is likely that each client will finish
at roughly the same time if they all do the same number of repeats.
With a long test period and a fixed number of repeats, some clients
will "pull ahead" in finishing all their repeats and so finish early.
This may have the effect of bringing down the average response time,
which is a misleadingly good result.
The -s option means that extra work will be added to the queue
of clients that are "ahead" to even things out.
---- more options ----
-ramp <milliseconds> clients will be started up with a delay
between each first request. For example with a ramp of 100 ms the first client
will start immediately, the second after 100 ms, the third 100 ms after that, etc.
-p <seconds> each client will pause the specified number of seconds between requests.
-od specifies the output directory. The default is a directory called stressOutput inside the current directory. A number of files will be created in the output directory. The output directory will be created if it does not already exist. Any pre-existing files in the output dir will be erased or overwritten.
-i <full load frequency> causes each response that is of
content-type text/html to be parsed for image tags. Each image will then
either be fully loaded or just checked for the modification
of (using a If-Modified-Since header).
For example, a frequency of 1 indicates to do a full load every
time. A frequency of 5 indicates to load once every 5 repeats.
A full load is always done the first time.
You should pick your full load frequency based on how many repeats you intend to mimic a real user. For example, suppose you expect the average user of your website to hit each page in your list of tasks (URLs) 5 times, and then leave the website. (For the sake of the example, assume you have specified the same frequency (column 2) for all the URLs.) Then you should specify 5 for the image full load frequency because the sixth repeat really represents a new user, who would not have the image cached in their browser.
The default behavior is to use persistent connections (keep-alive) for images (but not the initial page request) unless the -nk option is also specified (this option explained below). Regardless of the full-load frequency or persistent connections, like a real web browser, if the same image appears in a page multiple times, the load testing program only hits the HTTP server once for it during a given repeat. Separate time and size statistics will be kept on the images (treating all the images for a page as a collection) in addition to the main request (see section below about the data printed out when the program is finished).
Assuming that your web server(s) are on separate hardware from your application server(s), and that all images are static (served by your web server) specifying the -i will only effect the load level on your web servers. If you have already determined that your web servers are more than capable of handling the load of static images for your expected user traffic, you may wish to simply concentrate on load testing your application servers and avoid using the -i option, especially if you have a limited bandwidth test environment. If you temporarily ignore the images, you can increase the number of clients (-c) to push your application servers to their limit. However the scenaro just described may not be at all similar to your environment. Please evaluate your unique situation and make your own judgements.
-pc indicates that cookies should be printed out -- when ever a Set-Cookie header is received, that header will be printed out along with an updated list of all the cookies currently stored on the client side (after adjusting the list using the Set-Cookie header).
-td indicates that in addition to the Task Summary, Task Details should be printed -- this will list the times for every single request that was made. This option should not be used if -r or -s is specified with a large value (you will get a massive printout and possibly use of a lot of memory).
-nc indicates not to send cookies -- normally if the HTTP server(s) you are load testing tries to set cookies, this program (the client) will accept them and send them back to the server on successive requests. When this option is specified cookies will not be sent back to the server (unless you specify a Cookie header in the extraHeaders column in your script file). Note you can still use -pc with this option (because received cookies are still kept track of, they are just not sent back). However the server(s) may try constantly to set cookies on every request (because no cookies are being send back) so you may end up with many more cookies printed out.
-nk indicates not to use persistent connections (a.k.a. keep-alive) when loading images for an html response. This option will have no effect unless -i is also specified.
-nfr indicates not to follow redirects, instead they will be counted as bad requests.
-frl <integer> override the default follow redirect limit. The default limit is 3. You must specify a value that is 1 or greater.
-ds disables certificate checking while accessing https (secure) URLs - useful when you are testing a server that has an SSL certificate that is not signed by a trusted authority (a test certificate)
-lic <file> Normally the license file (if you are using one) is expected to be called "license.xml" and located in your working directory. However with this option you can specify a different path or name for the license file.
Example 1: java stress.url.URLStress urls.xml -c 4 -r 8 -i 5 Example 2: java stress.url.URLStress urls.xml -c 4 -s 10 -pc -ng
Whether you start a load test in graphical (TBD) or command line mode, progress indications are kept to the essentials in order to make the best use of the CPU of the machine. (more to be filled in here about indicators once they are finished).
When a load test is finished running, all the data collected will be saved in the output directory. The data is presented in a variety of formats, including plain text, HTML, and images (graphs).
In the future, the data will also be saved in a format that can easily be loaded into popular spreadsheet programs, for a wider options of graphical display than the load tester itself provides (see Features section).
The text output consists of the following sections:
Client Details
Client Summary
[If you specified -td, then Task Details]
Task Summary
Overall Summary
About the Task Summary:
(Explain about the breakout of images stats with the -i option,
especially full-load(fl) and not modified(nm).)
About the Overall Summary:
It's fairly self-explanatory, but note the following
total time for all requests (real) = 191,001 total time for all requests (req.) = 187,500 ratio = 0.9817
A ratio close to 1.0 (such as 0.9817) indicates that the CPU was under-utilized by the load test (which is good). If you see a ratio such as 0.6245, this means the CPU of the machine is being taxed heavily, and either the parameters of the load test should be changed, or a machine with a faster CPU or more CPU's is required to run the test. In a future release, you will also be able to make use of multiple machines for a load test (see Features section).
(More notes to be added here in a future release.)
Questions?
(HTML/image output is similar to the text output, but it's generatly easier to read due to the more powerful formatting offered by HTML/images.)
Questions?
Known issues apart from features already mentioned that are missing in the current release, or comments made elsewhere in this document.
Exception in thread "main" java.lang.OutOfMemoryErrorthen you need to run the load tester again, giving more memory to the Java Virtual Machiine - specify -Xmx512m or higher (this is not really an issue, more memory is simply required with long or intensive load tests).
java.net.BindException: Address already in use: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.<init>(Unknown Source)
at java.net.Socket.<init>(Unknown Source)
at net.HttpClient.openNewSocketIfNeeded(HttpClient.java:529)
at net.HttpClient.makeRequest2(HttpClient.java:222)
at net.HttpClient.makeRequest(HttpClient.java:126)
at stress.URLTask.execute(URLTask.java:195)
at stress.Client.doOneRepeat(Client.java:145)
at stress.Client.run(Client.java:114)
Probably somehow Java and/or Windows is making a bad choice about what
addresses to bind to instead of avoiding the addresses that haven't
been cleaned up yet (still in a TIME_WAIT state).
The bug is probably in Windows, not in Java JDK 1.4.2 because it has not been
reproduceable on Red Hat Linux Enterprise Linux 4.0.
Actually it may not be a bug but a purposeful limitation of Windows XP
- non-server versions of Windows are not meant to support large numbers of
TCP/IP connections.
In any case, this is not a problem that
can be fixed by altering any of the load testing code.
(Note this can happen as long as the load tester is run on Windows, even if
the target URLs are hosted by a Unix machine.)
*Update: This bug only seems to surface when the server side is also in Java - in particular using Tomcat 4.1.30 for testing, and hitting 10 static pages. But there is no problem if Tomcat is running on Windows and the load tester is running on RHEL 4.0.
Note to self - test cases:
url_me.txt -c 16 -s 20 [hitting 10 static pages served by my Java web server]
[and same url's, using Apache 2.0.48 this time]
If you find a bug/issue, please let me know how to reproduce it (including specific platform info and Java version), and I will attempt deal with it in an appropriate manner.
Instead of testing web pages (URLs) you can test Java methods in any number of classes. Here are some very quick instructions (to be expanded later).
Sample input file
<?xml version="1.0" ?>
<collection>
<class name="mypackage.MyClass">
<method name="setup" />
<method name="test1">
<param type="String" value="foo" />
</method>
<method name="test2" frequencey="2">
<param type="int" value="45" />
<param type="Integer" null="true" />
<param name="useDebug" type="boolean" value="true" />
</method>
</class>
<!-- (more class definitions) -->
</collection>
Each parameter you specify for a method must have one of the following types: String, boolean, Boolean, int, Integer, long, Long, float, Float, double, Double, char, Character.
Each parameter must either be specified as being null (say null="true") or must be given a value. Parameters that are primitive types (ex. int) cannot be null. The value given must be appropriate for the parameter type. For example, not give a value of "abc" for a type of "int".
The name attribute for parameters is optional - it's just there to help you make your test scripts more readable.
Running the test (example)
java stress.method.MethodStress methods.xml -c 4 -r 4
Note many of the options you can use for URLStress can also be used here (-ramp, -p, -od, -s, -lic)