siege HTTP REGRESSION TESTER

Content copied as is from http://www.joedog.org/Siege/Manual

SIEGE, AN HTTP REGRESSION TESTER & BENCHMARKING UTILITY by Jeffrey FulmerMay 2006


Siege is an http/https regression testing and benchmarking utility. It was designed to let web developers measure the performance of their code under duress, to see how it will stand up to load on the internet. It lets the user hit a web server with a configurable number of concurrent simulated users. Those users place the webserver "under siege." The duration of the siege is measured in transactions, the sum of simulated users and the number of times each simulated user repeats the process of hitting the server. Thus 20 concurrent users 50 times is 1000 transactions, the length of the test. Performance measures include elapsed time of the test, the amount of data transferred ( including headers ), the response time of the server, its transaction rate, its throughput, its concurrency and the number of times it returned OK. These measures are quantified and reported at the end of each run. Their meaning and significance is discussed below. Siege has essentially three modes of operation, regression, internet simulation and brute force. It can read a large number of URLs from a configuration file and run through them incrementally ( regression ) or randomly ( internet simulation ). Or the user may simply pound a single URL with a runtime configuration at the command line ( brute force ).
Invocation
The format for invoking siege is: siege options... siege supports the following command line options:
' -V '' --version'Print version information to the screen.
' -h '' --help'Print the help section. This presents a summary of the options discussed in this section of the manual.
' -C '' --config'Print the current configuration. This option reads your .siegerc file and prints the settings. You can change those settings by editing $HOME/.siegerc. If you don't have a .siegerc file, then you can generate one by running "siege.config"
' -v '' --verbose 'Verbose output. With this option selected, siege will print transaction information to the screen. This includes HTTP protocol type, the return code and the page it requested:HTTP/1.1 200 OK: /cgi-bin/whoohoo.cgi?first=Homer&last=simpsonThis option is especially useful for charting progress in regression or internet modes when the program is hitting a large number of assorted URLs.
' -g URL '' --get URL 'Get an HTTP transaction. Pull down headers from the server and display HTTP transaction. Great for web application debugging. [Example]
' -c NUM '' --concurrent=NUM 'Concurrent users ( requires argument ). This option allows the user to stress the web server with NUM number of simulated users. The amount is limited only by the computing resources available, but realistically a couple of hundred simulated users is equal to many times that that number in actual user sessions. The number you select represents the number of transactions your server is handling. It does NOT represent the number of concurrent sessions. Remember, real users take some time to actually read the page that they've requested....
' -i '' --internet 'This option is used with a configuration file, that is a file containing many URLs. With this option in place, each user randomly hits any one of the URLs in the file each time it hits the server. Much like you can't tell the users of your website which pages they can view, you have no control over which pages siege will hit in internet mode. With this option set, there is no guarantee that every page in the file will be hit.
' -t NUMm '' --time=NUMm 'TIME, allows you to run the test for a selected period of time. The format is "NUMm", where NUM is a time unit and the "m" modifier is either S, M, or H for seconds, minutes and hours. To run siege for an hour, you could select any one of the following combinations: -t3600S, -t60M, -t1H. The modifier is not case sensitive, but it does require no space between the number and itself.
' -f FILE '' --file=FILE 'The default configuration file, the file with all your URLs is SIEGE_HOME/etc/urls.txt. You can use this option to instruct siege to use a different configuration file: siege --file=serverb.txt
' - l '' --log 'This option instructs siege to log the statistics to SIEGE_HOME/var/siege.log. Each new statistics set is appended to the log.
' - m MESSAGE '' --mark=MESSAGE 'This option allows you to mark the log file with a separator, to differentiate your log file entries with header information. It is not necessary to use both the -m option and the -l option. -m assumes -l so it marks and logs the transaction. If the MESSAGE has spaces in it, make sure that you put it in quotes.
' -d NUM '' --delay=NUM 'Each siege simulated user is delayed for a random number of seconds between one and NUM. If you are benchmarking performance, it is recommended that you use a 1 second delay ( -d1 ). The default value is three (3 ). This delay allows for the transactions to stagger rather then to allow them to pound the server in waves.
Configuration File
Beginning with version 2.00, siege supports a settings configuration file in which you can store most of your command line options. This makes it much easier to invoke siege and it helps you ensure that every run in a series of runs has the exact same settings. Where is it? The siege configuration file is called .seigerc and it is located in the home directory directory of the user who installed siege. If you did not install siege but you want to use it, then run the command "siege.config". That will put a template .siegerc file in your home directory. You can edit the settings with your favorite editor. The .siegerc file which is generated by the siege.config utility is well documented with comments describing the directives. Those comments should be all you need to get started.
URL Format
Siege understands the following URL format: [protocol://] [servername.domain.xxx] [:portnumber] [/directory/file] Currently, siege only supports http and https protocols. HTTP is the default protocol and therefore does not require a protocol specification. Frankly siege allows you to cheat. The minimum requirement is this: servername. That's it. So if you're in the same domain as a server named shemp and shemp is in your host file or it is in DNS, then: siege -u shemp will stress http://shemp.yourdomain.net/index.shtml ( assuming that is the server specified index ). If you want to lay siege to https servers, then it is necessary to specify the protocol. In the above example, siege -u https://shemp/ Will lay siege to https://shemp.yourdomain.net/index.shtml given the assumptions stated above.
The URLs File
In order to run a regression test or an effective internet simulation, you are going to have to run through the URLs on the server you are testing. To accomplish that, place the URLs in a configuration file. The default file is SIEGE_HOME/etc/urls.txt. In that file list the URLs one per line:
comments behind hashes
http://homer.whoohoo.com/index.shtml http://homer.whoohoo.com/howto.jsp http://homer.whoohoo.com/cgi-bin/hello.pl?first=bart&last=simpson
and on and on....
Siege-2.06 and later supports both POST and GET directives. GET directives are constructed as shown above, but POST directives require the POST keyword. Construct POST directives as follows:
http://homer.whoohoo.com/cgi-bin/hello.pl POST name=homer http://homer.whoohoo.com/haha.jsp POST word=doh!&scope=ALL
When invoked without the URL option [ -u URL --url=URL ], siege looks for URLs in a file. It reads it into memory and runs through the URLs. Normally siege starts at the beginning of the file and works it way through it sequentially. If you specify internet mode [ -i --internet ], then it selects URLs randomly. An alternative file can be selected at run time with the [ -f FILE --file=FILE ] option.
Variables
Beginning with release 2.57, siege supports variable declaration and evalutation in both the .siegerc file and the urls.txt file. Siege employs variable syntax similar to UNIX shell. They are declared one per line in the file: varname=value To refererence a varaible, you must place it inside $() or ${}. In the example above, type $(varname) to access value. You can use variables to switch between two protocols with one quick edit in your urls.txt file. For example: PROT=https://$(PROT)eos.joedog.org/siege/index.php$(PROT)eos.joedog.org/wacky/index.php$(PROT)eos.joedog.org/scout/index.php$(PROT)eos.joedog.org/libping/index.php$(PROT)eos.joedog.org/gunner/index.phpNow, in order to switch between https and http, you need only edit one line in the entire file.
Log File
When siege is invoked with logging enabled [-l/--log], the program records the transaction in PREFIX/var/siege.log where PREFIX is the directory in which siege was installed. ( see the file INSTALL for instructions. ) The transaction logged is similar to standard output display at the end of every siege run. However, the information is arranged in comma separated text for easy import into a spread sheet. The logging option enables you to maintain history and chart progress over time. In order to group runs by conditions such as URL, server or even protocol, the -m "message"/--mark="message" option was added. This places the mark "message" in the log file. So that if you switched your testing from http to https, you could mark the log with "start HTTPS testing" -m/--mark assumes logging and makes the -l/--log option unnecessary.
Performance Statistics
Performance measures include elapsed time of the test, the amount of data transferred ( including headers ), the response time of the server, its transaction rate, its throughput, its concurrency and the number of times it returned OK. These measures are quantified and reported at the end of each run. The reporting format is modeled after Lincoln Stein's torture.pl script. This is a sample of siege output:
Ben: $ siege -u shemp.whoohoo.com/Admin.jsp -d1 -r10 -c25..Siege 2.65 2006/05/11 23:42:16..Preparing 25 concurrent users for battle.The server is now under siege...doneTransactions: 250 hitsElapsed time: 14.67 secsData transferred: 448000 bytesResponse time: 0.43 secsTransaction rate: 17.04 trans/secThroughput: 30538.51 bytes/secConcurrency: 7.38Status code 200: 250Successful transactions: 250Failed transactions: 0
Transactions is the number of server hits. In the example, 25 simulated users [ -c25 ] each hit the server with 10 repetitions [ -r10 ], a total of 250 transactions.
Elapsed time is the duration of the entire siege test. This is measured from the time the user invokes siege until the last simulated user completes its transactions. Shown above, the test took 14.67 seconds to complete.
Data transferred is the sum of data transferred to every siege simulated user. It includes the header information as well as content. Because it includes header information, the number reported by siege will be larger then the number reported by the server. In internet mode, which hits random URLs in a configuration file, this number is expected to vary from run to run.
Response time is the average time it took to respond to each simulated user's requests.
Transaction rate is the average number of transactions the server was able to handle per second, in a nutshell: transactions divided by elapsed time.
Throughput is the average number of bytes transferred every second from the server to all the simulated users.
Concurrency is average number of simultaneous connections, a number which rises as server performance decreases.
Successful transactions is the number of times the server returned a code less then 400. Accordingly, redirects are considered successful transactions.
Availabilty
New release annoucements are made on freshmeat.net. To sign up for new release information, click HERE. New versions are available for download via anonymous FTP, click HERE for downloads.
Machines
Multi-threaded siege was built and tested successfully on the following platforms: AIX( powerpc-ibm-aix4.2.1.0 ) Siege has been built and tested successfully using IBM C for AIX version 5. It currently does not support gcc on that platform.
GNU/Linux( i[56]86-pc-linux-gnu ) Siege was originally developed on SuSE GNU/Linux with gcc, there are no known issues on this platform.
HP-UX ( hppa2.0w-hp-hpux11.00 ) Siege has been built on this platform with both the HP ANSI C compiler and gcc.
Solaris( sparc-sun-solaris2.[678] ) Siege LOVES this platform. It was built and tested successfully using gcc.
Microsoft Windows( pc-i686-cygwin ) Siege was ported to Cygwin by Funkman. It runs nicely on all versions greater than or equal to 1.5.
Authors
Jeffrey Fulmer -- Designed and implemented Siege in his position as Webmaster for Armstrong World Industries
License information
Please consult the file, COPYING for complete license information. Copyright (C) 2000-2003 Jeffrey Fulmer Permission is granted to anyone to make or distribute verbatim copies of this document as received, in any medium, provided that the copyright notice and this permission notice are preserved, thus giving the recipient permission to redistribute in turn. Permission is granted to distribute modified versions of this document, or of portions of it, under the above conditions, provided also that they carry prominent notices stating who last changed them.