Home | Documentation |
Test Messenger
updated Sun Feb 16 2020 by Robert van Engelen
|
The Test Messenger tool is an advanced "fuzzer" software testing tool to test XML SOAP and REST Web services and client applications. Testing is performed by sending XML test messages (randomized in case of fuzzing) to services and client applications.
It is important to test Web services and client applications as part of a quality assurance process. Furthermore, the functional and non-functional requirements of an application should be verified and tested before the application is deployed in the field. Testing Web services with randomized message exchange patterns can help to pinpoint problems early on, before applications are deployed.
The benefit of the Test Messenger is the ability to use a set of message templates for unit and regression testing, and "fuzz testing" with randomization of XML attribute and element values in the message templates, including the generation of input errors, when specified. Randomization is driven by XML annotations in the message templates. These message templates are automatically generated by soapcpp2 with option -g
.
The Test Messenger tool is executed from the command line or from a shell script, such as a script intended to test services and clients. No coding is required to use the Test Messenger.
The testmsgr Test Messenger tool is located in the gSOAP distribution package under gsoap/samples/testmsgr
and is built with:
cd gsoap/samples/testmsgr make
To generate randomized XML messages with the Test Messenger, a SOAP/XML message template is required. The Test Messenger populates the template with random values to send the randomly populated SOAP/XML request message to a service for Server-side testing . Or the Test Messenger can be started as a stand-alone server to return the randomly populated response message template to a client for Client-side testing .
You can create a SOAP/XML message template manually or auto-generate it with soapcpp2 option -g
. For example, to auto-generate XML sample message templates from an interface header file file.h
(e.g. generated with wsdl2h from WSDLs), use soapcpp2 with option -g
:
soapcpp2 -g file.h
The soapcpp2 tool generates sample messages for the service operations defined in file.h
. Option -g
changes the sample message to message templates that contain placeholders of the form %[...]%
. A leading ???
qualifier indicates optional elements and attributes. Repeated elements and/or CDATA is indicated and enclosed in <__REPEAT>
...</__REPEAT>
. Element selections are indicated and enclosed as child elements in <__SELECT>
...</__SELECT>
. Element permutations are enclosed as child elements in <__PERMUTE>
...</__PERMUTE>
.
A placeholder value %[...]%
specifies an XSD-related type, enumeration, or numeric range. These placeholders are populated by the Test Messenger using random values that conform to the placeholder's specified value type.
While the sample messages generated by soapcpp2 (without option -g
) can be used to test clients and services as explained in the gSOAP user guide, the problem is that the content of the sample messages is always fixed. This provides only very limited testing capabilities. By contrast, the Test Messenger tool generates randomized messages based on SOAP/XML message templates for enhanced testing of client and service applications.
The Test Messenger is intended to test common messaging scenarios that are defined by SOAP/XML message templates. The Test Messenger does not add (random) XML attributes and elements that are not already in the message template. It can randomly remove attributes and elements with option -p
to test the robustness of an application to consuming partial and possibly invalid XML messages (invalid according to the schema of the XML message).
The Test Messenger has several options to further randomize the messages by randomly removing attributes and elements at a specific rate of chance, or to add random spacing to XML attribute and element values, and to continuously test a service until it fails.
To demonstrate the Test Messenger, assume we have the following services declared in the interface file.h
:
We run soapcpp2 on this interface file.h
with option -g
:
soapcpp2 -g file.h
This produces the following SOAP/XML template files among other source code files:
example.upload.req.xml
XML template upload request messageexample.download.req.xml
XML template download request messageexample.upload.res.xml
XML template upload response messageexample.download.res.xml
XML template download response messageThe example.upload.req.xml
file looks like this:
Note that XML values in this template are represented by value placeholders of the form %[...]%
. Also note that the person
element, the id
attribute and miles
are optional.
A leading ???
qualifier indicates optional elements and attributes, when the ???
qualifier appears after the opening tag of an element (i.e. in front of the element value or child elements) and in attribute values, respectively. For the full list of placeholders that can be used, see Placeholder values .
This message template is read, populated and then sent to the standard output by testmsgr as follows:
./testmsgr -r1 -i example.upload.req.xml
This produced a randomized SOAP/XML message to standard output from the example.upload.req.xml
template message. Option -r1
sets the random seed to 1
. Any seed value between 1 and 2147483647 can be specified with -r
. Option -i
displays the randomized XML with indentation.
To randomly retain or discard optional elements and attributes, use option -q
with a specified percentage to retain optional elements and attributes in the generated XML message:
./testmsgr -r1 -q50 -i example.upload.req.xml
This retains the optional element person
, the id
attribute, and the miles
element with a 50% chance each.
To send the randomized SOAP/XML message to a service endpoint, pass the endpoint URL as the last argument to the testmsgr command:
./testmsgr -r1 -i example.upload.req.xml http://www.example.com/service
For example, we can let soapcpp2 generate a simple echo test service to which we can send messages. Since we auto-generate a simple echo test service we have no server-side logic to return meaningful values for this example.
First we invoke soapcpp2 with option -T
to generate the server-side code for a simple test service:
soapcpp2 -g -SL -T file.h
This generates the SOAP/XML message templates and the following source code files:
example.wsdl
a WSDL to publish the serviceexample.nsmap
an XML namespace mapping table to include in your source codesoapStub.h
annotated copy of the source interface filesoapH.h
serialization functionssoapC.cpp
serialization functionssoapServer.cpp
service request dispatchersoapTester.cpp
auto-test echo serverNext, we compile these files together with the gSOAP library and the custom serializer for struct tm
to build a simple echo server:
c++ -I. -o echoserver soapTester.cpp soapServer.cpp soapC.cpp stdsoap2.cpp custom/struct_tm.c
Now run the server. For example with arguments 8080 as the port to bind to and 8192 to set the SOAP_XML_INDENT
engine context flag:
./echoserver 8192 8080
Finally, let's fire a randomized message to the simple echo service, using the auto-generated example.upload.req.xml
message template:
./testmsgr -r1 example.upload.req.xml http://localhost:8080
The testmsgr displays the response received from the server, which for the simple echo service upload request is the upload response with an id element.
To send a continuous stream of randomized message to the server, use option -c
:
./testmsgr -c -r1 example.upload.req.xml http://localhost:8080
This continous until the service fails. A test is reproduced by using its random seed. So if the service fails you can replicate the message with the random seed and option -r
with this seed value.
The Test Messenger tool can also send pruned XML messages with option -p
. Pruning means that attributes and elements are randomly discarded, which allows for testing the robustness of the service against incomplete and malformed SOAP/XML content:
./testmsgr -c -p80 -r1 example.upload.req.xml http://localhost:8080
Option -p80
means that 80% is retained and 20% of attributes and elements are randomly discarded from the SOAP/XML template example.upload.req.xml
.
When testing the auto-test echo service, the echo service displays failed requests and is silent for successful requests.
For the full list of testmsgr options, see Options .
To build testmsgr, install gSOAP and build testmsgr as follows:
cd gsoap/samples/testmsgr make testmsgr
This builds the command-line tool testmsgr in gsoap/samples/testmsgr
from where you can use it and/or copy it for use with your projects.
If you do not have the samples built, you can use soapcpp2 from the command line to generate the C++ code required for testmsgr:
cd gsoap/samples/testmsgr soapcpp2 -CSL ../../import/dom.h c++ -I../.. -o testmsgr testmsgr.cpp soapC.cpp ../../dom.cpp ../../stdsoap2.cpp
This builds the testmsgr command-line tool.
For users of Windows, visit download and installation to download testmsgr.exe
.
The testmsgr command takes a set of options, a SOAP/XML message file or a message template file (i.e. to randomize with option -r
), and an optional service endpoint URL.
To test services, run testmsgr against the service endpoint URL, for example when the service runs on the localhost on port 8080:
./testmsgr -r99 example.upload.req.xml http://localhost:8080
When using https endpoints, you must build testmsgr with compile-time flag WITH_OPENSSL
and link OpenSSL as follows:
soapcpp2 -CSL ../../import/dom.h c++ -DWITH_OPENSSL -o testmsgr testmsgr.cpp soapC.cpp ../../dom.cpp ../../stdsoap2.cpp -lssl -lcrypto
For CGI-based services that read SOAP/XML requests from standard input you can redirect the (randomized) message to the CGI-based service. For example:
./testmsgr -r99 example.upload.req.xml | ./echoserver
where echoserver is a CGI-based service (the echoserver built in the Introduction uses CGI by default).
To test clients, run testmsgr as a stand-alone server by specifying a port number instead of a service endpoint URL:
./testmsgr -r1 example.download.res.xml 8080
This runs the Test Messenger on port 8080. The Test Messenger discards request messages sent by clients and returns randomized responses using the SOAP/XML example.download.res.xml
response message template.
You can also use file redirection to test clients when your client uses the "http://" endpoint URL without a domain. By using this URL your client writes HTTP messages to standard output and reads HTTP responses from standard input. For example:
./testmsgr -r1 example.download.res.xml > response.xml ./client < response.xml > /dev/null
testmsgr [-aact] [-b] [-c] [-dnum] [-f] [-h] [-i] [-j] [-lmax] [-mmax] [-nlen] [-ofile] [-pnum] [-qnum] [-rseed] [-tsec] [-v] [-A] [-C] [-H] [-M] [-Oopt] [-Pper] [-Rrep] [-Ssel] [-Xuid:pwd] [-Yhost[:port[:uid:pwd]]] [infile|-] [URL|port]
where the testmsgr command-line options are:
Option | Description |
---|---|
-a act | add a HTTP SOAP Action header value "act" |
-b | add random spacing to randomized values when using -r |
-c | continuous testing until the service endpoint fails, when using a URL |
-d num | test a service endpoint num times or until it fails, when using a URL |
-f | exit on any HTTP error or SOAP fault when using -c or -d |
-h | display help message |
-i | indent XML with spaces when using -r |
-j | indent XML with tabs when using -r |
-l max | max number of randomized element repeats when using -r (default is 3) |
-m max | max length of randomized text values generated (default is 10) |
-n len | display the server response up to len bytes |
-o file | save to file instead of stdout |
-p perc | percentage of XML kept in general when using -r (default=100) |
-q perc | percentage of XML kept of optional/repeated indicators when using -r (default=100) |
-r seed | randomize XML from XML message templates (use soapcpp2 -g ) |
-t sec | socket idle timeout seconds (default=1), to test response time |
-v | verbose mode |
-A | enable SOAP with MIME attachments (SwA) |
-C | enable HTTP chunked transfers |
-H | add HTTP headers when no URL is specified |
-M | enable MTOM (application/xop+xml) |
-O opt | set the XML optional value indicator (default="???") |
-P per | set the XML element permutation indicator tag (default=__PERMUTE ) |
-R rep | set the XML element repetition indicator tag (default=__REPEAT ) |
-S sel | set the XML element selection indicator tag (default=__SELECT ) |
-X uid:pwd | connect with authentication credentials uid and pwd |
-Y host[:port[:uid:pwd]] | connect via proxy host, port, and proxy credentials uid and pwd |
infile | XML message or XML message template with indicator annotations |
- | instead of infile: read XML message or XML message template from standard input |
URL | optional endpoint URL of service to test |
port | optional stand-alone server port for clients to test against |
More specifically, infile
is an SOAP/XML message file (or message template file), -
specifies that the message should be read from standard input, and URL
is the optional service endpoint to send (randomized) messages to. When a port
number is specified instead of a URL, testmsgr runs as a stand-alone application to test clients that connect to it. A client application receives the (randomized) message generated by testmsgr from the specified SOAP/XML message file (or message template file).
-a act
Use HTTP SOAP Action header value "act"
with the SOAP request messages sent. This option is required for services that require a SOAP Action header.
-b
Add random blank spacing to randomized values when using option -r
.
-c
Continuous testing of a service endpoint until the service fails, i.e. until the server at the specified endpoint URL becomes unresponsive.
-d num
Test a service endpoint num
times or until the service fails, i.e. until the server at the specified endpoint URL becomes unresponsive.
-f
Exit on any HTTP error or on any SOAP fault by the service endpoint when using option -c
or -d num
, instead of continuing until the specified endpoint URL becomes unresponsive.
-h
Display help message.
-i
Indent XML with spaces when using option -r
.
-j
Indent XML with tabs when using option -r
.
-l max
Specifies the max number of randomized element repetitions for __REPEAT
and __SELECT
with min and max indicators, when using option -r
. The default is
-m max
Specifies the max length of randomized text values generated when using option -r
. The default is 255. The maximum value that can be specified is 18446744073709551615.
-n len
Display the server response up to len
bytes. -n0
removes the server responses. Omitting this option displays the full response.
-o file
Save output to file. This option is not applicable when URL
or port
are specified.
-p perc
Retain XML with the specified percentage chance, when using option -r
. XML attributes and elements are removed by chance as specified. For example, -p10
keeps only roughly 10% of the XML elements and attributes of the original message. The default is 100 (retain all XML).
-q perc
Retain XML qualified with ???
and elements repeated with the __REPEAT
indicator with the specified percentage chance, when using option -r
. For example, if a message has an element or attribute value ???%[[INT32]]
then -q10
keeps the element or attribute with a 10% chance. Likewise, optional XML elements enclosed in <__REPEAT>
as indicated with min
and max
bounds are kept with a 10% chance. The default is 100 (retain).
-r seed
Randomize the SOAP/XML message template given as the infile
argument. SOAP/XML message templates can be auto-generated with soapcpp2 option -g
. The seed
value is required and is used to reproduce test messages. Any positive seed value can be specified, from 1 up to 2147483647.
-t sec
Sets a socket idle timeout in seconds. The default is one second. A service is considered unresponsive when a socket timeout occurs while messages are exchanged.
-v
Enable verbose mode.
-A
Use SOAP with MIME attachments, also called SOAP with Attachments (SwA). This option generates a MIME message with SOAP Body and zero or more MIME attachments. When combined with option -r
, a MIME attachment is added for each element with binary content (base64Binary XML element) specified in the template with %[[BASE64]]%
. The template %[[BASE64]]%
value is replaced with an href
attribute referencing the MIME attachment with randomized content.
-C
Use HTTP chunked transfers.
-H
Add HTTP headers to the generated message even when no URL is specified on the command line.
-M
Use MTOM (application/xop+xml) messaging. This option generates a MIME message with SOAP Body and zero or more MTOM attachments in MIME format. When combined with option -r
, a MIME attachment is added for each element with binary content (base64Binary XML element) specified in the template with %[[BASE64]]%
. The template %[[BASE64]]%
value is replaced with an <xop:Include xmlns:xop="http://www.w3.org/2004/08/xop/include"/>
element with an href
attribute referencing the MIME attachment with randomized content.
-O opt
Sets the XML optional value indicator. The default is ???
. See Element repetition, selection and permutation indicators .
-P per
Sets the XML element permutation indicator tag. The default is __PERMUTE
. See Element repetition, selection and permutation indicators .
-R rep
Sets the XML element repetition indicator tag. The default is __REPEAT
. See Element repetition, selection and permutation indicators .
-S sel
Sets the XML element selection indicator tag. The default is __SELECT
. See Element repetition, selection and permutation indicators .
-X uid:pwd
Sets the basic or http digest authentication credentials.
-Y host:port:uid:pwd
Connect to the server through a proxy server. Sets the proxy host with optional proxy port and optional credentials.
A SOAP/XML message template file is required when sending randomized messages with testmsgr option -r
. The SOAP/XML message template file can be auto-generated with soapcpp2 option -g
on a interface file. This interface file is usually generated with wsdl2h from a (set of) WSDL and XSD file(s). Note that option -g
does not produce SOAP/XML message templates if the interface file does not declare any service operations.
Templates may contain placeholder values. Placeholder values may start with ???
meaning the value is optional, followed by the placeholder value type, an enumeration, or a numeric range. Values are inserted with testmsgr option -r seed
.
Element and attributes that are qualified as optional with ???
are randomly discared with options -r
and -q
, where -q
specifies the percentage of elements and attributes that are retained in the generated XML message. A zero percentage specified with -q0
removes all optional values. A value of 100 specified with -q100
retains all optional values (the default) in the generated XML message.
The ???
qualifier may appear after the opening tag of an element (i.e. in front of the element value or child elements) and in attribute values. For example:
where id="???%[[INT32]]"
is an optional attribute with a 32-bit integer value, <name>???%[[TEXT]]%</name>
is an optional element with text, <contact>???
is an optional element with subelements, and <data lang="%[[LANG]]%">???</data>
is an optional empty element with an attribute.
Enumeration placeholders can be used anywhere a value is expected and is used to provide an enumeration of possible values to choose from for element and attributes.
%[[A][B][C]]%
specifies a choice of values A
, B
or C
. The values can be anything but should not contain ][
or ]]%
. For example:
where completed="%[[false][true]]%"
enumerates the values false
and true
. Note that this enumeration is identical to the %[[BOOL]]%
placeholder type.
Pattern placeholders can be used anywhere a value is expected and is used to provide a pattern to generate randomized values.
A pattern placeholder %[['pattern']]%
specifies a pattern in the usual XML regular expression syntax, for example:
A quote (') in the pattern should be escaped with a backslash as in \', which the soapcpp2 tool automatically converts with option -g
to produce pattern facets.
The length of the generated string may be constrained with %[['pattern'[N:M]]]%
. The range N:M
specifies the number of characters.
\p{C}
are not yet supported in gSOAP versions 2.8.62-2.8.64.N:M
in %[['pattern'[N:M]]]%
should be viable given the pattern's inherent minimum and maximum lengths. For example, %[['[a-z]+'[0:10]]]%
is viable, but %[['id[0-9]{3}'[0:3]]]%
is not viable. There is a possibility that the generated pattern does not meet the [N:M]
constraints provided.A numeric range is integer-valued when the lower bound N
and upper bound M
are integer. Otherwise, the numeric range values are floating point. For example, %[[0:10]]%
speficies a range of integer values whereas %[[0.0:10.0]]%
specifies a range of floating point values. The bounds of a numeric range can be made exlusive by using (
and )
instead of [
and ]
.
%[[N:M]]%
specifies a range of integer or float values between N
and M
.
%[[N:M)]%
specifies a range of integer or float values between N
and M
(exclusive).
%[(N:M]]%
specifies a range of integer or float values between N
(exclusive) and M
.
%[(N:M)]%
specifies a range of integer or float values between N
(exclusive) and M
(exclusive).
For example:
where 0.00 is excluded from the float value range and 100000.00
is the inclusive maximum of the numeric range.
A decimal precision is specified with a printf-based format string, such as %.2f
to indicate 2 decimal places after the decimal period should be used. The decimal precision is added to the range in the template. For example, %[[0.0:999.0%.2f]]%
specifies a floating point range between 0.00 and 999.00 with 2 decimal places after the period. The total field width (total digits) can be specified with %[[0.0:999.0%6.2f]]%
for example.
The precision of a "limitless" floating point and double precision floating point numeric precision is specified with %[[FLOAT%.2e]]%
and %[[DOUBLE%.2e]]%
respectively. This produces numeric values within the range of the specified type formatted according to the specified printf-format string.
%[[BASE64]]%
and length-restricted %[[BASE64[N:M]]]%
specify xsd:base64Binary XSD values. The length restriction N:M
specifies the range of base64-encoded bytes.
%[[BOOL]]%
specifies xsd:boolean XSD values.
%[[DATE]]%
specifies xsd:date XSD values.
%[[DATETIME]]%
specifies xsd:dateTime XSD values.
%[[DOUBLE]]%
specifies xsd:double XSD values.
%[[DURATION]]%
specifies xsd:duration XSD values.
%[[ENTITY]]%
and length-restricted %[[ENTITY[N:M]]]%
specify xsd:ENTITY XSD values. The range N:M
specifies the number of characters.
%[[HEX]]%
and length-restricted %[[HEX[N:M]]]%
specify xsd:hexBinary XSD values. The length restriction N:M
specifies the range of encoded bytes, not the number of hexadecimal characters.
%[[ID]]%
and length-restricted %[[ID[N:M]]]%
specify xsd:ID XSD values. The range N:M
specifies the number of characters.
%[[IDREF]]%
and length-restricted %[[IDREF[N:M]]]%
specify xsd:IDREF XSD values. The range N:M
specifies the number of characters.
%[[FLOAT]]%
specifies xsd:float XSD values.
%[[INT8]]%
and %[[BYTE]]%
specify xsd:byte XSD values.
%[[INT16]]%
and %[[SHORT]]%
specify xsd:short XSD values.
%[[INT32]]%
and %[[INT]]%
specify xsd:int XSD values.
%[[INT64]]%
and %[[LONG]]%
specify xsd:long XSD values.
%[[LANG]]%
specifies xsd:language XSD values.
%[[NAME]]%
and length-restricted %[[NAME[N:M]]]%
specify xsd:Name XSD values. The range N:M
specifies the number of characters.
%[[NCNAME]]%
and length-restricted %[[NCNAME[N:M]]]%
specify xsd:NCName XSD values. The range N:M
specifies the number of characters.
%[[NMTOKEN]]%
and length-restricted %[[NMTOKEN[N:M]]]%
specify xsd:NMTOKEN XSD values. The range N:M
specifies the number of characters.
%[[QNAME]]%
specifies xsd:QName XSD values.
%[[TEXT]]%
and length-restricted %[[TEXT[N:M]]]%
specify xsd:string, xsd:normalizedString, xsd:anyURI, xsd:NOTATION, and xsd:token XSD values. The range N:M
specifies the number of characters.
%[[TIME]]%
specifies xsd:time XSD values.
%[[TOKEN]]%
and length-restricted %[[TOKEN[N:M]]]%
specify xsd:token XSD values. The range N:M
specifies the number of characters.
%[[UINT8]]%
and %[[UBYTE]]%
specify xsd:unsignedByte XSD values.
%[[UINT16]]%
and %[[USHORT]]%
specify xsd:unsignedShort XSD values.
%[[UINT32]]%
and %[[UINT]]%
specify xsd:unsignedInt XSD values.
%[[UINT64]]%
and %[[ULONG]]%
specify xsd:unsignedLong XSD values.
%[[URI]]%
specifies xsd:anyURI XSD values.
Three special XML elements __REPEAT
, __SELECT
and __PERMUTE
in message templates indicate XML element repetitions and selections, respectively. These elements are processed by the Test Messenger to generate child elements, after which the __REPEAT
, __SELECT
and __PERMUTE
XML elements are removed from the generated randomized XML messages.
These indicators require testmsgr option -r seed
.
Child elements and/or CDATA text in the __REPEAT
indicator element are repeated within a range specified by min
and max
attributes of the __REPEAT
element. The min
and max
attributes are optional and default to 1 when omitted.
Option -q
of testmsgr controls the percentage of child elements that are generated within the range. A zero percentage specified with -q0
generates the minimum number of child elements. A value of 100 specified with -q100
generates the maximum number of child elements. The maximum number of child elements (randomly) generated is bounded by option -l
(default is 3). Therefore, the maximum number of child elements is the greater of attribute min
and the lesser of attribute max
and the value lmax specified with option -l
:
min <= n <= maximum(min, minimum(max, lmax))
where n is the number of child elements generated with a random probability percentage specified by option -q
.
For example, to repeat an item
element with an integer value two or more times use the __REPEAT
indicator element with item
as a child element:
Assuming that testmsgr options -r1
, -l10
and -q50
are used, the generated XML message has at least two item
elements and at most 10 elements, with on average 50% of the elements kept between 2 (the min
minimum) and 10 elements (the maximum specified by -l10
), i.e. 6 elements on average. Each item
element has a random integer value.
Multiple child elements can be repeated as a group, which may include mixed CDATA content of any type, for example:
This repeats <product>
with optional <number>
up to 10 times, with mixed hexadecimal text included.
Note that the ???
qualifier can be used with child elements to make them optional in repetitions as shown in the example.
The testmsgr tool recognizes SOAP 1.1/1.2 arrays with array type and size attributes. These attributes are automatically changed in the randomized XML output. The array element should have one or more nested __REPEAT
indicators. Consider for example the following 2D array:
The SOAP-ENC:arrayType
attribute array dimension values are automatically adjusted in the XML output by testmsgr depending on the number of item
elements generated.
One child element of the __SELECT
indicator element is uniform randomly selected as an XML element for the generated randomized XML message.
For example, to select one element from the three choices of elements A
, B
or C
:
The generated XML message has either an A
element with an integer value, a B
element with a floating point value, or a C
element with a text value.
Note that the ???
qualifier can be used with child elements to make them optional in selections.
Child element of the __PERMUTE
indicator element are uniform randomly permuted in the generated randomized XML message.
For example, elements A
, B
or C
are permuted with:
The generated XML message has A
, B
and C
elements produced in any order.
Note that the ???
qualifier can be used with child elements to make them optional in permutations.
Copyright (c) 2018, Robert van Engelen, Genivia Inc. All rights reserved.