The WS-ReliableMessaging plugin

Table of Contents

• WS-ReliableMessaging Setup
• WS-ReliableMessaging Information Header Bindings
• WS-ReliableMessaging Overview
• Client-side Usage
  • Creating, Closing, and Terminating Message Sequences
  • Creating a Sequence without an Offer
  • Exchanging Messages in a Sequence
  • Exchanging Messages with Acknowledgements in a Sequence
  • Resending Non-Acknowledged Messages
  • Relaying Response and Fault Messages with WS-Addressing
  • Using Retry Loops to Improve Robustness of Message Sends
  • Example Client
• Server-side Usage
  • Setting up a WS-RM Destination Service
  • Handling Duplex Callback Service Operations
• WS-ReliableMessaging over HTTPS with Basic Authentication
• WS-ReliableMessaging over UDP with Timeouts
• WS-ReliableMessaging and WS-Security
• The wsrm Plugin and C++ Proxy and Server Objects
• Speed improvements with large number of sequences

WS-ReliableMessaging Setup

The material in this section relates to the WS-ReliableMessaging and WS-Addressing (2005) specifications.

To use the wsrm plugin:

1. Run wsdl2h -b -t typemap.dat on a WSDL of a service that requires WS-ReliableMessaging and WS-Addressing headers. The typemap.dat file included in the gSOAP package is used to recognize and translate header blocks. The -b option ensures that one-way response message operations are added for duplex communications that require a client-side callback service.
2. Run soapcpp2 -a on the header file produced by wsdl2h. To enable reliable-messaging and addressing-based service operation selection, you MUST use soapcpp2 option -a. This allows the service to dispatch methods based on the WS-Addressing action information header value (when the wsa plugin is registered).
3. (Re-)compile and link stdsoap2.c/pp or libgsoap, (dom.c/pp when needed), plugin/wsrmapi.c, plugin/wsaapi.c, custom/duration.c, and the soapcpp2- generated source files.
4. Use the wsrm plugin API functions described below. The wsrm plugin uses the wsa plugin to implement the WS-Addressing 2005 operations. Both must be registered. The wsrm plugin API is self-contained. There is no need to use the wsa plugin API, unless WS-Addressing-specific headers must be added to messages.
5. For WCF compatibility and interoperability, compile wsrmapi.c with compiler flag -DWITH_WCF for enhanced WCF compatibility and use compiler flag -DWITH_WCF_SIM to enable simulated WCF features channel instance and netrm BufferRemaining.

An example wsrm client/server application can be found in samples/wsrm.

A gSOAP service definitions header file with an #import "wsrm.h" to support WS-ReliableMessaging 1.1 or #import "wsrm5.h" to support WS-ReliableMessaging 1.0 (2005). The imports are automatically generated by wsdl2h for a set of WSDLs that use WS-ReliableMessaging as per WS-Policy. The wsdl2h-generated header file should be further processed by soapcpp2 to generate the binding code. The wsrmapi.h and wsrmapi.c implement the WS-ReliableMessaging API described in this document.

A wsdl2h-generated service definitions header file might include the following imports, where the wsrm.h is mandatory to support WS-ReliableMessaging:

#import "soap12.h"
#import "wsrm.h"

The wsrm.h header file is imported from import/wsrm.h by soapcpp2. The wsrm.h import can be manually added to enable WS-ReliableMessaging when needed. The gSOAP service definitions header file is processed with soapcpp2 to generate the client-side and/or server-side binding code.

Note that the wsrm.h and the WS-ReliableMessaging-dependent wsrx.h and wsa5.h header files are located in the import directory of the gSOAP package. These files define the WS-ReliableMessaging and WS-Addressing information header elements and types. The wsrx.h header file defines the WS-ReliableMessaging CreateSequence, CloseSequence, and TerminateSequence operations, as well as an one-way SequenceAcknowledgement operation to accept acknowledgements. The soap12.h header file enables SOAP 1.2 messaging.

For developers working on protocols: the WS-ReliableMessaging header blocks in wsrm.h were generated from the WS-ReliableMessaging schema with the wsdl2h tool and WS/WS-typemap.dat as follows:

wsdl2h -cgyex -o wsrm.h -t WS/WS-typemap.dat WS/WS-ReliableMessaging.xsd

This step is not needed to use the wsrm plugin.

Note

The WS-ReliableMessaging and WS-Addressing protocols do not use SOAP-RPC encoded id-ref attributes. To remove id-ref serialization e.g. when using document/literal SOAP/XML messaging, use the runtime SOAP_XML_TREE flag or compile the source code with WITH_NOIDREF.

WS-ReliableMessaging Information Header Bindings

To associate WS-ReliableMessaging and WS-Addressing information headers with service operations, the SOAP Header struct SOAP_ENV__Header must have been declared and contain the necessary header blocks to be transported with SOAP messages. The SOAP_ENV__Header for WS-ReliableMessaging and WS-Addressing is predefined in wsrm.h and imported into the gSOAP service definitions header file (this is automatically generated by wsdl2h). For each service operation in the gSOAP service definitions header file that uses WS-ReliableMessaging and/or WS-Addressing method-header-part directives are used.

For example, the following gSOAP service definitions header file illustrates a typical import and service operation definition of operation 'example' in service namespace 'ns':

#import "wsrm.h"

//gsoap ns service method-header-part: example wsa5__MessageID
//gsoap ns service method-header-part: example wsa5__RelatesTo
//gsoap ns service method-header-part: example wsa5__From
//gsoap ns service method-header-part: example wsa5__ReplyTo
//gsoap ns service method-header-part: example wsa5__FaultTo
//gsoap ns service method-header-part: example wsa5__To
//gsoap ns service method-header-part: example wsa5__Action

//gsoap ns service method-header-part: example wsrm__Sequence
//gsoap ns service method-header-part: example wsrm__AckRequested
//gsoap ns service method-header-part: example wsrm__SequenceAcknowledgement

//gsoap ns service method-action:          example urn:example/examplePort/example
//gsoap ns service method-output-action:   example urn:example/examplePort/exampleResponse

int ns__example(char *in, struct ns__exampleResponse { char *out; } *);

The wsa5 information headers are defined in wsa5.h and imported by wsrm.h (both are located in the 'import' directory). Here, all of the WS-Addressing and WS-ReliableMessaging information headers are bound to the ns__example operation request and response messages.

The method action directive is important for WS-Addressing, because WS-Addressing Action information headers must be included that are unique for each operation. The soapcpp2 option -a ensures that WS-Addressing Action header blocks (and HTTP Action headers) are processed at the receiving side, which means that the service dispatcher uses the Action together with the operation name to invoke the service operation at the destination. This also means that Action headers must be properly set by the client.

Note: the SOAP_ENV__Header struct can be declared in multiple files. The soapcpp2 tool gathers all members of the structs into the "final" SOAP_ENV__Header struct used by the gSOAP engine and your application. This is convenient when service-specific header blocks are combined with WS-ReliableMessaging and WS-Addressing header blocks or when WS-Security header blocks are added by the WSSE plugin.

WS-ReliableMessaging Overview

In this section a brief overview of WS-ReliableMessaging is given. For more details please refer to the WS-ReliableMessaging protocol or tutorials on the subject. The following introduces the basic concepts of WS-ReliableMessaging from a practical point of view.

WS-ReliableMessaging is useful to improve the reliability of one-way asynchronous messaging, for unreliable data gram messaging (SOAP-over-UDP), or to improve reliable delivery of responses relayed to other destinations, such as response messages that are relayed to destinations indicated by the WS-Addressing ReplyTo header. The protocol is also useful when multiple sources are sending messages that arrive out of order or must be flagged as an incomplete message collection when messages are missing as defined by the notion of a collection of related messages. Messages delivered out-of-order are not internally reordered automatically. The application logic should be designed to be robust to out-of-order delivery effects, for example by using a vector to collect the data elements received before the data is processed. Alternatively, the server-side soap_wsrm_check_and_wait calls can be used to let the current thread wait until preceding messages have been received and processed (by other threads). For single-thread processing with keep-alive, this will cause a performance bottleneck unless the message queue plugin is used. The message queue plugin (mq.c) provides an approach to buffer inbound messages for in-order processing.

WS-ReliableMessaging is not essential to improve the reliability of request-response message exchanges between two parties over HTTP, since a successful delivery of a request message can be inferred from the fact that a response was received for the request sent.

WS-ReliableMessaging "protects" message sequences, i.e. a collection of related messages. A WS-ReliableMessaging message sequence is created after which the sequence of messages is sent. The sequence is closed and terminated by the client after the last message. Either the message sequence is complete or not, and the resulting action to discard the incomplete message sequence or not depends on the chosen behavior. Duplicate messages (e.g. resulting from resends) are always discarded.

To create a new sequence, a client (WS-RM source) requests from the server (WS-RM destination) a unique (new) message sequence identification. The server responds with the identification to be used as well as other details, such as the expiration time of the sequence and the behavior implemented when a sequence was received incomplete:

• NoDiscard means that the sequence of messages is not discarded by the destination server when one or more messages are missing (or unacknowledged). That is, no acknowledged messages in the sequence will be discarded. The WS-RM plugin will reorder messages either by ignoring any out-of-order messages (when calling soap_wsrm_check) or wait for out-of-order messages to arrive in a timeout window (when calling soap_wsrm_check_and_wait).
• DiscardFollowingFirstGap means that the initial messages of the sequence are retained by the destination up to the first gap (a missing message) in the sequence. Does not allow for out-of-order message delivery.
• DiscardEntireSequence means that the entire sequence of messages will be discarded when there are one or more gaps (messages are missing). With the WS-RM plugin, messages may be accepted out-of-order.

When the client terminates the sequence, it first sends a sequence close request (or a last message with the older WS-RM 1.0) and then a sequence termination request to the destination server. The sequence close informs the server how many messages should have been received. The client can still resend messages after the close, but no new messages are supposed to be send. After the optional resends, the client requests termination of the sequence. The termination will be successful depending on the behavior when messages went missing, as was listed above.

The ensure reliable delivery, the WS-ReliableMessaging protocol allows the client to resend messages. Message resends are desirable when messages are lost in transit. Since the client has limited information on delivery success (message delivery acknowledgments can get lost as well), the client may resend more messages than necessary. This could lead to message duplication. However, messages that were already received by the server are discarded.

The client may request message delivery acknowledgements from the server. The server sends message receipt acknowledgements for all the messages it has received in the sequence back to the client, usually by piggy-backing them with the response of a subsequent message exchange. When the client is informed about successful delivery it reduces the number of resends the client will attempt.

Messages in a sequence are uniquely identified by their enumeration number in the sequence. Messages may be transmitted out of order. A missing message number indicates a gap in the message sequence. Message receipt acknowledgements consist of ranges of message numbers. Acknowledgements are normally sent to the source to help identify which messages should be resend.

With the WS-Addressing protocol, message responses and fault messages can be relayed to other destinations. The ReplyTo and FaultTo WS-Addressing header blocks are used for this purpose. The WS-ReliableMessaging protocol allows message acknowledgements to be relayed. The WS-ReliableMessaging AcksTo header block is used for this purpose.

In all, there are four types of communicating peers that are visible to the source (the client):

• The destination service. The WS-ReliableMessaging sequence is essentially controlled by this service. When a message is sent by the source to the destination service (over HTTP, TCP, or UDP), the WS-Addressing To information header may contain the endpoint address. The destination service normally returns message responses back to the client (HTTP request-response) or in duplex mode with one-way messages.
• Optionally, one of more ReplyTo destination services that accept response messages from the destination service. Rather than sending responses back to the client, the destination service relays them to another service. The WS-Addressing ReplyTo information header is used by the client to indicate the response relay target.
• Optionally, one or more FaultTo destination services that accept SOAP fault messages from the destination service. Rather than sending SOAP Faults back to the client, the destination service relays them to another service. The WS-Addressing FaultTo information header is used by the client to indicate the fault relay target.
• Optionally, one AcksTo destination service that accept WS-ReliableMessaging acknowledgements. Rather than sending acknowledgements piggy-backed with response messages back to the client, the destination service relays them to another service. The WS-ReliableMessaging AcksTo information header is used by the client when the sequence is created to indicate the acknowledgements relay target. The AcksTo cannot be changed after sequence creation to termination.

The destination service normally serves as ReplyTo, FaultTo, and AcksTo service, which is the default scenario with request-response message exchanges. Replies, faults, and acknowledgements can also be asynchroniously transmitted in duplex mode. In that case, the WS-RM source client acts as a callback service to accept messages on a port.

The practical aspects of message sequence creation, the message exchanges, the message relays, and sequence close/termination are presented for the client side first and then for each of the four types of destination servers.

Client-side Usage

Creating, Closing, and Terminating Message Sequences

A sequence is created, closed, terminated, and cleaned-up on the client side as follows, using a 'soap' context struct (use one 'soap' context per thread):

struct soap *soap = soap_new(); // Note: can use C++ proxy instead of 'soap'
soap_register_plugin(soap, soap_wsa);
soap_register_plugin(soap, soap_wsrm);

const char *destination = "..."; // WS-RM destination server address
const char *source = NULL;       // WS-RM source (NULL means current)
ULONG64 expires = 10000;         // 10000 ms to expire (10 seconds)
const char *id = NULL;           // id = NULL: generate a temp sequence ID
const char *opt_msg_id = NULL;   // WS-Addressing message ID (optional)

soap_wsrm_sequence_handle seq;   // a local handle to the sequence state

// Step 1: create a sequence
if (soap_wsrm_create_offer(soap, destination, source, id, expires, NoDiscard, opt_msg_id, &seq))
{
  soap_wsrm_seq_free(soap, seq);
  ... // error creating sequence
} 

// Step 2: exchange messages with WS-RM destination, request acks, receive acks, issue resends (see later) 
...

// Step 3: optionally close first before terminating
if (soap_wsrm_close(soap, seq, NULL))
{
  soap_wsrm_seq_free(soap, seq);
  ... // error closing sequence
}

// Step 4: optionally resend messages before terminating
if (soap_wsrm_nack(seq)) // any non-acks for messages sent?
  soap_wsrm_resend(soap, seq, 0, 0); // resend all non-acked messages

// Step 5: terminate
if (soap_wsrm_terminate(soap, seq, NULL))
{
  soap_wsrm_seq_free(soap, seq);
  ... // error creating sequence
}

// Step 6: cleanup
soap_wsrm_seq_free(soap, seq);

For duplex communication, the responses are sent to the WS-RM source's port, to which we need to listen via a callback service. There are two ways to do so, by polling or by running a separate callback server thread that accepts responses.

In the code below we use a polling approach to process responses sent asynchronously to the WS-RM source:

struct soap *soap = soap_new(); // Note: can use C++ proxy instead of 'soap'
soap_register_plugin(soap, soap_wsa);
soap_register_plugin(soap, soap_wsrm);

const char *destination = "..."; // WS-RM destination server address
const char *source = "...";      // WS-RM source address destination sends to
int source_port = ...;           // port of WS-RM source address
ULONG64 expires = 10000;         // 10000 ms to expire (10 seconds)
const char *id = NULL;           // id = NULL: generate a temp sequence ID
const char *opt_msg_id = NULL;   // WS-Addressing message ID (optional)
int retry;

soap_wsrm_sequence_handle seq;   // a local handle to the sequence state

struct soap *callback = soap_new(); // callback for polling

// Step 1: create a WS-RM source port binding for the callback to poll messages
if (!soap_valid_socket(soap_bind(callback, NULL, source_port, 100)))
  ... // error

// Step 2: create a sequence
if (soap_wsrm_create_offer(soap, destination, source, id, expires, NoDiscard, opt_msg_id, &seq))
{
  if (soap->error != 202) // Error != HTTP Accepted
  {
    soap_wsrm_seq_free(soap, seq);
    ... // error creating sequence
  }
}
// poll 10 times for 1 second until the sequence created response received
for (retry = 10; retry && !soap_wsrm_seq_created(soap, seq); retry--)
  if (callback_poll(callback, 1)) // 1 second poll
    ... // error
if (!retry)
  ... // error

// Step 3: exchange messages with WS-RM destination, request acks, receive acks, issue resends, and poll (see later) 
...

// Step 4: optionally close first before terminating
if (soap_wsrm_close(soap, seq, NULL))
{
  if (soap->error != 202) // Error != HTTP Accepted
  {
    soap_wsrm_seq_free(soap, seq);
    ... // error closing sequence
  }
}

if (callback_poll(callback, -500000)) // 500 ms poll
  ... // error

// Step 5: optionally resend messages before terminating
if (soap_wsrm_nack(seq)) // any nacks?
{
  soap_wsrm_resend(soap, seq, 0, 0); // resend all non-acked messages
  if (callback_poll(callback, -100000)) // 100 ms poll
    ... // error
}

// Step 6: terminate
if (soap_wsrm_terminate(soap, seq, NULL))
{
  if (soap->error != 202) // Error != HTTP Accepted
  {
    soap_wsrm_seq_free(soap, seq);
    ... // error creating sequence
  }
}

// only needed with WS-RM 1.1 to accept terminate response
if (callback_poll(callback, -500000)) // 500 ms poll
  ... // error

// Step 7: cleanup
soap_wsrm_seq_free(soap, seq);

The polling operation can be implemented as follows:

    int callback_poll(struct soap *soap, int timeout)
    {
      int poll = 5; // 5 poll cycles times max, ensures make progress
      soap->accept_timeout = timeout; // polling timout
      soap->recv_timeout = timeout;
      soap->send_timeout = soap->recv_timeout = 1; // 1 sec
      while (poll-- && soap_valid_socket(soap_accept(soap)))
      {
#if !defined(WITH_UDP) && defined(WITH_OPENSSL)
        // accept SSL/TLS requests, requires soap_ssl_server_context() to set up (not shown)
        if (soap_ssl_accept(soap) != SOAP_OK)
        {
          soap_print_fault(soap, stderr);
          fprintf(stderr, "SSL request failed, continue with next call...\n");
          continue;
        }
#endif
        soap_serve(soap);
        soap_destroy(soap);
        soap_end(soap);
      }
      if (soap->error == SOAP_STOP || soap->error == SOAP_EOF) // timed out
        return soap->error = SOAP_OK;
      return soap->error;
    }

In addition, callback service operations must be defined to handle faults and, when applicable, one-way message responses sent by the destination.

Fault messages are accepted by the callback and processed by the service operation that can be customized to your needs:

int SOAP_ENV__Fault(struct soap *soap, 
  _QName                   faultcode,             // SOAP 1.1
  char                    *faultstring,           // SOAP 1.1
  char                    *faultactor,            // SOAP 1.1
  struct SOAP_ENV__Detail *detail,                // SOAP 1.1
  struct SOAP_ENV__Code   *Code,                  // SOAP 1.2
  struct SOAP_ENV__Reason *Reason,                // SOAP 1.2
  char                    *Node,                  // SOAP 1.2
  char                    *Role,                  // SOAP 1.2
  struct SOAP_ENV__Detail *Detail                 // SOAP 1.2
)
{
  soap_send_empty_response(soap, 202);
  if (!detail)
    detail = Detail;
  if (detail && detail->__type == SOAP_TYPE__wsrm__Identifier)
  {
    // the sequence id is in the Fault Detail __type and fault members
    char *id = (char*)detail->fault;
    // we opt to treat all faults fatal, so let's terminate the sequence
    soap_wsrm_sequence_handle seq = soap_wsrm_seq_lookup_id(soap, id);
    if (seq)
    {
      soap_wsrm_error(soap, seq, wsrm__SequenceTerminated);
      soap_wsrm_seq_release(soap, seq);
      return soap->error;
    }
  }
  return SOAP_OK;
}

This cleanup of memory resources may be performed at any time in the sequence of message exchange or afterwards when desired. The sequence state is maintained independent of these cleanup operations.

A graceful sequence termination may fail when the delivery of a sequence of messages is incomplete or when the lifetime of the sequence expired (the hard limit on the lifetime is SOAP_WSRM_MAX_SEC_TO_EXPIRE, this macro can be changed).

The WS-RM destination determines the failure based on the final sequence state and the sequence behavior. The behavior is set to NoDiscard by default, which means that the sequence is not discarded when transmission gaps appeared in the messages and the sequence is incomplete. The desired behavior can be specified with a sequence creation offer as explained in the next section.

If the source ReplyTo and AcksTo addresses differ in your project, then use the following:

const char *destination = "..."; // WS-RM destination server address
const char *replyto = NULL;      // WS-RM ReplyTo
const char *acksto = NULL;       // WS-RM AckstTo
ULONG64 expires = 10000;         // 10000 ms to expire (10 seconds)
const char *id = NULL;           // id = NULL: generate a temp sequence ID
const char *opt_msg_id = NULL;   // WS-Addressing message ID (optional)

if (soap_wsrm_create_offer_acksto(soap, destination, replyto, acksto, id, expires, DiscardEntireSequence, opt_msg_id, &seq)))
  ... // error

Expiration times should not be infinite, because sequences are kept in memory until expired even when the sequences are closed and terminated. This permits the detection of duplicate sequences. The limit on the lifetime is SOAP_WSRM_MAX_SEC_TO_EXPIRE. This macro can be changed.

Instead of polling, we can implement a callback server thread that accepts responses. This also permits keep-alive connections, because keep-alive connections are handled in soap_serve which runs a loop to process all incoming messages until timeout or close. This would block the polling method and also blocks an iterative callback server. Therefore, we must use multiple threads to handle keep-alive connections in a callback server.

struct soap *soap = soap_new(); // Note: can use C++ proxy instead of 'soap'
soap_register_plugin(soap, soap_wsa);
soap_register_plugin(soap, soap_wsrm);

const char *destination = "..."; // WS-RM destination server address
const char *source = "...";      // WS-RM source address destination sends to
int source_port = ...;           // port of WS-RM source address
ULONG64 expires = 10000;         // 10000 ms to expire (10 seconds)
const char *id = NULL;           // id = NULL: generate a temp sequence ID
const char *opt_msg_id = NULL;   // WS-Addressing message ID (optional)
int retry;

soap_wsrm_sequence_handle seq;   // a local handle to the sequence state
THREAD_TYPE tid;                 // callback server thread ID

struct soap *callback = soap_new(); // callback server context
soap_set_mode(callback, SOAP_IO_KEEPALIVE); // optional, see comments below

// Step 1: create a WS-RM source port binding for the callback server
if (!soap_valid_socket(soap_bind(callback, NULL, source_port, 100)))
  ... // error
THREAD_CREATE(&tid, (void*(*)(void*))callback_server, (void*)callback);

// Step 2: create a sequence
... // see earlier client code that uses polling

The callback server thread can be implemented as follows, in which we spawn more threads when keep-alive is enabled (optional in the code above):

void *callback_server(void *ctx)
{
  struct soap *soap = (struct soap*)ctx;

  soap->accept_timeout = 10; // server quits after 10 seconds of inactivity
  soap->recv_timeout = 10;   // 10 sec timeout
  soap->send_timeout = 1;    //  1 sec timeout

  while (soap_valid_socket(soap_accept(soap)))
  {
    if (soap->keep_alive)
    {
      // a keep-alive callback server must be multi-threaded, otherwise blocks on open socket
      THREAD_TYPE tid;
      struct soap *tsoap = soap_copy(soap);
      THREAD_CREATE(&tid, (void*(*)(void*))callback_server_process_request, (void*)tsoap);
    }
    else
    {
#if !defined(WITH_UDP) && defined(WITH_OPENSSL)
      // accept SSL/TLS requests, requires soap_ssl_server_context() to set up (not shown)
      if (soap_ssl_accept(soap) != SOAP_OK)
      {
        soap_print_fault(soap, stderr);
        fprintf(stderr, "SSL request failed, continue with next call...\n");
        continue;
      }
#endif
      // serve one response at a time
      if (soap_serve(soap) && soap->error != SOAP_STOP && soap->error != SOAP_EOF)
        soap_print_fault(soap, stderr);
      else if (soap->error == SOAP_EOF && !soap->errnum)
        continue; // messaging IO timed out
      else
        soap_wsrm_dump(soap, stdout);
    }
  }

  return NULL;
}

void *callback_server_process_request(void *ctx)
{
  struct soap *soap = (struct soap*)ctx;

  THREAD_DETACH(THREAD_ID);

#if !defined(WITH_UDP) && defined(WITH_OPENSSL)
  // accept SSL/TLS requests (optional)
  if (soap_ssl_accept(soap) != SOAP_OK)
  {
    soap_print_fault(soap, stderr);
    fprintf(stderr, "SSL request failed, continue with next call...\n");
  }
  else
#endif
  {
    if (soap_serve(soap) && soap->error != SOAP_STOP && soap->error != SOAP_EOF)
      soap_print_fault(soap, stderr);
    else if (soap->error == SOAP_EOF && !soap->errnum)
      ; // messaging IO timed out
    else
      soap_wsrm_dump(soap, stdout);
  }

  soap_destroy(soap);
  soap_end(soap);
  soap_free(soap);

  return NULL;
}

Creating a Sequence without an Offer

To enable a destination server to produce a reliable message response sequence, you need to create a sequence with an offer. Otherwise, response messages are not tracked and delivery is not verified. A sequence of two-way message exchanges should use the offer mechanism. If all messages are one-way from source to destination then the sequence offer mechanism is not needed. A sequence can be created without an offer as follows:

const char *destination = "..."; // WS-RM destination server address
const char *source = NULL;       // WS-RM source (NULL means current)
ULONG64 expires = 10000;         // 10000 ms to expire (10 seconds)
const char *opt_msg_id = NULL;   // WS-Addressing message ID (optional)

if (soap_wsrm_create(soap, destination, source, expires, DiscardEntireSequence, opt_msg_id, &seq))
  ... // error

Expiration times should not be infinite, because sequences are kept in memory until expired even when the sequences are closed and terminated. This permits the detection of duplicate sequences. The limit on the lifetime is SOAP_WSRM_MAX_SEC_TO_EXPIRE. This macro can be changed.

Exchanging Messages in a Sequence

Each message exchange with the WS-RM destination should be preceded with a soap_wsrm_request or soap_wsrm_request_acks call to set the required WS-RM information headers for the message send operation or request-response exchange.

For example, consider the 'example' operation defined previously and suppose we invoke the 'example' operation in a sequence (after creation and before closing):

const char *exampleRequestAction = "urn:example/examplePort/example";
const char *exampleRequestMessageID = NULL; // optional WS-Addressing ID
struct ns__exampleResponse response;

const char *endpoint = soap_wsrm_to(seq);
if (endpoint)
{
  if (soap_wsrm_request(soap, seq, exampleRequestMessageID, exampleRequestAction))
    ... // error: out of memory

  if (soap_call_ns__example(soap, endpoint, exampleRequestAction, &response))
    soap_print_fault(soap, stderr); // an error occurred
  else
    ... // process the response
}

To generate WS-Addressing message IDs, use soap_wsa_rand_uuid, for example:

const char *exampleRequestAction = "urn:example/examplePort/example";

if (soap_wsrm_request(soap, seq, soap_wsa_rand_uuid(soap), exampleRequestAction))
  ... // error: out of memory

The soap_wsrm_request takes the sequence handle and optional WS-Addressing message ID and mandatory WS-Addressing action string (this string must match the method-action defined in the gSOAP service definition header file). It produces a WS-RM header block with the message number incremented by one for the invocation. Messages are enumerated from one and included in the WS-RM header to allow the destination to determine which messages were received in the sequence (for acknowledgements) and to ignore duplicate messages.

The remote invocation soap_call_ns__example uses the endpoint provided by soap_wsrm_to(seq) for the WS-RM destination address, which was set by soap_wsrm_create or by soap_wsrm_create_offer. Because the address may change due to a redirect, we encourage the use of soap_wsrm_to for the WS-RM destination address. When used at the server side, soap_wsrm_to(seq) is the ReplyTo address provided by the client's request message, which may be none (NULL) for a one-way call. Therefore, it is wise to check the return value of soap_wsrm_to(seq).

A C++ proxy object (generated by soapcpp2 option -j) that invokes a service operation should reset the destination address explicitly by setting the soap_endpoint member string before each operation invocation.

Exchanging Messages with Acknowledgements in a Sequence

Before sending a message, a WS-RM request should be issued using soap_wsrm_request_acks. This informs the WS-RM destination to return message delivery acknowledgements back to the sender (piggy-backed in the header of the response message), unless the AcksTo is set to target an acknowledgement service endpoint in which case the acknowledgements are sent to.

if (soap_wsrm_request_acks(soap, seq, opt_msg_id, exampleRequestAction))
  ... // error
if (soap_call_ns__example(soap, endpoint, exampleRequestAction, &response))
  if (soap->error != 202) // Error != HTTP Accepted
    ... // error

When duplex messaging is used via a callback, a polling operation will be needed afterwards:

if (callback_poll(callback, -500000)) // 500 ms poll
  ... // error

The soap_wsrm_close returns acknowledgements automatically, so requesting intermediate acknowledgements is not required to issue a final soap_wsrm_resend to resend all non-acknowledged messages, e.g. after soap_wsrm_close. See also futher below.

Resending Non-Acknowledged Messages

All non-acknowledged messages in a sequence that were previously sent can be resend (from the internal sender-side cache of sent messages in a sequence) as follows:

soap_wsrm_resend(soap, seq, 0, 0);

To resend a range of non-acknowledged messages, say between 3 and 7, use:

soap_wsrm_resend(soap, seq, 3, 7);

Or all messages after message number 3:

soap_wsrm_resend(soap, seq, 3, 0);

Resends should be used with care, since in the worst case when no acknowledgements have been received all messages up to the last will be resend (and ignored by the WS-RM destination when a message is received more than once).

Note that when an AcksTo destination service address was set with soap_wsrm_create or soap_wsrm_create_offer, then the acknowledgements will not be returned to the sender (client). In this case message resends are performed for all messages sent, since these have not been acknowledged.

It is permitted to issue resends between creation and termination of a sequence, including after a sequence close (as long as no new messages are sent after close). The sequence close provides acknowledgement information to limit the number of messages that need to be resend.

To find out if none, some, or all messages have been acknowledged, use:

ULONG64 nack = soap_wsrm_nack(seq);
if (nack == 0)
  ... // all sent messages have been acknowledged
else if (nack == soap_wsrm_num(seq))
  ... // none of the sent messages have been acknowledged
else
  ... // some sent messages have been acknowledged

Relaying Response and Fault Messages with WS-Addressing

WS-ReliableMessaging is important when messages are relayed, and especially when relayed over UDP.

The ReplyTo and FaultTo destination service endpoints can be specified for each message as follows:

const char *replyto = source; // endpoint of WS-RM source
const char *faultto = "..."; // endpoint of fault processing service
const char *endpoint = soap_wsrm_to(seq);

if (endpoint)
{
  if (soap_wsrm_request_acks(soap, seq, NULL, exampleRequestAction)
   || soap_wsa_add_ReplyTo(soap, replyto)
   || soap_wsa_add_FaultTo(soap, faultto))
    ... // error: out of memory
  if (soap_call_ns__example(soap, endpoint, exampleRequestAction, &response))
  {
    if (soap->error != 202)
      ... // error
  }
}

An optional source address information header can be added with soap_wsa_add_From:

const char *from = "..."; // endpoint of the client (could be any URI)

if (soap_wsrm_request_acks(soap, seq, NULL, exampleRequestAction)
 || soap_wsa_add_From(soap, from))
  ... // error: out of memory

Adding a From address is optional and usually not required with WS-ReliableMessaging.

Using Retry Loops to Improve Robustness of Message Sends

A potential problem with reliable message delivery with resends can fail when the initial send was not successful and message ordering is relevant. Resending will lead to an unordered message delivery.

Also, the wsrm plugin records all sent messages when the send operation was not interrupted. A problem occurs when the message cache contains incomplete messages and these messages cannot be resend. This section presents an additional mechanism to ensure messages are cached properly for automatic retransmission.

Besides network failues, a request-response message exchange can also appear to fail due to a non SOAP_OK returned, such as a benign "HTTP 202 Accept" To distinguish fatal send errors from errors returned by the peer, the soap_wsrm_check_retry function can be used as follows to only retry the message exchange (or send) when needed:

const char *endpoint = NULL;
if (soap_wsrm_request_acks(soap, seq, NULL, exampleRequestAction))
  ... // error
while ((endpoint = soap_wsrm_to(seq)) != NULL && soap_call_ns__example(soap, endpoint, exampleRequestAction, &response))
{
  if (soap->error == 202)
  {
    // request was accepted by destination (HTTP 202 Accept)
    break;
  }
  else if (soap->error == SOAP_NO_TAG) // empty <Body>
  {
    // request was accepted by destination, acks are returned
    break;
  }
  soap_print_fault(soap, stderr);
  if (soap_wsrm_check_retry(soap, seq))
    break; // do not continue
  sleep(1); // wait a second to give network a chance to recover
}

if (soap->error == SOAP_OK)
  ... // response can be processed

Note that the soap_wsrm_request is only invoked once in the above to increment the message enumeration.

The loop retries transmissions a maximum of SOAP_WSRM_MAX_RETRIES iterations before giving up.

Note that this mechanism does not replace acknowledgements for delivery. Delivery acknowledgements are verified with soap_wsrm_nack.

Example Client

The following code shows an example WS-RM client fragment that combines the concepts introduced in the previous sections for a request-response scenario:

struct soap *soap = soap_new(); // Note: can use C++ proxy instead of 'soap'
soap_register_plugin(soap, soap_wsa);
soap_register_plugin(soap, soap_wsrm);

struct ns__exampleResponse response;
const char *exampleRequestAction = "urn:example/examplePort/example";

const char *destination = "..."; // WS-RM destination server address
const char *source = NULL;       // WS-RM source (current)
ULONG64 expires = 60000;         // 1 minute sequence lifetime

soap_wsrm_sequence_handle seq;

if (soap_wsrm_create_offer(soap, destination, source, NULL, expires, DiscardEntireSequence, NULL, &seq))
{
  soap_wsrm_seq_free(soap, seq);
  ... // error creating sequence
} 

if (soap_wsrm_request_acks(soap, seq, NULL, exampleRequestAction))
  ... // error
while (soap_call_ns__example(soap, destination, exampleRequestAction, &response))
{
  if (soap->error == 202)
  {
    // request was accepted by destination (HTTP 202 Accept)
    break;
  }
  else if (soap->error == SOAP_NO_TAG) // empty <Body>
  {
    // request was accepted by destination, acks are returned
    break;
  }
  soap_print_fault(soap, stderr);
  if (soap_wsrm_check_retry(soap, seq))
    break; // do not continue
  sleep(1); // wait a second to give network a chance to recover
}

if (soap->error == SOAP_OK)
  ... // response can be processed

if (soap_wsrm_close(soap, seq, NULL))
{
  soap_wsrm_seq_free(soap, seq);
  ... // error closing sequence
} 

soap_wsrm_resend(soap, seq, 0, 0); // resend non-acked messages

if (soap_wsrm_terminate(soap, seq, NULL))
{
  soap_wsrm_seq_free(soap, seq);
  ... // error
} 

soap_wsrm_seq_free(soap, seq);

soap_destroy(soap);
soap_end(soap);
soap_free(soap);

A duplex mode client that accepts responses via a callback:

struct soap *soap = soap_new(); // Note: can use C++ proxy instead of 'soap'
soap_register_plugin(soap, soap_wsa);
soap_register_plugin(soap, soap_wsrm);

struct ns__exampleResponse response;
const char *exampleRequestAction = "urn:example/examplePort/example";

const char *destination = "..."; // WS-RM destination server address
const char *source = "...";      // WS-RM source address destination sends to
int source_port = ...;           // port of WS-RM source address
ULONG64 expires = 60000;         // 1 minute sequence lifetime
int retry;

soap_wsrm_sequence_handle seq;

struct soap *callback = soap_new(); // callback for polling

// WS-RM source port binding for the callback to poll messages
if (!soap_valid_socket(soap_bind(callback, NULL, source_port, 100)))
  ... // error

if (soap_wsrm_create_offer(soap, destination, source, NULL, expires, DiscardEntireSequence, NULL, &seq))
{
  if (soap->error != 202)
  {
    soap_wsrm_seq_free(soap, seq);
    ... // error creating sequence
  }
} 

// poll 10 times for 1 second until the sequence created response received
for (retry = 10; retry && !soap_wsrm_seq_created(soap, seq); retry--)
  if (callback_poll(callback, 1)) // 1 second poll
    ... // error
if (!retry)
  ... // error

if (soap_wsrm_request_acks(soap, seq, NULL, exampleRequestAction))
  ... // error
while (soap_call_ns__example(soap, destination, exampleRequestAction, &response))
{
  if (soap->error == 202)
  {
    // request was accepted by destination (HTTP 202 Accept)
    break;
  }
  else if (soap->error == SOAP_NO_TAG) // empty <Body>
  {
    // request was accepted by destination, acks are returned
    break;
  }
  if (soap_wsrm_check_retry(soap, seq))
    break; // do not continue
  sleep(1); // wait a second to give network a chance to recover
}

if (callback_poll(callback, -500000)) // 500 ms poll
  ... // error

if (soap_wsrm_close(soap, seq, NULL))
{
  if (soap->error != 202)
  {
    soap_wsrm_seq_free(soap, seq);
    ... // error closing sequence
  }
}

// Resend messages marked as non-acked (as an option)
for (retry = 2; retry && soap_wsrm_nack(seq); retry--)
{
  soap_wsrm_resend(soap, seq, 0, 0); // 0 0 means full range of msg nums
  if (callback_poll(callback, -500000)) // 500 ms poll
    ... // error
}

if (soap_wsrm_terminate(soap, seq, NULL))
{
  if (soap->error != 202)
  {
    soap_wsrm_seq_free(soap, seq);
    ... // error closing sequence
  }
} 

if (callback_poll(callback, -500000)) // 500 ms poll
  ... // error

soap_wsrm_seq_free(soap, seq);

soap_destroy(callback);
soap_end(callback);
soap_free(callback);
soap_destroy(soap);
soap_end(soap);
soap_free(soap);

Server-side Usage

To set up a WS-ReliableMessaging compliant server, register the wsa and wsrm plugins with the soap context (or with the C++ proxy object generated by soapcpp2 option -j):

struct soap *soap = soap_new(); // Note: use C++ proxy->soap instead of 'soap'
soap_register_plugin(soap, soap_wsa);
soap_register_plugin(soap, soap_wsrm);

The following subsections detail the differences between the types of WS-RM destination services.

Setting up a WS-RM Destination Service

For duplex communications, it is advisable to send acknowledgements to the peer, which are normally piggy-backed on messages to the peer. If no messages are sent to the peer, explicit acknowledgements can be sent:

if (soap_wsrm_acknowledgement(soap, seq, NULL))
  ... // error

To sent acknowledgements to all peers with open sequences, use:

int timeout = -10000; // 10 ms
if (soap_wsrm_pulse(soap, timeout))
  ... // error

where timeout is in seconds (pos value) or microseconds (neg value). Depending on the message exchange scenarios, it may be advisable to have the server send periodic acknowledgement updates as follows:

soap->accept_timeout = -200000; // 200ms intervals
for (;;)
{
  // server loop, accept next message
  if (!soap_valid_socket(accept()))
  {
    // error or timeout?
    if (soap->errnum)
    {
      soap_stream_fault(std::cerr);
      exit(1); // may want to exit, but trying to continue is also possible
    }
    else
    {
      // timeout occurs after 200ms
      // send acks to peers (optional), take 10 ms per message 
      soap_wsrm_pulse(soap, -10000); // 10 ms
      // sleep(1); // must do this with UDP: since accept() returns immediately
    }
  }
  else
  {
    ... // serve
  }

Each service operation that supports WS-ReliableMessaging and WS-Addressing should use the soap_wsrm_check, soap_wsrm_sender_fault, soap_wsrm_receiver_fault, and soap_wsrm_reply functions as follows:

int ns__example(struct soap *soap, char *in, struct ns__exampleResponse *response)
{
  const char *ResponseAction = "urn:example/examplePort/exampleResponse";

  // fatal service operation-specific errors (before soap_wsrm_check())
  if (!database) // suppose we need a database, if there is none terminate
    return soap_wsrm_receiver_fault(soap, "No database!", NULL);

  // check for WS-RM/WSA and set WS-RM/WSA return headers and protocol errors
  // note: use soap_wsrm_check_and_wait() with NoDiscard behavior to queue out-of-order messages
  if (soap_wsrm_check(soap))
    return soap->error;

  // check for non-fatal service operation-specific errors
  if (!in || !*in) // sender did not put anything in the 'in' string: fault
    return soap_wsrm_sender_fault(soap, "No string content!", NULL);

  response->out = ...

  // return normally, relaying the response to the ReplyTo service when needed
  return soap_wsrm_reply(soap, NULL, ResponseAction);
}

An error produced by soap_wsrm_sender_fault or soap_wsrm_receiver_fault before soap_wsrm_check is considered fatal, it will terminate the sequence and the sender (client) will not be able to continue the sequence transmissions. While the faults preduced after soap_wsrm_check allow the sequence to continue.

Handling Duplex Callback Service Operations

To set up a callback to accept responses in a duplex scenario, we set up the server in the same way as the destination server. Service opertions should not use soap_wsrm_reply. Because response messages are sent (as if these were request messages), the service must define the appropriate one-way operations and gSOAP service definitions bindings.

For example, the one-way response message of the ns__example operation is defined as follows in the gSOAP service definitions header file:

//gsoap ns service method-header-part: exampleResponse wsa5__MessageID
//gsoap ns service method-header-part: exampleResponse wsa5__RelatesTo
//gsoap ns service method-header-part: exampleResponse wsa5__From
//gsoap ns service method-header-part: exampleResponse wsa5__ReplyTo
//gsoap ns service method-header-part: exampleResponse wsa5__FaultTo
//gsoap ns service method-header-part: exampleResponse wsa5__To
//gsoap ns service method-header-part: exampleResponse wsa5__Action

//gsoap ns service method-header-part: exampleResponse wsrm__SequenceAcknowledgement

//gsoap ns service method-action: exampleResponse urn:example/examplePort/exampleResponse

int ns__exampleResponse(char *out, void);

The one-way response service operations are automatically generated with wsdl2h option -b.

Note that when these definitions are combined with the previous definition for ns__example, there is no need to define the ns__ExampleResponse struct any longer as this is implied by the ns__exampleResponse function content.

The server operation implementation is for example:

int ns__exampleResponse(struct soap *soap, char *out)
{
  // check WS-RM/WSA headers and protocol errors and send 202 Accept back to peer
  // note: use soap_wsrm_check_send_empty_response_and_wait() with NoDiscard behavior to queue out-of-order messages
  if (soap_wsrm_check_send_empty_response(soap))
    return soap->error;
  ... // process the 'out' content, invoke callback etc
  return SOAP_OK;
}

WS-ReliableMessaging over HTTPS with Basic Authentication

The HTTPS client and server are set up as shown in the gSOAP documentation and examples. There are no additional API calls needed to support WS-RM with HTTPS.

Note that the WS-RM destination service may also relay messages to other HTTPS services, thus the WS-RM destination acts as a receiver (server) and sender (client). Therefore, the WS-RM destination server's SSL context should be set to authenticate the other servers:

if (soap_ssl_server_context(soap,
  SOAP_SSL_DEFAULT,
  "server.pem",     // keyfile (server)
  "password",       // password to read the key file (server)
  "cacert.pem",     // cacert file to store trusted certificates (client)
  NULL,             // optional capath
  NULL,     // DH file name or DH param key len bits, NULL: RSA
  NULL,             // file with random data to seed randomness
  argv[1]   // unique server identification for SSL session cache
))
{
  soap_print_fault(soap, stderr);
  ... // handle error
}

Here, the cacert.pem file contains certificates to authenticate the ReplyTo, FaultTo, and AcksTo services when HTTPS is used.

The client side sets up the SSL context with the soap_ssl_client_context as instructed in the documentation and by the examples. Multi-threaded HTTPS clients and servers must register mutex locks with OpenSSL

To use Basic Authentication at the client side, set the userid and passwd values:

soap->userid = "..."; // Basic Auth user id
soap->passwd = "..."; // Basic Auth password
if (soap_wsrm_request_acks(soap, seq, NULL, exampleRequestAction)
  ... // error: out of memory
if (soap_call_ns__example(soap, endpoint, exampleRequestAction, &response))
{
  if (soap->error == 401)
    ... // authentication failed for the userid/passwd pair
  else
    ... // other error
}

At the server side add the authentication check to the service operation before soap_wsrm_check to terminate the sequence when an authentication failure occurs. For example:

int ns__example(struct soap *soap, char *in, struct ns__exampleResponse *response)
{
  if (!soap->userid || !soap->passwd || strcmp(soap->userid, "...") || strcmp(soap->passwd, "..."))
  {
    soap->authrealm = "..."; // optional to set HTTP WWW-Authenticate: Basic realm="..."
    return 401; // HTTP 401 Unauthorized
  }
  if (soap_wsrm_check(soap))
    return soap->error;
  ...

Here, we check only one userid-passwd pair though normally we could search for valid credentials in an authentication store.

Note: never use Basic Authentication over HTTP because the password is sent in the clear. You must use HTTPS to encrypt the HTTP authentication information and message content. HTTP Digest Auth is preferred for this reason, because Digest Auth verifies the digest of a userid-passwd rather than require the password to be exchanged or stored in cleartext.

WS-ReliableMessaging over UDP with Timeouts

The use of UDP is automatic at the client side using the "soap.udp://" protocol. Therefore, endpoints should use a "soap.udp://" URL to connect. (when using an already opened socket, the SOAP_IO_UDP flag must be used, see the documentation.)

Note that UDP datagram messages should not exceed 8K, which is usually a size that UDP datagrams can support. To reduce the message size, we recommend compression (-DWITH_GZIP compile flag to enable ZLIB and use libgsoapssl.a or libgsoapssl++.a for OpenSSL and ZLIB compression combined).

The code of an UDP-enabled server is identical to an HTTP/TCP server except that the soap_accept call is disabled and unnecessary.

When message responses are not returned to the client, the client may block indefinitely when it expects a response. Therefore we recommend the use of send and receive timeouts:

struct soap *soap = soap_new();

const char *destination = "soap.udp://...";
const char *source = NULL;
ULONG64 expires = 60000;         // 1 minute sequence lifetime

soap->send_timeout = soap->recv_timeout = 1; // 1 second to timeout

if (soap_wsrm_create_offer(soap, destination, source, NULL, expires, NoDiscard, NULL, &seq))
 ... // an error occured

if (soap_wsrm_request(soap, seq, NULL, exampleRequestAction))
  ... // an error occured
if (soap_call_ns__example(soap, ...))
{
  if (soap->error == SOAP_EOF && soap->errnum == 0)
    ... // a timeout occured
  else
    ... // an error occured
}

Note that the WS-Addressing ReplyTo and the use of NoReply do not return response message from the server. However, acknowledgements will be returned when acknowledgements were requested (unless acknowledgements are relayed with AcksTo).

struct soap *soap = soap_new();

soap->send_timeout = soap->recv_timeout = 1; // 1 second to timeout

const char *destination = "soap.udp://...";
const char *from = "..."; // some identifying URI
const char *source = from;
ULONG64 expires = 60000;         // 1 minute sequence lifetime

soap->send_timeout = soap->recv_timeout = 1; // 1 second to timeout

if (soap_wsrm_create_offer(soap, destination, source, NULL, expires, NoDiscard, NULL, &seq))
 ... // an error occured

if (soap_wsrm_request_acks(soap, seq, NULL, exampleRequestAction)
 || soap_wsa_add_NoReply(soap)
  ... // an error occured
if (soap_call_ns__example(soap, ...))
{
  if (soap->error == SOAP_EOF && soap->errnum == 0)
    ... // a timeout occured
  else if (soap->error == SOAP_NO_TAG)
    ... // ack was received and recorded
  else
    ... // an error occured
}

In this case an acknowledgement will be returned and the timeout reflects a possible network packet loss.

WS-ReliableMessaging and WS-Security

WS-Security can be combined with WS-ReliableMessaging using the WS-Security plugin.

Both plugins must be registered at the client and server side. These APIs are independent.

The wsrm Plugin and C++ Proxy and Server Objects

The WS-ReliableMessaging plugin is developed to support C and C++. To support C++ server objects generated with soapcpp2 option -j (or -i), run soapcpp2 again:

soapcpp2 -A -pwsrx import/wsrx.h

This generates wsrxClient.cpp and wsrxServer.cpp that are also needed with the WS-ReliableMessaging plugin and code.

To dispatch the wsrm service operations when received: Suppose we have a myService server class generated by soapcpp2 option -j, which is used to process requests with the serve() member function (also generated):

myService service;
if (soap_invalid_socket(service.bind(NULL, port, 100)))
  ... // error
for (;;)
{
  if (!soap_valid_socket(service.accept()))
  {
    service.soap_stream_fault(std::cerr);
    exit(1);
  }
  if (soap_begin_serve(service.soap) == SOAP_OK)
  {
    if (service.dispatch() == SOAP_NO_METHOD)
    {
      if (soap_serve_request(service.soap) != SOAP_OK)
      {
        soap_send_fault(service.soap);
        service.soap_stream_fault(std::cerr);
      }
    }
    else if (service.soap->error)
      service.soap_stream_fault(std::cerr);
  }
  service.destroy();
}

Speed improvements with large number of sequences

Compile wsrmapi.c with -DSOAP_WSRM_FAST_LOOKUP to speed up sequence lookups for a large number of concurrent message sequences or sequences with a long time to live. An internal hash table is used, which will grow to accommodate the total number of concurrent sequences. The hash table entries are reused but (temporary) empty entries are never deallocated to avoid allocation overhead, which means that leak detectors may complain about space not being freed.