MTOM vs. Streaming vs. Compression – Large attachments over WCF

Above question pops up when one is about to do a large transfer of data (images for instance) using WCF. Let me try answer this starting with basics.

Bandwidth & Buffer – There are 2 considerations to large transfers. First – you want to transfer as minimal as possible in terms of size (bytes) to avoid bandwidth cost which normally matters a lot when you are over WAN paying for it & Second – whether you want to transfer the entire message (read the entire image in memory on client & send it to server) or you want to stream it byte by byte. Streaming sometimes is necessary as buffering can adversely affect performance of your server in case of multiple clients (e.g. 20 clients concurrently transferring a 100 MB image, which would take up to 2 GB of your server’s RAM). So coming to title of this post – MTOM is related to Bandwidth while Streaming is related to Buffering. Let’s dig in bit more.

MTOM (Message Transmission Optimization Mechanism) – WCF supports 3 encodings (in context of WCF, encoding means converting a WCF message (serialized XML InfoSet) into bytes) – Text, MTOM & Binary (JSON & POX are also possible – webHttpBinding). All Http Bindings (Basic, WS, Dual, etc.) support Text / MTOM encoding, Text being the default one. Text/MTOM are preferred in WS-* interoperability scenarios. To switch to MTOM encoding all you need to do is just select it as shown below:

<wsHttpBinding>
        <binding messageEncoding=”Mtom” />
</wsHttpBinding>

why MTOM? Problem with Text Encoding is it uses base 64 encoding format which can inflate the message size by 30%. This can be a heavy penalty while carrying large binary attachments. Enter MTOM!!! MTOM avoids base 64 encoding for binary attachments keeping the overall size of message in control. Moreover, MTOM is based on open specifications & hence is largely interoperable. Coming to binary encoding of WCF (TCP/Pipe/MSMQ) though it’s best in terms of performance it’s not interoperable. Some people are also averse to TCP etc. because of firewall constraints & need of Sticky Sessions (Load balancing with transport sessions). I would strongly recommend to do a performance test on all of them in your environment and then take a decision.

Streaming – Streaming (BasicHttp, Tcp, Pipe) can be a good solution when you don’t want to increase the load on your servers though unlike buffering this doesn’t allow you to leverage on WCF’s message based security & reliability (how do you ensure that entire stream is transferred and not broken in between?). In case, latter two are your requirements and you want to limit the memory usage on Server, there is a chunking channel sample on MSDN. When you want to use streaming though, your OperationContract should use only one instance of Stream class (details here) in parameter list or as return type.
E.g. Stream PlaySong();
Unfortunately above still uses a buffered mode. PlaySong API is as good as returning a ‘Byte array’ in buffered mode. To enable the Streamed mode, you need to select it at Binding level, as shown below:

<basicHttpBinding>
        <binding name=”streamedHttp” transferMode=”Streamed” />
</basicHttpBinding>

Compression – WCF’s extensible channel architecture allows us to easily plug-in a compression channel. So, how about not using MTOM or binary, and just applying compression on what we are about to transfer? First compression doesn’t come for free, it costs a lot in terms of CPU. You need to weigh the CPU cost of compression / decompression vs. Latency cost (i.e. is bandwidth a bottleneck?). For Binary encoding, I think it doesn’t make sense (I would encourage you to do your own test, but it didn’t show me much difference), for MTOM encoding I would prefer sending an already offline compressed attachment (i.e. a compressed .bmp instead of .bmp) & for Text encoding, yes, it may make sense. Say, you want to send 10000 customers over WAN (though you shouldn’t be doing that) and you need to use Text for interoperability reasons. I recommend to use compression by all means for such scenarios.

Below are the important Knobs one might have to configure depending on their message transfer requirements.

<customBinding>
        <binding name=”LargeMessageOverHttp”>
<!–Encoders–>
          <textMessageEncoding>
            <readerQuotas maxStringContentLength=”” maxArrayLength=””
  maxBytesPerRead=”” maxDepth=”” maxNameTableCharCount=”” />
          </textMessageEncoding>
<!–Transport–>
          <httpTransport maxBufferPoolSize=”” maxBufferSize=”” maxReceivedMessageSize=”” />         
        </binding>
</customBinding>

maxArrayLength
The maximum allowed array length. The default is 16384.

maxBytesPerRead
The maximum allowed bytes returned for each read. The default is 4096.

maxDepth
The maximum nested node depth. The default is 32.

maxNameTableCharCount
The maximum characters allowed in a table name. The default is 16384.

maxStringContentLength
The maximum string length returned by the reader. The default is 8192.

maxBufferPoolSize
The maximum size of the buffer pool. The default is 524,288 bytes.

maxBufferSize
The maximum size, in bytes, of the buffer. defaults to 65536.

maxReceivedMessageSize
The maximum allowable message size that can be received. The default is 65,536 bytes.

Hope above gives some clarification 🙂 .