Niraj Bhatt – Architect's Blog

Ruminations on .NET, Architecture & Design

MTOM vs. Streaming vs. Compression – Large attachments over WCF

Above question pops up when one is about to do a large transfer of data (images for instance) using WCF. Let me try answer this starting with basics.

Bandwidth & Buffer – There are 2 considerations to large transfers. First – you want to transfer as minimal as possible in terms of size (bytes) to avoid bandwidth cost which normally matters a lot when you are over WAN paying for it & Second – whether you want to transfer the entire message (read the entire image in memory on client & send it to server) or you want to stream it byte by byte. Streaming sometimes is necessary as buffering can adversely affect performance of your server in case of multiple clients (e.g. 20 clients concurrently transferring a 100 MB image, which would take up to 2 GB of your server’s RAM). So coming to title of this post – MTOM is related to Bandwidth while Streaming is related to Buffering. Let’s dig in bit more.

MTOM (Message Transmission Optimization Mechanism) – WCF supports 3 encodings (in context of WCF, encoding means converting a WCF message (serialized XML InfoSet) into bytes) – Text, MTOM & Binary (JSON & POX are also possible – webHttpBinding). All Http Bindings (Basic, WS, Dual, etc.) support Text / MTOM encoding, Text being the default one. Text/MTOM are preferred in WS-* interoperability scenarios. To switch to MTOM encoding all you need to do is just select it as shown below:

<wsHttpBinding>
        <binding messageEncoding=”Mtom” />
</wsHttpBinding>

why MTOM? Problem with Text Encoding is it uses base 64 encoding format which can inflate the message size by 30%. This can be a heavy penalty while carrying large binary attachments. Enter MTOM!!! MTOM avoids base 64 encoding for binary attachments keeping the overall size of message in control. Moreover, MTOM is based on open specifications & hence is largely interoperable. Coming to binary encoding of WCF (TCP/Pipe/MSMQ) though it’s best in terms of performance it’s not interoperable. Some people are also averse to TCP etc. because of firewall constraints & need of Sticky Sessions (Load balancing with transport sessions). I would strongly recommend to do a performance test on all of them in your environment and then take a decision.

Streaming – Streaming (BasicHttp, Tcp, Pipe) can be a good solution when you don’t want to increase the load on your servers though unlike buffering this doesn’t allow you to leverage on WCF’s message based security & reliability (how do you ensure that entire stream is transferred and not broken in between?). In case, latter two are your requirements and you want to limit the memory usage on Server, there is a chunking channel sample on MSDN. When you want to use streaming though, your OperationContract should use only one instance of Stream class (details here) in parameter list or as return type.
E.g. Stream PlaySong();
Unfortunately above still uses a buffered mode. PlaySong API is as good as returning a ‘Byte array’ in buffered mode. To enable the Streamed mode, you need to select it at Binding level, as shown below:

<basicHttpBinding>
        <binding name=”streamedHttp” transferMode=”Streamed” />
</basicHttpBinding>

Compression – WCF’s extensible channel architecture allows us to easily plug-in a compression channel. So, how about not using MTOM or binary, and just applying compression on what we are about to transfer? First compression doesn’t come for free, it costs a lot in terms of CPU. You need to weigh the CPU cost of compression / decompression vs. Latency cost (i.e. is bandwidth a bottleneck?). For Binary encoding, I think it doesn’t make sense (I would encourage you to do your own test, but it didn’t show me much difference), for MTOM encoding I would prefer sending an already offline compressed attachment (i.e. a compressed .bmp instead of .bmp) & for Text encoding, yes, it may make sense. Say, you want to send 10000 customers over WAN (though you shouldn’t be doing that) and you need to use Text for interoperability reasons. I recommend to use compression by all means for such scenarios.

Below are the important Knobs one might have to configure depending on their message transfer requirements.

<customBinding>
        <binding name=”LargeMessageOverHttp”>
<!–Encoders–>
          <textMessageEncoding>
            <readerQuotas maxStringContentLength=”” maxArrayLength=””
  maxBytesPerRead=”” maxDepth=”” maxNameTableCharCount=”” />
          </textMessageEncoding>
<!–Transport–>
          <httpTransport maxBufferPoolSize=”” maxBufferSize=”” maxReceivedMessageSize=”” />         
        </binding>
</customBinding>

maxArrayLength
The maximum allowed array length. The default is 16384.

maxBytesPerRead
The maximum allowed bytes returned for each read. The default is 4096.

maxDepth
The maximum nested node depth. The default is 32.

maxNameTableCharCount
The maximum characters allowed in a table name. The default is 16384.

maxStringContentLength
The maximum string length returned by the reader. The default is 8192.

maxBufferPoolSize
The maximum size of the buffer pool. The default is 524,288 bytes.

maxBufferSize
The maximum size, in bytes, of the buffer. defaults to 65536.

maxReceivedMessageSize
The maximum allowable message size that can be received. The default is 65,536 bytes.

Hope above gives some clarification :) .

About these ads

11 responses to “MTOM vs. Streaming vs. Compression – Large attachments over WCF

  1. Pingback: WCF Serializers – XmlSerializer vs. DataContratSerializer vs. NetDataContractSerializer « Niraj Bhatt – Architect’s Blog

  2. Pingback: May 2010 be your Best Year so far!!! « Niraj Bhatt – Architect’s Blog

  3. Manish January 12, 2010 at 4:03 am

    Hi,
    I have similar issue with WCF(dotnet 3.5) hosted on
    IIS6. By binding is wsHttpbinding.

    The client communicates through ssl.

    I am having lot of issues during concurrent request for file uploading and downloading from the webservice.
    Currently my api consumes and publishes byte array.
    What are your suggestion to solve the issue?

  4. nirajrules January 12, 2010 at 10:34 pm

    Hi Manish,

    Can you please elaborate on issues you are facing? What’s your threading model on WCF Service?

  5. Manish January 13, 2010 at 1:28 am

    Here is my service configuration file. I want to send and receive byte array from the service. What is the best way to accomplish it?

  6. nirajrules January 13, 2010 at 2:21 am

    I am not sure if wordpress allows you paste configuration file. I need your actual scenario and constraints to recommend anything. In case your security permits and as you are saying you have lot of users I assume you would like to keep your memory usage in control on server. So streaming looks like a good option.

  7. Pingback: What are Encodings? « Niraj Bhatt – Architect's Blog

  8. prashant October 19, 2011 at 10:34 pm

    So using WSHttpBinding there is no way I can use streaming or chunking? In other words if I have to use streaming then I am bound to use BasicHttpBinding…Am I right??Please let me know

  9. Miguel October 5, 2012 at 2:27 am

    Do you know how to MTOM encode just a property of an object?
    In your sample what if you had an object :

    public class UploadData
    {
    public string FileName { get; set; }
    public byte[] Content { get; set; }
    }

    and this object was inside another object let’s say:

    public class RequestObject
    {
    public UploadData[] UploadFiles { get; set; }
    }

    and you wanted to upload the files using this object instead of a stream so your contract would look like:

    [OperationContract]
    ResponseObject UploadFile(RequestObject request);

    Is it possible to serialize each Content as a separate part in the mtom encoded soap message?0

    file1.xml

    file2.xml

    –MIMEBoundaryurn_uuid_FILE1
    Content-Type: text/plain
    Content-Transfer-Encoding: binary
    Content-ID:

    This is the content of doc 1

    –MIMEBoundaryurn_uuid_FILE1–

    –MIMEBoundaryurn_uuid_FILE2
    Content-Type: text/plain
    Content-Transfer-Encoding: binary
    Content-ID:

    This is doc 2

    –MIMEBoundaryurn_uuid_FILE2–

    • Miguel October 5, 2012 at 2:29 am

      Oops.. didn’t escape the xml.. let’s see if this works:

      <s:envelope><s:header></s:header>
      <s:body>
      <n:RequestObject>
      <n:UploadData>
      <n:FileName>file1.xml</n:FileName>
      <n:Content>
      <xop:Include href="cid:1.urn:uuid:FILE1"
      xmlns:xop="http://www.w3.org/2004/08/xop/include"/&gt;
      </n:Content>
      </n:UploadData>
      <n:UploadData>
      <n:FileName>file2.xml</n:FileName>
      <n:Content>
      <xop:Include href="cid:1.urn:uuid:FILE2"
      xmlns:xop="http://www.w3.org/2004/08/xop/include"/&gt;
      </n:Content>
      </n:UploadData>
      </n:RequestObject>
      </s:body>
      </s:envelope>

  10. Pingback: Transferring large payloads of data (Serialized Objects) using wsHttp in WCF with message security | Ask Programming & Technology

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 178 other followers

%d bloggers like this: