Niraj Bhatt – Architect's Blog

Ruminations on .NET, Architecture & Design

Performance Testing – Response vs. Latency vs. Throughput vs. Load vs. Scalability vs. Stress vs. Robustness

Normally I find quite a bit of ambiguity when people talk about performance tests, some restrict it to response time whereas some use it to cover a gamut of things they are testing or measuring. In this post, I will put across few thoughts on contrasting between them. Ideally a lot depends on what you are trying to measure. The terms that you will frequently hear in this arena are – Response Time, Latency, Throughput, Load, Scalability, Stress, Robustness, etc. I will try explaining these terms below also throwing some light on how can you measure them.

Response Time – Amount of time system takes to process a request after it has received one. For instance you have API and you want to find how much time that API takes to execute once invoked, you are in fact measuring response time. So how do we measure them? Simple use a StopWatch (System.Diagnostics) – start it before calling API & stop it after API returns. The duration arrived here is quite small so a preferred practice is to call that API in sequential loops say 1000 times, or pass variable load to the API if possible (input/output varies from KBs/MBs/GBs e.g. returning customer array of varied lengths).

Latency – In simplest terms this is Remote Response time. For instance, you want to invoke a web service or access a web page. Apart from the processing time that is needed on the server to process your request, there is a delay involved for your request to reach to server. While referring to Latency, it’s that delay we are talking about. This becomes a big issue for a remote data center which is hosting your service/page. Imagine your data center in US, and accessing it from India. If ignored, latency can trigger your SLA’s. Though it’s quite difficult to improve latency it’s important to measure it. How we measure Latency? There are some network simulation tools out there that can help you – one such tool can be found here.

Throughput – transactions per second your application can handle (motivation / result of load testing). A typical enterprise application will have lots of users performing lots of different transactions. You should ensure that your application meets the required capacity of enterprise before it hits production. Load testing is the solution for that. Strategy here is to pick up a mix of transactions (frequent, critical, and intensive) and see how many pass successfully in an acceptable time frame governed by your SLAs. How to measure it? You normally require a high end professional tool here like Visual Studio Team System (Load Testing feature). Of course, you can try to simulate load through your custom made applications /code but my experience says custom code are good to test for response times; whereas writing custom code for load testing is too much of work. A good load testing tool like VSTS allows you to pick a mix of transactions, simulate network latency, incorporate user think times, test iterations, etc. I would also strongly recommend this testing to be as close as possible to real world with live data.

Scalability – is the measure of how your system responds when additional hardware is added. Does it take new increased load by making use of added resources? This becomes quite important while taking into consideration the growth projections for your application in future. Here we have two options – scale vertically/up (better machine) or horizontally/out (more machines), latter is usually more preferred one. A challenge to scale out is to ensure that your design doesn’t have any server affinity, so that a Load balancer can adjust load across servers. Measuring scalability can be done with help of load balancing tools with a software/hardware NLB in place ensuring system is able to take on new load without any issues. One can monitor performance counters to see whether actual request load has been balanced/shared across servers (I plan to cover NLB in a future post).

Stress testing – Many people confuse this or relate it to load testing. My take which I have found easy to explain is, if you find yourself running tests for more than 24 hours you are doing a stress test (precise would be your production time i.e. duration before you take your machine offline for a patch, etc.). Motivation behind stress test is to find out how easily your system can recover from over loaded (stressed) conditions. Does it limp back to normalcy or gives up completely? Robustness an attribute that is measured as part of stress testing relates to long running systems with almost negligible down time. A simple example here could be memory leak. Does your system release memory after working at peak loads? Another, what happens if a disk fails due to constant heavy I/O load? Does your system lose data? Finding and addressing such concerns is motivation behind stress testing.

I will look forward to read your thoughts on above :) .

About these ads

23 responses to “Performance Testing – Response vs. Latency vs. Throughput vs. Load vs. Scalability vs. Stress vs. Robustness

  1. raghu November 19, 2009 at 3:41 pm

    I wanted a quick difference between Response Time and Throughput. I got it here. :)

    Thank you for the definitions.

  2. zubin December 8, 2009 at 11:23 am

    Good to get abstract information about terminology..
    thanks lot…

  3. Pingback: May 2010 be your Best Year so far!!! « Niraj Bhatt – Architect’s Blog

  4. Sachin Chaudhary February 3, 2011 at 11:25 am

    Good Presentation and Information.

  5. Nagesh March 10, 2011 at 11:42 pm

    Really your explanaion is very easy to understand. Thanks for that. Will you be able to provide good performance material or can u plz suggest good blog to learn performance stuff

    • vijay June 16, 2011 at 12:43 am

      I was working on Oracle RAC and came across throughput,response time and scalability.
      Nice to see clear definitions and difference amongst them. Thanks a lot.

  6. Pingback: Scalability: Links, News and Resources (1) « Angel “Java” Lopez on Blog

  7. Kriti Tiwari August 22, 2011 at 9:54 am

    Hi,

    Thanks for above information.
    Could you please tell me consequences of increase in think time; i.e. User Load will increase if think time is increased or not?

  8. yakub mohammad September 14, 2011 at 2:23 pm

    Complex information in simple terms thanks

  9. Ben October 6, 2011 at 3:20 pm

    Great post! Thanks for the overview. Does anybody know good literature covering performance metrics like throughput and response time? I would like to have a look at this in more depth, but can’t find any books.
    Thanks!

  10. vinodh kumar October 20, 2011 at 4:44 pm

    Can u please tell me the Difference between application response time and Server response time

  11. vinodh kumar October 20, 2011 at 4:46 pm

    Can u please let me know the Difference between application response time and Server response time

  12. priyaa December 13, 2011 at 12:00 pm

    thanq its very useful…..

  13. prashanth February 22, 2012 at 12:37 pm

    Hello,
    Good write up. I just have couple of questions:
    For testing the performance of any website(say, CNN.com), what is the ideal bench mark values which we need to look for:

    1) Throughput
    2) Latency
    3) Response time
    4) CPU utilization time

  14. Balakrishnan Ramasankar February 22, 2012 at 5:04 pm

    awesome explanation with precise information

  15. Alixhah March 17, 2012 at 5:05 am

    what is the effect of scalability on throughput,PDR,access delay and power consumption?

  16. George Hicks June 15, 2012 at 1:53 am

    I appreciate the comment about server affinity, and would like to add that there are instances where server affinity is important for best possible performance. Semi-sticky sessions, where server affinity is encouraged but load balancing is also allowed.

  17. Rahul D July 8, 2012 at 8:02 pm

    enlightening …

  18. Tobi October 18, 2012 at 5:07 am

    what is the best testing tool to test the above mentioned 1) Throughput
    2) Latency
    3) Response time
    4) CPU utilization time

  19. Prasanth RJ December 27, 2012 at 1:16 pm

    crisp, clear and very informative.. Thx for sharing.

  20. Caio January 23, 2013 at 6:41 pm

    Great stuff! Thanks for sharing.

    I’d like to mention that there are free softwares out there that can perform most, if not all, these types of measurements. For example, instead of writing a code to simulate a load test, use JMeter that is a lot more capable and accurate. And it works on Windows, Mac, Linux…

  21. Rajalakshmi May 8, 2013 at 7:14 pm

    Got the proper difference for the performance testing terms in very quick time.

  22. Pingback: Performance, throughput, scalability | Windhood's Weblog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 160 other followers

%d bloggers like this: