Verint Architecture

This is an offbeat post. Recently I was attending Verint training which happens to be a COTS product for managing your Call Center requirements. These requirements include the ones like Call Recording, Screen Capture, Quality management of calls / Scoring, Reports, etc. In this post I will jot down the basics of Verint Architecture I could collect from session.

Verint like any other Call Center Management related product taps into your incoming PSTN network via PBX and CTI interface. CTI interface raises events on various activities, for instance whenever there is an incoming call to your contact center. From here Verint takes over and starts recording the call. Verint also supports Screen capture in case your business needs one.

Verint has a tiered architecture in form of – Acquisition Server, Hub & Data Center. Verint distinguishes between Hub & Data Center for high volume scenarios, otherwise it bundles both together into something called AIO (All-In-One) Server.

Acquisition Server is where modules like VAM (Voice Acquisition Module) & SAM (Screen Acquisition Module) run. Recorded calls are stored here temporarily, and corresponding call detail is captured (Verint calls it as Contact detail). The recorded call files are moved from Acquisition Server to Hub Server via FTP. And the Contact details which are stored in SQL Server are transferred to Hub Server via MSMQ. There is a purging service on Acquisition Server which can purge recorded calls data on disk space available, duration, etc.

Hub Servers appeal seems to be there from a Latency (performance) standpoint. They get Contact details & recorded voice calls from Acquisition Server. Acquisition server has a relatively small storage and relies on Hub for larger storage. All the recorded voice calls needs to be backed up (archived) from Hub Server itself. But unlike them Contact details are synced with Data Center again via MSMQ. I am not quite sure of the motivation in handling these two differently. Verint Portal is also deployed on Hub Server.

Data Center Servers are the ones that have entire Contact database available with them. They also have reporting tools, data warehouse, application database (scoring for instance), speech analytics etc. as part of their system. Data Center seems to handling everything apart from recorded calls which are archived at Hub level.

Food For Thought – Do you take your bugs to stretcher directly?

A critical bug comes to your support team. They try reproducing it and are unable to do so. Business guys get impatient. They escalate issue to the group President. You get a call in midnight from your boss. He tells that he is on a fire call with his boss and his boss’s boss. Your credibility is at stake.

You try understanding the error. Error turns out to be the ubiquitous one – “Object Reference not set to an instance of a Object”. You pull out the source code but don’t find any valid reason for the error. Debug builds are rolled out, PDBs are shipped, Logging is all over the place. A team member leaves for the site visit hoping to find something relevant. You start flipping through Release It, start browsing through Tess’s Blog entries – hoping to get some breakthrough. You start preparing your arsenal – WinDbg + SOS, Memory Profilers, Fiddler, Wireshark, CLR Profiler, etc. You want to somehow get through this one hoping it might result in a promotion. Suddenly a thought pops up in your mind – could this be because of missing Service Packs? Unfortunately, not all .NET Service Packs are as popular as .NET 3.5 SP1. You hit a bullseye. Production confirms they don’t have .NET 2.0 Service Pack 2 on their machine (.NET seems to have an innate trouble with handling circular references). Plans are chalked out, approvals are sought. Things start falling back to normalcy. Of course, the promotion you dreamed of never comes through .

This calls for an interesting analogy. When you visit a doctor he checks you with a stethoscope and doesn’t take you directly to OT. We as developers are technically motivated to do latter. We want to analyze every bug by taking dumps or using profilers. Alas, that would only benefit Airlines using which we do emergency travel.

(P.S. Framework detector is a good tool to troubleshoot such issues. Word – “You” is used here to create thrill and hide my infallible mindset ).

Cross Site Scripting (XSS) vs. Cross Site Request Forgery (CSRF)

This post not only highlights the difference between above but also gives bare bones steps (using ASP.NET Web Forms ) to reproduce them. Though there are more convoluted samples out there, having these bare bones steps can help in better understanding of these terms.  Both XSS and CSRF are type of Web attacks.

Cross Site Scripting

Description: Cross Site Scripting a relatively older attack talks about injecting malicious scripts in web pages which then would served to other users over a period of time. The malicious scripts in turn gains access to page content and start misusing it. A simple example could be someone entering a malicious JavaScript function in comments section of a webpage. When other users try to fetch that page they would also fetch malicious JavaScript and that can be devastating.

Solution: ASP.NET Web Forms can counter attack this by checking incoming requests for malicious scripts. ValidateRequest an attribute of Page Directive helps in preventing any incoming malicious scripts. Although ValidateRequest does a good job many don’t consider it foolproof. And a common recommendation is to use Server.HtmlEncode() for every output. Example:

Response.Write (“<script>alert(‘Niraj’)</script>”);
Response.Write(Server.HtmlEncode(“<script>alert(‘Niraj’)</script>”)); // Or HttpUtility.HtmlEncode

Simple Steps To Reproduce: Create a ASP.NET Page with TextBox, Submit Button & Label. Go to Button’s click event in code behind file and assign the content of textbox to label (without HtmlEncode). Run the application and and type script tag with alert message in textbox. Click on Submit button. You, on receiving response, would see that label is blank while an alert pops up on your screen. Go to CodeBehind file and wrap textbox’s text within HtmlEncode. Run the application again and output would be as per your expectations.

Cross Site Request Forgery

Description: Cross Site Request Forgery (also called One-Click) attack is lesser known though much easier to implant. Normally, a HTTP request is enough to get website into doing something. But most often that Website won’t allow your request to get through due to lack credentials. So how can a hacker lull you into providing your credentials? Ubiquitous way – a hacker can send you mail telling you won a grand prize and to claim it click on a link in his mail. You click and in case you are having persistent (though not necessary) authentication cookie (.ASPXAUTH) from site that hacker wants to manipulate, hacker would latch on it, use your credentials and send a HTTP request to that site.

Solution: A preferred approach to counter attack this is bundle some user specific secret into the request which hacker won’t have access to. This secret is added while the page is created on server & is cross verified on every postback. ASP.NET Web Forms does it with help of ViewStateUserKey property while ASP.NET MVC leverages Html.AntiForgeryToken() method and ValidateAntiForgeryToken attribute.

Simple Steps to Reproduce: Create a web site with couple of pages. Go to web.config, set the authentication mode to Forms & specify the loginUrl as one of the pages. Go to that login page & place a button ‘Login’. Go to code behind file & in click event of ‘Login’ button add a single line -

FormsAuthentication.RedirectFromLoginPage(“YourName”, true);.

This will add a persistent cookie & redirect the user to requested page. In second page go to .aspx @Page directive and ensure it has – EnableViewStateMac=”true” attribute. That’s it, browse this page. You will get login page due to forms authentication. Click on Login Button to login. You will come to your requested page. Click on View -> Source of this page & copy the entire source (CTRL + C). Create a directory on your machine and create a text file in it and paste source in that. Rename the text file to dummy.aspx. Configure a virtual directory for dummy.aspx using Inetmgr. Edit dummy.aspx and add couple of lines after closing </form> tag.

<script> document.form1.submit(); </script> (N.B. form1 is the default name of form generated by ASP.NET.)

Last step is to change ‘action’ attribute in form tag of dummy.aspx to point to the page where request should be posted.

<form method=”post” … action=”http://localhost/SomeVirtualDirectory/YourSecondPage.aspx”>

That’s it. Open your inetmgr and try browsing dummy.aspx. Things should work fine and you would be redirected to your second page. N.B. you won’t be redirected to login page as you already have a persistent authentication cookie created. This is how attack is launched. Instead of going to inetmgr, you will open the link by clicking on Prize URL sent by hacker and the page would be hosted on hacker’s machine instead of yours.

To counter attack this go to YourSecondPage.aspx and add below method:

protected override void OnInit(EventArgs e)
{
base.OnInit(e);
//Add a user specific key to viewstate which will be verified later by ASP.NET
ViewStateUserKey = Session.SessionID;
}

If you try to browse now from inetmgr you would get an error stating – “Validation of viewstate MAC failed”. Prevention in place (N.B. you might get the same error while deploying your application on Web Farm, a solution for such a scenario can be found here).

There is also a general recommendation to allow interactions with your site via HTTPS only inorder to to prevent attacks like Session Hijacking, etc. Not doing so would give hackers easy access to session cookie – “ASP.NET_SessionID”. Finally parameterized queries are recommended to prohibit SQL injection attacks, server side validations should always be done, and custom cookies must set HttpOnly property to true.

Any other techniques you always use? Hope above helps to some extent .

Food For Thought – Do you accept password as clear text?

There are some subtle bugs we developers invite due to our backgrounds. I recently saw a configuration screen in an application (normally used by only admin) asking for connection string:

Preferred approach would have been:

Of course, connection timeout is missing in both .

Food For Thought – Does your product / application performance improve with every release?

This year I plan to write few small blog entries that can stir bit of thinking. I had written a similar post couple of years back. Windows 7 is easily the best OS from Microsoft so far. Personally, though its way better than Vista, there are still some areas of improvisation. My IE still hangs, Outlook still gets stuck. That beside Microsoft has done a great thing on improvising performance yet adding new features.

That’s ideally what should happen to your application / product. I have seen robust products which give in over a period of time. The reason stated normally is increased features. Not acceptable. Ideally your product should improve on performance with release while adding on features too. VS.NET 2010 has few issues on this front. So what happens with your product / application? I would love to hear your experiences.

May 2010 be your Best Year so far!!!

Happy New Year, Guys. Phew!!! What a year of blogging… Although blog has seen substantial traffic increase above 1000% compared to last year, what mattered more was a year of self satisfying blogging. If you have any thoughts of improvisation like changing my blog theme, blogging style, no. of posts, using polls, etc. I would love to hear on that. As an experiment I had enabled ratings for my posts. I was hoping people to rate and leave comments whenever a post didn’t meet their expectations but unfortunately that doesn’t happen always. Luckily, none of my rated posts have an average rating of less than 3.5, which is encouraging. Like last year below are statistics for top ten posts of 2009:

1) MVC vs. MVP vs. MVVM – Little did I realize while writing this post that it would top the charts surpassing everything. With 19 votes and average rating of 4.5 clearly this post is a chart buster. This shows that you needn’t be esoteric with your blogging, plain simple grounded thoughts are good enough to bring people to your blog. So if you are thinking to start a blog but don’t have ground breaking ideas to share with others, you can still start and be as useful as anybody else. Hoping to get better of this post in 2010.

2) ControlTemplate vs. DataTemplate vs. HierarchicalTemplate vs. ItemsPanelTemplate – common point of confusion for WPF programmers.

3) Resolving XmlDictionaryReaderQuotas Error For WCF Compression using GZipEncoder – one of the frequent encountered errors by WCF developers – XmlDictionaryReaderQuotas.

4) NHibernate Lessons Learned – covers some field knowledge from one of the projects we were working on. Handy reference.

5) RAID 0 vs. RAID 1 vs. RAID 4 vs. RAID 5 vs. RAID 10 – Proud of this one. I don’t deal with infrastructure setup daily but was able to put this across in a concise way. In fact references to this post where from some prominent IT forums.

6) Username Authentication over basicHttpBinding with WCF’s ChannelFactoryInterface – Again a frequently used scenario but lesser known.

7) MTOM vs. Streaming vs. Compression – Large Attachments Over WCF – we all want to transfer as little as possible but just can’t away with it. This post highlights how can you transfer large data with WCF contrasting the options available.

8 ) Logging Best Practices – Only post from 2008 that made into top 10. A widely discussed topic and a post which can be improvised.

9) Inside WPF’s DependencyObject and DependencyProperty – Only post that I liked so much and imported it from old blog.

10) Snapshot vs. LogShipping vs. Mirroring vs. Replication vs. Failover Clustering – Last but not a least other ‘versus’ post that highlights important differences between techniques part of almost every enterprise setup.

Few other posts that liked but which didn’t make it to top list – Load Balancing vs. Failover Clustering, Tower Servers vs. Rack Servers vs. Blade Servers, WCF Serializers, Performance Testing, .NET Threads, …

That’s it from last year. May, year 2010 be the best year by far for all of us. Keep Rocking!!!

SQL Server Reporting Services (SSRS) Architecture Overview

This is level 100 for people trying to figure out how SQL Server Reporting Services (SSRS) works and what makes it work. Honestly, I haven’t found SSRS architecture explained that clearly. I have given up in past, as the search engines weren’t leading to any easy to understand sources. Few days back I had to give a management presentation on migrating to SSRS. So with my back on wall I had little option but to dive in. During the journey I came across few distilled facts that I am sharing in this blog post. Hope you find them useful .

SSRS is an optional package which you can select to install while installing SQL Server. SSRS in turn is made up of number of components. The simplest diagram I could find that describes these components and their deployment was from TechNet

As you can see in above diagram when you install SSRS it creates Report Server Databases in your SQL Server Instance. These databases are ReportServerDB and ReportServerTempDB which are used to store report configurations and other things including Caching, Session, etc. that improvise the overall performance. You have an option of installing other components like Report Manager and Report Server on the same machine where SQL Server instance is running or you can install them on a different server (typical enterprise setup). An important thing to note here is if you opt for latter you would end up paying for 2 or more SQL Server licenses.

Let’s focus on other components. The best concise write up you are to found on Report Server, Report Manager & Report Designer happens to be on Wiki.

As it turns out there are 4 distinguish components of Reporting Services:

1) Report Server: A collection of Web Services (.asmx) which allows LOB applications to interact with Report Server database.

2) Report Manager: An ASP.NET web based application that in turn interacts with Report Server Web Services. This is a great tool for the Report Designers in your team. They can create reports (see next point as to how), deploy them to Report Server Database, and test them using Report Manager. Report Manager also allows configuring security associated with Reporting Services.

3) Report Designer: There would be few guys in your team whom you may want to designate as Report Designers. Report Designers can design reports using VS.NET Business Intelligence projects (Report Server Project). Once they are done with designing their reports then can publish them to Report Server and test them using Report Manager. There is also a tool called as Report Builder available which is targeted at business users who want to generate custom reports on fly. Report Builder is a ClickOnce application, intuitive and easy to use but doesn’t support all the options available with VS.NET.

4) Background Processing: This is a windows service (ReportingServicesService.exe) component targeting report processing, scheduling (auto generated reports), subscriptions (mailers), etc.

SSRS 2005 created virtual directories for Report Server & Report Manager, but SSRS 2008 leverages the OS level HTTP listener making SSRS independent of IIS. This allows bundling of Report Server & Report Manager within ReportingServicesService.exe.

There is another term you would run into while talking about SSRS – RDL. RDL stands for Report Definition language. This is an XML file which stores query information, data source information, etc. which are required to generate report. There is another type of report definition – RDLC (Report Definition Language Client-side) which don’t store any of above configurations. RDLC is a client side component (VS.NET Application Projects) to which you can pass data (e.g. via DataSet) coming from any of data sources. RDLC can be useful for scenarios like implementing custom pagination (SSRS 2005 pagination by default is client side pagination).

Inputs/Thoughts/Suggestions/Corrections?

PDB and ClickOnce

Bit of history first. PDB – Program Debug Database is essential sometimes during debugging. By default, stack trace points to the function where problem lies and don’t include line numbers at which errors are thrown. Sometime this becomes quite critical for a serious production issue. Best practice seems to build PDBs during your build process (both Debug / Release), exclude them while creating your installers, and ship them to production when you need to diagnose your code. Another related thing you might want to keep in mind while debugging some critical issue – you can create a debug build by turning off code optimization (project properties -> Build Tab -> Optimize Code (uncheck)). This helps you to get an accurate stack trace devoid of any code optimizations like inline functions. Let’s get back to PDBs, the topic of this post. Enabling PDB effect normally is done by turning them on (Project Properties -> Build -> Advanced -> Debug Info = pdb-only) and copying generated PDBs to deployment directory. But when you are using ClickOnce things are different. In ClickOnce, the assemblies are downloaded the client’s local machine and then executed (deriving the benefit of auto update). So how do you ensure that the client download PDBs with assemblies? Fortunately VS.NET simplifies this for us. Steps are below:

1) Go the publish tab of your project file and click on “Application Files” button.

2) By default PDB files are not bundled for publish. You need to check “Show All Files” check box and then you would get to see PDB. PDB again is excluded by default, so include them. And you are all set to get line numbers and file names with your stack trace.

Hope this helps .

Explicit Interface Implementation and WCF

You are inheriting from 2 contracts to create a contract. Both contracts that you are inheriting from have a method with same name and signature. You don’t have much control over these interfaces as they are tightly coupled with your subsystem. You create an implementation class with help of explicit interface implementations and dispatch calls to subsystem. Now you have a challenge of invoking the right implementation from client side? Will WCF work even when we can’t have any config files? Fortunately YES, WCF supports such scenarios. Below is a code sample on same, hope it helps you when you have a sharp knife against your throat :

//Contracts
[ServiceContract]
public interface IIPV4Location
{
[OperationContract]
string GetLocationId();
}

[ServiceContract]
public interface IIPV6Location
{
[OperationContract(Name = "IIPV6Location")] //A must for WCF to distinguish
string GetLocationId();
}

[ServiceContract]
public interface ILocation : IIPV4Location, IIPV6Location
{ }

//Server Side
class LocationImpl : ILocation
{
string IIPV4Location.GetLocationId()
{
return "Ipv4";
}

string IIPV6Location.GetLocationId()
{
return "Ipv6";
}
}

class Program
{
static void Main(string[] args)
{
ServiceHost host = new ServiceHost(typeof(LocationImpl), new Uri("http://localhost:9999/"));
var behavior = new ServiceMetadataBehavior();
behavior.HttpGetEnabled = true;
host.Description.Behaviors.Add(behavior);

host.AddServiceEndpoint(typeof(ILocation), new BasicHttpBinding(), "Location");

host.Open();
Console.WriteLine("Running...");
Console.ReadLine();
host.Close();
}
}
//Client Side
class Program
{
static void Main(string[] args)
{
ILocation location = new ChannelFactory(new BasicHttpBinding(),
"http://localhost:9999/Location")
.CreateChannel();

var ipv4Location = ((IIPV4Location)location).GetLocationId();
Console.WriteLine(ipv4Location);

var ipv6Location = ((IIPV6Location)location).GetLocationId();
Console.WriteLine(ipv6Location);
}
}

Load Balancing vs. Failover Clustering

A pending post from long time!!! At distance both look quite similar and are point of confusion for many. Rationale though is making Load Balancing address scalability while Failover Clustering address high availability. Load Balancing is all about improvising performance while Failover Clustering is improvising uptimes mitigating system failures. Another difference is, you would find Load Balancing happening at web/application servers (stateless hopefully) and failover clustering at database servers (state full). Industry seems to be using word “Cluster” (set of connected nodes) for both – but with different intents of Load & Failover.

Both are also separate things in terms of configuration & setup. For instance, Windows 2003 (currently that’s what we have in our production) has separate options for Load Balancing & Clustering. Windows recommendation is not to mix both, i.e. you shouldn’t cluster machines for failover which are already configured for load balancing.

Setting up load balancing is simple – you need couple of machines connected to a common network and an additional IP where clients would connect to. This Virtual IP where the requests are made by clients, is in turn is used for Load Balancing nodes that part of this cluster (load balancing cluster).

Setting up failover clustering on the other hand is little complex. You need 2 networks a public and private (hear beat), a shared drive (called Quorum), and an additional Public IP (in addition to minimum – 2 public and 2 private IPs that 2 systems will have). Remember, creating a failover cluster at Windows level is a primary requirement to build a failover SQL Server cluster. Reason to create a Windows level cluster is install required cluster services and create cluster groups (logical collection of nodes). You can select a cluster group (obviously at least 2 nodes should be part of this group) and configure SQL Server Cluster or anything else on top it. SQL Server Cluster would require an additional IP, another shared disk for installation & database files (this disk is a shared resource for chosen cluster group), domain account (that has administrative privileges on all nodes) & group on that domain which that account has complete access. Shared Quorum and Shared Disk are normally part of SAN storage. I have also come across quite a few implementations using Starwind or similar tools to create these shared iSCSI targets in form of virtual disks (.img). It might be helpful to know that Windows 2003 doesn’t have the iSCSI initiator built-in and you can download the same from here.

Hope above helps to some extent .