Recovering from Windows Azure Virtual Machines failures – Unable to establish RDP connection

Your ability to recover from failures is quintessential while working with any cloud platform and same applies to Windows Azure Virtual Machines. Being in preview mode you might experience some glitches around this offering. For instance, there were times when I would setup a VM and next day I wasn’t able to RDP (Remote Desktop) into it. It looked like my hard work of setting up the VM would go down the drain. But those initial struggles filled my knowledge gaps. While the stability of VMs is dramatically improvising as we approach the General Availability, below are few pointers to deal with such failures. Hope you find them useful. 

a. Restart VM: First is to restart your VM. That should do the trick whenever your VM becomes unresponsive.

b. Delete VM: In case restart doesn’t help or restart operation is failing you can try deleting and re-creating VM. If you are attempting a manual delete via Azure Portal, you should delete the underlying Cloud Service too. Cloud Service – is the container holding your VM and would become visible when you delete the VM (will detail this in an upcoming post). Deleting your Cloud Service would allow you to reuse your DNS name (as a best practice ensure that your VM and DNS names are prefixed with unique identifiers). I have seen few developers struggle and provide different names to their VMs every time they delete it, failing to realize that they need to delete the Cloud Service too. Below Figure shows how you can create a new VM by selecting an existing disk (New -> Compute -> Virtual Machine -> From Gallery)

c. Resize VM: An alternative to deleting your VM is to change the size of your VM. This is less intrusive but might not be feasible in all the scenarios.

d. Unlock VHD / Disk: At times, after you delete your VM, the portal might show that Disk is still attached to the deleted VM. This would result into a quandary because you can’t reuse the Disk to create a new VM until it’s detached. The easier option to resolve this is to delete the disk object. Deleting the Disk object would still retain the underlying blob. After deletion you can re-create the disk object and use it to spawn a new VM. If you can’t delete the disk then your only option is to break the blob lease manually via PowerShell or using Storage Client Library (refer to this MSDN forum link for further information).

Please do leave a comment, if you have used other approaches.

Dependency Inversion, Dependency Injection, DI Containers and Service Locator

Here is a post I have been planning to do for years . Yes, I am talking about Dependency Injection, which by this time, has most likely made its way into your code base. I got to recollect few thoughts around it in a recent discussion and hence writing them down.

Dependencies are the common norm of object oriented programming, helping us to adhere to software principles like Single Responsibility, Encapsulation, etc. Instead of establishing dependencies using direct references between classes or components, they are better managed through an abstraction. Most of the GOF patterns are based around this principle which is commonly referred as ‘Dependency Inversion’. Using Dependency inversion principle we can create flexible Object Oriented designs, making our code base reusable and maintainable.

To further enhance value proposition of the dependency inversion, you can pass the dependencies via a constructor or a property from the root of your application, instead of instantiating them within your class. This would allow you to mock / stub your dependency and make your code easily testable (read unit testing).

E.g.

IA ob = new A(); //Direct instantiation
public D (IA ob) { … } //constructor injection
public IA ob { get; set } //property injection – create object and set the property

Entire software industry sees value in above approach and most people refer to this entire structure (pattern) as Dependency Injection. But beyond this things get little murkier.

Confusion starts for programmers who see above as a standard way of programming. To them Dependency Injection (DI) is the automated way of injecting dependencies using a DI container (DI container at times is also referred as IoC (Inversion of Control) container. As Martin Fowler points out in his classic article, IoC is a generic term used in quite a few cases – e.g. the callback approach of Win32 programming model. In order to avoid confusion and keep this approach discernible, the term DI was coined. In this case, IoC refers to inversion of dependencies creation, which are created outside of the dependent class and then injected.) If you are surprised by this statement, let me tell you 90% of people who have walked to me to ask if I was using DI, wanted to know the about the DI container and my experience with it.

So what the heck is this ‘DI container’? A DI container is a framework which can identify constructor arguments or properties on the objects being created and automatically inject them as part of object creation. Coming from the .NET world, the containers I use include StructureMap, Autofac and Unity. DI containers can be wired up using few lines of code at the starting of your program or you can even specify configuration in a XML file. Beyond that, containers are transparent to the rest of your code base. Most containers also provide AOP (Aspect Oriented Programming) functionality and its variants. This allows you to bundle cross cutting concerns like database transactions, logging, caching, etc. as aspects and avoid boilerplate code throughout the system (I have written a CodeProject article on those lines). Before you feel I am over simplifying things, let me state if you haven’t worked with DI containers in past you are likely to be faced with a learning curve. As is the case with most of the other frameworks, a pilot is strongly recommended. As a side note, the preferred injection rule is – unless your constructor requires too many parameters (ensure you haven’t violated SRP), you should resort to constructor injection and avoid property injection (see fowler’s article for a detailed comparison between the two injection types).

Finally, let’s talk about Service Locator an alternative to DI. A Service Locator holds all the services (dependencies) required by your system (in code or using a configuration file) and returns a specific service instance on request. Service Locator can come in handy for scenarios where DI container is not compatible with a given framework (e.g. WCF, ASP.NET Web APIs) or you want more control over the object creation (e.g. create the object late in the cycle). While you can mock service locator, mocking it would be little cumbersome when compared to DI. Service Locator is generally seen as an anti-pattern in DI world. Interestingly, most DI containers offer APIs which can allow us to use them as Service Locator (lookup for Resolve / GetInstance methods on the container).

Below sample shows you StructureMap – a DI container, in action (use NuGet to add StructureMap dependency).

class Program
{
static void Main(string[] args)
{
var container = new Container(registry =>
{
registry.Scan(x =>
{
x.TheCallingAssembly();
x.WithDefaultConventions();
});
});
var ob = container.GetInstance<DependentClass>();
ob.CallDummy();
}
}

public interface IDependency
{
void Dummy();
}

public class Dependency : IDependency
{
public void Dummy()
{
Console.WriteLine("Hi There!");
}
}

public class DependentClass
{
private readonly IDependency _dep;
public DependentClass(IDependency ob)
{
_dep = ob;
}
public void CallDummy()
{
_dep.Dummy();
}
}


I will try to post some subtle issues around DI containers in future. I hope above helps anyone looking for a quick start on DI and associated terms .

Controlling Windows Azure VM lifetimes

Most of the Cloud computing resources are billable on an hourly basis and it’s important that you release these resources when you no longer need them. This typically applies to Windows Azure Virtual machines not running 24×7, example – there could be business workloads which requires an application to be available only twelve hours on week days. To limit running costs most users stop their VMs, just to realize that Windows Azure bills for VMs that are in a stopped state. So, your only option to control costs is to delete the VMs. But wouldn’t deleting VM cause any issues?

The answer is both No and Yes. When you delete a VM you are just deleting VM instance but the underlying OS and data disks are still intact (in fact, you still keeping paying for their storage which luckily is quite negligible). Hence, you can easily resurrect your VM without much harm. It’s important to note that when you delete the VM, you still retain the underlying Cloud Service container and its associated Site URL. The issue you might face though when you delete and re-create the VM, is the public IP address change. I work in an organization with strict IT security rules and locked down access. Static IP was necessary for me to raise an outbound RDP access request with my IT team. But with IP changing everyday it was definitely turning into a challenge. In end the solution I adopted was to create an extra small VM running 24×7 and bounce from there to other VMs.

To delete a VM you can use PowerShell cmdlets. PowerShell cmdlets allow you to export your VM configuration, delete the VM and then re-create VM using exported configuration.

Export-AzureVM -ServiceName ” -Name ” -Path ‘c:\vmconf.xml’

Remove-AzureVM -ServiceName ” -Name ”

Import-AzureVM -Path ‘c:\vmconf.xml’ | New-AzureVM -ServiceName ” -VNetName ‘’ -DnsSettings ‘’

Export configuration as shown above is stored in a XML file. It contains various properties of a VM including Endpoints, disks, VM size, subnet, etc. Below snapshot from Azure Portal shows an empty Cloud Service Container post deletion of the VM. Currently there is no cost associated with an empty cloud service. It’s important to note that when you retain Cloud Service you also retain the underlying DNS URL.

Creating PivotTable and PivotChart using VSTO

Extending the previous discussion on PivotTables, in this blog post I will show how you can create a PivotTable and PivotChart using Visual Studio Tools for Office (VSTO). After populating data into excel worksheet, generating PivotTables is quite a common task for VSTO developers. This can be easily broken down into below steps:

a) Create Table Range where Pivot Table would be created
b) Create Chart Range where Pivot Chart would be placed
c) Determine Data Range that would act as input for Pivot (generally you might want to create many pivot tables out of this range)
d) Create a PivotCache object using Data Range as Input
e) Determine Data Range that would act as input for a pivot table
f) Generate Pivot Table and set the orientation of Pivot fields
g) Create chart with using Pivot Table as input

Let’s have a quick look at the C# source code below:


//Step A, B
Worksheet worksheet = Globals.Factory.GetVstoObject(
Globals.ThisAddIn.Application.ActiveWorkbook.ActiveSheet);

var chartRange = worksheet.Range["G3", "J15"];
var pivotTableRange = worksheet.Range["G20", "J30"];
//Step C
var listObject = FindListObject("demoDataList");
var end = listObject.DataBodyRange.get_End(Microsoft.Office.Interop.Excel.XlDirection.xlDown).Row;
var dataRangeForPivot = worksheet.Range["A1", "W" + end.ToString()];
//Step D
Excel.PivotCache pivotCache = this.Application.ActiveWorkbook.PivotCaches().Add(Excel.XlPivotTableSourceType.xlDatabase, dataRangeForPivot);
//Step E
Microsoft.Office.Interop.Excel.Range dataRangeForPivotTable = worksheet.Range["H35", "H" + end.ToString()];
//Step F
Microsoft.Office.Interop.Excel.PivotTable pivotTable = pivotCache.CreatePivotTable(dataRangeForPivotTable, @"DTable", dataRange, Type.Missing);
Microsoft.Office.Interop.Excel.PivotField demoField = ((Microsoft.Office.Interop.Excel.PivotField)pivotTable.PivotFields(8));
demoField.Orientation =
Microsoft.Office.Interop.Excel.XlPivotFieldOrientation.xlRowField;
demoField.Orientation =
Microsoft.Office.Interop.Excel.XlPivotFieldOrientation.xlDataField;
demoField.Function = Microsoft.Office.Interop.Excel.XlConsolidationFunction.xlCount;
//STEP G
Chart chart = worksheet.Controls.AddChart(chartRange, "DChart");
chart.ChartType = Microsoft.Office.Interop.Excel.XlChartType.xlColumnClustered;
chart.SetSourceData(pivotTableRange);
chart.ChartTitle.Text = "Demo Analysis";

Confusing? I hope not

Pivot, PivotTable and PowerPivot

OK, I am not a data guy. Or let me say I am not an Excel (MS-EXCEL) guy. I have always believed that if you are good at PowerPoint (along with Visio) you probably are an architect and if you are good at Excel you are most likely to be a manager . But not knowing Excel well, has always given me a feeling of – “something is missing”. After all, it’s the same data, but the way few people present it, they just make it look so good. While it takes time to develop those skills, understanding Pivot and related features is usually a good step in that direction.

Excel primarily is a spreadsheet tool in which you can dump some data. At times though, data is not everything. For e.g. your retail transactional system captures the sales across product lines, but if a manager needs to predict his stocks for next fiscal based on historic sales, he got to make some sense out of that data. That’s where you got to Pivot the data. The word ‘Pivot’ as Wiki generally describes it – is center point of any rotational system. You have seen them in action while helping your kids on a seesaw. Pivot table on other hand is a data summarization tool found in spreadsheet softwares like Excel. If you have to bridge this tool and the word ‘Pivot’ – you can think you are bending your data to a specific viewpoint using functions like sort, total, average, count, etc. Let’s see it with help of a trivial example. Consider below the sample data in Excel. To bend it to monthly stock view you can click on Insert tab and insert a Pivot Table. This would give you an empty pivot table and a PivotTable field list. Drag and drop your fields to specific areas to bend your data to a viewpoint (N.B. you can have multiple fields for each area and control them via expand / collapse buttons and all your charting knowledge works as is with Pivot tables).

PowerPivot as the name suggests elevates the Pivot table feature to next level making it full-fledged self-service BI tool. An Excel Add-In, PowerPivot connects to almost any external data sources including SQL Server Analysis Services. Once fetched, data stays inside the workbook and using excellent compression techniques the file size on the disk is still relatively small. You can share your PowerPivot workbooks via SharePoint and configure refresh cycles for your workbook to ensure your team always makes their decisions based on recent data. The experience of pivoting though remains same as shown earlier.

Hope that helps .

Passing Parameters to Hadoop Streaming

This weekend I would be presenting @ BDOTNET UG meet on topic – “Big Talk: Hadoop on Azure”. In case you are around and plan to attend here’s the facebook event link. In this talk I would help you get started with Hadoop and show how you can leverage it with Windows Azure. With that, let’s focus on the subject of this blog post.

Hadoop Streaming allows you to write and run MapReduce jobs in language of your choice. For Azure and Microsoft world this would be mostly C#. You can create your programs / executable in C#, read input from Console and write output to Console. Mapper task would feed input lines to your executable via console (standard input) and also collect output via console (standard output). It converts output into key value pairs. Reducer task on other hand converts key value pairs into input lines, feeds it to your executable (via console), and collects the output (via console) converting it back to key value pairs. For scenarios where you need only mapper you can emit reducer and set ‘numReduceTasks’ to zero as shown below:

call hadoop.cmd jar hadoop-streaming.jar -files "hdfs://10.186.36.85:9000/example/apps/Mapper.exe" -mapper "Mapper.exe" -input "asv://account/inputdata/account.data" -output "/example/data/StreamingOutput/mywc" -numReduceTasks=0

IP address in above case is that of Namenode (you can get by executing following command from Javascript console – #cat file:///apps/dist/conf/core-site.xml).

Now at times, your mapper program would need additional parameters to carry out its operations e.g. say you want mapper to filter data on few attributes. So, how do can we pass these attributes to Mapper executable? Simple – pass them as command line parameters and in your program read them from args.

call hadoop.cmd jar hadoop-streaming.jar -files "hdfs://10.186.36.85:9000/example/apps/Mapper.exe" -mapper "Mapper.exe param1 param2 param3" -input "asv://account/inputdata/ account.data" -output "/example/data/StreamingOutput/mywc" -numReduceTasks=0

static void Main(string[] args)
{
string line;

string parameterOne = args[0];
string parameterTwo = args[1];
string parameterThree = args[2];
…

Hope this helps!

What is DMZ?

A very brief introduction. DMZ is an element which most of architects miss out in their deployment architectures (except few who run their designs through IT pros). The word stands for “Demilitarized Zone”, an area often found on perimeter (outside) of a country’s border. These areas are typically not guarded (under treaties between two or more countries).

In IT domain we refer to DMZ as a separate network. So, why to create separate networks or DMZs? Simplest answer is to enhance security. For example, consider hosting the public facing websites of your business. You might want to host these sites inside a DMZ, separate from your corporate network. In case security of your site is compromised, your corporate network is still safe. Many architects also prefer hosting web and database servers under different DMZs to ensure there is no comprise on data, in case a hacker breaks into their web servers. Like elsewhere, you can use routers to transfer data to DMZ networks. While DMZ is separate network, you must have enough defense packed into it. An enterprise firewall (with redundancy, of course) is a minimum recommendation. Enterprises are also known to have multiple firewalls for each of their DMZs / networks. Below is a simplistic diagram of DMZ deployment.

Hope this helps !

Presenting at Great Indian Developer Summit 2012

Readers, I have been blogging quite less lately but really working on bouncing back soon. Meanwhile, I am speaking at the Great Indian Developer Summit (GIDS) 2012 on Windows Azure Access Control Service Usage Patterns. My session is on 17th April, 10 a.m. IST. Infact, this is my 4th presentation at GIDS conference, having presented at last three. If you are attending the summit, do drop by my session; I am committed to provide maximum ROI on your time investment. Below is the session snapshot

“Access Control Service (ACS) is perhaps the most powerful but least understood aspect of Windows Azure. While developers / architects understand it’s value proposition they are often left confused with surrounding acronyms and buzzwords like Active / Passive federation, SWT, SAML, ADFS, WIF, WS-Trust, WS-Federation, OAuth, OAuth WRAP, etc. This session distills the facts along with the underlying business motivation helping you with your moment on ACS. Having built the initial base session advances to focus on typical usage patterns of Access Control Service within enterprises. These common recurring implementation themes would further simply the mapping of ACS to your LOB applications. Attend this session to walk out with real implementation knowledge on ACS.”

Here’s the presenter billboard (speciality of GIDS )

Looking forward to meet you in person…

Live!!! PluralSight Course – “Windows Azure Diagnostics”

Friends, My new course for PluralSight – “Windows Azure Diagnostics” just got published. It has been quite a gap since I last did the flyweight pattern module for PluralSight. Windows Azure Diagnostics (WAD) to me is a very important aspect of Windows Azure. I can confidently say hadn’t it been for WAD, I wouldn’t have succeeded developing a single application on Windows Azure. On top of that most the articles and materials I came across on this topic talked about only imperative code based setup. I confess WAD was a distant topic for me, and for quite some time I dreaded it, not quite understanding what it stands for and how to work with it. Finally, having got my Ahaa moment , I thought of sharing this knowledge with you all. Among others, course talks about WAD integration with Enterprise Library Logging Application Block and Log4Net , including configuring the logging verbosity at runtime. Course also covers DB Trace Listener, WAD PowerShell Cmdlets, On-Demand Transfer and IntelliTrace.

Do drop me a line (niraj@indiamvps.net), if you happen to look at the course. What you learnt, how useful it was to you, what could have been better (I personally felt there is room for voice quality and demo alignments)?. Though, I am still happy with overall content and course  . If you would like to see any other courses around Windows Azure or other Microsoft technologies, please let me know that as well.Until next time, happy learning…

ConnectionTimeout vs CommandTimeout

ConnectionTimeout a property of Connection class in ADO.NET, is the time you would wait, for connecting to a given database, before flagging a connection failure. Default value is 30 seconds.

new SqlConnection().ConnectionTimeout = 10;

CommandTimeout a property of the Command class in ADO.NET, is the time you would wait, for a command (query, stored procedure, etc.) to return result set, before flagging an execution failure. Default value for CommandTimeout too is 30 seconds.

new SqlConnection().CreateCommand().CommandTimeout = 10;

Unlike ConnectionTimeout which are part of connection string, command timeouts are defined separately (hardcoded or as appSettings). Setting both of them to 0 (zero) would result in indefinite wait period which is generally considered a bad practice. You ideally want to set both of them to in accordance to your performance SLAs. Though at times, while running data heavy background jobs / workflows you may want to set your CommandTimeout to a larger value.