Niraj Bhatt – Architect's Blog

Ruminations on .NET, Architecture & Design

Category Archives: Windows Azure

Azure ExpressRoute Primer

What is Azure ExpressRoute?
ExpressRoute is an Microsoft Azure service that lets you create private connections between Microsoft datacenters and infrastructure that’s on your premises or in a colocation facility. ExpressRoute connections do not go over the public Internet, and offer higher security, reliability and speeds with lower latencies than typical connections over the Internet.

How to setup ExpressRoute Circuit?
ExpressRoute circuits are resources within Azure subscriptions. But before you setup Expressroute connection (or circuit as it’s normally referred as), you need to make decisions about setup parameters.
1) Connectivity Option – You can establish connection with Azure cloud either by extending your MPLS VPN (WAN), or you can leverage your Colocation provider and it’s cloud exchange or roll out an point-to-point ethernet your self. Most large enterprises would use the first option, medium size enterprise running in COLO would go with second option, and the last is more specialized scenario warranting higher level security
2) Port Speed – Bandwidth for your circuit
3) Service tier / SKU – standard or premium (more on this later)
4) Location – You might get multiple options for this depending on your choice for connectivity (#1). E.g. MPLS providers have multiple peering locations from which you can pick the one closet to you
5) Data Plan – Limited plan with pay as you go egress charges or unlimited plan with higher cost irrespective of egress volume

After you make these five choices, you can fire up PowerShell on your VM to execute ‘New-AzureDedicatedCircuit’. Remember to select the right Azure subscription (Add-AzureAccount / Select-AzureSubscription) where you want the circuit to be created. Please note you would need to import ExpressRoute module if not already (Import-Module ‘C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ServiceManagement\Azure\ExpressRoute\ExpressRoute.psd1’

New-AzureDedicatedCircuit -CircuitName $CircuitName -ServiceProviderName $ServiceProvider -Bandwidth $Bandwidth -Location $Location -sku Standard

As soon as this completes you will get a service key which is kind of a unique identifier for your circuit. At this step only your billing starts, as we have only completed Azure side of things. Now you need to work with your Network Service Provider, provide your service key and ask them to complete their side of configuration to Azure. This would also involve setting up a BGP session at your end. Once this done you are all set to leverage expressroute and connect the circuit to azure virtual networks – with the traffic flowing over private connection.

Connecting ExpressRoute Circuit to Azure Virtual Network
Once the circuit is configured it’s relatively straight forward to connect it to virtual network. Once again PowerShell is your friend. But before firing the below command ensure your VNET and the virtual gateway is created.

New-AzureDedicatedCircuitLink -ServiceKey “***” -VNetName “MyVNet”

ServiceKey parameter uniquely identifies your circuit. As circuits are part of the Azure Subscription (wish there was a way to view them in portal) your VNET should be part of the same subscription. This lead to the question – Can we connect expressroute circuits to VNETs across subscriptions? Answer is yes.

Connecting ExpressRoute Circuit to Azure Virtual Network across subscriptions
As we know circuit is part of a subscription, so as a subscription admin you will have to grant rights to other subscription admins so that they can link their VNETs to your circuit. Here’s the PowerShell cmdlet to do that.

New-AzureDedicatedCircuitLinkAuthorization -ServiceKey “***” -Description “AnotherProdSub” -Limit 2 -MicrosoftIds ‘devtest@contoso.com’

This commands allows 2 VNETs from AnotherProdSub to connect to the ExpressRoute circuit. You might see the last parameter MicrosoftId replaced by AzureAD Id (not sure what IDs are supported right now)

Once you have the authorization, you can query the servicekey from your subscription and link your VNET as appropriate.

Get-AzureAuthorizedDedicatedCircuit #This will get details of the circuit including ServiceKey

New-AzureDedicatedCircuitLink –servicekey “***” –VnetName ‘APSVNET’ #Link VNET in another subscription

Remember you can only connect 10 VNETs per circuit. Though this is a soft limit but you can grow only few folds. If you need to create 100 VNET instance you need to look at ExpressRoute Premium.

What is ExpressRoute Premium?
Premium tier for enterprises that need more VNETs per circuit, need their circuit to span geo-political region or have more than 4000 route prefixes. You will pay around 3000 USD more for the premium features, when compared to standard edition with same bandwidth.

How much it costs?
Express route costs boil down to price you pay to Microsoft and your service provider.
To Microsoft it’s
monthly fee depending on the port speed
Bandwidth consumed (unless you are in unlimited data where you flat 300 USD)
Virtual Gateway which you would provision in your VNET (mandatory for expressroute & S2S VPN)

To Network Service Provider:
It’s one time setup fee for the circuit
Bandwidth charges (how much data goes through their cloud to Microsoft)

How long does it take to setup connection?
Well it depends. If you already have a network service provider or an exchange provider supporting Azure, it shouldn’t take more than a day (excluding paperwork). Otherwise this can turn out to be a project in itself.

Can we use ExpressRoute to connect to office 365?
Answer is yes, but it actually depends on your provider. Apart from connecting to Azure VNET, expressroute allows you to establish public peering and Microsoft peering to route your Azure PaaS (public services) and Office 365 traffic over the private network. For more details refer to this link. Public peering allows you to route your traffic to public services like Azure Services and Azure SQL database over private tunnel.

Using a Single Windows Azure Active Directory tenant for All EA Azure Subscriptions

As you know by now Windows Azure Active Directory is at the root of every Azure subscription.

 Image

But in an EA setup you typically have multiple subscriptions and you definitely don’t want to create a different WAAD tenant for every other subscription. So here’s what you can do (there might be other ways too of achieving this). You can first create a Shared account and under that a Shared Subscription. Also create the WAAD tenant you want to use and ensure your shared subscription is under that WAAD tenant. In that WAAD tenant create all the account administrators.

Image

Now go to your EA portal, and add new accounts specifying the account administrators you just created. That’s it – next when you create subscriptions for those newly created accounts, these subscriptions will be by default part of the same WAAD tenant under which you created your shared subscription.

Image

It can’t get any easier, isn’t it 🙂 ?

Windows Azure Portals and Access Levels

When you sign up for Windows Azure you get a subscription and you are made the Service administrator of that subscription.

Image

While this creates a simple access model, things do get little complicated in an Enterprise where users need various levels of access. This blog post would help you understand these access levels. 

Enterprise Administrator
Enterprise Administrator has the ability to add or associate Accounts to the Enrollment and can view usage data across all Accounts. There is no limit to the number of Enterprise Administrators on an Enrollment.
Typical Audience: CIO, CTO, IT Director
URL to GO: https://ea.windowsazure.com

Account Owner
Account Owner can add Subscriptions for their Account, update the Service Administrator and Co-Administrator for an individual Subscription, and can view usage data for their Account. By default all subscriptions are named as ‘Enterprise’ on creation. You can edit the name post creation in the account portal. Under EA usage, only Account Administrators can sign up for Preview features. Recommendation for accounts to be created is either on functional, business or geographic divisions, though creating a hierarchy of accounts would help larger organizations.
Typical Audience: Business Heads, IT Divisional Heads
URL to GO: https://account.windowsazure.com

Service Administrator
Service Administrator and up to nine Co-Administrators per Subscription have the ability to access and manage Subscriptions and development projects within the Azure Management Portal. The Service Administrator does not have access to the Enterprise Portal unless they also have one of the other two roles. It’s recommended to create separate subscriptions for Development and Production, with production having strict restricted access.
Typical Audience: Project Manager, IT Operations
URL to GO: https://manage.windowsazure.com

Co-Administrators
Subscription co-administrators can perform all tasks that the service administrator for the subscription can perform. A co-administrator cannot remove the service administrator from a subscription. The service administrator and co-administrators for a subscription can add or remove co-administrators from the subscription.
Typical Audience: Test Manager, Technical Architect, Build Manager
URL to GO: https://manage.windowsazure.com

That’s it! With above know-how you can create an EA Setup like below

Image

Hope this helps 🙂

Azure Benefits for MSDN subscribers

Friends, hope you are aware of this great offer. Click on the image below to sign up 🙂

MSDNAzureOffer2

Recovering from Windows Azure Virtual Machines failures – Unable to establish RDP connection

Your ability to recover from failures is quintessential while working with any cloud platform and same applies to Windows Azure Virtual Machines. Being in preview mode you might experience some glitches around this offering. For instance, there were times when I would setup a VM and next day I wasn’t able to RDP (Remote Desktop) into it. It looked like my hard work of setting up the VM would go down the drain. But those initial struggles filled my knowledge gaps. While the stability of VMs is dramatically improvising as we approach the General Availability, below are few pointers to deal with such failures. Hope you find them useful. 

a. Restart VM: First is to restart your VM. That should do the trick whenever your VM becomes unresponsive.

b. Delete VM: In case restart doesn’t help or restart operation is failing you can try deleting and re-creating VM. If you are attempting a manual delete via Azure Portal, you should delete the underlying Cloud Service too. Cloud Service – is the container holding your VM and would become visible when you delete the VM (will detail this in an upcoming post). Deleting your Cloud Service would allow you to reuse your DNS name (as a best practice ensure that your VM and DNS names are prefixed with unique identifiers). I have seen few developers struggle and provide different names to their VMs every time they delete it, failing to realize that they need to delete the Cloud Service too. Below Figure shows how you can create a new VM by selecting an existing disk (New -> Compute -> Virtual Machine -> From Gallery)

Azure VM Disks

c. Resize VM: An alternative to deleting your VM is to change the size of your VM. This is less intrusive but might not be feasible in all the scenarios.

d. Unlock VHD / Disk: At times, after you delete your VM, the portal might show that Disk is still attached to the deleted VM. This would result into a quandary because you can’t reuse the Disk to create a new VM until it’s detached. The easier option to resolve this is to delete the disk object. Deleting the Disk object would still retain the underlying blob. After deletion you can re-create the disk object and use it to spawn a new VM. If you can’t delete the disk then your only option is to break the blob lease manually via PowerShell or using Storage Client Library (refer to this MSDN forum link for further information).

Please do leave a comment, if you have used other approaches.

Controlling Windows Azure VM lifetimes

Most of the Cloud computing resources are billable on an hourly basis and it’s important that you release these resources when you no longer need them. This typically applies to Windows Azure Virtual machines not running 24×7, example – there could be business workloads which requires an application to be available only twelve hours on week days. To limit running costs most users stop their VMs, just to realize that Windows Azure bills for VMs that are in a stopped state. So, your only option to control costs is to delete the VMs. But wouldn’t deleting VM cause any issues?

The answer is both No and Yes. When you delete a VM you are just deleting VM instance but the underlying OS and data disks are still intact (in fact, you still keeping paying for their storage which luckily is quite negligible). Hence, you can easily resurrect your VM without much harm. It’s important to note that when you delete the VM, you still retain the underlying Cloud Service container and its associated Site URL. The issue you might face though when you delete and re-create the VM, is the public IP address change. I work in an organization with strict IT security rules and locked down access. Static IP was necessary for me to raise an outbound RDP access request with my IT team. But with IP changing everyday it was definitely turning into a challenge. In end the solution I adopted was to create an extra small VM running 24×7 and bounce from there to other VMs.

To delete a VM you can use PowerShell cmdlets. PowerShell cmdlets allow you to export your VM configuration, delete the VM and then re-create VM using exported configuration.

Export-AzureVM -ServiceName ” -Name ” -Path ‘c:\vmconf.xml’

Remove-AzureVM -ServiceName ” -Name ”

Import-AzureVM -Path ‘c:\vmconf.xml’ | New-AzureVM -ServiceName ” -VNetName ‘’ -DnsSettings ‘’

Export configuration as shown above is stored in a XML file. It contains various properties of a VM including Endpoints, disks, VM size, subnet, etc. Below snapshot from Azure Portal shows an empty Cloud Service Container post deletion of the VM. Currently there is no cost associated with an empty cloud service. It’s important to note that when you retain Cloud Service you also retain the underlying DNS URL.

Passing Parameters to Hadoop Streaming

This weekend I would be presenting @ BDOTNET UG meet on topic – “Big Talk: Hadoop on Azure”. In case you are around and plan to attend here’s the facebook event link. In this talk I would help you get started with Hadoop and show how you can leverage it with Windows Azure. With that, let’s focus on the subject of this blog post.

Hadoop Streaming allows you to write and run MapReduce jobs in language of your choice. For Azure and Microsoft world this would be mostly C#. You can create your programs / executable in C#, read input from Console and write output to Console. Mapper task would feed input lines to your executable via console (standard input) and also collect output via console (standard output). It converts output into key value pairs. Reducer task on other hand converts key value pairs into input lines, feeds it to your executable (via console), and collects the output (via console) converting it back to key value pairs. For scenarios where you need only mapper you can emit reducer and set ‘numReduceTasks’ to zero as shown below:

call hadoop.cmd jar hadoop-streaming.jar -files "hdfs://10.186.36.85:9000/example/apps/Mapper.exe" -mapper "Mapper.exe" -input "asv://account/inputdata/account.data" -output "/example/data/StreamingOutput/mywc" -numReduceTasks=0

IP address in above case is that of Namenode (you can get by executing following command from Javascript console – #cat file:///apps/dist/conf/core-site.xml).

Now at times, your mapper program would need additional parameters to carry out its operations e.g. say you want mapper to filter data on few attributes. So, how do can we pass these attributes to Mapper executable? Simple – pass them as command line parameters and in your program read them from args.

call hadoop.cmd jar hadoop-streaming.jar -files "hdfs://10.186.36.85:9000/example/apps/Mapper.exe" -mapper "Mapper.exe param1 param2 param3" -input "asv://account/inputdata/ account.data" -output "/example/data/StreamingOutput/mywc" -numReduceTasks=0

static void Main(string[] args)
{
string line;

string parameterOne = args[0];
string parameterTwo = args[1];
string parameterThree = args[2];

Hope this helps!

Presenting at Great Indian Developer Summit 2012

Readers, I have been blogging quite less lately but really working on bouncing back soon. Meanwhile, I am speaking at the Great Indian Developer Summit (GIDS) 2012 on Windows Azure Access Control Service Usage Patterns. My session is on 17th April, 10 a.m. IST. Infact, this is my 4th presentation at GIDS conference, having presented at last three. If you are attending the summit, do drop by my session; I am committed to provide maximum ROI on your time investment. Below is the session snapshot

“Access Control Service (ACS) is perhaps the most powerful but least understood aspect of Windows Azure. While developers / architects understand it’s value proposition they are often left confused with surrounding acronyms and buzzwords like Active / Passive federation, SWT, SAML, ADFS, WIF, WS-Trust, WS-Federation, OAuth, OAuth WRAP, etc. This session distills the facts along with the underlying business motivation helping you with your moment on ACS. Having built the initial base session advances to focus on typical usage patterns of Access Control Service within enterprises. These common recurring implementation themes would further simply the mapping of ACS to your LOB applications. Attend this session to walk out with real implementation knowledge on ACS.”

Here’s the presenter billboard (speciality of GIDS 🙂 )

Looking forward to meet you in person…

Live!!! PluralSight Course – “Windows Azure Diagnostics”

Friends, My new course for PluralSight – “Windows Azure Diagnostics” just got published. It has been quite a gap since I last did the flyweight pattern module for PluralSight. Windows Azure Diagnostics (WAD) to me is a very important aspect of Windows Azure. I can confidently say hadn’t it been for WAD, I wouldn’t have succeeded developing a single application on Windows Azure. On top of that most the articles and materials I came across on this topic talked about only imperative code based setup. I confess WAD was a distant topic for me, and for quite some time I dreaded it, not quite understanding what it stands for and how to work with it. Finally, having got my Ahaa moment , I thought of sharing this knowledge with you all. Among others, course talks about WAD integration with Enterprise Library Logging Application Block and Log4Net , including configuring the logging verbosity at runtime. Course also covers DB Trace Listener, WAD PowerShell Cmdlets, On-Demand Transfer and IntelliTrace.

Do drop me a line (niraj@indiamvps.net), if you happen to look at the course. What you learnt, how useful it was to you, what could have been better (I personally felt there is room for voice quality and demo alignments)?. Though, I am still happy with overall content and course 🙂  . If you would like to see any other courses around Windows Azure or other Microsoft technologies, please let me know that as well.Until next time, happy learning…

Demystifying Access Control Service

Having presented quite a few sessions on Claims Based Idenitity and Access Control Service I still see quite a few participants confused on how to get started. While most of them are able to understand the underlying business motivation, they are list lost admist all the new terms like SAML, SWT, OAuth, WRAP, WIF, ACS, ADFS, Claims, Active / Passive Federation, etc. Let’s get them in turn.

At the heart of these offerings is the below simple block diagram which drives the key concept of –  Relying on a trusted External Entity (Identity Provider) for – Authenticating users and Providing user attributes (claims) to saves our services / applications from Identity nightmares. So, for your Service or your Application, you establish trust with an identity provider by sharing a X509 certificate or a shared secret. Clients or users of your service / application no longer connect with you for authentication, rather they authenticate with an identity provider to get back a claims token. That token is then presented to your service / application. As your service / application has a trust established with that IP they can validate the trust of the incoming token and then use the claims bundled in it to authorize the level of access for client / user.

Above flow becomes little complex when RP needs to trust multiple identity providers (for instance you are offering a multi-tenant service for individuals and coporates with each of them having a different identity provider). Not only RP would have to establish trust with all these providers but the Identity Providers too would have to register this RP to issue tokens on request. This is where a mediator called federation provider comes into picture. RP get registered only with federation provider and federation provider in turn is registered and trusted by various identity providers. Hence it simplifies many-to-many relationship into easily manageable one-to-one. Access control service (ACS) is a federation provider hosted on Windows Azure.

You will also find two federation terms used quite frequently – Passive & Active federation. Passive federation is assoicated with web applications (rather web browsers) where authentication happens via a set of redirects. Active federation is assoicated with Web Services and clients that explicity get authenticated.

Once a client authenticates with Identity Provider he get a token back. There are two token formats supported by Access Control Service – SAML and SWT. SAML exchange happens over WS-* protocols while SWT tokens are usually transferred over OAuth WRAP / OAuth 2.0 protocols (details here). You are most likely to use SWT tokens for RESTful services hosted in Azure. You can find more details about these token formats explained here

ACS would accept either of the token formats as input and would also return either of them as outputs. For instance there could be scenarios where you get a SAML token from ADFS (corporate identity provider), you use that to get authenticated with ACS and ACS returns you back a SWT token which in turn is used to access a protected REST Service (offered by a business partner).

Let’s start with SWT tokens. To see SWT tokens and OAuth WRAP in action protecting a WCF REST Service (active federation) I would recommend you have  a look at the ACS samples. Establishing trust between RP & ACS is quite simple – you exchange a shared secret. When client authenticates with ACS he gets back a SWT token which integrated HMAC256 hash. Relying part checks the authenticity of hash and claims inside the token on incoming tokens  to allow access to requesting client. You should see that retrieving SWT tokens using WRAP protocol from ACS and sending it to a RP is quite simple, infact you would hardly need anything beyond HTTP client library.

//Requesting a SWT token with username / password from Access Control Service

var client = new WebClient();
client.BaseAddress = string.Format(“https://{0}.{1}”, serviceNamespace, acsHostName);

var values = new NameValueCollection();
values.Add(“wrap_name”, unamepass.Username); /*Service Identity to be specified in Access Control Service*/
values.Add(“wrap_password”, unamepass.Password);
values.Add(“wrap_scope”, ConfigurationManager.AppSettings[“relyingpartyname”]);

byte[] responseBytes = client.UploadValues(“WRAPv0.9/”, “POST”, values); /*SWT token is received in raw format*/

Handling WS-*/SAML on the other hand is little more complex. Luckily for us Microsoft provides nice Visual Studio integreated tooling (FedUtil – Add STS Reference) for working with WS-*/SAML tokens along with WIF (Windows Identity Foundation) SDK. This tooling can be leveraged by both services and applications. In fact all you need to establish trust with identity providers is federation metadata and tooling generates that for you in form a file called FederationMedata.xml. WIF SDK, in addition, provides web controls for federated sign in and sign in status (Look at this article in case you want to extend WIF for handling SWT tokens).

Tooling also sets you up by plugging necessary modules in your RP’s web.config namely – WSFederationAuthenticationModule and SessionAuthenticationModule. Former helps you validate authenticity of incoming token while later establishes a session between client and relying party (FedAuth Cookie) so that token validation doesn’t become an overhead for every operation invoked. You can also add an additional module to this called ClaimsAuthorizationModule which let’s you invoke your custom ClaimsAuthorizationManger class as shown below.

<microsoft.identityModel>    
<service>
<claimsAuthorizationManager type=”WebApplication4.CustomAuthorizationManager”/>
</service>
</microsoft.identityModel>

public class CustomAuthorizationManager : ClaimsAuthorizationManager    
{        
public override bool CheckAccess(AuthorizationContext context)        
{      
//…      
return base.CheckAccess(context);        
}    
}

HTH!!!