Creating an Exchange 2010 DAG Fails with “Access Denied–Server Side Error”


Just a quick blurb today.  Was working on setting up a cross site DAG with a customer today.  Kept getting an issue where when we tried to add two existing mailbox servers to the DAG, both would fail with an “access denied” error:


A server-side database availability group administrative operation failed. Error: The operation failed. CreateCluster errors may result from incorrectly configured static addresses. Error: The Cluster service couldn’t access the Microsoft Failover Cluster Virtual Miniport network adapter. Verify that other network adapters are working and check Device Manager for errors associated with this adapter. If the configuration for this adapter has changed, you may have to reinstall the Failover Clustering feature on this computer. Learn more at Error: 1062. —> Microsoft.Exchange.Cluster.Replay.DagTaskNetFtProblemException:

Turns out, in the past, this customer had wiped out their local administrators on the Exchange servers with a group policy run amuck!  They added things like Domain Admins in, but didn’t add in “Exchange Trusted Subsystem”.  We added that in, DAG creation worked like a charm. 

Posted in Uncategorized | Leave a comment

Exchange 2010 Cluster Loses Quorum and Unexpectedly Fails Over Databases


I was working on a client’s Exchange 2010 setup, working on a specific issue.  The client had a multi site stretched DAG across two datacenters.  One datacenter was the “primary”, and then they had a second disaster recovery datacenter.  In the primary datacenter they had two mailbox servers, MBX01 and MBX02 each with a copy of the database, and the secondary had a single mailbox server, DRMBX03 with a copy of the database.

The client was experiencing an issue where databases would suddenly failover within the primary site from MBX01 to MBX02, and report that the cluster lost quorum. 

I took a look at the cluster log, which can be generated by running the command:

Cluster log /g /copy:LogFolder /span:120

The span entry specifies the amount back in minutes that the log is generated for.  Just a hint, if you run this from the root of the C: drive, it will copy the logs to the C:\LogFolder location.

Within that folder you’ll find a separate log for each of the servers in the DAG, in our case MBX01, MBX02 and DRMBX01:


When I opened the logs, I began to see 1226 and 1236 errors in the log:




These errors are specifically handled by the following hotfix for 2008 R2 Failover Clusters (which is what Exchange 2010 runs on top of):

These were recently released and talked about by the Exchange Team, along with these two other hotfixes:

After applying each of these hotfixes to EACH and EVERY single DAG node and rebooting, the issue was resolved.  These hotfixes are recommended to ANYONE running Exchange 2010 on 2008 R2, regardless if your seeing the issues or not.

Posted in Uncategorized | 1 Comment

How to Import Users via CSV in Exchange 2010

Create an csv file with the necessary information across the top row of the file as such:


The top row is going to coordinate with the S_.value that you are going to use in the following Exchange Shell command:

Import-CSV “C:\Mailboxes.csv” | foreach {new-mailbox –Name $ –Alias $_.alias –UserPrincipalName $_.userprincipalname –Database $_.Database –OrganizationalUnit $_.organizationalunit –password (ConvertTo-SecureString $_.password –AsPlainText –force)}


And you should see the mailbox’s created below:


That’s it.  You can see how the values map with their respective column names.  You can add as many users as you want, and change it so they go to different database’s.

You can even create an automated job to export from your production servers, and them import them to your DEV Exchange Servers for testing. 

Posted in exchange 2007, Exchange 2010 | Tagged , , | 8 Comments

How to View Disconnected Mailbox’s and Purge Disconnected Mailboxes from Exchange 2010


To view disconnected mailbox’s, essentially mailboxes that have been deleted from their user accounts, you need to first ensure that Exchange has gone through and cleaned the database.  This is done to ensure that it marks that mailbox as deleted.  If your database is MDB36, run the following command:

Clean-MailboxDatabase MDB36


Exchange gives no result from the command.  But now you can view Disconnected mailboxes through the “Disconnected Mailbox” view in the EMC:



You can also view it in the shell by running the following command:

Get-MailboxStatistics –Database MDB36 | where {$_.disconnectdate –ne $null}


And you will receive the following output:


By default, Exchange 2010 keeps disconnected mailbox’s in the DB for 14 days.  But say you want to remove this mailbox now and return it’s white space to use in the DB.  You need to remove the mailbox from the shell. 

You can do this by getting the GUID for the mailbox by running the command:

Get-MailboxStatistics –Database MDB36 | where {$_.disconnectdate –ne $null} | select displayname,MailboxGUID


And you will receive the following output:


Now run the following command to remove the mailbox:

Remove-Mailbox –Database MDB50 –StoreMailboxIdentity 7b40b106-5941-4de0-9fce-27ede21c474e


You’ll receive a confirmation prompt, just accept it, and your all set:



Posted in Exchange 2010, High Availability | Tagged , | Leave a comment

User’s Cannot Log into Exchange 2007 or Exchange 2010 OWA with UPN Suffix


Lets say you have the following environment:


ExchangeResource.corp hosts all the Microsoft Exchange 2010 servers, and linked mailbox accounts.  The actual user accounts are stored in the Tailspin.corp and Mantech.corp forests.  The Tailspin.corp and Mantech.corp forests have a one way forest trust with ExchangeResource.corp so that users in the Tailspin.corp and Mantech.corp forests can access their linked mailboxes in the ExchangeResource.corp domain.

Now to make things easy on the users, you set the OWA directory to use UPN suffix names instead of Domain\user:


This will allow users in Tailspin to login using username@tailspin.corp and users in Mantech to use username@mantech.corp.

Everything works fine, but then you add a UPN suffix to each individual forest that makes the UPN suffix match the users email address. Below is an example shown with the user Tom Jones in the Tailspin forest:


Now users in Tailspin login using and users at Mantech login using

A user goes to login with the new UPN and is greeted with an error message that they could not login:


But using the old UPN still works fine, so what’s going on?

Well, if we check the event logs of the DC in the ExchangeResource.corp domain we find EventID 6034 for LsaSrv in the security event log:


The DC is telling us that it does not know how to route the suffix.  It notes that it has been added to the forest tailspin.corp, as it learns it through the forest trust, but that the name suffix is not enabled.  It does very nicely tell us how to fix this.  Go to Active Directory Domains and Trusts->Right click on ExchangeResource.corp->Properties


Go to the Trusts tab.  Here you will see all the forests that you have trusts with.  Highlight the tailspin.corp forest and click on properties:


Navigate to the Name Suffix Routing tab:


Here we can see the new suffix has been added, it even has a status of “New”, but the Routing is disabled.  Highlight the suffix and then click Enable:


If you do not see the new suffix you created listed here, simply click the Refresh button and it should appear.

After hitting apply both names should be enabled:


Now if a user try’s to login, they should be all set!


Keep in mind you will need to do this each time you add a new UPN Suffix to one of the domains that are being trusted by ExchangeResource.corp.


Posted in Client Access, Exchange 2010, Hosting, Resource Forest | Tagged , , , | 1 Comment

Exchange 2010 Database’s Fail to Replicate or Seed


Recently had a colleague that ran into an issue with an Exchange 2010 migration.  He could fail over the mailbox databases with no issue to DR, but that’s where the trouble started.

The production database would start to report that there was a high copy queue length that would increase as more activity occurred on the DB.  The production database pure and simple was not receiving the transaction logs from the newly activated database in DR. 

The setup was simple, two nodes, one in production, one in DR with a FSW.  The nodes were all-in-one 2010 boxes, with one NIC for MAPI and one NIC for replication.

My colleague also informed me that he had some trouble initially seeding the database.  All roads pointed to an issue with replication.  We quickly checked his replication network settings and found the following setup:


His DAG network had both replication networks for the separate sites under one object.  Once we moved them to their own separate networks:


Everything went back to normal!

Till the next time!

Posted in Exchange 2010, High Availability | Tagged , , | Leave a comment

Delayed Email or Message Rescans with BES 5 and Exchange 2010


I recently ran into an issue where users were reporting that messages to their Blackberry handhelds were being delivered in clumps, and were also extremely delayed.  These are symptoms of a Blackberry Server rescan.  We were running BES 5.0.2 and Exchange 2010 SP1.  We upgraded the BES to 5.0.2 MR4, but the issue remained.

This essentially means that the BES server cannot keep up with email delivery, and thus cannot deliver the emails as they arrive.  It is forced to queue them, and deliver them in chunks.

I had already configured the Exchange 2010 throttling policy as per Blackberry’s documentation:

BES 5 and Exchange 2010 Throttling Policy

So throttling wasn’t the issue.  The issue ended up being that the BES server was only creating one mailbox agent in the BES server.  This has to do with the DAG model in Exchange 2010, and the way around it is to create static mailbox agents for EVERY user, or force the BES server to create multiple mailbox agents.  We can do this with two registry keys.  The first one:

HKLM\Software\Research In Motion\Blackberry Enterprise Server\Dispatcher

If it is not there, create a DWORD named MaxUsersPerAgent and set the decimal value to the maximum number of users per agent, in my case 40 users:


The second registry edit is:

HKLM\Software\Research In Motion\Blackberry Enterprise Server\Agents

If it is not there, create a new DWORD with the name MaxMailboxesPerSession and set the decimal value to the value that you want the maximum number of mailboxes to be piped through a single MAPI session.  This is separate from the agent above.  For instance, I set mine to 35, which means at the 36th mailbox, the BES will create a new MAPI session.


After you make those changes, restart your BES server.  After making the above changes, your performance should increase, and you should see extra instances of the BlackberryAgent.exe process:


Just an FYI, SQL Express and MSDE are limited to five agents on the server.  For more agents, you will need to be using SQL Standard or higher, and edit the key:

HKLM\Software\Research In Motion\Blackberry Enterprise Server\Agents\NumAgents and change its decimal value to a number higher than 5:


Again, this value will be ignored higher than a value of 5 on SQL Express and MSDE.  If you have 1000 users, and you set the MaxUsersPerAgent value to be 40 as above, your agent breakdown will be as follows:

BlackberryAgent (1) = 40 Users

BlackberryAgent (2) = 40 Users

BlackberryAgent (3) = 40 Users

BlackberryAgent (4) = 40 Users

BlackberryAgent (5) = 840 Users

So be careful to set the MaxUsersPerAgent value appropriately for your environment if your limited to the number of agents.

As for the BES server itself, we saw slightly higher memory utilization due to the extra agents (around 400 MB), but much lower CPU usage.

Hope you guys find this helpful!

Posted in Blackberry, Exchange 2010 | Tagged , | 4 Comments

Users Receive a Login Prompt After a Database Failover in Exchange 2010


When performing a database *over in Exchange 2010, especially a planned one, it is suppose to be as seamless as possible to the end user.  An Outlook user for example, will receive a small pop up informing them that the connection to Exchange has been lost, and their Outlook may hang for a couple of seconds before reconnecting and resuming normal behavior.

I recently ran into an issue where this was not the case.  Users would call the helpdesk as soon as a database failover was initiated with complaints that their outlook was prompting them for a login:

Jan. 2601 09.54

Well, needless to say this is not expected behavior.  After a little troubleshooting that involved a packet capture, it seemed that the Outlook clients were making an HTTPS call to the CAS servers at that moment.  Turns out it was an attempt to connect to them over Outlook Anywhere, and this was the reason of the login prompt.  When I checked the Outlook client, they in fact had the Outlook Anywhere Settings enabled.  This was due to Autodiscovery.  To check the settings in Outlook 2007 navigate to Tools->Account Settings->Change->More Settings->Connection  If yours is enabled, it will look like the following:

Jan. 2603 09.55

Under Exchange Proxy Settings, you’ll find the settings enabled:

Jan. 2604 09.55

Since these were internal clients, and had no need to use Outlook Anywhere, simple unselect the Connect to Microsoft Exchange using HTTP:

Jan. 2606 09.58

This will disable the Outlook Client from connecting.  The only issue is, if you use automatic profile generation through Group Policy, this leverages autodiscovery, so it will continue to put the setting back.  You can do one of two things.  The first is to delete the Outlook Anywhere provider using the Remove-OutlookProvider command, which is NOT recommended.  This will stop Autodiscovery from publishing Outlook Anywhere GLOBALLY. 

The second is to use Group Policy.  Create a blank GPO named something like Disable Outlook Anywhere Settings.  Download the Outlook Anywhere ADM template from here, and import it into the template under User Settings.  You’ll now have the Outlook Anywhere (RPC/HTTP) options available in Group Policy:

Jan. 2608 11.23

The only value you need to edit here is the RPC/HTTP Connection Flags setting:

Jan. 2609 11.24

Edit the setting, set it to Enabled and No Flags

Jan. 2610 11.25

This will disable the Connect to Microsoft Exchange Using HTTP in the outlook clients after its been applied, notice how its greyed out:

Jan. 2611 11.25

Once this GPO has applied to all your users, you should now be able to failover databases without the users receiving a log in prompt. 

Posted in Client Access, Exchange 2010, High Availability, Outlook Anywhere | Tagged , | 4 Comments

Issue Adding RDM LUN to Exchange 2010 Server Using VMWare vSphere and NetApp

**UPDATED on January 6, 2012**

Thanks to both user comments and NetApp themselves, we have determined that there is an easier way to add disks to members of a DAG without removing them from the DAG.  You can simply stop the Windows Clustering service (run the command “net stop ClusSvc” from the command line).  Obviously you should move any active mailbox databases on this DAG member to another DAG member before doing this, as stopping the clustering service will cause the databases to dismount and move on their own.  From there, you should be able to add the disk’s, start the clustering service, and your DAG member will automatically return to a normal operating mode, participating in the DAG.

Thanks to all who sent this in!!

**Original Article**

I recently ran into an issue where I was unable to add an RMD LUN to a Windows Guest running on VMWare vSphere.  Here is my setup.

I had a Windows 2008 R2 guest that was running Exchange Server 2010 SP1.  The guest was a Mailbox server that was a member of a Database Availability Group.  I was attaching the LUN’s to iSCSI RDM’s that were based on a NetApp FAS 3140 running ONTAP 7.3.2.  The guest was running version 6.3 of Snapdrive.

The guest had 12 iSCSI RDM’s working properly for month’s, but the issue arose when I tried to add more.  I would be able to select the volume, create the LUN, size, the mounting location of the LUN.  The issue was when in Snapdrive I was presented with where to store the RDM file for the VM.  See the screen below:


The problem was the console starting freezing up, and generally not responding.


After several minutes, I eventually received an error stating there was an “error in fetching number of vmfs datastores”


I tried all the basic’s, re-installing Snapdrive, upgrading to Snapdrive 6.3 PP1, rebooting the host, stopping and starting the service.

Turns out there is a bug in Snapdrive that causes the error above, when the Guest is a member of a Windows Cluster.  Since all DAG members utilize Windows Clustering, this applied to me.   The resolution was easy.

I moved all the databases off of the server in question.  Then, in Exchange Management Tools, I went to Organization Config->Mailbox and selected the Database Availability Tab.  Right click your dag and select Manage Database Availability Group Membership:


Right click the server in question, select Delete and then the manage button.  This will remove the server from the DAG.

[This will not cause any issue with the existing databases as we’ll see below]

Now, go back into Snapdrive and add your LUN’s, all should be working now.

After your done, add the server back to the Database Availability group, almost the same way you removed it, this time select Add, and then select the previously removed server and add it back.

Next, for each MailboxDatabase that the server has copies of, run this command in the Exchange Management Shell:

Add-MailboxDatabaseCopy –Identity MB01 –MailboxServer NYDAGNODE1

Or in the EMC, go to Organizational Configuration->Mailbox and right click each Mailbox Database and select Add Database Copy.  Then select your server.

Since the server still has copies of the Mailbox Databases, it will start to resynchronize with the DAG, and bring itself up to date.  That way you won’t need to reseed your entire DB which can take some time.

Hope someone finds this useful!

Posted in Uncategorized | 31 Comments

Migrating Exchange 2007 ActiveSync to Exchange 2010. And why your Android may work but your Apple iphone / ipad may not.


When doing a migration from Exchange 2007 to Exchange 2010, one of the biggest item’s you need to watch out for is the migration of the ActiveSync environment, and be aware of how it can affect your end users.  You should also be aware of potential issues depending on the TYPE of active sync device you are using, as some will work, and other’s will have issues.

First we’ll start with the migration.  Our current Exchange 2007 ActiveSync environment is as follows:


Here, we have one internet facing site, the NY site.  There is a DNS record for the CAS server in NY, that points to the IP of the CAS server on the internet.  We have set the –externalurl to and the –internalurl to  Note that newyork2007 is the NETBIOS name of the CAS server in NY.  We set both of these values with the following command:

Set-ActiveSyncVirtualDirectory –Identity “newyork2007\Microsoft-Server-ActiveSync” –InternalURL –ExternalURL

In London, we have set the InternalURL attribute to the local name of the server, but leave the ExternalURL attribute blank.  We do not populate the ExternalURL attribute because London is not accessible directly from the internet. 

Setting the –internalurl attribute updates the SCP in active directory, so that any system that query’s AD itself, will be able to return the internal URL the user should access.  For instance, in our above scenario, LON07 the user configures his active sync device from the internet.  He put’s in as the server address, which is the external DNS name of the NY CAS server.  The NY CAS server, as part of the Active Sync process, query’s Active Directory for the home mailbox of LON07, and then determines which site LON07 is in.  Since LON07 is not in the same site as the NY CAS, the NY CAS then returns the value for ExternalURL.  If we had entered a value here, such as, the users device would be redirected to it (as long as the device supported auto discovery, more on that later), and the user would connect, as long as that was configured properly.  In our case, since there isn’t, the NY CAS uses the InternalURL entry to determine what address the NY CAS should use to proxy on behalf of the LON07 user.  Essentially the NY CAS connects to the London CAS, and returns the Active Sync info to the users device, all seamless to the LON07 user.

Now, we start to introduce Exchange 2010 to the equation.  Microsoft’s high level recommendation is to create a new namespace, called and have this entry point to the 07 CAS, and slide the 2010 CAS into the existing namespace.   See the below diagram:


So we would need to reconfigure the –ExternalURL and –InternalURL attributes of the NY 07 CAS servers, as well as the NY 2010 CAS servers.  They can all be done by changing the values of the command listed earlier in this article.  The logic here is the same as 07-07 proxy.  If the NYC07 user enters in into his/hers server address on their ActiveSync device, the 2010 CAS server will query AD, and determine that he is a 2010 user, but in the same AD site.  It will then query to see if an ExternalURL setting is populated, in which case ours is.  That users device, if it supports activesync, will automatically be redirected to and their profile loaded, all seamless to the end user.

If LON07 enters in the NY 2010 CAS server query’s Active Directory, finds his mailbox is in another site, and checks to see if there is an ExternalURL entry.  Since their isn’t, like before, it proxies the connection to the London 07 CAS server, all seamless to the end user. 

Now, this is all great, but what happens if your device does not support auto discovery?  Some active sync devices don’t work properly with auto discovery, and in that case, Microsoft recommends that you manually change their profile to point to  Maybe not even that, but for security purposes you don’t allow external devices to use auto discover to determine the settings.  In this case, you again have to manually point those devices to  If you have any significant number of users, this can be insanely time consuming. 

Let me show you an example.  I had configured everything as it was in the above diagram.  I was configuring his active sync on an Apple iPad, a device that supports activesync.  Problem was, his account wasn’t working.  The following log file was taken from the NY 2010 CAS server for the NYC07 user, they are located at c:\inetpub\logs\LogFiles\W3SVC1:


Sep. 2801 15.56 Here, we can see that the NY 2010 CAS server is telling the device that it has the wrong URL, and is redirecting it to  This is because the device has advertised that it can do auto discover.  In our example, since auto discover is disabled, or because the device doesn’t handle auto discover properly, the user was getting a connection to server error on the iPad.

Now, with NO changes, let’s try configuring the same user, but not using the iPAD, but using Touchdown for the Android.  Now, all work’s without issue, here is the log files:


Sep. 2802 15.56

In this case, the configuration worked without issue.  Notice how it also says  This is because since Touchdown did not advertise to Exchange that it could do auto discovery, Exchange knew it would have to proxy the connection back to the NY 07 CAS server to allow this to complete successfully.

The funny thing is, if we were to configure LON07 on the iPAD it would work fine.  Why?  Because since the London 07 CAS server does NOT have a value for ExternalURL, Exchange knows it HAS to proxy to London 07 CAS for all London users.

So, we want the same behavior for the NY users on 07.  To do so, we simply need to clear the ExternalURLvalue on the NY 07 CAS server.  We would do so with the following command:

Set-ActiveSyncVirtualDirectory –Identity “newyork2007\Microsoft-Server-ActiveSync” –InternalURL –ExternalURL $null

This would wipe out the ExternalURL value.  The downside to this, is that auto discover for this URL would not be included, so if you used Outlook Anywhere, or other devices to connect using Auto discover, it would cause issues.  If you didn’t though, for instance you disable auto discover, this fixes all your issues.  Now when you try to connect NYC07’s mailbox to the iPad, since there is no ExternalURL entry for the NY 07 CAS server, the NY 2010 CAS server is forced to proxy:


Sep. 2803 15.56

Now, all existing 07 users will continue to have access to their mailbox’s via active sync and will not need any changes when their mailbox’s are moved to 2010.

Posted in ActiveSync, Client Access, exchange 2007, Exchange 2010, Security | Tagged , | 5 Comments