How to Manage a Datacenter Failure or Disaster Recovery Scenario in Exchange 2010 – Part 3

In Part 3 of this series, we are going to discuss how to failback to your primary site, after whatever condition that caused it to go offline in the first place occurred.

So when we last left off, we have activated all of our databases in our disaster recovery site. I should you how had to stop the mailbox servers in the primary active directory site, which then allowed us to activate our disaster recovery site.

So now, let’s say the flood that occurred in the New York site has been fixed, all the water’s been removed, and thankfully, it did not damage our equipment. We are ready to move everything back to NYC. We begin by powering on the servers in NY. Since our DAG is in DAC mode, the NY servers do not try to mount their copies of the databases. Instead, the begin copying over any log files so as to bring their copies of the databases up to date with the DR site. Wait until all the database’s are reporting a status of healthy:

Now, remember, we told the DAG that the NY servers were down with the Stop-DatabaseAvailabilityGroup command from Part 2:

Notice how NYMB01 and NYMB02 are both listed as Stopped Mailbox Servers. You will get this error in the console if you try to activate the database on one of those two servers:

What we have to do is start these servers in the DAG, making them viable copies for activation again. We do this with the Start-DatabaseAvailabilityGroup command. Again, we could use with the –MailboxServer command and specify each server, separating them with a comma, or we can specify the whole site with the –ActiveDirectorySite option as below:

Start-DatabaseAvailabilityGroup DAG1 –ActiveDirectorySite NYC

Now if we run the Get-DatabaseAvailabilityGroup command, all servers are listed as started

Now, Microsoft recommends dismounting the databases that are going to be moved back to the primary datacenter site. This will mean that you will need to get a maintenance window to perform this action, as the databases will be offline. Keep in mind that you do not NEED to dismount the databases; it is just recommended by Microsoft.

Now, we can activate the Database. You can do this either by using the shell with the Move-ActiveMailboxDatabase MDB01 –ActivateOnServer NYMB02 command:

Or through the EMC with the Activate a Database Copy Wizard:

Now, the database should be activated. One issue that I have seen pop up is that the Catalog Index is corrupted. This is the index for the Full Text Index Search. If this is the case, you may have to reseed the Catalog Index with the following command:

Update-MailboxDatabaseCopy MDB02NYMB02 –SourceServer DRMB01 –CatalogOnly

This command will reseed the content index from the server DRMB01. You should now be able to activate the database copy.

Continue by moving all active database copies to their respective servers. If you dismounted the databases as stated above, now mount these databases.

Now, our clients are still connected, but they are still connecting through DRHT01.nygiants.com because it is set as the RpcClientAccessServer:

We simply need to run the command Get-MailboxDatabase | Set-MailboxDatabase –RpcClientAccessServer NYHT01.nygiants.com to change this setting over:

If we check the properties of the databases, we see that NYHT01.nygiants.com is again set as the Rpc Client Access server:

Now, our Outlook clients will be connecting through NYHT01 for their access:

Well, that’s it!

In this three part series, I should you how to activate the backup datacenter, in the event that the primary datacenter became unavailable. We discussed the theory behind the process, mainly Datacenter Activation Coordinator, the actual work needed to do it, as well as how to failback when the primary site came back online.

If you have any questions, feel free to email me at ponzekap2 at gmail dot com.

How to Manage a Datacenter Failure or Disaster Recovery Scenario in Exchange 2010 – Part 2

 

In the first article of this series, we went over some of the premises of Exchange 2010, Database Availability Groups or DAG’s, and Database Activation Coordinator.  We discussed our test environment, as well as how, in theory, an Exchange 2010 DAG handles the failure of a datacenter.

In this article, I’ll show you how to actual activate your disaster recovery site, should your primary site go down.

One of the first thing’s we need to take into consideration is how the clients, most importantly Outlook, connect to the DAG.  We spoke in the first article how there is a new service in Exchange 2010 Client Access Servers, called the Microsoft Exchange RPC Client Access Service.  Outlook clients now connect to the Client Access Server, and the Client Access Server connects to the Mailbox Server.  This means Outlook clients don’t connect directly to the Mailbox Server’s anymore.  This becomes significant in a disaster recovery situation.  

Lets look at the output of the command:

Get-MailboxDatabase | Select Name,*rpc*

 

The RpcClientAccessServer value on a particular Mailbox Database indicates that connections to this database, are passed through to the Client Access Server listed.  So, for all these databases above, all Outlook connections have to go through NYHT01.nygiants.com.  (If you have more than one Client Access Server per site, which you should for redundancy and load balancing, you can create a cluster using either Windows Network Load Balancing, or a hardware load balancer, and change this value to point to the cluster host name).  Lets look at our Outlook Client, and we notice that all connections are being passed to NYHT01.nygiants.com:

Alright, now, we are ready to start failing over servers!  My test user, Paul Ponzeka, is located on a mailbox database named MDB02 which is running on server NYMB02:

So, to simulate a datacenter failure, we’ll just pull the power on ALL NY servers:

NYDC01

NYHT01

NYMB01

NYMB02

NY-XP1 (client machine)

So now, all of our NY machines are off, lets check the status of the Databases on our DR servers located in Boston:

Well, that’s not good huh?  Notice how the DB’s for the two copies in NY are listed as ServiceDown under copy status?  Also note that DRMB01, the Mailbox Server in Boston’s copy of the database is Disconnected and Healthy.  The reason the DB is not mounted, is because of the DAC mode enabled on the DAG, which we discussed in Part 1 of this series. DRMB01 dismounts ALL its Mailbox Databases because there are not a majority number of DAG members available, this it can’t make quorum. It dismounts to prevent a possible split brain scenario, but in this case, we REALLY need to get this activated.  How do we do this?  What we need to do is remove the NY server members as being active participating members of the DAG.  We do this through the Exchange Shell.

If we note the current status of the DAG with the command:

Get-DatabaseAvailabilityGroup | select Name,*server*

Notice that the Servers value lists DRMB01, NYMB01 and NYMB02 has servers in the DAG, and lists them again as StartedMailboxServers?  Well, we need to tell the DAG, that NYMB01 and NYMB02 are no longer “started” or operational.  We need to use the following command for that:

Stop-DatabaseAvailabilityGroup.

Our command can specify each server that’s down, one by one:

Stop-DatabaseAvailabilityGroup –Identity DAG1 –MailboxServer NYMB01,NYMB02

Or, since we lost the entire NY site, we can specify that the entire NY site:

Stop-DatabaseAvailabilityGroup –Identity DAG1 – ActiveDirectorySite NYC

Now, since our Mailbox Servers in NY are actually unreachable, we want to specify the –ConfigurationOnly option at the end of this command.  Otherwise, the command attempts to actually stop the mailbox services on every mailbox server in the NYC site, and causes the command to take an extremely long time to complete:

Now, if we re-run the command:

Get-DatabaseAvailabilityGroup | select Name,*server*

We notice that the NY mailbox servers, NYMB01 and NYMB02 are both now listed as Stopped Mailbox Servers.

Next, ensure that the clustering service is stopped on DRMB01:

Now, we want to tell the DR site, it should restart with the new settings (both the NY servers missing):

Restore-DatabaseAvailabilityGroup –Identity DAG1 –ActiveDirectorySite DR

You will see a progress bar, indicating its adjusting Quorum and the cluster for the new settings.  Don’t be alarmed if you see an error regarding the command not being able to contact the downed mailbox servers.

If we return to the Exchange Management Console, all our databases in the DAG have been mounted on DRMB01!

Great right?  But our clients are still having trouble connecting.  What’s the problem?

The reason is that NYHT01.nygiants.com is still listed as the RPC Client Access Server for this database:

We have two choices.  First, is to change the DNS record for NYHT01.nygiants.com to be a CNAME for DRHT01.nygiants.com.  Or the second, and faster method, is changing the RPC Client Access Server to be DRHT01.nygiants.com with the following command:

Set-MailboxDatabase MDB02 –RpcClientAccessServer DRHT01.nygiants.com

To do it for every DB that you failed over is simple:

Now, back to the Outlook client and WHOILA!

Now all your messaging services are back and running in your Disaster Recovery site, with limited downtime for your end users.

In this article we discussed how to fail over a datacenter to your backup or disaster recovery datacenter, should your primary go offline.

In the next and final part of this series, I’ll show you have to fail back to the primary datacenter site, which some admins think is even more terrifying than failing over!

Stay Tuned!

How to Manage a Datacenter Failure or Disaster Recovery Scenario in Exchange 2010 – Part 1

 

Exchange 2010 introduced several high availability and disaster recovery features, the one that receives the most publicity is the Database Availability Group (or DAG for short) feature.  In short a DAG allows replication of a mailbox, to other servers in the DAG, that can be activated automatically within 30 seconds, restoring user access to their mailbox’s.  For more information see my article series on DAG’s here.

The automatic failover is great for High Availability within a datacenter, or even across a datacenter.  For instance, consider the following diagram:

Here, the green copies are the Active Copy, they are the one’s users are actually accessing for their mailbox’s.  The yellow and the red are copies that can be activated, should the Active Copy go offline.  Consider the possibility that MDB01 on server NYMB01 goes offline, the copy on NYMB02 would be activated within 30 seconds automatically.  Next, the drive holding the database MDB01 on server NYMB02 fails, causing THIS copy to go online.  In this case, the copy of MDB01 on DRMB01 in Boston would be activated with 30 seconds, and users would be able to access their mailbox’s, across the WAN link to Boston!  This is all part of the design of the DAG, and is great from a High Availability standpoint. 

But, as we know, High Availability and Disaster Recovery are COMPLETELY separate.  High Availability means to provide your users with high uptime, or access to the application.  Disaster Recovery is the ability of the application to function when a catastrophic event happens, such as destruction of a datacenter or worse, the building holding the datacenter.  This last part, is what we will cover in these articles.  To do so, there is a feature of DAG’s that we need to talk about, and that is the Datacenter Activation Coordinator, or DAC.

DAC is a setting on a DAG that has three or more member mailbox servers, that are extended to multiple sites.  So, a higher level view of our Exchange environment is below:

NYMB01, NYMB02 and DRMB01 are all part of the same DAG, lets call it DAG1, and all these servers are located in the NYGIANTS.COM domain.

Now, our DAG fits the criteria for DAC mode, which is three or more member servers, spread across multiple Active Directory Sites.  So, now what is DAC mode?

DAC mode is, quite simply, a mechanism to prevent the possibility of a split brain in your exchange environment.  Consider the following scenario.  As per the first diagram, you have MDB01 and MDB02, and both active copies run in NY on NYMB01 and NYMB02 respectively.  NYHT01 is running the file share witness.  A file share witness is a server that only participates if the DAG has an even number of servers, it’s used to “break” any tie in voting regarding if a server is down or not.  The NY site is connected via WAN connection to Boston, where DRMB01 hosts replica’s of MDB01 and MDB02.  Say there is a cut to the WAN connection, and for whatever reason, NY and Boston can no longer communicate, but neither side is truly offline.  The Boston side, since it can no longer connect to the NY server’s, assumes they are down, and mounts the database copies it has of MDB01 and MDB02, and marks them as active.  Since NY is still operational, it STILL has its copies of MDB01 and MDB02 mounted and active.  This is a split brain scenario, both sites believe that they are the rightful owner of the database, and have thus mounted their respective DB’s.  This would cause a divergence in data.  For example, if outside user, sends an email to a user at nygiants.com, and its received in NY, it would get delivered to his mailbox in NY.  If another user sends the same user at nygiants.com an email, and it gets received by Boston, it would get delivered to that’s users mailbox in Boston.  Each mailbox is different, which is a huge problem, this is the issue with a split brain scenario, and is what DAC was built to protect against. 

DAC does this by preventing the DR servers from mounting their databases.  DAC requires that a majority set of the DAG members be available for the DAG to be able to make an operational decision, in this case the DR servers mounting their database.  A DAG that has the majority of its member servers is said to have Quorum.  So, in our previous example where the line was severed, DR would NOT mount its database’s.  Why not?  Because the DAG consisted of 3 total members, NYMB01, NYMB02 and DRMB01.  What this means is that according to DRMB01, its the only surviving server, which is 1 out of 3, and is not a majority, hence it cannot mount its database. Now, if you look at the first diagram, you will notice that MDB03, is green on DRMB01, meaning that the active copy of MDB03 is running on DRMB01.  Well, what happens in this scenario, where the WAN connection was cut?  Wont one of the NY servers mount MDB03?  Since DRMB01 has MDB03 already mounted, wont this cause the EXACT split brain scenario we are trying to avoid?  No.  Why not?  Remember how I said that the DAG needs to be able to make Quorum?  Well, in this case, since DRMB01 cannot make Quorum, it is forced to dismount any database that it has running.  In the event log, you’ll see the following message:

So, DRMB01 dismounts MDB03, which is mounted and activated in NY.  This is how the split brain scenario is avoided. 

So what does this mean if there really is a need for a datacenter failover?  At one site I work at, there was a broken pipe in the tenant above them, causing a flood that threatened to destroy their datacenter.  If the datacenter had been destroyed, how do we activate DR?  We’ll go over that in Part 2 of this series.

For this article, we discussed mainly the theory and thought process behind DAG’s, Datacenter Activation Coordinator, and the concept of Quorum with regards to the cluster. In the next article, we’ll jump in and do an actual datacenter failover. 

How to Apply Permissions to Public Folder and All Sub Folders in Exchange 2007/2010 Using Exchange Management Shell

 

If you have a public folder that your working on, and you need to apply permissions to it using the Exchange Management Shell, its pretty easy.  The command is:

Add-PublicFolderClientPermission –Identity “Foldername” –user UserName –AccessRights PublishingEditor

For instance, to add the user pponzeka to the folder IT with the Publishing Editor permission, the command would be the following:

This works great, but what if we have several subfolders under IT, and we want to apply the same user permissions to all of the subfolders as well?  A utility called PFDAVADMIN that was available from Microsoft used to allow you to do this, and it still works with Exchange 2007.  But, since the protocol it uses, WebDAV is no longer available in Exchange 2010, we no longer have this option.  Plus, the shell is easier to use anyway!

So, we have the IT public folder, and three subfolders:

So, first, in the Exchange Management Shell, if we attempt to list the public folder IT, this is the result of what we’ll see.  The command used is Get-PublicFolder –Identity “\IT”

That’s odd, we know there are three folders underneath, why doesn’t it list these?  We need to add the –Recurse option to our command, so that it looks in the root, and everything underneath.  So the command should be Get-PublicFolder –Identity “\IT” –Recurse

Notice the Parent Path?  IT has \ listed, which means its a Top Level Folder in Public Folders.  The other three have \IT listed, which means they are sub folders of IT.

So, back to the top.  The permission to add permission on a public folder was:

Add-PublicFolderClientPermission –Identity “Foldername” –user UserName –AccessRights PublishingEditor

So in our case it was:

Add-PublicFolderClientPermission –Identity “IT” –User pponzeka –AccessRights PublishingEditor

So, now, how do we apply these permissions to the root folder, in this case IT, and all three subfolders, in this case Documents, Emails and Plans?  Well, we use the piping command to pipe the entire list of folders to the Add-PublicFolderClientPermission command.

Get-PublicFolder –Identity “\IT” –Recurse | Add-PublicFolderClientPermission –User pponzeka –AccessRights PublishingEditor

Note that we don’t need to specify the Identity  in the Add-PublicFolderClientPermission because we piped that to it with the | command.

And there you go.  The user account has been given these rights to the root folder, IT, and all its subfolders.  This works for any number of subfolders, and you can also use the same method to remove access rights. 

Issues with IPv6 and Transport Servers using DNS Delivery in Exchange 2007 and General Troubleshooting Steps

 

I recently ran into a little bit of an issue at a company I was working with regarding their Exchange 2007 installation.  At the company, we had installed 2 Edge Transport servers in their main site in NY, and 1 Edge Transport server at their secondary location in Boston. 

One of their user’s complained that when they sent email to a specific domain, say xzy.com, they were getting delivery delayed messages, and eventually full blown NDR messages:

As we can see by the NDR, the message expired.  This happens when Exchange cannot connect to a certain destination mail server (such as the receiving company’s server is down), or there is no path to the destination.  The 1st is usually the fault of the receiving organization, the second is usually a configuration problem on your end.  Needless to say, we set out to troubleshoot.

Some basic stuff out of the way first.  All three Edge Servers were configured identical, and all experienced the same exact behavior.  All use Send Connectors, configured to deliver email using DNS, and thus connect directly to the destination domain.

The first thing I did, was the old telnet to the email server trick, and see if I can send email manually through telnet.  First, you need to figure out the receiving email server for the domain in question.  You can do this using the nslookup command.  In a command prompt, enter the command nslookup and hit return.  This will cause the machine your working off of to connect to its default DNS server, in NSLOOKUP mode.

Now, how do we find the server or servers that are responsible for receiving email on behalf of the organization?  Why, by searching for MX records of course!  In NSLOOKUP, tell the DNS server that you only want to know the MX records for a domain.  You do this by setting the query to MX only with the set q=mx command:

Next, enter the domain you are having trouble with, such as espn.com:

All the entries listed, are servers that receive email on behalf of espn.com.  Next, we want to see if we can send email to those servers, using telnet.  Which server should we pick?  Honestly, since they are all the same priority, to be really thorough, you can do all.  If one had a lower priority, start with that one, and work your way back up through the highest one.

Open up another command prompt, and run the command telnet nameofserver.domain.com 25 and hit enter:

This is your greeting from the remote server.  Now, to send email, you must issue several commands.  First, introduce yourself using the helo  yourserver.yourdomain.com command.  Replace yourserver.yourdomain.com with the external DNS name of the server your connecting from.  If you don’t, the remote server may tell you that you forged your name:, and it may reject your email:

Correct:

Incorrect:

The whole process will end up looking like this:

Okay, we’ll this means email was successfully sent from my Edge Transport server, to their email servers.  So that means that their wasn’t any connectivity issue between the servers.  Hmmm.  Next, on to Connectivity Logging!

Connectivity logging is not turned on by default, but can be configured by right clicking on the server in the console, and going to properties->Log Settings:

Check the box to “Enable Connectivity Logging” and then select browse for the location. 

When I checked the log, this is what I found for the domain in question:

The 72.21….. IP address is actually my DNS server for this particular Edge Transport server, and as you can see, it’s returning a DNS failure for that domain.  Well, that’s odd, being with the NSLOOKUP we already determined that we could resolve names for that domain on this particular server.

As a test, I changed DNS servers to be ones from a completely different company and……same problem.

We’ll, now we now why the emails are expiring, they cannot figure out how to get to the domain in question, so they sit in the queue until they expire and the server generates an NDR.

So we know its a problem with name resolution, but have no idea why it would be failing.  The next step I took was to use a network analyzer like WireShark to see what was happening.  Here is a sample capture for the domain in question that was giving me problems:

Well, well well, here we have a DNS failure of Standard Query AAAA.  So, what’s going on here?  We’ll it seems that the server is performing ONLY an IPv6 DNS lookup, and not doing a IPv4 lookup.  Since the domain doesn’t have an IPv6 record for their domain, the query is failing. 

I reached out to Microsoft and it seems that this is a fairly common issue with Exchange 2007 servers working on Windows Server 2008 machines, as they come installed with IPv4 and IPv6.  Microsoft does not have an official fix for it, but does have three potential solutions for the problem.

1. Place a regular SMTP server ahead of the Edge Transport server, and have the Edge Server forward all outgoing mail to this server as a smart host. (this is in my opinion the worst solution)
2. Create a new Send Connector for that particular domain as an address space.  Set all email to forward through this connector as a smart host and add all MX records as potential smart hosts for this connector (this works very well, with the only potential problem being if the problem is spread across a lot of domain’s, there is a lot of work for the administrator)
3. On the Send Connector, check the “Use the External DNS Lookup on the transport server”

Even if you do not configure specific External DNS servers on the Edge or Hub Transport servers, this will cause the server to only perform a standard IPv4 lookup, which will resolve the issue.

Hopefully this article can help you when troubleshooting email delivery issue’s, giving you some of the steps you can take to track down the specific issue your users are having.

Exchange 2010 Archive Mailbox and Retention Policies – Part 1

 

PST’s and Mailbox Limits.  Those two items keep Exchange Administrator’s up at nights.  One of the biggest battles between the end users and the administrator is mailbox size and retention.  It’s pretty well known that the bigger the mailbox get’s, usually the worse the performance gets.  Admins have been trying for years to get users to get their mailbox size down, and users always fight back.  With the dramatic changes that the Exchange Team has made to the Mailbox Store in Exchange 2010, it is now possible to have huge mailboxes’ (10GB+) to support the growing need’s of our end users.

Problem is, we still want to get the best performance possible out of our mailbox’s.  For archive purposes, the only “built in” solution from Microsoft was to export data to PST files.  PST files though are not managed by the organization, are prone to corruption, not accessible through OWA, and are generally messy.  Company’s were forced to move towards 3rd party utilities to satisfy their archiving and compliance needs.

One of the new features in Exchange 2010 is the Archive Mailbox feature.  Simply put, the Archive Mailbox is an extra mailbox assigned to the user, that’s meant to hold older, less accessed data.  For instance a user could place every email older than 1 year in the archive folder, keeping only new constantly accessed data in the primary mailbox.  The Archive Mailbox is integrated into the users Outlook Profile in Outlook 2010:

As well as Outlook Web App (the new name for Outlook Web Access in Exchange 2010):

There are some caveat’s to the Archive Mailbox feature:

1. Only Outlook 2010 supports the integration of the Archive Mailbox in the user profile
2. The archive mailbox is placed in the SAME database as the primary mailbox

Word on the street is that Microsoft is working on a plug in or an update to Outlook 2007 that will allow the older version of Outlook to access the Archive Mailbox.  Still, many companies have an Office 2003 installation that will be hard to uproot to 2010 for this feature.

The second point I think is the one that really hurts this feature.  The fact that the archive mailbox is part of the same database as the primary mailbox.  I understand that in 2010, the IOPS requirement has been dramatically lowered, and now the organization can run on SATA or Tier-2 disk’s, but many company’s want to segregate their data onto different tier’s and speed of storage.  I think this aspect of the feature will end up hurting Microsoft.  Let’s see if they make a change in the future. 

I digress.  The Archive Mailbox is essentially meant to replace the use of PST’s.  The best part is you can drag data from existing PST’s into the archive, just like you can drag messages from your inbox to the archive.  But the real power is setting this up to be done with policy’s, called Retention Policy’s. 

As administrator’s, we can control the length of time that users are allowed to keep items in their primary mailbox.  This is an extension of the Management Folder’s feature from Exchange 2007.  In 2007 however, users had to place email’s in certain folder’s, be it inbox, or a created folder, so that the policy’s were enacted properly.  2010 uses the concept of Retention Policy’s as well as Retention Policy Tag’s to enforce these settings. 

Retention Policy Tag’s are classification’s that are set for a folder or a type of item.  For instance you can create a Retention Policy Tag for the inbox, and set it to archive anything over 90 days.  You could also set a Retention Policy Tag for any calendar item over 180 days to delete and not archive. 

You would then group this tag’s into a Retention Policy, which would then be applied to a mailbox.

By default, Exchange 2010 comes with two retention policy’s, Default Archive Policy, and Arbitration. 

The Default Archive Policy comes with predefined tags:

1. Personal never move to archive
2. Personal 1 year move to archive
3. Personal 5 year move to archive
4. Default 2 year move to archive

You can check these settings by entering the following powershell command:

Get-RetentionPolicy *default* | fl

Every mailbox that is archive enabled is automatically assigned the default retention policy.  You can determine the retention policy by running the command Get-Mailbox jsmith | select Name,RetentionPolicy:

So according to the Default Archive Policy, the user can tag items as one of the three personal setting’s, and then have it moved to archive after the specified time, and anything not tagged will be moved to the archive after 2 years.  Retention Policy’s base the time or age of the message on when the item was delivered, or if the item wasn’t delivered (such as calendar item or post), when the item was created.

The Arbitration Retention Policy is used for moderated mailboxes.  Moderated mailboxes are used for users who mail must pass through a manager for approval first, or for group membership approval.  This policy should only be used in conjunction with these type of mailboxes.

Next, we’ll create a user and assign him an Archive Mailbox

Creating the user is straightforward and exactly the same as normally creating a mailbox, except for one thing.  Towards the end of the mailbox creation period, you’ll be prompted with this screen:

You can create an archive mailbox at the mailbox creation time, or after the mailbox is created by right clicking the user and selecting

One thing to keep in mind is you cannot mix Retention Policy’s and Managed Folder Mailbox Policy’s.

Notice how the icon changes for Archive Enabled mailbox’s:

If we navigate to the each user in ADSIEDIT, you’ll see some other difference’s in available attributes.  The three in particular are msExchArchiveGuid, msexchArchiveName, and msExchMailboxtemplateLink.

Notice how the user on the right the value’s are blank, and how they are populated on the John Smith user on the left.  Also, notice the msEchMailboxTemplateLink value.  This is the retention policy applied to this particular user. 

In this first part, we discussed the new Online Archive Mailbox feature of Exchange 2010, its concept and purpose, and then discussed how administrator’s can apply them using Retention Policy’s.  In part two of this article.  We’ll get in and get our hands dirty, and start applying policies, and seeing how Exchange manages and move’s item’s accordingly.

Uncovering the New Move Request Feature in Exchange 2010

 

One of the biggest issue’s any administrator faces with any type of Exchange upgrade or migration is the prospect of mailbox moves.  The issue comes from the fact that a mailbox move, from the moment it’s started, makes the mailbox unavailable to the end user until it’s completed.

This comes from the way that mailbox moves are done in Exchange 2000/3/7.  The mailbox move is actually a MAPI copy operation.  The reason’s behind this are clear, Exchange need’s to lock in the mailbox, then copy it, to ensure that no data is lost. The user is stuck waiting for the mailbox move to complete, meaning they can’t work.  This ends up in mailbox moves being forced to happen late at night in bunches, through the use of scripting or other methods. 

One of the exciting new features of Exchange 2010 is online mailbox move’s, which are called Mailbox Move Requests.  This new feature allows administrators to move mailbox’s while the user is actively using it.  How does this work you ask?  The magic lies in a new service that is installed on Exchange 2010 Client Access Servers, the Microsoft Exchange Mailbox Replication Service:

 

Exchange 2010 no longer performs a MAPI copy operation to perform any of it’s mailbox moves.  Now, when a move request is created, the Microsoft Exchange Mailbox Replication Service or (MRS) is responsible for replicating the mailbox from it’s source, to it’s destination. An initial replication occurs, and then incremental updates happen until the service can complete the move.  Once the service completes’ the sync, the user receives a message to restart outlook.  They restart, their mailbox is again available.  The only downtime was for them to restart Outlook, not bad at all.

The nice part about this, is that mailbox move’s from Exchange 2007 benefit from this feature when moving to an Exchange 2010 mailbox.  Keep in mind that you must be at service pack level 2 with Exchange 2007 for it and 2010 to interoperate.

Let’s take a look at actually performing one of these.  In our environment we have two Exchange 2007 servers, DEVLOCALHT1, which is a Hub Transport/Client Access Server, and DEVLOCALMB1, which is a Mailbox Server.  Then we have three Exchange 2010 servers.  DEVLOCALHT2010, which is a Hub Transport/Client Access Server, and then DEVLOCALNODE1 and DEVLOCALNODE2 which are both Mailbox Servers.  The Mailbox Servers are joined into a Database Availability Group or DAG.  If your not sure what a DAG is, see my article series:

DAG – Part1

DAG – Part2

DAG – Part 3

DAG – Part 4

Our user, creatively named Paul Ponzeka, has an Exchange 2010 mailbox. We currently have two Mailbox Database’s in Exchange 2010, MDB1 and MDB2:

Currently Paul Ponzeka’s mailbox is located on MDB1, and we’ll move him to MDB2.  In the Exchange 2010 EMC, navigate to Recipient Configuration->Mailbox.  Select Paul Ponzeka, right click and select New Local Move Request:

Don’t let the language fool you, if your moving a mailbox anywhere within the SAME Active Directory Forest, its a Local Move.  If your moving the mailbox Cross-Forest’s, its a Remove Move.

Select the target mailbox database:

After selecting next, you’ll see a familiar screen regarding what to do regarding corrupted items.  You can choose to have the move fail, or to ignore up to a certain amount of items.

After you select next, you’ll have a summary page, select the “New” button, and as always, it will provide you with the powershell command for the equivalent.

Now, if you navigate to Recipient Configuration->Move Request, you’ll see the move request, and it’s progress:

You can also get the progress through the Exchange Management Shell by running the command:

Get-MoveRequest

Also, in the Event Viewer of DEVLOCALHT2010, the Client Access Server handling the move, we will note that Event ID 1102 is logged regarding the start of the move:

Once the move is complete, Event ID 1107 will be logged:

And if we notice back in the EMC, the status of the move:

And the shell:

Now, the client, who’s Outlook has been open the whole time, will receive the following message:

Once the client restart’s outlook, their mailbox is back, and they’ve only been out of Outlook from the time the move completed, till when they restart.

The move’s work EXACTLY the same way as from an Exchange 2007 SP2 mailbox with no difference. 

The new online mailbox moves, through the Move Mailbox Request feature of Exchange 2010 will allow administrators to move mailbox’s with minimal downtime and interruption to the end users.

The “Require That All Senders are Authenticated” Does Not Work As Expected in Exchange 2007

 

I recently ran across an issue where one company was setting the “Require that all senders are authenticated” checkbox on a particular mail distribution group in Exchange 2007.

Most organizations will do this, to ensure that only users of the internal Exchange organization can email this group, effectively stopping external users from emailing it.  This is particularly useful if the group is an extremely broad group such as “ALLUSERS” or something similar.

The problem was that even with this checked, the group was still receiving emails from outside senders.  What gives?  Well, I started doing a little digging, and this is what I found.

The company had followed this article by Scott Landry of the Exchange Team.  The articles discusses two ways to allow application servers to relay through Exchange 2007, applications such as SharePoint. 

If you read Scott’s article, his tone indicates that these applications are really internal applications.  He also lists two method’s for doing so.  The first being to set the receive connector to externally secured, and the second with manually applying permissions using powershell.  The company who was having this issue, had used the first method, eternally secured, but was using it to allow a 3rd party SMTP server that was used for spam filtering on it’s DMZ, to relay into Exchange.  So what’s the problem?  Essentially the method they used, gave TOO much permission to the 3rd party server.  Let’s take a look.

Scott’s article states to create a new receive connector, and set this connector to allow the IP address of the 3rd party server.  So, assuming our third party server has an IP address of 192.168.1.5, the connector would look like the following:

Next, the article states to edit the authentication and permissions tab’s, so that “Exchange Servers” are set on the Permission Groups tab:

  and then “Externally Secured” is selected for authentication:

After this, the 3rd party server would be allowed to relay.  So you ask, what’s the problem.  Well, by setting the above, most importantly the Exchange Servers on the Permission Groups, you are essentially telling Exchange 2007, that any email that comes in this connector, should be treated the exact same as if it originated from an Exchange 2007 server.  This means that the emails will bypass anti-spam settings, as well as appear as if they were sent from inside the organization.  Scott lays out exactly what permissions are set:

MS Exchange\Externally Secured Servers {ms-Exch-SMTP-Accept-Authoritative-Domain}
MS Exchange\Externally Secured Servers {ms-Exch-Bypass-Anti-Spam}
MS Exchange\Externally Secured Servers {ms-Exch-Bypass-Message-Size-Limit}
MS Exchange\Externally Secured Servers {ms-Exch-SMTP-Accept-Exch50}
MS Exchange\Externally Secured Servers {ms-Exch-Accept-Headers-Routing}
MS Exchange\Externally Secured Servers {ms-Exch-SMTP-Submit}
MS Exchange\Externally Secured Servers {ms-Exch-SMTP-Accept-Any-Recipient}
MS Exchange\Externally Secured Servers {ms-Exch-SMTP-Accept-Authentication-Flag}
MS Exchange\Externally Secured Servers {ms-Exch-SMTP-Accept-Any-Sender}

Now, any email from this connector, will bypass the “Require That All Senders are Authenticated” check box, because according to Exchange 2007, they are authenticated!

So, how do you fix it?  It couldn’t be easier, Scott’s article already tells us how.  He lists Option 2, grant the relay permission to Anonymous Users on the connector.

If we change the permission group, so that only “Anonymous” is checked, and remove “Externally Secured” from the authentication tab:

And then run one little powershell command:

Get-ReceiveConnector “3rd Party Relay” | Add-ADPermission –User “NT Authority\Anonymous Logon” –ExtendedRights “ms-Exch-SMTP-Accept-Any-Recipient”:

That’s it.  Now, the third party server can send emails into the organization, but they are treated as outside emails.  This means they will be subject to size limit’s, anti-spam, as well as being rejected if they are sent to a group that has the “Require That All Senders Are Authenticated” setting checked.

Creating a Database Availability Group in Exchange 2010 – Part 4

 

In part 3 of this series, we configured the replication networks for the DAG, as well as add database copies to the DAG.  In Part 4, the final part of this series, I’ll show you how to move the active copy between different servers, and how this affect’s the end user. 

Remember way back in Part 1 of this series, I mentioned that Outlook client’s now connect to a Client Access Server to connect to their respective mailbox’s, instead of connecting directly to the Mailbox server?  We’ll, part of the reason why, is that in Exchange 2010, Microsoft wanted the failovers to be less than 30 second’s, no matter what.  By moving the Outlook client to connect to the CAS server’s, you can meet this threshold.  Client Access Server’s are notified about a database move, and thus can redirect their client’s accordingly.

Take our user, Paul A. Ponzeka, who’s mailbox is is located on the Accounting database we created in part 3.  The active copy of the database is on NYDAGNODE1. 

When we create his profile in Outlook, look at the server he connects to:

Notice how the Outlook client connects to the CAS server, and not the mailbox server, which in this case would be NYDAGNODE1.  Everything, except public folder connections, are proxies through a CAS server.

Okay, so now let’s move the active copy to the London server, LNDAGNODE1. We can do this through the console, or the shell.  In the console, navigate to Organization Configuration –>Mailbox->Database Management.  Simply right click on the database in question, and select “Move Active Mailbox Database”.  Fill out the menu, with which server you want to move it to.  The override mount dial allows you to specify an override for the amount of data loss your willing to accept.  We don’t have a need to change this, so just leave it be and hit move:

So now the system will move the database, and if you note, in our example, it took 6 seconds, also note the command shell you could have run to get the same effect:

Now the accounting database will go to Mounted on LNDAGNODE1 and healthy under copy status on NYDAGNODE1:

There you go, all set!  Again, the threshold will be 30 seconds according to Microsoft for the failovers!

So in wrapping, I wanted to touch on a couple of the core differences between Exchange 2010 DAG’s, and CCR clusters, both normal and stretched.

First, in 2007 CCR clusters, when you perform the initial seeding, you need to have every transaction log for that particular storage group, otherwise seeding will fail.  This is no longer true with 2010.  You can add a DAG at ANY POINT.  Second, in 2007, once you moved the active copy to the passive node, you had to re-seed the previously active copy from the passive node before that node could ever be brought back on line.  This is again no longer true in 2010.  You can move back and forth without reseeding.

Third, CCR’s were limited to just 2 nodes, you are limited to 16 members of a DAG in 2010!  In 2007, even if you were doing a stretched cluster with 2008, both nodes had to be members of the same AD site.  This had to do with the nodes having to speak to the same set of Hub Transport servers to avoid data loss.  With 2010 and the inclusion of the new Shadow Transport feature, this is not needed.  So you can have any number of DAG members, in any number of different AD sites.  The outlook client’s will just be redirected to a CAS server in the local site of the now active mailbox server copy, again in under 30 seconds!

Another huge difference, is that since the database’s are no longer members of the servers, the failovers are on a per database level.  What this means is if a server in NY, has three databases, and all of them are active, and you have a situation where only one of the database’s go offline, only that database will be activated on another copy in the DAG, leaving the previous two untouched!

In this four part series, we went over the theory of the Database Availability Group feature of Exchange 2010, as well as how to configure and test the DAG.  Overall, it is a great improvement for both the High Availability, and Site Resiliency model for Exchange. 

Creating a Database Availability Group in Exchange 2010 – Part 3

 

In Part 2 of this series, we created the Database Availability Group, and added both NYDAGNODE1 and LNDAGNODE1 to it.

In Part 3 of this series, we are going to configure the network’s properly, and create some database’s for the DAG.

As we noted last time, by default, all networks for every node in a DAG is configured for replication.  We only want replication to occur over certain networks, mainly 172.16.1.0 for NY and 172.17.1.0 for London.  If we navigate to Organization Configuration –> Mailbox and select the Database Availability Group tab and select DAG01, we see all the networks listed.

Just a side note, if you right click the label DagNetwork01 for example, you can rename it to something more descriptive.

Now, for the two production network’s, when your in the property’s page, un-check the “Replication Enabled” check box:

Now, it states for both NY and LN Production, that replication is disabled.  This will ensure that all replication occurs over a dedicated network.

Now, it’s time to add some database’s to the DAG!  Move on to the Database Management tab.  You will note two database’s here, both of which are the default one for each server. 

We can add these existing database’s to the DAG, by right clicking on each of them and selecting “Add Mailbox Database Copy”.  You will select any free server to add a copy, in our case:

Select Add, and it will add NYDAGNODE1, as a replica for the Database.  Note the preferred list sequence number.  This indicates that NYDAGNODE1’s copy of this database, should be the second database activated, should something happen to Preferred List Sequence Number 1, which is the original copy on LNDAGNODE1.

The powershell command we could have run is listed:

Now note, we have one database that’s listed as Copy Status Mounted, and one who’s Copy Status is Healthy.  The Healthy means it’s not in production and is a replica. 

Note how it lists the servers that are hosting the database, as well as the Copy Queue Length, Replay Queue Length, as well as the Preferred List Sequence Number.  The copy queue length is how many transaction logs are waiting to be copied to the node, the replay is how many are waiting to be played into the database on that node, and the list sequence is what is the preferred next copy of the database Exchange should activate, if the currently mounted one becomes unavailable.

Adding a new mailbox database is very similar.  You create a new mailbox database, and select a node to host the first copy:

And then just add extra copies as we did above.  All servers that are in the same DAG should have the same drive letter or mount point configuration.  This is because all copies will have the same path to the EDB File, as well as the Transaction Log and System Files path’s.  Also, since Mailbox Database’s are objects of the organization now, you need to ensure that their names are unique throughout the entire Exchange Organization.

So, that’s it for the third part of this series.  In this part, we configured the networks for replication, and we added copies of existing database’s, as well as created new database’s and copies into our DAG.

In the next part, I’ll show you how to fail over to different copies of the Mailbox Database’s, and it’s impact on the end user.