Manually create FileWitness Share

When creating the DAG in Exchange 2010, Microsoft recommends that the FSW is located on a HUB server in the environment. One of the reasons is that these servers are the Exchange owners, and the Exchange admins have full control. It is recommended to place the FSW share in a third location or in the primary location.

To provide failover functionality to the FSW, you can use file cluster, Force cluster @File Share Witness and Force Quorum Guidance or the old and no longer recommended method of DNS CNAME.

At one of my customers, we have placed the FSW on a file cluster, since this gives us the full
redundancy in the case of a datacenter failure. The customer has two datacenters, but they are in a single site.

And Microsoft recommends you to add the Exchange Trusted Subsystem group in the local admin group of the server holding the FSW. But this might not be compliant to security rules. This is as far as I have understood just so the set-databaseavailability command can create the folder and shares.

To pre-create the FSW share you need the following:

- Create a folder etc. D:\FilesWitness\DAGNAME
- Give the owner permission to Exchange Trusted Subsystem
- Give the Exchange Trusted Subsystem Full Control (NTFS)
- Share the folder with the following DAGNAME.FQDN (If you try a different share name, it won't work. This is somehow required)
- Give the DAGNAME$ computeraccount Full Control (Share)

When you've done this, you can run the
set-databaseavailabilitygroup -witnessserver CLUSTERSERVER - witnessdirectory D:\Filewitness\DAGNAME

You'll get the following warning message:
WARNING: Specified witness server Cluster.fqdn is not an Exchange server, or part of the Exchange Servers security group.WARNING: Insufficient permission to access file shares on witness server Cluster.fqdn. Until this problem is corrected, the database availability group may be more vulnerable to failures. You can use the set-databaseavailabilitygroup cmdlet to try the operation again. Error: Access is denied

This is expected, since the cmdlet tries to create the folder and share, but don't have the permissions to do this.
When this is done, the FSW should be configured correct. To verify this, the following files should be created:

- VerifyShareWriteAccess- Witness

This should do the trick, and the FSW is ready to work, and follow the active datacenter.
There could be a slightly drawback with using a file cluster as FSW. If the file cluster gets in a split brain scenario, then the Exchange solution could get affected. A result of this could be two active database servers on the same database, resulting in corrupt data. But this is theoretically, and in special circumstances.

Rune

Exchange 2010 DAG datacenter failure

One of the neat features in Exchange 2010 is the DAG. This seems to have the most built-in features for solutions where the datacenters are spread across AD sites, but alot of companies out there don't have multiple AD sited.
One customer I'm at right now don't, and due to this we are trying out the datacenter failure scenario. And to get the solution up and running, my experience is that you have to do some manal steps, and its important to do this in the right order. This is descibed below.

One datacenter fails, and the Cluster does not get MajorityNodeSet.
Datacenter contains 2 MBX servers and 1 HUB/CAS witch is the FSW for the DAG.

To recover from this failure, this are the steps:
- Stop Cluster service on remaining DAG members I secondary datacenter
- On one DAG member do a net start clussvc /forcequorum
o In my case, the databases got mounted already here

To change the FSW on the DAG now will not work, since the DAG can’t communicate with the failed DAG servers.
To remove the affected DAG members and change FSW, you have to complete the following:
- Start cluster admin and evict the failed DAG member servers
- Remove all database replication
o Get-mailboxdatabasecopystsatus -server failedmbxserver
o Remove-mailboxdatabasecopy databasename\affectedMBX
- Remove-databaseavailabilitygroupserver DAGNAME –mailboxserver failedmbxserver

Now you will be able to change the FSW for the DAG.
- Set-databaseavailabilitygroup DAGNAME –witnessserver FQDN

Now everything should be cleaned and in order.

One thing I noted is when doing these steps in the wrong order, running the remove-databaseavailabilitygroupserver DAGNAME before evicting the nodes from the cluster.
The databases that had a copy to the failed MBXserver, got the following errors in their properties. This was a hell to clean up.

Get-mailboxdatabase databasename fl

Server : ActiveMBXServer
MasterServerOrAvailabilityGroup : FAILEDMBXServer
MasterType : Server

To cleanup this, I had to rejoin the failed MBXservers to the DAG,and enable database replication.
Even though the replication seemed to work fine, it didn’t. It wasn’t possible to switch the active database between the servers in the DAG due to the properties on the database.
Then remove the old failed MBXserver from the DAG to a single MBX server with the existing database. Then rejoin the MBXserver to the DAG, and then the properties was OK.

When a database is member of the DAG, the correct way this should be is:

Server : ActiveMBXServer
MasterServerOrAvailabilityGroup : DAGNAME
MasterType : DatabaseAvailabilityGroup

This is my experience with the datacenter failure, but I'll post more later.
Rune