Routing Group Connector - Loop condition 2010

Today I’ve been troubleshooting a looping condition on a Exchange 2003 -> 2010 transition. The mail flow from Exchange 2010 -> 2003 is working fine, and with a new SMTP connector for internet mail the Exch2010 is able to send to the internet.

But one thing does not work, and that is mail flow between the Exc2003 and Exc2010. No NDRs, nothing is reported back from the system, but the mails are located in the “Messages queued for deferred delivery”queue. When the SMTP service is stopped, I receive a NDR for the messages saying it’s a loop:

Your message did not reach some or all of the intended recipients.
Subject: Test 11
Sent: 5/01/2010 3:34 PM
The following recipient(s) could not be reached:
user@domainname.com on 5/01/2010 3:34 PM
A configuration error in the e-mail system caused the message to bounce between two servers or to be forwarded between two recipients. Contact your administrator.


The exchange 2003 organization does not have other routing group connectors, and is not using Smarthost for outgoing emails. The Exc2010 RoutingGroupConnector created during install has been deleted and recreated to see if that did the trick, but it didn’t.

It turns out that the Exc2003 cluster had 2 SMTP virtual servers, where as one was stopped due to some old errors, and the on the routinggroupconnector this was the SMTP server used. I could not find any powershell parameters to define a different SMTP virtual server, so I had to use ADSIEDit to change the default SMTP virtual server on the connector. Thanks to Winroute for giving me a red cross on the RoutingConnector, and sending me in the direction of multiple SMTP servers.

After changing this and wait for the Exchange to pick up the config from AD, mail flow is now working both ways between Exc2003<->Exch2010.

MapiExceptionNoSupport: IExchangeFastTransferEx.TransferBuffer failed hr=0x80040102, ec=-2147221246

These last couple of days I've been migrating a customer from Exchange 2003 to Exchange 2010 SP1.
All users were moved successfully, except one. On this user we got some strange error messages in the move-requeststatistics command and on the HUB transport responsible for the move.
The complete error message is posted at the bottom.

On this user the move-request failed after just some MB, and the troubleshooting begun.
These are the steps I took:

• Use Exmerge on 2003 for .pst migration - Failed. The error message in the exmerge logfile is as follow:
  Error copying messages from folder '\Inbox' (MAPI_W_PARTIAL_COMPLETION) [22:14:27] Trying to copy messages in folder '\Inbox', individually. To avoid duplicates in the target store, messages will be merged and not copied into the target store. [22:14:27] Error getting Contents Table. (CopyMessagesToDestFolder)
• Migration to another Exchange 2003 mailbox store - OK
• Migration to Exchange 2010, gives the error message below.
• Export to .pst from Outlook - OK

The solution for me was to use PFDAVAdmin and to reset the DACL on the mailbox. Tried the Check DACL state, before the Fix, and it said it was ok. But apparently not.

When the DACL was reset, the move-request went successfull.

Complete Error message:
Request domain.fqdn/OU/name' (f71639ea-0868-4527-b71e-7b6311359e98) failed.
Error code: -2147221246
MapiExceptionNoSupport: IExchangeFastTransferEx.TransferBuffer failed (hr=0x80040102, ec=-2147221246)
Diagnostic context:
Lid: 55847 EMSMDBPOOL.EcPoolSessionDoRpc called [length=470]
Lid: 43559 EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=645][latency=0]
Lid: 23226 --- ROP Parse Start ---
Lid: 27962 ROP: ropFXDstCopyConfig [83]
Lid: 27962 ROP: ropTellVersion [134]
Lid: 27962 ROP: ropFXDstPutBufferEx [157]
Lid: 17082 ROP Error: 0x80040102
Lid: 31329
Lid: 21921 StoreEc: 0x80040102
Lid: 27962 ROP: ropExtendedError [250]
Lid: 1494 ---- Remote Context Beg ----
Lid: 1238 Remote Context Overflow
Lid: 32762 dwParam: 0x3FD60040
Lid: 30106 StoreEc: 0x471
Lid: 18842 StoreEc: 0x471
Lid: 20234
Lid: 3625 StoreEc: 0x471
Lid: 25066
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x668F0040
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x674A0014
Lid: 21970 StoreEc: 0x8004010F PropTag: 0x36138000
Lid: 31473 StoreEc: 0x8004010F
Lid: 39287
Lid: 17916 StoreEc: 0x80040102
Lid: 32252 StoreEc: 0x80040102
Lid: 7915 StoreEc: 0x80040102
Lid: 20320
Lid: 4559 StoreEc: 0x80040102
Lid: 21802
Lid: 19994 StoreEc: 0x80040102
Lid: 20202
Lid: 3305 StoreEc: 0x80040102
Lid: 32762 dwParam: 0x3FE00102
Lid: 32762 dwParam: 0x675C0102
Lid: 32762 dwParam: 0x67510003
Lid: 26346
Lid: 4073 StoreEc: 0x80040102
Lid: 28570 StoreEc: 0x80040102
Lid: 29738
Lid: 3401 StoreEc: 0x80040102
Lid: 1750 ---- Remote Context End ----
Lid: 26849
Lid: 21817 ROP Failure: 0x80040102
Lid: 22630
Context:
--------
Operation: IMapiFxProxy.ProcessRequest
OpCode: TransferBuffer
DataLength: 404
--------
Operation: IMapiFxProxy.ProcessRequest
OperationSide: Target
Primary (f71639ea-0868-4527-b71e-7b6311359e98)
OpCode: TransferBuffer
DataLength: 404

Iphone OS4 - Exchange Activesync issues

Apple has created a hotfix for issues regarding OS4.0 and activesync.
http://support.apple.com/kb/TS3398

This hotfix needs to be installed on the iPhone.

Unable to access mailbox after migration from Exchange 2003 to Exchange 2010

I migrated 400 users this weekend to Exchange 2010. The migration seems like the migration went well, since all the move requests came out 100% completed and no one reported any errors. But something didn’t go successfully as reported…..


Right now I have completed troubleshooting an issue where a migrated user was not able to access his mailbox. The users who by the move request report went successful, but not all of the AD attributes were updated.

This is an excerpt of the migration log:
Status : Completed
StatusDetail : Completed
SourceVersion : Version 0.0 (Build 7638.0)
SourceDatabase : exchange2003server\Storage Group\DBname
TargetVersion : Version 14.0 (Build 639.0)
TargetDatabase : 2010-databasename
MoveServerName : fqdn.servername
TotalMailboxSize : 2.951 GB (3,168,187,842 bytes)
TotalMailboxItemCount : 21074
BytesTransferred : 2.717 GB (2,917,612,755 bytes)
ItemsTransferred : 20958
PercentComplete : 100

I then did a get-mailbox username and this gave me the following result:

And this is strange, the log says it completed successfully, but AD says it’s on the Ex2003 server.

I then did | fl to get some more info, and the recipient type and recipient type details returned usermailbox, the Exchange version on the user was 0.10(14.0.100.0). And all these values were correct compared to the migration log.
The only attributes that was wrong on the user object was serverlegacyDN and Servername where it pointed to the Ex2003 server and Ex2003 administrative group.

This pointed me to something going wrong in Active Directory, at the current moment.
I went in to Active Directory with ADSI edit and changed the following attributes
HomeMDB and mdexchhomeservername to the new Exchange 2010 paths.

After changing this to the correct values, the mailbox worked as normal, and the get-mailbox username returned the correct value on servername and serverlegacyDN

Save all attachments does not work after moving to Exchange 2010

Today I troubleshot an issue with Outlook 2007 and the save all attachment function

When we tried to save all attachment from the file->Save as-> save all attachment, nothing happened. We were able to save attachment from the mail (right click save as…).

I tested to find out what causing this problem, and I was able to reproduce it. It seems like if the attachment has been sent from outside the Exchange organization this happens, but if I send the attachment within my Exchange organization the save all attachments works.

After verifying this, I thought this must be an bug.
I did some googling and found this link: http://social.technet.microsoft.com/Forums/en/exchange2010/thread/69f951f4-5ef3-46a3-8391-3e1e4c669c20

They say it’s a known bug in Exchange 2010, and that they are working on a KB article, which has not yet been published (KB980940).

iPhone OS4.0 - ActiveSync

After iPhone OS 4 has been released, many of our users complain that the ActiveSync does not longer work against Exchange 2010.
This seems to be a bug in the ActiveSync somehow, but we are working on this issue to see if we can find a solution, or if we have to wait for Apple to fix it.

SCOM And Exchange 2010, the Extest_ account gets locked out

For the past 1 month we've been troubleshooting an issue with our SCOM and Exchange 2010 monitoring.
The monitonring with the SCOM Management Pack works fine for a long time, but suddenly the extest_ account get locked out. This is right after the automatic password change routine on the Exteset_ account.

After talking to MS premier support, they have now found a bug in how the SCOM does it authentication, and that the authentication is multiple ways.

I've got an workaround, and that was to disable the following SCOM monitors, this did not work for us, but you might try it to remove the annoying SCOM alerts. This is the response we got from Microsoft:

"In the authoring section of the OpsMgr console, “Outlook Server Availability” to your scope and disable the 6 monitors that start with “KHI: HTTP Connectivity Against Local Server”. This should disable the instance of test-outlookconnectivity that causes this problem. We’ll be addressing issue, but this should serve as a workaround for now"

Users unable to access Exchange Control Panel(ECP, OWA)

Today I troubleshooted an issue on Exchange 2010.
When the users clicked the Options button to access the Exchange Control Panel, the OWA page just reloaded and nothing happend.
Also when I tried to access the url for the ECP https://url/ECP I came to the OWA page.

One configuration change we did to remove the timezone and language prompt for our OWA users, was to set the defaultclientlanguage on the OWAVirtualdirectory.

When the OWAvirtualdirectory -defaultclientlanguage is set to something else than 0, this issue will occure.

I have now send a bug report to Microsoft, to find out if this is by design, or if its a bug.

An IIS directory entry couldn't be created. The error message is Access is denied.. HResult = -2147024891

Today I did a Exchange 2007 to Exchange 2010 migration. we had to keep one of the Exchange 2007 CAS serveres for the Microsoft Transporter Suite (Lotus Notes integration).

When doing an get-owavirtualdirectory and other cmdlets involving all servers in the organisation, I got the following error message:

The solution for this error messsage is to add the Exchange trusted subsystem to the local admin group on Exchange 2007 servers.

How to change placement of queues and log files in Exchange 2010 HUB and CAS

As a general best practise its recommended to change the default paths for logfiles, to avoid them filling up your System disk.

Exchange 2010 uses the default installation path for all logs and mail queues for HUB and CAS.
To change these you can modify the following files in the Exchange 2010 install folder \bin.
The paths I normally change is the ones specified below:

Edgetransport.exe.config

• add key="QueueDatabasePath" value = "D:\Exchange\Transport Roles\Data\Queue"
• add key="QueueDatabaseLoggingPath" value = "D:\Exchange\Transport Roles\Data\Queue"
• add key="IPFilterDatabasePath" value = "D:\Exchange\Transport Roles\Data\IPFilter"
• add key="IPFilterDatabaseLoggingPath" value = "D:\Exchange\Transport Roles\Data\IPFilter"
• add key="TemporaryStoragePath" value = "D:\Exchange\Transport Roles\Data\Temp"

microsoft.exchange.addressbook.service.exe

• add key="LogFilePath" value="Drive\Folder"

microsoft.exchange.rpcclientaccess.service.exe

•  add key="LogPath" value="D:\Exchange\RPC Client Access\"

POP3 IMAP config files are located in \v14\ClientAccess\PopImap

Change the following files to set the new log folder for the services:

  

Microsoft.Exchange.Imap4.exe.config

• add key="TemporaryStoragePath" value = "D:\Exchange\ClientAccess\PopImap\Temp"
• add key="LogPath" value="D:\Exchange\ClientAccess\Imap4" 

Microsoft.Exchange.Pop3.exe.config

•  add key="TemporaryStoragePath" value = "D:\Exchange\ClientAccess\PopImap\Temp"
•  add key="LogPath" value="D:\Exchange\ClientAccess\Pop3"

Always backup your files before you edit them, and to make the change active. You have to restart the services related to the config file.
Regarding NTFS permissions, be sure to grant the correct user on the folder. Some services acts as network service.

Test-ActiveSyncConnectivity - The remote server returned an error: (403) Forbidden

Today I troubleshoot an issue where the test-activesyncconnectivity didn't work.
The following error was given when running the test:

PS] C:\Windows\system32>Test-ActiveSyncConnectivity
CasServer LocalSite Scenario Result Latency(MS) Error
--------- --------- -------- ------ ----------- -----
CASServer Default-Fi... Options Success 15.60
CASServer Default-Fi... FolderSync Failure [System.Net.WebExcept...

With the | FL we can see a more specific error message:

RunspaceId : 64768a1f-b8cc-49cd-bd76-ee70de43c728
LocalSite : Default-First-Site-Name
SecureAccess : True
VirtualDirectoryName :
Url :
UrlType : Unknown
Port : 0
ConnectionType : Plaintext
ClientAccessServerShortName : CASSERVER
LocalSiteShortName : Default-First-Site-Name
ClientAccessServer : CASSERVER.FQDN
Scenario : FolderSync
ScenarioDescription : Issue a FolderSync command to retrieve the folder hierarchy.
PerformanceCounterName : DirectPush Latency
Result : Failure
Error : [System.Net.WebException]: The remote server returned an error: (403) Forbidden.


HTTP response headers:


MS-Server-ActiveSync: 14.0
Content-Length: 5355
Cache-Control: private
Content-Type: text/html; charset=utf-8
Date: Wed, 26 May 2010 13:50:19 GMT
Server: Microsoft-IIS/7.5
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET


UserName : extest_3650257c296a4
StartTime : 26.05.2010 15:50:20
Latency : -00:00:01
EventType : Error
LatencyInMillisecondsString :
Identity :
IsValid : True

The reason for this is becase the default ActiveSync Mailbox Policy has been changed and does not support non- provisionable devices.

Solution is to create a new ActiveSync mailbox policy which allow non-provisionable devices and assign this to the extest_3650257c296a4 user.

Eventid 200 Database headers have been successfully validated.... Dirty shutdown

Today I've troubleshoot some issues with backup of an Exchange 2010 DAG installation. Some databases were partially backed up according to the backup software (Symantec Netbackup).

I came over the information event:

Source: Storage Group Consistency Check
Event ID: 200
Level: Information
Instance X: Database headers have been successfully validated. All Databases are in a dirty shutdown state. Tobring these databases to a clean shutdown state, log generation xxxxx(0x0xxxx) to xxxxxx (0x0xxxxx) will be required.

As it says this is only for information, and there is no need to be worried and is as expected when the database is online.

When the database is online it has logs in memory or open logs etc. that are not committed yet.

So, don’t worry, this is expected, and this check is done on online backup.

On the other hand, if your database is offline and you get Dirty Shutdown when running eseutil /mh (DBpath+filename) then you might have a bigger issue, and have to do a repair / restore and log replay job.

Exchange 2010 - Service Pack1 scheduled this year

MSexchangeteam has now announced that Microsoft will relase Service Pack1 for Exchange 2010 this year.
The service pack will include both fixes and new features.

Some of the new features are:

• Archiving. You will now be able to create the archive mailbox in a different mailboxdatabase than the database where the user mailbox are located. This finally gives the archive feature the benefit of being located on slow and cheap storage.
• Multi-Mailbox  search feature - improved
• Outlook Web Application - new features
• Active-Sync - new features
• New managmenet UI - extended commands

Read more on http://msexchangeteam.com/archive/2010/04/07/454533.aspx

Exchange 2010 performance monitoring/baselining - templates

For baselining or performance monitonring your Exchange 2010, you need to add alot of counters.
I've created the templates based on the http://technet.microsoft.com/en-us/library/dd335215.aspx which described the counters needed.

The following templates has been created:
- Common Server
- Mailbox server
- CAS server
- HUB server

These templates may be downloaded here:
http://cid-214999a8ebe4fcb4.skydrive.live.com/self.aspx/.Public/Exchange2010/Exchange%202010%20-%20PerformanceMonitoringTemplatest.zip

Regards,

Rune

Manually create FileWitness Share

When creating the DAG in Exchange 2010, Microsoft recommends that the FSW is located on a HUB server in the environment. One of the reasons is that these servers are the Exchange owners, and the Exchange admins have full control. It is recommended to place the FSW share in a third location or in the primary location.

To provide failover functionality to the FSW, you can use file cluster, Force cluster @File Share Witness and Force Quorum Guidance or the old and no longer recommended method of DNS CNAME.

At one of my customers, we have placed the FSW on a file cluster, since this gives us the full
redundancy in the case of a datacenter failure. The customer has two datacenters, but they are in a single site.

And Microsoft recommends you to add the Exchange Trusted Subsystem group in the local admin group of the server holding the FSW. But this might not be compliant to security rules. This is as far as I have understood just so the set-databaseavailability command can create the folder and shares.

To pre-create the FSW share you need the following:

- Create a folder etc. D:\FilesWitness\DAGNAME
- Give the owner permission to Exchange Trusted Subsystem
- Give the Exchange Trusted Subsystem Full Control (NTFS)
- Share the folder with the following DAGNAME.FQDN (If you try a different share name, it won't work. This is somehow required)
- Give the DAGNAME$ computeraccount Full Control (Share)

When you've done this, you can run the
set-databaseavailabilitygroup -witnessserver CLUSTERSERVER - witnessdirectory D:\Filewitness\DAGNAME

You'll get the following warning message:
WARNING: Specified witness server Cluster.fqdn is not an Exchange server, or part of the Exchange Servers security group.WARNING: Insufficient permission to access file shares on witness server Cluster.fqdn. Until this problem is corrected, the database availability group may be more vulnerable to failures. You can use the set-databaseavailabilitygroup cmdlet to try the operation again. Error: Access is denied

This is expected, since the cmdlet tries to create the folder and share, but don't have the permissions to do this.
When this is done, the FSW should be configured correct. To verify this, the following files should be created:

- VerifyShareWriteAccess- Witness

This should do the trick, and the FSW is ready to work, and follow the active datacenter.
There could be a slightly drawback with using a file cluster as FSW. If the file cluster gets in a split brain scenario, then the Exchange solution could get affected. A result of this could be two active database servers on the same database, resulting in corrupt data. But this is theoretically, and in special circumstances.

Rune

Exchange 2010 DAG datacenter failure

One of the neat features in Exchange 2010 is the DAG. This seems to have the most built-in features for solutions where the datacenters are spread across AD sites, but alot of companies out there don't have multiple AD sited.
One customer I'm at right now don't, and due to this we are trying out the datacenter failure scenario. And to get the solution up and running, my experience is that you have to do some manal steps, and its important to do this in the right order. This is descibed below.

One datacenter fails, and the Cluster does not get MajorityNodeSet.
Datacenter contains 2 MBX servers and 1 HUB/CAS witch is the FSW for the DAG.

To recover from this failure, this are the steps:
- Stop Cluster service on remaining DAG members I secondary datacenter
- On one DAG member do a net start clussvc /forcequorum
o In my case, the databases got mounted already here

To change the FSW on the DAG now will not work, since the DAG can’t communicate with the failed DAG servers.
To remove the affected DAG members and change FSW, you have to complete the following:
- Start cluster admin and evict the failed DAG member servers
- Remove all database replication
o Get-mailboxdatabasecopystsatus -server failedmbxserver
o Remove-mailboxdatabasecopy databasename\affectedMBX
- Remove-databaseavailabilitygroupserver DAGNAME –mailboxserver failedmbxserver

Now you will be able to change the FSW for the DAG.
- Set-databaseavailabilitygroup DAGNAME –witnessserver FQDN

Now everything should be cleaned and in order.

One thing I noted is when doing these steps in the wrong order, running the remove-databaseavailabilitygroupserver DAGNAME before evicting the nodes from the cluster.
The databases that had a copy to the failed MBXserver, got the following errors in their properties. This was a hell to clean up.

Get-mailboxdatabase databasename fl

Server : ActiveMBXServer
MasterServerOrAvailabilityGroup : FAILEDMBXServer
MasterType : Server

To cleanup this, I had to rejoin the failed MBXservers to the DAG,and enable database replication.
Even though the replication seemed to work fine, it didn’t. It wasn’t possible to switch the active database between the servers in the DAG due to the properties on the database.
Then remove the old failed MBXserver from the DAG to a single MBX server with the existing database. Then rejoin the MBXserver to the DAG, and then the properties was OK.

When a database is member of the DAG, the correct way this should be is:

Server : ActiveMBXServer
MasterServerOrAvailabilityGroup : DAGNAME
MasterType : DatabaseAvailabilityGroup

This is my experience with the datacenter failure, but I'll post more later.
Rune