Written by Support InfoBridge
When using the Exchange Online Synchronizer, you might end up with a crashing service and error messages in the logfile & Windows Eventlog.
The message you will notice is: Unable to receive folder notification events, Exchange server down?
This indicates that the Synchronizer is trying to open more connections than Exchange / Office 365 is allowing. This is called throttling.
The preferred method would be to create a custom throttling policy without limitations. If your company allows for this then follow this KB article: How to disable Exchange Throttling
Please note that Office 365 is not included in that article. The reason behind this: Office 365 has no options or possibilities to influence the number of connections to Office 365. It is a set value, locked on 20 connections per user.
Exchange Server Down?
The ‘Exchange Server Down?’ message is usually caused by Exchange throttling. This article provides some background information, explanation of new settings and recommendations.
In the past years a lot of customers have switched from the Exchange Synchronizer v2 to the Exchange Online Synchronizer, which is now the recommended product for server based synchronization for Exchange 2007 SP1 and up. The new product no longer uses Extended MAPI to communicate with Exchange but uses Exchange Web Services, or EWS for short. This switch had several benefits:
- No more ‘ICS file corruption’;
- Future proof;
- Support for Office 365 (Exchange Online) and Hosted Exchange.
However, by using EWS a new factor came into play: client throttling policies.
Client throttling policies were introduced by Microsoft in Exchange to ensure a good performance of the Exchange server by limiting the bandwidth and resources clients can use. Each client has a budget for the amount of resources it can use and once the client exceeds this budget, throttling kicks in. The server will first start to delay the requests coming from the client which, in case of the Synchronizer, will slow down synchronization. If the client continues to breach the throttling limit the server will eventually block all incoming request for the client for a while. For an external application like the Synchronizer it would seem that the server is not available anymore, hence the message ‘Exchange Server Down?’.
You can imagine that the Synchronizer easily breaks the throttling limits as all requests are executed by the ‘syncuser’. The limits are designed with a single user accessing it’s mailbox in mind, but the Synchronizer might access dozens of mailboxes every second, depending on the number of users that are linked and the amount of changes made by those users.
To make things worse: In Exchange 2010 SP1 and up throttling policies are enabled by default.
The previous solution
The solution for this problem up until now was fairly simple and straightforward: create a new throttling policy that has no limits at all and assign this policy to the ‘syncuser’. This method is described in the InfoBridge knowledgebase in this article.
Whilst this solution works in most cases there are several problems with this method:
- Some (usually larger) organisations do not allow custom throttling policies;
- Most Hosted Exchange providers do not allow customers to either change throttling policies or access Exchange Management Shell at all;
- Office 365 (Exchange Online) currently does not support configuration of throttling policies.
The new solution
In the Exchange Online Synchronizer three new options were introduced which should minimize throttling issues.
However, keep in mind that each option is designed with a specific area / cause of throttling in mind so just setting all options is not recommended: each option will have an impact on the performance of the Synchronizer.
The three options are added to the technical settings screen of the Synchronizer Admin:
* Optimize for High Server Load environments
This option enables response time monitoring of EWS requests. When the (weighted moving) average of the requests is larger than 5 seconds the Synchronizer will throttle its requests to the Exchange server by adding small delays.
* Use Incremental Change Synchronization Detection only
With this option the Synchronizer will only use ICS to detect changes. Each user will be checked (in sequential order) for changes through ICS. In between each user there is a delay of 3 seconds.
* Limit number of concurrent server requests
The Synchronizer will execute requests simultaneously up to the maximum specified. Additional requests will be queued when the maximum is reached.
The value must be equal to the value of ‘EWSMaxConcurrency’ in the throttling policy of the sync user. That value is by default set to 20 and this is also the default value in the Synchronizer.
Setting it to 0 in the Synchronizer will mean that there is no limit. This corresponds to the behaviour of synchronizer versions prior to 3.2.141.
To get the ‘EWSMaxConcurrency’’ for the policy of the ‘syncuser’, execute the following Exchange Management Shell commands:
- Get-ThrottlingPolicyAssociation “domain\syncuser”
- Remember the ‘ThrottlingPolicyId’
- get-throttlingpolicy ENTER_POLICY_ID_HERE
- Check the value ‘EWSMaxConcurrency’
‘Optimize for High Server Load environments’ and ‘Limit number of concurrent server requests’ (when set to the default value) have the smallest impact on the user experience. The Synchronizer will just take a bit longer to synchronize items from Exchange to SuperOffice and from SuperOffice to Exchange.
‘Use Incremental Change Synchronization Detection only’ has the biggest impact on the synchronization speed. This is due to the fact that new items will be synchronized through ICS instead of through events from Exchange. The other way around, changes from SuperOffice, is unaffected though.
We recommend turning on ‘Use Incremental Change Detection only’ in synchronizers connected to Office 365 environments. Also make sure that the 'Limit number of concurrent requests' is set to 20 (this is the default setting in both the Synchronizer and the default Office 365 throttling policy).
The new options will definitely solve a lot of throttling related issues. However, if a customer wants very fast synchronization and has the capacity on the Exchange Server for this, we recommend turning all throttling options in the Synchronizer off, setting ‘Limit number of concurrent server requests’ to 0 and making a throttling policy for the ‘syncuser’ with all throttling parameters set to null as described in this knowledge base article.