Exchange Server: Connector And Performance Issues?

Where and how are connector restrictions configured? In Microsoft Exchange 2000/2003 Server, connector restrictions are configured on the connector's Delivery Restrictions tab. The restriction checking functionality is controlled by a registry key that is set on the Exchange 2000/2003 bridgehead that is the source for the connector that is being checked. If you need to […]

Where and how are connector restrictions configured? In Microsoft Exchange 2000/2003 Server, connector restrictions are configured on the connector's Delivery Restrictions tab. The restriction checking functionality is controlled by a registry key that is set on the Exchange 2000/2003 bridgehead that is the source for the connector that is being checked. If you need to apply a restriction to a connector which restricts who is able to send data to the designated link or connector, you must manually add the restriction checking registry value. So after you configure the Delivery Restrictions option on a connector, the CheckConnectorRestrictions regkey must also be configured for restrictions to work.

Please see the following article for more information:

277872 XCON: Connector Delivery Restrictions May Not Work Correctly

If you configure the delivery restrictions on a connector by using Microsoft Exchange 2000 Server or Microsoft Exchange Server 2003 and forget to add the registry key, the restriction settings will not be applied. Additionally, the following event ID will also be generated:

Type: Warning
Event ID: 957
Source: MSExchangeTransport
Category: Routing Engine/Service
Description:
Connector restrictions (by the group or by the user) are present in the organization. However, restriction checking is disabled. Set the registry value HKLM\SYSTEM\CurrentControlSet\Services\RESvc\Parameters\CheckConnectorRestrictions to 1 (DWORD) and restart resvc and smtpsvc to enable restriction checking on local machine

Connector restriction checking is turned off by default because it can significantly affect performance to expand distribution groups and check the restrictions for each message that passes through the system. If needed, turn on this setting only where it is necessary (for example on the bridgehead server for the restricted connector).

Symptoms of performance problems

  • Mail delivery is extremely slow on an Exchange 2000 Server or on an Exchange Server 2003.
  • Mail is backed up in "Messages Awaiting Directory Lookup" (MADL) queue and / or "Messages Waiting to be Routed" Queue. Messages may back up in either queue or both queues and it may even cause backups in Pre Submission queue and / or Local Delivery Queue depending on the load on the server.

Cause of performance impact: This problem may occur if you configure connector restrictions to reject or accept messages based on distribution group, universal security group membership or even based on individual users. For example, on the SMTP Connector's Delivery Restrictions tab, you have configured "By default, messages from everyone are" Rejected and then you have added Distribution Groups or users under "Accept messages from" list.

When a sender sends an e-mail message through the connector that is configured with a restriction that rejects or accepts messages based on membership of a particular distribution group or security group, Exchange 2000/2003 Server must expand that group to make sure that the sender is not a member of the restricted group i.e. to determine whether each user account is permitted to send the e-mail message. The results of this group expansion are not cached by Exchange Server and must be performed each time. If you send a message to a group that contains many recipients, and if each of those recipients is also configured with a delivery restriction to reject messages from the members of a distribution group that contains many members, Exchange 2000/2003 Server must expand the restricted distribution group one time for each member of the group to which you sent the message. Also, if a failure that can be retried occurs during this process, Exchange Server stops the group expansion process, and then retries the connection an hour later. This causes the messages to be held in the categorizer queues, delays message processing, puts load on transports' Advanced Queuing and SMTP components, and eventually causes system queues to start backing up.

This problem may also occur if you configure delivery restrictions on the user or group to reject messages based on distribution group or universal security group membership. For example, you click "From everyone except" under Message restrictions on the Exchange General tab of the user account or distribution group properties, and then add a distribution group to the exception list.

Solution: The Microsoft recommended solution is to use flat DGs in combination with RestrictionMethod registry key and hotfix outlined in KB895407. Note that this solution is only effective if:

1. The hotfix and registry key are applied, and

2. All DG's used in the restriction must be flat. If they are not then the nested groups will not be used in the restriction checking logic.

RestrictionMethod regkey in conjunction with hotfix greatly enhances Connector Restriction checking performance. This regkey / solution helps with and changes the way restrictions are checked both per-recipient and per-connector (it affects both categorizer and routing restriction checking performance).

This solution is ONLY for Exchange Server 2003 SP2 servers or Exchange 2003 post SP1 servers. For Exchange 2000 Servers, every possible effort should be made to upgrade to SP2. At the bare minimum, the server which is doing connector restrictions should be upgraded to Exchange Server 2003 SP2. This is the preferred and recommended solution even for Exchange Server 2000 due to the extremely positive performance gain for connector restrictions checking capabilities of Exchange Server 2003 SP2.

Note: A flat DL is a DL which has only users as direct members of it. i.e there are no sub-DLs as members of the flat DL.

Workarounds: Like mentioned above, every possible effort and emphasizes should be made that RestrictionMethod regkey with Flat DL solution is the recommended resolution for this problem. However, you may have valid reasons not to apply this solution on Exchange 2003 Servers, and may not want to upgrade the Exchange 2000 Server to Exchange 2003 Server. In this situation, we have several workarounds / solutions available, and we are presenting all options here to help resolve this issue. Please keep in mind there is not a single solution below which is preferred or is recommended over others. Each Exchange environment is unique and different; a solution which works best for one may not be feasible for another. You should understand the pros and cons of each possible workaround, based on a good understanding of your environment, and then decide upon which solution / workaround to implement. Below is the list of possible workaround/solutions, which apply to both Exchange 2000 and Exchange 2003 Servers.

1. Design a dedicated Routing Group (RG) for Connector Restrictions with Routing Group Connector (RGC) scoped to local Routing Group.

Please see this KB for more details:

329171 XADM: Mail Delivery Is Slow if Recipients Are Configured with Delivery Restrictions Based on Group Membership

2. Use dedicated, powerful Exchange Servers with dedicated and powerful GC Servers.

This configuration would help keep the connector restriction related load on the designated Exchange and GC servers. A general guidance cannot be given here as it all depends on many factors including sizes of DG, number of DGs configured for restriction, how often messages are sent thru the restriction checking connector, network speed and network load, etc. You should size your servers accordingly based on your needs.

3. Configure individual Users and Not DLs for restrictions (up to 1000 per connector).

Please see the following article (Method 2):

812298 Mail delivery is slow after you configure delivery restrictions that are based on a distribution list

Note: This workaround adds to Routing packet bloat, so if you have a large company, especially with Hub and Spoke topology, this might lead to more bandwidth use for routing packet exchanges.

4. Limit the number of servers that you enable restriction checking on to just the source BH servers if there are no alternate routes.

Please see the following article (Method 1):

812298 Mail delivery is slow after you configure delivery restrictions that are based on a distribution list

A variation of this workaround is simplified restrictions implemented ONLY at the HUB routing group.

The positives of this workaround are that there is no impact at the remote RGs since they do not have connector restrictions configured. Additionally, we only have one distribution group to look up (restricted senders), and it also provides centralized administration.

The negatives includes there is additional overhead at the HUB site, and it doesn't appear to improve performance to needed levels and may require an additional hardware investment in HUB site to implement effectively.

5. SMTP Sender Filtering: Another solution for customers that want to block internet email for a large group to the internet by using DG based connector restrictions is to use Sender Filtering. With the release of IMF we can block by sender filtering starting with Exchange 2003 SP1. If you are using Exchange 2003 BH servers, then we can setup Sender filtering on the BH servers only.

This works at the SMTP protocol layer and has a lot less overhead. What we can do is setup all users that are prohibited from sending to the internet to have a primary STMP address of user@domain.loc. Then we can block senders with the email addresses of "@domain.loc" from talking to the Internet SMTP BH servers using Sender Filtering in Exchange 2003 SP1 or above. This will not disrupt any internal email, as long as the Internet BHs are only used for Internet connectivity. If any non-internet email from the blocked users must pass through these servers then it would be prevent as well.

Exchange 2003 SP1 will block the restricted sender with 554 5.1.0 Sender Denied. The message will be NDR back to the sender with the same error in the NDR.

The pro of this approach is that it doesn't utilize connector restrictions, thus performance is much improved.

However, a large customer may feel this is impractical because of recipient policies and RUS updates; no way to enforce a new restriction without issuing an 'Apply Policy Now'.

6. Custom event sink using a custom attribute: A custom event sink\solution can also be used to address this issue. Several customers has been using it successfully. Basically the customer can have a bit on the user object (using a custom attribute) and write the sink to check this attribute. Many customers have done it themselves (using just vb and introducing caching with a 5 minute timeout).

The advantage of this workaround is that it doesn't utilize connector restrictions, thus performance is much improved. However, disadvantage is that it would require substantial redesign and testing. It is not "out of the box" and would put customer into a semi-unsupported configuration, and an additional investment in development/maintenance of code and additional administration is needed.

7. Custom event sink querying a local text file: This is similar to the sink solution mentioned in #6 above.  However in this case event sink/custom code does not query AD at all (this might be an additional advantage), instead it queries a local text file. This solution has been used with success by some of our customers, before the Exchange 2003 hotfix came out, and it may be a viable solution if you don't want to maintain separate flat DLs for restrictions.

How does all this apply to Exchange 2007? In Exchange 2007, this performance issue due to connector restrictions is resolved as there are no connector restrictions.  Restrictions are not placed on connectors as was done in Exchange 2000/2003. Restrictions can now be applied in the core transport via Transports Rules. Connector restrictions as such are completely gone (since the concept of internal connectors is pretty much gone) and it's up to transport rules to enforce restrictions in Exchange 2007.

Exchange Team Blog

Microsoft, Exchange Server, Connector, Performance, Knowledgebase