Introduction

The purpose of this article is to provide high level overview of hierarchy synchronization process in public folders and provide some troubleshooting steps to monitor and troubleshoot hierarchy related issues. The information presented here applies to public folder deployment in Exchange on-premises as well as Exchange Online, with any differences called out.

Before we get started, these are some of the terms used in public folder world:

Public folder hierarchy sync

In simple terms, the public folder hierarchy is the skeleton or logical structure in which the permissions and other properties of public folders are arranged. The hierarchy makes it possible for clients to determine which PF mailbox stores the actual content of the public folder. The hierarchy is synchronized across all public folder mailboxes, to ensure a single PF mailbox is not burdened with serving the hierarchy to all clients.

Frequent synchronization of the hierarchy is critical to ensure end users always get an up to date view of the hierarchy.

Here’s what can happen when the hierarchy is not in sync. Public folder mailbox EPF1 is the primary hierarchy mailbox and has two folders in the hierarchy. A new PF mailbox EPF2 is created. EPF2 won’t show the complete public folder structure until the hierarchy is synced:

Let’s discuss how the public folder synchronization works and the components involved.

Pull mode

In pull mode, hierarchy replication is driven by secondary hierarchy mailboxes. At regular interval, the secondary hierarchy mailbox connects to the primary hierarchy mailbox to pull the changes in hierarchy. This process is accomplished with the help of the ServiceHost process (running on each server) and the MRS proxy service (running on the server hosting the primary mailbox). The frequency of connection depends on whether clients were connected to the mailbox or not.

This workflow is executed with the following frequencies:

Additionally, the following situations also trigger sync operations between a secondary mailbox and primary:

1. A user connected to a secondary mailbox performs a hierarchy update / change: The operation is performed in the primary hierarchy mailbox and then immediately replicated to the PF mailbox that is serving hierarchy for the user who made the change.
2. An update is done in a folder whose content is hosted in a mailbox other than the primary mailbox: The operation is first applied in primary (all write hierarchy client operations get proxied there) and then replicated immediately to the mailbox hosting the affected folder.
3. An admin user invokes the Update-PublicFolderMailbox with the InvokeSynchronizer parameter.
4. A new mailbox is created by Migration or AutoSplit (in Exchange Online only): These processes will cause a full hierarchy sync to be executed in the new mailbox.

The process described above was the default model for full as well as incremental hierarchy synchronization in the early days of modern public folders and served well as long as the number of public folder mailboxes was not high. However, there were some limitations with this model:

• Secondary mailboxes had no way to know if there had been a change in hierarchy. In other words, secondary mailboxes would keep connecting to the primary at defined intervals, even if there was no change in hierarchy.
• More secondary mailboxes meant more sync jobs, thus more connections competing for the limited resources in the primary mailbox.
• Larger hierarchies mean more data during sync (larger sync states and more folders to transfer) which meant longer execution times and higher resource consumption (memory, IO, CPU and network).
• As the size of the user base increases it means more hierarchy mailboxes (and consequently a higher number and frequency of sync jobs) and this usually leads to more hierarchy churn (thus requiring longer sync jobs as more data is processed).

To address these problems, a new “push mode” of hierarchy sync was introduced.

Push mode

The push mode of hierarchy sync is driven by primary PF mailbox, mainly for the incremental synchronization of hierarchy. This change was introduced in Exchange Server 2016 CU2 on-premises and in Exchange Online around the same time.

In push mode, the primary PF mailbox proactively:

• Notifies secondary mailboxes when there have been changes.
• Sends any data the secondary mailbox may need to apply the recent changes, limiting the need for it to contact primary.

Pull mode is still used for:

• Initial sync of newly created PF mailboxes
• Full sync requests triggered by an administrator
• Fallback mechanism in case of push notification being lost, or when mailboxes become out of sync

In push mode, the primary PF mailbox drives the incremental hierarchy sync jobs, using transport to send email messages to a hidden distribution list containing the different public folder mailboxes present in the hierarchy.

Some key properties of this distribution list:

• It is a hidden Dynamic DL.
• It has ShowInAddressList set to false.
• Its identity will be persisted in the organization object so mailboxes can refer to it.
• You should normally find only one such dynamic distribution group created in an Exchange environment.

The hidden DL created for sending push notification messages cannot be seen in Exchange Online but you can use the following command to view the hidden DL in Exchange on-premises:

Get-DynamicDistributionGroup -IncludeSystemObjects pub*

Secondary mailboxes will use both modes of sync, pull mode for initial full sync and push mode for incremental hierarchy sync, depending on the sync state they are in. The first sync is performed using pull mode, after that, they will not contact primary mailbox repeatedly for incremental syncs. Instead, they wait for the hierarchy sync notification email and apply the hierarchy changes once the email is received. If the secondary hierarchy mailbox doesn’t receive a push notification message within 10 minutes, it will fall back to pull mode and perform the incremental sync by contacting the Primary mailbox.

Secondary mailboxes will also use the pull mode if an administrator triggers hierarchy sync.

Hierarchy mailbox assignment

Exchange by default automatically assigns a default public folder mailbox on all user mailboxes, load balancing between all available and assignable public folder mailboxes.

A public folder mailbox having following properties is considered available for automatic assignment. In the following example, both public folder mailboxes are considered available for automatic assignment:

An admin can exclude specific PF mailbox from being automatically assigned by configuring IsExcludedFromServingHierarchy to $True

Example:

Set-Mailbox -PublicFolder epf1 -IsExcludedFromServingHierarchy $True

An admin can override the system assignment by using this command:

Set-Mailbox <username> -DefaultPublicFolderMailbox <PFMailboxName>

Example:

Set-Mailbox cloud1 -DefaultPublicFolderMailbox epf1

The public folder mailbox serving hierarchy for the user can be found using this command:

Get-Mailbox |ft name,DefaultPublicFolderMailbox,EffectivePublicFolderMailbox

The admin assigned mailboxes appear under “DefaultPublicFolderMailbox”; whereas the system assigned PF mailbox will be displayed under “EffectivePublicFolderMailbox” and DefaultPublicFolderMailbox will be blank.

For larger deployments, avoid manually assigning DefaultPublicFolderMailbox on users, as it may overload the assigned PF mailbox with lots of concurrent connections. As we learned earlier, the system assigns DefaultPublicFolderMailbox on users in such a way that connections are load balanced between public folder mailboxes available for serving hierarchy. Follow the best practices guidelines here for public folder mailbox placement and assignment.

Making sure hierarchy is healthy

Here are some tips and tricks to help keep your public folder hierarchy in top shape.

Full hierarchy sync

Admins can use the following command to ensure that the public folder mailbox has received the initial full sync of hierarchy and confirm that the PF mailbox is ready to serve hierarchy. The value of True returned here indicates the mailbox has received its first full sync of hierarchy

Get-Mailbox -PublicFolder |ft Name,IsHierarchyReady

Diagnosing hierarchy sync issues

If you suspect a specific PF mailbox is not receiving hierarchy (perhaps users tell you they can’t see a folder they know to exist), you can use the following command to get diagnostic information about hierarchy:

Get-PublicFolderMailboxDiagnostics <pfmailboxname_notreceiving_hierarchy> -IncludeHierarchyInfo

Here are some other ways to view data;

To compare the hierarchy between PF mailboxes

$P=Get-PublicFolderMailboxDiagnostics <Primary_pfmailboxname> -IncludeHierarchyInfo
$S=Get-PublicFolderMailboxDiagnostics <pfmailboxname_notreceiving_hierarchy> -IncludeHierarchyInfo

Then, you can check the output of “HierarchyInfo” from both the mailboxes and compare:

$p.HierarchyInfo
$s.HierarchyInfo

If you find the hierarchy information is not the same, you can use following command to view the time of the last sync:

$S.AssistantInfo.LastAttemptedSyncTime.LocalTime

This command tells you the last time sync failed; a gibberish value (that’s a technical term now!) indicates sync has never failed:

$s.AssistantInfo.LastFailedSyncTime.LocalTime

The following command will give you a detailed failure message from the last sync failure, a blank output indicates sync has never failed:

$s.AssistantInfo.LastSyncFailure

You can explore the other values of AssistantInfo, SyncInfo & HierarchyInfo blocks.

In case the need arises to contact Microsoft Support for assistance, you should export the report to XML format and send it along.

Get-PublicFolderMailboxDiagnostics <pf mailbox failing to sync> -IncludeHierarchyInfo |Export-Clixml epf2.xml

Other useful information

The following are some things to know related to the hierarchy sync of public folders:

Push notification emails and journaling

Since push notifications are an email message delivered using transport they will be included in journaling if enabled in an organization. In such a scenario, you can use scoped journaling and exclude the SMTP address of primary public folder mailbox from the journaling rule (as that is a sender of push notification messages).

Public folder mailbox is a top sender

You may observe that the public folder mailbox shows in the “Top Sender and recipient report” in the Security & Compliance Center.

This is currently normal and not a sign of a problem, because the primary public folder mailbox sends a large number of push notification emails to secondary mailboxes via transport. The next version of the report will exclude the public folder hierarchy sync emails.

Multiple dynamic DL’s created

As discussed earlier, push notification emails are sent to an Exchange-managed dynamic distribution group. Normally, only one dynamic distribution group is enough for the push notification emails. However, in environments that have multiple domain controllers within a single AD site with AD replication issues between the DCs, multiple dynamic distribution groups may get created unexpectedly. The issue happens because of a race condition and time lag between AD replication between DC’s in same site. Please follow steps in this KB to resolve this.

Conclusion

We hope this gives you better understanding of modern public folder hierarchy and some tools you can use to perform the troubleshooting. We would love to hear any feedback you may have on the topic.

The Public Folder Team

As you are aware, we released the Microsoft Hybrid Agent for Exchange Server as a preview on Feb. 5^th, 2019. The blog post for that release is located here.

We have been busy working on a few improvements for the Hybrid Agent over that past two months and wanted to let you know of some new features now available in the latest build of the Hybrid Configuration Wizard.

The three enhancements for the Hybrid Agent are:

1. Multi Agent installation and configuration
2. Agent status views
3. Registration/Usage of a load balancer for the internal URL instead of a specific Exchange Client Access Server

Multi Agent Deployment

Option 1 – Use the Hybrid Configuration Wizard to install additional agents

For existing Hybrid Agent preview customers who would like to install additional Hybrid Agents for redundancy, simply download the latest version of the Hybrid Configuration Wizard (HCW) and open the application on the machine where you would like to install an additional Hybrid Agent.

1. Like previous HCW runs, start the application, select Next.
2. Select a desired server to execute against, select Next.
3. Provide credentials to sign into your Office 365 tenant, select Next.
4. The HCW will gather configuration information, select Next when complete.
5. Select the default option provided for either Full or Minimal, select Next.
6. Select Exchange Modern Hybrid Topology, Next.
7. Agree to the Terms, Next.
8. A new page will be shown that will provide you with the status of your existing / previously installed agent. Make sure the status of the existing agent is accurate before proceeding to the next step. Select Install and additional agent option, Next.

Example:

The HCW will install the additional Hybrid Agent. When the installation is complete, you can open the Microsoft Windows Services console from the machine and verify the service / agent is installed and running (look for Microsoft Hybrid Service - mshybridsvc). At that point, you can either re-run HCW if you wish to make further changes to your hybrid config, or simply cancel the wizard.

You can repeat this step on each machine where you would like an additional Hybrid Agent installed.

Option 2 – Manually download & install additional agents

A second option for installing additional agents is outside the HCW itself and is done by downloading and manually installing the agent on the desired machine.

1. Go to https://aka.ms/hybridagentinstaller.
2. Save the MSHybridService.msi to a location on your machine.
3. From that machine, open a Windows Command console as Administrator and run:
4. Msiexec /i MSHybridService.msi to install the hybrid agent. You will be prompted for your tenant Global Admin credentials.
5. After the installation is complete, you can open the Microsoft Windows Services console from the machine and verify the service / agent is installed and running.

You can repeat this step on each machine where you would like an additional Hybrid Agent installed.

Checking the Status of Your Hybrid Agents

Option 1 – Get status via the Hybrid Configuration Wizard (HCW)

1. Start the HCW application, select Next.
2. Select a server in your Exchange org, select Next.
3. Provide credentials to sign into your Office 365 tenant, select Next.
4. The HCW will gather configuration information, select Next when complete.
5. Select default option provided for either Full or Minimal, select Next.
6. Select Exchange Modern Hybrid Topology, Next.
7. Agree to the Terms, Next.
8. A new page will be shown that will provide you with the status of your existing installed agents.

Cancel the HCW app when you are done.

Option 2 – Get status via the Hybrid Management PowerShell Module

With each installation of the Hybrid Agent, we install the Hybrid Management PowerShell module in the directory …\Program Files\Microsoft Hybrid Service\ on the machine where the agent is installed. By default, this module is not imported and so you will need to import it before you can use it. This module also requires the Azure module for PowerShell if not already installed. Please install the PackageManagement modules first and then see this article for how to install the Azure module.

To import the Hybrid Management module, run the following from a Windows PowerShell prompt as Administrator:

Import-module .\HybridManagement.psm1

After that you can run:

Get-HybridAgent -credential (get-credential) to view agent status:

Note: the ‘id’ value provided in the above snip is the agent identity and not your unique tenant guid assigned to the route.

Directing your Hybrid Agent(s) to the load balancer instead of a specific server

If you would like your Hybrid Agent(s) to direct requests to your load balancer instead of a specific Exchange Client Access Server, this can be done with the Hybrid Management PowerShell module. In preview, the Hybrid Agent supports routing requests to the load balancer for Exchange Server 2013 – 2019 Client Access Servers. You cannot use this if you only have Exchange Server 2010 CAS deployed.

Follow the steps above to import the Hybrid Management module for PowerShell.

From PowerShell, then simply use the Update-HybridApplication cmdlet to change the value of the internalURL (targetURI parameter) from a specific server to your load balancer endpoint.

Before running this cmdlet you will need to get the unique guid value (e.g. 12345678-1234-1234-1234-1111111111111) of your tenant’s endpoint from either your MRS configuration or the Organization Relationship object (TargetSharingEPR). Example:

PS C:\PowerShell> Get-OrganizationRelationship ((Get-OnPremisesOrganization).OrganizationRelationship) | Select-Object TargetSharingEpr
TargetSharingEpr
----------------
https://e399b770-3b66-407b-a71a-96ef80a67714.resource.mailboxmigration.his.msappproxy.net/EWS/Exchange.asmx

Or

PS C:\PowerShell> Get-MigrationEndpoint "Hybrid Migration Endpoint - EWS (Default Web Site)" | Select-Object RemoteServer
RemoteServer
------------
e399b770-3b66-407b-a71a-96ef80a67714.resource.mailboxmigration.his.msappproxy.net

To be clear, the ID parameter required in this cmdlet is not the agent ID, but the guid listed in your custom registered endpoint retrieved using the method above.

Example:

We really hope you are enjoying and benefitting from using the Hybrid Agent. We see a lot of usage and we’re working hard to improve it all the time.

Thank you!

Exchange Hybrid Team

Several weeks ago we added a new Microsoft 365 Roadmap item announcing our intent to add ability to migrate Google G Suite calendars and contacts to the ability to migrate mail to Office 365 using our native migration tools.  We're excited to say that this functionality has started rolling out!  You can expect to see the new features light up for your tenant in the coming weeks.

Since this is a new flavor of migration to Office 365, let's take a look at what is now becoming available and answer some frequently asked questions about it.  Before diving into any migration project, it is important to answer some basic questions about it:

1. What is the size of the organization you are trying to migrate?
2. Based on your business requirements, would you like to migrate using a "cutover" (everyone at once) or "staged" (longer-term coexistence) approach?
3. What is the (approximate) average mailbox size that you'd be migrating?
4. What kind of data do you need to migrate - like mail, calendar and contacts?
5. Given the answers to the above questions, does this migration tool fit my needs?

Answering these questions will give you some idea of the scope of your project. Then you just need to find out how to get started. And you can read all about that here.

Before you actually start clicking though, let's cover some of the terms you'll need to understand and an overview of how the new G Suite migration flow will work:

Intro to Mailbox Replication Service (MRS)

Microsoft Exchange Mailbox Replication Service (MRS) is a component that handles mailbox import, export, migration and restoration for Exchange and Office 365.  Migrations are managed using individual requests.  MRS is the principal mechanism used to move mailbox data from one place to another.  In context of moving data from G Suite to Office 365 MRS is used to move mailbox data, including messages, contacts, and calendar items.  Each mailbox being migrated from G Suite to Office 365 has its own request that will be processed by MRS.

What is a migration batch?

For better usability and scheduling at scale, another component called the Migration Service provides the ability to submit migration requests for batches of users. Behind the scenes, the Migration Service uses the Mailbox Replication Service to manage the per-mailbox requests.  Grouping a set of users into a batch is primarily for management purposes and does not impact the speed or throughput of your migrations based on batch size.  For your migration, you could choose to have one batch of all users or split the users into multiple batches - the choice is yours.

Give me an overview; how do migration batches work?

How is mail data migrated?

Mail data is currently migrated using IMAP protocol.  Authentication happens using domain-wide delegated access using a Service Account that is under your control.

How fast can I migrate my data?

For mail data there is a throughput limitation of 2 GB per mailbox per day. This limit is enforced by G Suite.

For contacts and calendar data, this completely depends on the quota restrictions for your tenant's service account on the Google G Suite side (because a different protocol is used to migrate calendars and contacts; please see documentation).

I reached my 2GB limit for the day (for mail) - will the migration continue the day after?

Yes - it will automatically continue once there is capacity to migrate more data, until the 2GB/day quota is reached.

What data will NOT be migrated over?

• Mail: Vacation Settings or Automatic reply settings as well as Filters or Rules
• Meeting Rooms: Room bookings
• Calendar: Shared calendars, cloud attachments, Google Hangout links and event colors
• Contacts: A maximum of three email addresses per contact are migrated over
• Contacts: Gmail tags, contact URLs and custom tags are not migrated over

Will this also support migrating data from Google Vault to Office 365?

No.

What is the smallest batch size and is there an optimal batch size?

A migration batch can have a single user, with no current upper limit.  There is no magic number for the best migration speed or efficiency.  We recommend creating batches per-department or another logical grouping for your organization.

What is the size of the largest email I can migrate from G Suite to Office 365?

The limitation is based on the transport configuration for your organization.  The default limit is 35MB. Please see this article on how this limit can be increased.

I tried migrating and saw mention of bad items; what are those?

Bad items are data conversion failures that may be encountered during the syncing phase.  Should you encounter any bad items, you can see these in the migration report on the user and batch levels.

How do I track progress of migrations?

You can view the progress and reports via the Exchange Admin Center or use the Get-MigrationBatch PowerShell cmdlet.

Mail on Office 365 doesn't support labels.  How is mail translated to folders?

In order to provide the best mail experience in Outlook as well as account for the Focused/Other view, we translate most labels to folders.  Mail with the 'Starred' label will become flagged.  Mail with the 'Important' label will influence its Focused/Other designation.  Mail with other labels will be copied into a folder with the corresponding label name, and mail with no other labels will be moved to the Archive folder.  This means that some items that had multiple flags in G Suite will appear in multiple folders in Office 365.

I started off a batch in EAC, why doesn't my migration make progress?

Please make sure that you went through all of the pre-requisite steps on the G Suite side and have provisioned users on the O365 side.  Also check the per-user error messages if you see users with a status of 'Failed'.

My messages have completed copying but my migration is stuck at 95%. Help!

This could be expected.  Check the status of migration batch; if it reached the 'Synced' status, you can choose to complete the migration batch.

What does completion do?

For G Suite migrations, there's an option to complete a batch.  During completion, an additional incremental sync of data is performed, followed by a "switchover".  During switchover, a forwarding address is applied to the source G Suite mailbox to forward emails to the configured Target Delivery Domain.  Also, any forwarding address applied to the target O365 user is removed.

Do I really need to "Complete" my batch?

Depends - If you're taking the cutover migration route, you don't need to complete your batch(es). If you're taking the staged migration route (i.e., migrating a subset of users sequentially over a longer period of time), you'd want to complete certain batches if you'd like the automatic mail forwarding rules to reverse (Forward email from Gmail to Exchange Online instead of Exchange Online to Gmail).

How can I tell if this feature is available in my tenant?

As an Admin, you will see the G Suite (Gmail) migration option under the Migration tab in the Exchange Admin Center (EAC) or will have access to the -Gmail parameter for New-MigrationBatch CMDlet.

We really hope you enjoy using this new set of tools and it makes your migration from G Suite to Office 365 even easier. We really want to hear your feedback, so please do leave us a comment, and feel free to ask any questions here too. We have plans to make this migration toolset even better over time, so keep an eye on the blog.

The Exchange Migration Team

As you might or might not know, in Exchange Online we have an upper limit of 100 migration batches. You can see this limit for MaxNumberOfBatches in the Exchange Online PowerShell Get-MigrationConfig:

(To understand the MaxConcurrentMigrations limit of 300, you can check this blog post written by one of our migration experts, Brad Hughes.)

100 batches created at the same time should be enough for all O365 multi-tenant customers. But, as we have seen in support, sometimes there are support tickets opened because customers would see the error that says: “The maximum number of migration batches is already running. Please remove a batch before you add another one”.

The main reason why this number of batches limit is in place is that having more than 100 batches causes performance problems and could cause outages for other customers in the multi-tenant system. Additional reading on resourcing for mailbox moves can be found here.

Let’s say that you want to migrate 50,000 mailboxes to Exchange Online: you could create 50 batches with 1000 users each, and you would still have 50 batches available.

If you need to complete all the migrations of users from Batch_1 at the same time, you can easily set -CompleteAfter at the migration batch level. Once this batch is completed, you could delete it to create room for another one.

In real world, though, we have seen that our customers rarely want to complete large amount of mailbox migrations at the same time. This is usually the reason why large migration batches are not created but rather smaller batches are preferred. That way, all the migrated users from one migration batch at a time can be completed together.

In this scenario, our recommendation is to have large batches for initial sync and if you need to complete the migration for users at a separate date, then you would complete the migration for groups of users in smaller batches, as needed. So how would you create small completion batches for already synced users from larger synced batches?

Suppose you want to complete migration of users A, B and C, who reached Synced status and they are all contained in a large batch of 1000 users called Batch_1.

To do this, first ensure that you have fewer than 100 batches created at the moment:

(Get-MigrationBatch).count

Then, proceed with PowerShell commands to get the users from the larger batch Batch_1 into the new smaller batch CompletionABC. Then complete this smaller batch and optionally, remove the completed migration batch:

New-MigrationBatch -Name CompletionABC -UserIds EmailAddressOfUserA, EmailAddressOfUserB, EmailAddressOfUserC
Complete-MigrationBatch CompletionABC
Remove-MigrationBatch CompletionABC

Another method we sometimes see people using is setting -CompleteAfter on the move requests directly using Set-MoveRequest.  We don't recommend this method because the Migration Service may overwrite the per-request CompleteAfter setting with the one from the batch without notice. This method of individually completing the move requests should only be done when you’ve started the migration through PowerShell, using New-MoveRequest cmdlet, without creating migration batches to contain those move requests.

I hope this post gives you a better idea of how to manage migration batches with many users being migrated and not have to run into the limit of number of batches.

See you in the cloud!

Special thanks to Brad Hughes and Nino Bilic who contributed to this blog post and their precious help in my daily support job.

Mirela Buruiana

To prepare for supporting on-premises Exchange Server 2019, I felt it necessary to build out a basic lab. I've used a default installation lab to great effect during my career supporting Exchange, both for demonstration purposes and troubleshooting customer environments by comparing my default configuration to their custom settings. This is something I've looked forward to with each new version of the product.

A welcome feature of Exchange Server 2019 is the addition of Windows Server 2019 Core to the list of supported operating systems. I have precious little experience with Windows Server Core since it was introduced, so I first navigated to Bhalchandra Atre's excellent blog post Deploy Exchange Server 2019 on Windows Server Core and read it verbatim. My reason for this post is to expand on a few items that I encountered during my installation process that I found little documentation for, especially in the context of installing Exchange on Windows Server Core.

After spinning up a domain controller, a Windows 10 client and two Windows Server 2019 Server Core installs, I ran smoothly through the steps detailed in Bhalchandra’s post, downloading the latest files for installation noted in his post:

• Exchange Server 2019 CU1 (or later) .ISO
• Visual C++ Redistributable Packages for Visual Studio 2013 (vcredist_x64.exe)

The UCMA 4.0 Runtime is still a prerequisite for installing Exchange 2019, but there’s no need to download it. As Bhalchandra points out, the UCMA installable is located under the “UCMARedist” folder on the Exchange Server 2019 .ISO.

I created the VMs with four volumes - one each for the OS, Exchange installation, databases and logs. Before going to install, I needed to initialize and format the Exchange installation and DB/Log volumes, a task easily done in PowerShell on Windows Server Core. Run Get-Disk to enumerate your disk numbers, then build your command lines. I’m sure there's a more efficient command line construction to accomplish this, but these three steps per disk ran smoothly for me:

Set-Disk -Number 1 -IsOffline $False
Get-Disk 1 | Initialize-Disk -PartitionStyle GPT
Get-Disk 1 | New-Partition -UseMaximumSize -DriveLetter F | Format-Volume -FileSystem NTFS -NewFileSystemLabel EXINSTALL
Set-Disk -Number 2 -IsOffline $False
Get-Disk 2 | Initialize-Disk -PartitionStyle GPT
Get-Disk 2 | New-Partition -UseMaximumSize -DriveLetter G | Format-Volume -FileSystem REFS -NewFileSystemLabel DB -SetIntegrityStreams $false
Set-Disk -Number 3 -IsOffline $False
Get-Disk 3 | Initialize-Disk -PartitionStyle GPT
Get-Disk 3 | New-Partition -UseMaximumSize -DriveLetter L | Format-Volume -FileSystem REFS -NewFileSystemLabel Logs -SetIntegrityStreams $false

You have the option to connect the Disk Management MMC from a regular Windows Server or Windows 10 client to Server Core systems to manage storage in a familiar fashion. As Server Core installs become more common in your environment, I recommend understanding your remote management options as detailed in Manage a Server Core server at Microsoft Docs.

Next, I had to learn how to configure the paging file on a Server Core installation. Microsoft Docs to the rescue again - Configure memory dump files for Server Core installation:

wmic computersystem set AutomaticManagedPagefile=False
wmic pagefileset where name="c:\\pagefile.sys" set InitialSize=32768,MaximumSize=32768

How did I come by that size? The minimum recommended RAM for the mailbox role is now 128 GB. The paging file guidance is defined as "a size equal to 25% of installed memory" according to the  Exchange Server system requirements, a notable change to the guidance of RAM + 10 MB to a maximum of 32778 that’s been recommended since April of 2014. Additionally, the paging file may now grow as large at 64 GB if you assign the maximum recommended RAM of 256 GB.

Next, I downloaded the latest HealthChecker.ps1 script and processed the output for any recommendations before putting the new Exchange installs into service. I regularly use and endorse this script with customers, and I highly recommend a monthly maintenance task of running the latest version of the script in your Exchange environment. Check out the available command line switches too, especially the -BuildHtmlServersReport to generate a very useful HTML output of the findings across multiple servers.

With the two servers prepped for installation, I pasted the unattended command line into the first server to begin the installation:

Setup.exe /m:install /roles:m /IAcceptExchangeServerLicenseTerms /InstallWindowsComponents

In my enthusiasm to get started, I pasted in this overly simplistic command line on the first server which neglected to leverage the options to name and locate the first database and its transaction logs. It's easy enough to move Exchange databases, even renaming the .edb file while it's making the trip, but I had another thought. At some point in the future, Exchange 2019 will need to be properly uninstalled and there's no Control Panel in Windows Server Core. That's where I picked up my third valuable experience worth mentioning here - how to uninstall Exchange from Windows Server Core. You currently have one option recommended by the product group to do this: the command line.

I proceeded with the installation of the second Exchange 2019 server with the correct command line, rebooted it, then moved the Administrator mailbox and the -Arbitration mailboxes from the first database to the second one to prepare for the uninstallation of the first server. I pasted in this command line to start the uninstallation process:

Setup.exe /m:uninstall /roles:m /IAcceptExchangeServerLicenseTerms

The process failed at the prerequisite check, complaining of mailboxes still present on the server. The error reminded me to move or remove user mailboxes, system mailboxes, archive mailboxes and audit log mailboxes. Not the first time I’ve forgotten that one... Once I moved the audit log mailbox, the uninstall processed smoothly.

Important: You may find generalized guidance detailing how to uninstall software on Windows Server Core using MSIEXEC. Attempting to uninstall Exchange using this method will result in an incomplete removal of the server and is therefore not recommended. The product group supports both the GUI and command line options to install Exchange Server 2019, and currently just the command line option for uninstallation.

Here are the unattended installation commands I used and recorded in my build document:

Setup.exe /m:install /roles:m /MdbName:DB1 /DbFilePath:"G:\DB1\db1.edb" /LogFolderPath:"L:\DB1" /IAcceptExchangeServerLicenseTerms /InstallWindowsComponents
Setup.exe /m:install /roles:m /MdbName:DB2 /DbFilePath:"G:\DB2\db2.edb" /LogFolderPath:"L:\DB2" /IAcceptExchangeServerLicenseTerms /InstallWindowsComponents

Note: I didn't run /PrepareAD first as a standalone command because I was logged in with the necessary credentials to extend the schema on a server in the same site and domain as the schema master.

After the installations completed, I ran through all the standard tasks of getting a new Exchange server ready to host mailboxes using the same methods as before with Exchange 2013/2016.

Windows Server Core management options were mentioned earlier, but I want to specifically call out the Windows Admin Center as an excellent way to manage all your recent vintage Windows Servers, and it’s especially useful for Server Core. If you have not tried it yet, you may be missing out on a utility that will significantly improve your daily Windows Server administration tasks.

Final thought - if you're building out a lab for testing and would like to populate it quickly with real world users and data, I highly recommend Aaron Guilmette's awesome script Create Realistic Lab Users. Run it with -TheWholeShebang parameter and watch it go!

Butch Waller
Senior Premier Field Engineer

With the introduction of Unified Retention & Retention Labels in the Security and Compliance, many customers have questions on the differences between Unified Retention, Retention Labels and MRM retention, configurable parameters and other common scenarios.

Retention or Unified Retention

Retention or Unified Retention is available in Office 365 Security and Compliance portal.

Unified retention policy in Office 365 can help you achieve all these goals. Managing content commonly requires two actions:

1. Retaining content so that it can't be permanently deleted before the end of the retention period.
2. Deleting content permanently at the end of the retention period.

With a retention policy, you can:

• Decide proactively whether to retain content, delete content, or both - retain and then delete the content.
• Apply a single policy to the entire organization or just specific locations or users.
• Apply a policy to all content or just content meeting certain conditions, such as content containing specific keywords or specific types of sensitive information.

SCC Retention provides true retention, you can use a single SCC retention policy to perform both deletion and retention and at the same time a single policy can be applied across different workloads.

For more details, refer Overview of Retention Policies

Retention Labels

Retention Labels is available in Office 365 Security and Compliance portal.

Retention labels in Office 365 can help you take the right actions on the right content. With retention labels, you can classify data across your organization for governance, and enforce retention rules based on that classification.

With retention labels, you can:

• Enable people in your organization to apply a retention label manually to content in Outlook on the web, Outlook 2010 and later, OneDrive, SharePoint, and Office 365 groups. Users often know best what type of content they're working with, so they can classify it and have the appropriate policy applied.
• Apply retention labels to content automatically if it matches specific conditions, such as when the content contains:
  • Specific types of sensitive information.
  • Specific keywords that match a query you create.
  • The ability to apply retention labels to content automatically is important because:
  • You don't need to train your users on all of your classifications.
  • You don't need to rely on users to classify all content correctly.
  • Users no longer need to know about data governance policies - they can instead focus on their work.
• Apply a default retention label to a document library in SharePoint and Office 365 group sites, so that all documents in that library get the default retention label.
• Implement records management across Office 365, including both email and documents. You can use a retention label to classify content as a record. When this happens, the label can't be changed or removed, and the content can't be edited or deleted.

Retention setting in Labels and Unified Retention is same. A single retention labels policy to perform both deletion and retention and at the same time a single policy can be applied across different workloads.

There are different ways to monitor the usage of Retention Labels using Data Governance Dashboard, Label Activity Explorer (Available with E5 only), Content Search, Audit log

For more details, refer Overview of Retention Labels

Messaging Records Management (MRM)

Messaging Records Management aka Retention Policy is available in Exchange on-premises as well as in Exchange online and available in Exchange Admin Center (EAC).

You can use retention policies to enforce basic message retention for an entire mailbox or for specific default folders. Although there are several strategies for deploying MRM, here are some of the most common:

• Remove all messages after a specified period.
• Move messages to archive mailboxes after a specified period.
• Remove messages based on folder location.
• Allow users to classify messages.
• Retain messages for eDiscovery purposes.

When you implement MRM policies that remove messages from mailboxes after a specified period it also retains them in the Recoverable Items folder for In-Place eDiscovery purposes, even if the messages were deleted by the user or another process.

In Exchange Server and Exchange Online, MRM is accomplished through the use of retention tags and retention policies.

• Assigning retention policy tags (RPTs) to default folders, such as the Inbox and Deleted Items.
• Applying default policy tags (DPTs) to mailboxes to manage the retention of all untagged items.
• Allowing the user to assign personal tags to custom folders and individual items.

Messaging Record Management policy itself doesn’t perform any retention. You need to use a time-based In-Place Hold or Litigation Hold to preserves messages that were deleted for long period of time than the Single Item Recovery period.

In this post, we will be referring Messaging Records Management (MRM) as EAC based Retention.

For more details, refer Messaging records Management

Next, we will answer some of the frequently asked questions around Retention Policies in the SCC and EAC.

Deletion and Retention options for Retention. What do they really do?

While creating Unified Retention policy or Retention Labels, the settings below, may not be as clear for some customers. Let’s take a deeper look:

Option: “Yes, I want to retain”

This option means retain content in user’s mailbox (mail folders and Recoverable Items folder) wherever they are located for specified x days/months/years. You also get an option to retain them forever. This setting also applies to content in folders in archive mailbox and its Recoverable items folders.

Content deleted from user’s mail folders will be moved to Recoverable items folder and content which is already existing in Recoverable items folder (when policy is applied), will be retained for x days/months/years. In short retention will make sure that the content will not be purged completely from the mailbox for specified number of days/months/years

What happens to content when the retention period for emails is expired? It depends on what’ option is selected next;

“Do you want us to delete it after this time?”

If “Yes” is selected, MFA does the job of cleaning the expired contents from user’s mail folders and from the Recoverable items folders. This also includes expired content in archive mailbox and its recoverable items folders.

If “No” is selected, Managed Folder Assistant (MFA) will not clean the expired content (move to recoverable items folder) which exists in user’s mailbox folders. But the expired content in Recoverable items folder older than Single Item recovery period (14 days) will be cleaned, provided there is no other hold applied to this mailbox to retain the content longer.

To identify other holds on the mailbox, refer How to identify the type of hold placed on an Exchange Online mailbox.

Option: “No just delete content that’s older than”

This option indicates delete content in user’s mailbox (users’ mail folders and Recoverable Items folder) which is older than configured x days/months/years, wherever it is located. This also includes content in folders in the archive mailbox and its Recoverable items folder.

With this option selected, expired content from user's mail folders and Recoverable items will be deleted permanently (provided that there is no other hold configured to retain content for longer period.)

For more details refer Deleting content that's older than a specific age

Let’s discuss some of the common scenarios.

Retain and Delete content in the entire mailbox.

If you are planning to use Unified Retention and your requirement is that the mailbox should not hold any content older than 1 year.

You can create a SCC Retention as shown below so that any data which is older than 1 year would be deleted from the user's mail folders and Recoverable items folders.

This option makes sure than there is no content in the mailbox older than 1 year, both in users mail folder and Recoverable items folder, this also includes content in archive mailbox. The expired content is not immediately purged from the mailbox instead it is retained for some more days, it could be because other holds and because of DelayHoldApplied on the mailbox.

Retain the deleted content for a longer period.

If you are planning to use SCC Retention and your requirement is that the content from user's mail folders older than 1 years needs to be deleted and the deleted content need to be retained for 7 years for eDiscovery or recovery.

One of the ways to achieved this is by creating two SCC Retention policies

One policy to delete email older than 1 year:

Another policy to retain data for 7 years:

How is the retention period specified calculated?

The retention period calculation for different types of items varies and is documented in below article.

For more details How retention age is calculated

Above article applies both the EAC based retention and SCC Retention

Principles of retention.

A mailbox can have multiple Unified Retention or Retention Labels policies applied either implicitly or explicitly. At times in order to meet your compliance requirement, a given mailbox can be subjected to multiple policies, in such cases it’s important to understand which action take precedence, which is explained nicely using “Principles of retention”

For more detail on “Principles of retention” refer Overview of retention policies

Should I use the EAC based retention or SCC Retention?

It really depends on your retention requirements.

With introduction of auto-expanding archive feature, it is important that you move your old emails from primary mailbox to archive mailbox this includes emails from the user’s folders and Recoverable Items folder of primary mailbox, so that Primary mailbox doesn’t exceed the mailbox quota limits.

For auto-expanding archiving feature refer Auto-expanding archiving feature

Automate moving emails to the archive.

What if you want to automate moving emails older than 2 years from primary to archive, the only option to do this currently is using Default Policy tag or Personal tag in MRM 2.0 as these are the only retention tags which support move to archive action.

SCC Retention or even Retention Labels doesn’t provide us the same option of moving emails to archive mailbox. So, in this case EAC based retention is the only option (currently). This is probably the only advantage of using EAC based retention.

Does it mean that I can apply EAC based retention and SCC Retention to the same mailbox?

Yes, You can.

It's important note that a given mailbox can have only one EAC based retention with multiple tags and at the same mailbox can have multiple SCC Retention policies and Retention labels policies.

I would recommend using EAC based retention to meet your archiving (mailbox) needs and SCC retention for your retention needs.

But what about emails in the Recoverable Items folder in Primary mailbox?

As Recoverable items has its own quota, in order to prevent it from being full, you can opt to archive emails from your primary mailbox’s recoverable items to archive mailbox’s recoverable items. There is a special tag called “Recoverable Items tag” in EAC based retention which only support the move to archive action can move emails from Recoverable items folder of Primary mailbox to Recoverable items folder of Archive mailbox.

So, if you are planning to use EAC based retention for archiving purpose and SCC retention to meet your retention needs, your sample policies should look as below.

With above EAC based retention policy in place, emails (as well as other items) older than 180 days in users mail folders will be moved to archive mailbox, at the same time deleted content in Recoverable items of Primary mailbox will be moved to Recoverable items of archive mailbox after 14 days.

Also, when you are planning to use SCC retention along with EAC based retention policy it is important to understand how precedence works in EAC based retention like;

• Default Policy tag (DPT) with move to Archive action always overwrites the Retention Policy tag (RPT) or the Personal tag (PT), when the age limit for retention of DPT is lower than of RPT or PT.
• Explicitly assign tag wins over an implicit tag

It’s important to plan your policies & test the policies on test mailboxes to understand the behavior.

Organizations share a common goal of having consistent approach to categorize, classify important content from its creation, retention and disposal. In achieving this goal it's critical that administrators and Information Management teams carefully plan and test their data governance strategy.

Hope this post helps.

Big Thanks to Linda Harrell (Supportability PM - Information Protection) & Bhalchandra Atre (Supportability PM - Exchange) for reviewing this post.

Vikas Soundade

As mentioned back in March – we are moving! Starting today, we are starting our ‘freeze’ period where we do not expect to create any new posts on this blog. We’ll start again once we have moved to our new home. To give you an idea of what’s going to happen:

• We will take a snapshot of current blog state; posts, comments, graphics, etc.
• We are going to import this to Tech Community blogs production environment and do some optimization
• We will then work on enabling blog post URL redirection
• Finally, we will turn the original blog (this one) off
• We’ll make sure that short URLs like http://aka.ms/ehlo are redirected

We are aware of a few formatting issues that we will have at Tech Community site, and will work on sorting that out, but there should be no really bad surprises. In any event, all the above should happen by the end of next week. If there are issues that you run into, links appear broken or you find yourself in a different looking site – this is what is going on.

As a reminder: if you want to keep asking questions or commenting on our posts, you will need to register with the Tech Community site because we cannot migrate blog user accounts. Additional plus is that there is a bunch of communities where you can then get engaged in too, if you are so inclined!

See you on the other side!

The Exchange Team Blog Team

We have moved this blog to Microsoft Tech Community. The EHLO announcement on the new blog was already posted.

Our new blog home page is here. The new RSS feed URL is here.

We are now working on redirects for posts on TechNet blog as well as various short URLs before we turn this site off. We do not expect to post to the TechNet blog anymore (this is likely the last post). See you in Tech Community!

The Exchange Team Blog Team