Office 365 unencrypted PST file upload feature

In this blog article you can read about my recent experience using the Office 365 unencrypted PST file upload feature. The procedure that I followed is described in the following Microsoft TechNet Library article:

We have a hybrid Exchange 2013 server environment. Until now almost all of our mailboxes have been on-premises. I had been trying to migrate my mailbox to Exchange Online over the last week using various methods. Before I began the PST upload/import procedure, I had a working mailbox in Office 365 containing about half of the content that I wanted to migrate.

Overall, I was satisfied with the whole upload and import process.

In my case, it took approximately 2 hours to upload a single 3.5 GB PST file to Microsoft Azure.

After the upload, the import process took roughly 4.5 hours to import 3.5 GB of data into a single Office 365 mailbox (approx. 0.77 GB/hour). Approximately 26,000 items were imported and 18,000 items were skipped because they were already in the mailbox. I don’t know what effect the skipped items had on the import performance.

Following is a walk-through of the procedure with comments and an example. I’ve also shown how to use the updated Microsoft Azure Storage Explorer tool instead of the tool listed in the TechNet article that is no longer actively maintained. The various steps that are shown below correspond to the steps listed in the TechNet article.

Step 0: Before you begin

This step contains instructions about how to assign the correct permissions to import PST files to Office 365 mailboxes. Since I was performing the procedure as an Office 365 global admin, I decided to assign the “Mailbox Import Export” role to the “Organization Management” role group in Exchange Online. Try not to skip this bit; in my case I “tested” the process by not performing this step until after I submitted the import job. The effect was that I received an error message saying that the import process would be delayed for an hour until after I had assigned the correct permissions. I was quite impressed that the process was robust enough to do this.

I recommend that you create a new folder in which you will store the PST file/s that are going to be uploaded. The reason for this is that the AzCopy program will upload all files in the source location (and optionally subfolders).

I created a new folder in which to store the PST file/s (e.g. “C:\Temp\O365PSTUpload“) and then shared it as follows:

  • Share name: \\FS1\O365PSTUpload
  • Share permission: I gave the Everyone group Read permissions to this share.

Step 1: Download the PST Import tools

The following figure shows the “Upload files over the network” window. I downloaded and installed the PST Import tools (i.e. AzCopy) by clicking on “Download tool“.

01 Upload files over the network

Step 2: Obtain the storage key and upload URL for PST Import

In this step you obtain the storage key and upload URL, then save them somewhere for future reference. In my case, I created a text file called “Office365PSTImportInfo-20160309.txt” and copied the key and URL into the file.

Tip: There is a section titled “More information” right at the end of the TechNet article that contains a good example of what the file should contain.

The following figure shows what happens when you click on “Show key“. The key should appear in the box within a few minutes.

02 Storage Account Key

Note: In case the key doesn’t appear in a few minutes, simply close the “Upload files over the network” window, go back to the “Import files to Office 365” window, click on “New job +“, and then click “Upload files over the network“. Then go to the start of Step 2. The storage key should only take a few seconds to display.

Step 3: Upload your PST files to Office 365

I recommend using the text file that you created in Step 2 to work out what the command will be before pasting it into the command prompt. This will also allow you to save a record of the command that you used for future reference.

Here is an example of what my file (Office365PSTImportInfo-20160309.txt) looked like:

Storage account key:
yaNIIs9Uy5g25Yoak+LlSHfqVBGOeNwjqtBEBGqRMoidq6/e5k/VPkjOXdDIXJHxHvPoGoFH5NcVUJXHwu9ZxQ==

Network upload URL:
https://3c3e5952a2764023ad14984.blob.core.windows.net/ingestiondata/

AzCopy.exe /Source:<Location of PST files> /Dest:<Network upload URL + PST file path> /Destkey:<Storage account key> /V:<Log file location>

EXAMPLE
AzCopy.exe /Source:"\\FS1\O365PSTUpload" /Dest:"https://3c3e5952a2764023ad14984.blob.core.windows.net/ingestiondata/FS1/O365PSTUpload" /Destkey:"yaNIIs9Uy5g25Yoak+LlSHfqVBGOeNwjqtBEBGqRMoidq6/e5k/VPkjOXdDIXJHxHvPoGoFH5NcVUJXHwu9ZxQ==" /V:"C:\Temp\O365PSTUpload.log"

After you enter the command in a command prompt, the PST files that you specified will be uploaded to the Azure storage account. In my case the initial upload timed out. I seamlessly resumed the upload by simply running the same command again. The contents of my log file (O365PSTUpload.log) showing both upload runs was as follows:

[2016/03/09 15:18:26.818+02:00] >>>>>>>>>>>>>>>>
[2016/03/09 15:18:26.880+02:00][VERBOSE] 5.0.0 : AzCopy /Source:\\FS1\O365PSTUpload /Dest:https://3c3e5952a2764023ad14984.blob.core.windows.net/ingestiondata/FS1/O365PSTUpload /Destkey:****** /V:C:\Temp\Office365PSTUpload.log
[2016/03/09 15:18:30.354+02:00][VERBOSE] Start transfer: PLapornik-20160309.pst => PLapornik-20160309.pst
[2016/03/09 17:14:14.400+02:00][VERBOSE] Transfer FAILED: PLapornik-20160309.pst => PLapornik-20160309.pst.
[2016/03/09 17:14:14.400+02:00][ERROR] PLapornik-20160309.pst: The client could not finish the operation within specified timeout.
The client could not finish the operation within specified timeout.
[2016/03/09 17:14:14.447+02:00] Transfer summary:
-----------------
Total files transferred: 1
Transfer successfully: 0
Transfer skipped: 0
Transfer failed: 1
Elapsed time: 00.01:55:47
[2016/03/09 17:49:22.211+02:00] >>>>>>>>>>>>>>>>
[2016/03/09 17:49:22.227+02:00][VERBOSE] 5.0.0 : AzCopy /Source:\\FS1\O365PSTUpload /Dest:https://3c3e5952a2764023ad14984.blob.core.windows.net/ingestiondata/FS1/O365PSTUpload /Destkey:****** /V:C:\Temp\Office365PSTUpload.log
[2016/03/09 17:49:36.137+02:00][VERBOSE] [PROMPT] Incomplete operation with same command line detected at the journal directory "C:\Users\Administrator\AppData\Local\Microsoft\Azure\AzCopy", do you want to resume the operation? Choose Yes to resume, choose No to overwrite the journal to start a new operation. (Yes/No) : Y
[2016/03/09 17:49:37.053+02:00][VERBOSE] Start transfer: PLapornik-20160309.pst => PLapornik-20160309.pst
[2016/03/09 18:06:38.929+02:00][VERBOSE] Finished transfer: PLapornik-20160309.pst => PLapornik-20160309.pst
[2016/03/09 18:06:38.976+02:00] Transfer summary:
-----------------
Total files transferred: 1
Transfer successfully: 1
Transfer skipped: 0
Transfer failed: 0
Elapsed time: 00.00:17:16

On my Internet connection, the transfer rate to upload the PST file using AzCopy was approximately 1.6 GB/hr.

Step 4: View a list of the PST files uploaded to Office 365

The TechNet Library article shows how to verify that the PST file got uploaded using the Azure Storage Explorer tool. This tool is not being actively maintained. Instead, I have shown how to do the same thing using the Microsoft Azure Storage Explorer tool (a different tool) that is being actively maintained by Microsoft. Microsoft Azure Storage Explorer is supported by Microsoft and can be downloaded from http://storageexplorer.com A direct link to download the tool is http://go.microsoft.com/fwlink/?LinkId=708343

After installing and launching the application, I closed the initial prompt that asks you for an account.

03 Microsoft Azure Storage Explorer 1

I right-clicked on “Storage Accounts” then clicked on “Attach External Storage…

04 Microsoft Azure Storage Explorer 2

In the “Attach External Storage” window, I entered the following information:

05 Microsoft Azure Storage Explorer 3

  1. Account name: Use the numeric portion of the network upload URL that you obtained in Step 2. For example, if the network upload key is https://3c3e5952a2764023ad14984.blob.core.windows.net/ingestiondata/, you would use 3c3e5952a2764023ad14984 for the account name.
  2. Account key: Use the storage account key that you obtained in Step 2. For example, yaNIIs9Uy5g25Yoak+LlSHfqVBGOeNwjqtBEBGqRMoidq6/e5k/VPkjOXdDIXJHxHvPoGoFH5NcVUJXHwu9ZxQ==.

Finally, I browsed through the storage accounts hierarchy until I was able to confirm that the PST file had been uploaded.

For example, 3c3e5952a2764023ad14984 (external) –> Blob Containers –> ingestiondata –> FS1 –> O365PSTUpload

Here is a screenshot from my storage account (sensitive information has been obscured).

06 Microsoft Azure Storage Explorer 4

Step 5: Create the PST Import mapping file

The next step is to create a mapping file. The following is an example of the mapping file that I used for a single mailbox:


Workload,FilePath,Name,Mailbox,IsArchive,TargetRootFolder,SPFileContainer,SPManifestContainer,SPSiteUrl

Exchange,FS1/O365PSTUpload,PLapornik-20160309.pst,peter@contoso.com,FALSE,/,,,

In my case I decided to import the PST file into the root folder of my mailbox. I did this because I wanted mail items to be imported back into their original places.

Step 6: Create a PST import job in Office 365

To create the import job you need to go back to the “Upload files over the network” screen, tick the options to indicate that you have uploaded the PST file/s and created a mapping file, then click on “Next“.

As shown in the example, you will now be prompted to specify a name for your import job. The name must consist of 2-64 lowercase letters, numbers, hyphens and underscores. It must start with a letter, No spaces are permitted. I used “importplapornikpst”.

07 Import job name

The next step is to upload and verify the mapping file. As shown below, I selected my mapping file, clicked on “Validate“, agreed to the terms and conditions, then clicked on “Finish“.

08 Upload and validate mapping file

Now that you have successfully submitted the import job, you can close the “Upload files over the network” window.

Viewing the status of the import job

If you go to the “Import data to Office 365” window, you can view the status of the import job. You can also click on “View details” for more detailed information.

09 Import job status

The following screenshot shows an example of the amount of detail provided in the detailed view.

10 Import job status details

Post import report

The following figure provides an example of what you might see after the PST import has completed. In my case, the “Missing skipped item” column referred to emails that were skipped because they were already imported in my mailbox.

11 Post import report

I was able to see more details about skipped items by clicking on “Download report“. This allowed me to cross-reference emails in my mailbox with the list of emails that were downloaded as a CSV file. A small sample from the report is shown below.


Subject,Created on,Folder,Skipped item reason,,
SSO for ISA Server (DITS),03/10/2016 18:28,Sent Items,"Item is missing in a folder. Folder name: Sent Items, folder ID: 000000000D60CE1C58423E458AF905D442289DC962840000, item ID: 000000000D60CE1C58423E458AF905D442289DC9E44A2E00.",,
Office 365 for Meice,03/10/2016 18:28,Sent Items,"Item is missing in a folder. Folder name: Sent Items, folder ID: 000000000D60CE1C58423E458AF905D442289DC962840000, item ID: 000000000D60CE1C58423E458AF905D442289DC9044B2E00.",,
FW: Cytram,03/10/2016 18:28,Sent Items,"Item is missing in a folder. Folder name: Sent Items, folder ID: 000000000D60CE1C58423E458AF905D442289DC962840000, item ID: 000000000D60CE1C58423E458AF905D442289DC9244B2E00.",,

Additional Notes

After you import information into a mailbox using the Office 365 unencrypted PST file upload feature, the mailbox is automatically placed on retention hold. This is not the same as as a litigation hold or in-place hold. A retention hold suspends the processing of any retention policies. Just remember to take the mailbox off retention hold to resume normal processing of any retention policies.

6 Comments

  1. Johan

    Hi,
    Trying to access the BLOB according to your guide above, but the explorer doesn’t recognize the account key.
    The Accountkey is much longer than yours, maybe Microsoft has changed it after you had released this tutorial?

    Reply
    1. Peter Lapornik (Post author)

      Hi Johan, thanks for letting me know about this issue. I just had a look at the TechNet Library documentation (last modified 2016-04-27) and discovered that what was known as the “secure storage account key” is now known as the “shared access signature (SAS) key”. I’ll review the entire article later this week and post anything new that I discover.

      Reply
      1. Johan

        Hi Peter,
        That would be great. I did some exploring myself, but untill now I didn’t find any solution.
        Hope you can!
        Regards
        Johan

        Reply
  2. Aykut

    I have the same issue. Key is too long.
    Regards
    Aykut

    Reply
  3. Johan

    Hello,
    Sice 06/15/2016 has Microsoft released an update of Storage Explorer, so it’s again possible to review all the files uploaded on Azure.
    Microsoft has also released a manual (howto), see https://support.office.com/en-gb/article/Use-network-upload-to-import-PST-files-to-Office-365-103f940c-0468-4e1a-b527-cc8ad13a5ea6.

    Regards
    Johan

    Reply
  4. Irfan

    What does missing skipped items means?

    Reply

Leave a Comment

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.