SharePoint / Office 365 New Migration API

New SharePoint / Office 365 Migration API

Introduction
In Microsoft Ignite 2015 conference;  new SharePoint / Office 365 Migration API is also announced. The new Migration API will enable the users to migrate contents (all sorts of contents, files, permissions, metadata etc.) from on-premises SharePoint Farm or local/share drives etc. to Azure BLOB Storage (a.k.a Azure Temporary Storage) as packages in batches and pushed in batches and finally Timer Jobs of SharePoint Online pulled the package into the Office 365, SharePoint Online and OneDrive for Business.

With this new API; the speed of migrating Gigabytes (GBs) of data is almost five (5 times) than the old way of migration and no worries for CSOM call throttling.

The flow of migration is as follows

Step by Step Guide

Step 1 -  Install SharePoint Online Management Shell

Go to the SharePoint Machine and uninstall all previous versions of SharePoint Online Management Shell, then download it using this link and install it 

https://www.microsoft.com/en-us/download/details.aspx?id=35588

Once it is downloaded; run it "As Administrator" and execute this command in SharePoint Online Management Shell window

$cred = (Get-Credentials "")

This will pop-up a window to get the password. Please provide the password.

This $cred variable will be used later in the migration.

Step 2 -  Add 2 new folders in your local SharePoint machine

These folder will only contain XML files so will not occupy lots of space

Folder 1 Name: c:\temp\TempPackage

Folder 2 Name: c:\temp\FinalPackage

Step 3 -  Create Azure Storage Account

Create Azure Storage account and make note of the Account Name and Key for later use

1- Go to https://manage.windowsazure.com and login




2 - Create a new Azure Storage for temporary storage and keep account username and key value for later use

Step 4 -  Create New Migration Package and Upload to Azure Storage

Migration package can be created in 2 ways

• From Local Fileshare

Folders and files in the local file share can directly uploaded to the Azure Storage using below PowerShell command

 New-SPOMigrationPackage
-SourceFilesPath $sourceFiles 
-OutputPackagePath $targetPackage

where

$srcFiles = {Source Files Path}
$targetPackage = {Target Package Path}

Example:

PS C:\> $azurePkg = Set-SPOMigrationPackageAzureSource -SourceFilesPath '\\fileserver\fileshare'
-OutputPackagePath 'd:\temp\migrationpackage_source'

• From On-Premises SharePoint Server

From On-Premises SharePoint Server; it is required to go through few extra steps. First use Export-SPWeb.  

The Export-SPWeb cmdlet exports a site, list, or library. 

Export-SPWeb [-Identity]  

-Path  

[-AppLogFilePath ] 

[-AssignmentCollection ] 

[-CompressionSize ] 

[-Confirm []] 

[-Force ] 

[-HaltOnError ] 

[-HaltOnWarning ] 

[-IncludeUserSecurity ] 

[-IncludeVersions ] 

[-ItemUrl ] 

[-NoFileCompression ] 

[-NoLogFile ] 

[-UseSqlSnapshot ] 

[-WhatIf []]

where

[Identity] = GUID or URL of SharePoint Site. It is required.

Path = Folder Path of the Export File. Since NoFileCompression property is used; this must be a folder path. It is required.

ItemUrl = URL of Web Application, GUID or SharePoint object to be exported. It is optional/

Please get other details from

https://technet.microsoft.com/en-us/library/ff607895.aspx

Example:

PS C:\>Export-SPWeb http://spsite -path "d:\temp\exportpackagefilename.cmp"

Note:

The package named exportpackagefilename.cmp is NOT  READY for migration; 

So it is required to make it ready for migration to target destination using below command

ConvertTo-SPOMigrationTargetedPackage -TargetWebUrl $targetWeb 
-SourceFilesPath $sourceFiles
-SourcePackagePath $sourcePackage
-OutputPackagePath $targetPackage
-TargetDocumentLibraryPath $targetDocLib
-Credentials $creds

Examples:

PS C:\>$targetPackage =  ConvertTo-SPOMigrationTargetedPackage -TargetWebUrl http://spsite                           -SourceFilesPath \\fileserver\sharedfolder\sourcefilesFolder-SourcePackagePath c:\temp\TempPackage-OutputPackagePath c:\temp\FinalPackage -TargetDocumentLibraryPath "SPDocumentsLibName"-Credentials $creds

OR In a SubFolder destination

PS C:\> $targetPackage = ConvertTo-SPOMigrationTargetedPackage 
-TargetWebUrl https://spsite
-SourceFilesPath \\fileserver\sharedfolder\sourcefilesFolder 
-SourcePackagePath c:\temp\TempPackage-OutputPackagePath c:\temp\FinalPackage\-TargetDocumentLibraryPath "SPDocumentsLibName"-TargetDocumentLibrarySubFolderPath "Libray/SubFolder"-Credentials $creds






For more details

https://technet.microsoft.com/en-us/library/mt143609.aspx







The above PowerShell command will convert the XML [ generated in temporary folder (c:\temp\TempPackage) when executed Export-SPWeb command ] and saves/copy new targeted migration package metadata files to the final package folder [ (c:\temp\FinalPackage) ]




Note: You might need to provide password for the credentials



Uploading On-Premises SharePoint Package to Azure Storage




On-Premises SharePoint can be uploaded to Azure Storage using below PowerShell command



Set-SPOMigrationPackageAzureSource 
-SourceFilesPath $sourceFiles 
-SourcePackagePath $targetPackage 
-AccountName $azureAccountName 
-AccountKey $azureAccountKey
-AzureQueueName $azureQueueName 





Example:

PS C:\> $azurePkg = Set-SPOMigrationPackageAzureSource -SourceFilesPath $sourceFiles -SourcePackagePath $targetPackage -AccountName $azureAccountName -AccountKey $azureAccountKey-AzureQueueName $azureQueueName 





Step 5 - Submit Azure Package




Now the package is uploaded to the Azure Storage; the next big step is to submit the package.







Submit-SPOMigrationJob 



-TargetWebUrl $targetWeb 


-MigrationPackageAzureLocations $azurePkg 

-Credentials $creds









Example:

PS C:\> Submit-SPOMigrationJob
              
-TargetWebUrl = https://spsite
              -MigrationPackageAzureLocations = $azurePkg
              -Credentials $creds

Note for Credentials:
SharePoint Online Credentials that has admin rights on Final Destination Site




Step 6 - Migration from Azure Storage to Destination (SPO, O365, OD4B etc.)







Once the package is submitted; now actual migration is happening between Azure Storage and Destination (SharePoint Online, Office 365, OneDrive for Business etc.). This migration process is based on Azure Timer Job which is a queue and serves on first come first server basis in a parallel fashion and not impacting or preventing other timer jobs executed at the same time. 




The real-time progress of the timer job can be viewed on Azure Storage Account queue and logs can be viewed in the Manifest Container of Azure BLOB Storage.




Note:

All the above process and scripts can be saved in a single PowerShell script (.PS1) file and re-use any time.





$Office365LoginID = "admin@bhmconsultings.onmicrosoft.com"

$adminSite = "https://bhmconsultings-admin.sharepoint.com"

$sourceFilePath = "C:\migration\SourceData\"

$packagePath = "C:\migration\SourcePackage"

$targetWebUrl = "https://bhmconsultings.sharepoint.com/sites/{target site name}"

$targetPackage = "C:\migration\TargetPackage"

$targetLibrary = "{Taregt Library Name. For examples 'Shared Documents'}"

$azureStorageAccountName = "{Azure Storage Account Name}"

$azureStorageAccountKey = "{Azure Storage Account Key}"

$azureStorageQueueName = "{Azure Storage Queue Name}"




#Get Credentials 

$credentials = Get-Credential {Office365 Login ID}




#Create package and upload to Azure Temporary Storage

New-SPOMigrationPackage 

-SourceFilesPath $sourceFilePath 

-OutputPackagePath $packagePath




#For On-Premises SharePoint migration ONLY 

ConvertTo-SPOMigrationTargetedPackage 

-TargetWebUrl $targetWebUrl 

-SourceFilesPath $sourceFilePath 

-SourcePackagePath $packagePath 

-OutputPackagePath $targetPackage 

-TargetDocumentLibraryPath $targetLibrary 

-Credentials $credentials




#Ready package for migration from Azure Storage to Destination

$SourcePackageInAzureStorage = Set-SPOMigrationPackageAzureSource 

               -SourceFilesPath $sourceFilePath 

               -SourcePackagePath $targetPackage 

               -AccountName $azureStorageAccountName 

               -AccountKey $azureStorageAccountKey 

               -AzureQueueName $azureStorageQueueName







#Submit above Package for Migration to Destination

Submit-SPOMigrationJob 

-TargetWebUrl $targetWebUrl 

-MigrationPackageAzureLocations $SourcePackageInAzureStorage-Credentials $credentials 







Migration Benefits and Limitations





The biggest benefit of using SharePoint New Migration API is SPEED which is 5 - 10 times faster (depending on the complexity) than using any exiting technology who are not using 

New APIs.



There are some minor limitations as well.

1. Need to manage Azure Storage for temporary storage of the package until migration is done and related cost.

2.Trust Microsoft Azure to send migration package for temporary storage



3. Azure Storage Limit = 500 TB



4. Max Size of Single BLOB container, queue, table = 500 TB



5. Target throughtput for single BLOB = 500 requests per second OR 60 MB per second



6 . See SharePoint Online and OneDrive for Business limitations here





Happy SharePoint Migration.