New SharePoint / Office 365 Migration API
Introduction
In Microsoft Ignite 2015 conference; new SharePoint / Office 365 Migration API is also announced. The new Migration API will enable the users to migrate contents (all sorts of contents, files, permissions, metadata etc.) from on-premises SharePoint Farm or local/share drives etc. to Azure BLOB Storage (a.k.a Azure Temporary Storage) as packages in batches and pushed in batches and finally Timer Jobs of SharePoint Online pulled the package into the Office 365, SharePoint Online and OneDrive for Business.
In Microsoft Ignite 2015 conference; new SharePoint / Office 365 Migration API is also announced. The new Migration API will enable the users to migrate contents (all sorts of contents, files, permissions, metadata etc.) from on-premises SharePoint Farm or local/share drives etc. to Azure BLOB Storage (a.k.a Azure Temporary Storage) as packages in batches and pushed in batches and finally Timer Jobs of SharePoint Online pulled the package into the Office 365, SharePoint Online and OneDrive for Business.
With this new API; the speed of migrating Gigabytes (GBs) of data is almost five (5 times) than the old way of migration and no worries for CSOM call throttling.
The flow of migration is as follows
Step by Step Guide
Step 1 - Install SharePoint Online Management Shell
Go to the SharePoint Machine and uninstall all previous versions of SharePoint Online Management Shell, then download it using this link and install it
Once it is downloaded; run it "As Administrator" and execute this command in SharePoint Online Management Shell window
$cred = (Get-Credentials "
This will pop-up a window to get the password. Please provide the password.
This $cred variable will be used later in the migration.
Step 2 - Add 2 new folders in your local SharePoint machine
These folder will only contain XML files so will not occupy lots of space
Folder 1 Name: c:\temp\TempPackage
Folder 2 Name: c:\temp\FinalPackage
Step 3 - Create Azure Storage Account
Create Azure Storage account and make note of the Account Name and Key for later use
1- Go to https://manage.windowsazure.com and login2 - Create a new Azure Storage for temporary storage and keep account username and key value for later use
Step 4 - Create New Migration Package and Upload to Azure Storage
Migration package can be created in 2 ways
- From Local Fileshare
Folders and files in the local file share can directly uploaded to the Azure Storage using below PowerShell command
-SourceFilesPath $sourceFiles
-OutputPackagePath $targetPackage
where
$srcFiles = {Source Files Path}
$targetPackage = {Target Package Path}
Example:PS C:\> $azurePkg = Set-SPOMigrationPackageAzureSource -SourceFilesPath '\\fileserver\fileshare' -OutputPackagePath 'd:\temp\migrationpackage_source'
- From On-Premises SharePoint Server
From On-Premises SharePoint Server; it is required to go through few extra steps. First use Export-SPWeb.
The Export-SPWeb cmdlet exports a site, list, or library.
Export-SPWeb [-Identity]
-Path
[-AppLogFilePath ]
[-AssignmentCollection ]
[-CompressionSize ]
[-Confirm []]
[-Force ]
[-HaltOnError ]
[-HaltOnWarning ]
[-IncludeUserSecurity ]
[-IncludeVersions ]
[-ItemUrl ]
[-NoFileCompression ]
[-NoLogFile ]
[-UseSqlSnapshot ]
[-WhatIf []]
where
[Identity] = GUID or URL of SharePoint Site. It is required.
Path = Folder Path of the Export File. Since NoFileCompression property is used; this must be a folder path. It is required.
ItemUrl = URL of Web Application, GUID or SharePoint object to be exported. It is optional/
Please get other details from
Example:
PS C:\>Export-SPWeb http://spsite -path "d:\temp\exportpackagefilename.cmp"
Note:
The package named exportpackagefilename.cmp is NOT READY for migration; So it is required to make it ready for migration to target destination using below command
ConvertTo-SPOMigrationTargetedPackage -TargetWebUrl $targetWeb
-SourceFilesPath $sourceFiles
-SourcePackagePath $sourcePackage
-OutputPackagePath $targetPackage
-TargetDocumentLibraryPath $targetDocLib
-Credentials $creds
Examples:PS C:\>$targetPackage = ConvertTo-SPOMigrationTargetedPackage -TargetWebUrl http://spsite -SourceFilesPath \\fileserver\sharedfolder\sourcefilesFolder-SourcePackagePath c:\temp\TempPackage-OutputPackagePath c:\temp\FinalPackage -TargetDocumentLibraryPath "SPDocumentsLibName"-Credentials $creds
OR In a SubFolder destination
PS C:\> $targetPackage = ConvertTo-SPOMigrationTargetedPackage -TargetWebUrl https://spsite -SourceFilesPath \\fileserver\sharedfolder\sourcefilesFolder -SourcePackagePath c:\temp\TempPackage-OutputPackagePath c:\temp\FinalPackage\-TargetDocumentLibraryPath "SPDocumentsLibName"-TargetDocumentLibrarySubFolderPath "Libray/SubFolder"-Credentials $creds
For more detailsThe above PowerShell command will convert the XML [ generated in temporary folder (c:\temp\TempPackage) when executed Export-SPWeb command ] and saves/copy new targeted migration package metadata files to the final package folder [ (c:\temp\FinalPackage) ]Note: You might need to provide password for the credentialsUploading On-Premises SharePoint Package to Azure StorageOn-Premises SharePoint can be uploaded to Azure Storage using below PowerShell commandSet-SPOMigrationPackageAzureSource -SourceFilesPath $sourceFiles -SourcePackagePath $targetPackage -AccountName $azureAccountName -AccountKey $azureAccountKey -AzureQueueName $azureQueueNameExample:PS C:\> $azurePkg = Set-SPOMigrationPackageAzureSource -SourceFilesPath $sourceFiles -SourcePackagePath $targetPackage -AccountName $azureAccountName -AccountKey $azureAccountKey-AzureQueueName $azureQueueName
Step 5 - Submit Azure PackageNow the package is uploaded to the Azure Storage; the next big step is to submit the package.Submit-SPOMigrationJob-TargetWebUrl $targetWeb-MigrationPackageAzureLocations $azurePkg-Credentials $credsExample:PS C:\> Submit-SPOMigrationJob -TargetWebUrl = https://spsite -MigrationPackageAzureLocations = $azurePkg -Credentials $creds
Note for Credentials: SharePoint Online Credentials that has admin rights on Final Destination Site
Step 6 - Migration from Azure Storage to Destination (SPO, O365, OD4B etc.)Once the package is submitted; now actual migration is happening between Azure Storage and Destination (SharePoint Online, Office 365, OneDrive for Business etc.). This migration process is based on Azure Timer Job which is a queue and serves on first come first server basis in a parallel fashion and not impacting or preventing other timer jobs executed at the same time.The real-time progress of the timer job can be viewed on Azure Storage Account queue and logs can be viewed in the Manifest Container of Azure BLOB Storage.Note:All the above process and scripts can be saved in a single PowerShell script (.PS1) file and re-use any time.$Office365LoginID = "admin@bhmconsultings.onmicrosoft.com"$adminSite = "https://bhmconsultings-admin.sharepoint.com"$sourceFilePath = "C:\migration\SourceData\"$packagePath = "C:\migration\SourcePackage"$targetWebUrl = "https://bhmconsultings.sharepoint.com/sites/{target site name}"$targetPackage = "C:\migration\TargetPackage"$targetLibrary = "{Taregt Library Name. For examples 'Shared Documents'}"$azureStorageAccountName = "{Azure Storage Account Name}"$azureStorageAccountKey = "{Azure Storage Account Key}"$azureStorageQueueName = "{Azure Storage Queue Name}"#Get Credentials$credentials = Get-Credential {Office365 Login ID}#Create package and upload to Azure Temporary StorageNew-SPOMigrationPackage-SourceFilesPath $sourceFilePath-OutputPackagePath $packagePath#For On-Premises SharePoint migration ONLYConvertTo-SPOMigrationTargetedPackage-TargetWebUrl $targetWebUrl-SourceFilesPath $sourceFilePath-SourcePackagePath $packagePath-OutputPackagePath $targetPackage-TargetDocumentLibraryPath $targetLibrary-Credentials $credentials#Ready package for migration from Azure Storage to Destination$SourcePackageInAzureStorage = Set-SPOMigrationPackageAzureSource-SourceFilesPath $sourceFilePath-SourcePackagePath $targetPackage-AccountName $azureStorageAccountName-AccountKey $azureStorageAccountKey-AzureQueueName $azureStorageQueueName#Submit above Package for Migration to DestinationSubmit-SPOMigrationJob-TargetWebUrl $targetWebUrl-MigrationPackageAzureLocations $SourcePackageInAzureStorage-Credentials $credentialsMigration Benefits and LimitationsThe biggest benefit of using SharePoint New Migration API is SPEED which is 5 - 10 times faster (depending on the complexity) than using any exiting technology who are not usingNew APIs.There are some minor limitations as well.1. Need to manage Azure Storage for temporary storage of the package until migration is done and related cost.2.Trust Microsoft Azure to send migration package for temporary storage 3. Azure Storage Limit = 500 TB 4. Max Size of Single BLOB container, queue, table = 500 TB 5. Target throughtput for single BLOB = 500 requests per second OR 60 MB per second 6 . See SharePoint Online and OneDrive for Business limitations here Happy SharePoint Migration.
