Hadoop

PowerShell script wrappers using the Microsoft Azure AzCopy.exe tool

Use case

We are working on building data lake in Azure using Azure container, ADF, Azure DWH, Databricks and many other services of Azure. After ingesting wide variety of datasources using API, on premise databases, flate files, reporting servers, we come to know that clients have some requirement to push files in Azure Blob storage. Users can locate the files on their local system but dont know what actual folder hierarchy in Azure to run the process. This requirement extents to help them testing their processes in UAT and Dev environments.

Introduction

AzCopy is a command-line utility that you can use to copy blobs or files to or from a storage account. 

This article helps you to understand the workings and how to parameterized AzCopy.exe to Export and Import Azure files. You can find Powershell Script to copy local files to Blob storage Account using AzCopy at my git that uses azCopy to move files from an on-premise directory to azure storage.

https://github.com/mkjmkumar/migrate_files_to_azure_azcopy

How to Download AzCopy

To download Azure azcopy windows executable visit Microsoft website https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10

Authenticate your User ID

Open Powershell terminal and locate to the azcopy directory from Above. Now Authenticate yourself using below command and follow instructions

#.\azcopy.exe login –tenant-id xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Permission Error and configure script

In case you are getting permission error, that means user-id does not have permission to Azure Storage account or SAS key expired, to get fresh SAS key. In Case your SAS key expired go to storage account “mystorageaccount” and left-hand pane search for “Shared access signature” and replace this key with $SAS variable in PowerShell script.

How to execute Powershell script

Once you have made the changes to ps file locate azcopy executable use below command to execute your PowerShell file:-

“C:\copytoazure.ps1”

Demo

PS C:\AzCopy> .\copytoazure.ps1
Please authenticate using .\azcopy login to avoid permission error...
Current Directory..
C:\FMAG\TempAZ\Sombody.txt
1-datasource1
2-datasource2
3-datasource3
4-datasource4
Choose Datasource from Above...: 2
Your Choice is Envestnet
-------------------------------
1-xxxxx/mf_1/
2-xxxxx/mf_2/
3-xxxxx/sma_1/
4-xxxxx/sma_2/
5-xxxxx/manage_1/
-------------------------------
Please Choose entity name...: 3
Your Choice is 3-xxxxx/sma_1/ Folder!
Target Location =
https://wwwwwwwwwwwwww.blob.core.windows.net/wwwwwwww-zone/wwwwwww/www_wwww/?wwwwwwwwwwwwwwwwwwww

Output

Job e726bb1f-fdb0-bc4f-7f02-ea2670aa93f4 has started Log file is located at: C:\xxxxx.azcopy\e726bb1f-fdb0-bc4f-7f02-ea2670aa93f4.log


0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 
0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 
0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 
0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total,


Job e726bb1f-fdb0-bc4f-7f02-ea2670aa93f4 summary Elapsed Time (Minutes): 2.1104 
Total Number Of Transfers: 1 
Number of Transfers Completed: 1 
Number of Transfers Failed: 0 
Number of Transfers Skipped: 0 
TotalBytesTransferred: 19 
Final Job Status: Completed

Source Code

#Replace below folder location with your azcopy.exe folder location
cd C:\FMAG\AzCopy\azcopy_windows_amd64_10.3.4\azcopy_windows_amd64_10.3.4
#log file
$logfile="C:\AzCopy\azcopy_windows_amd64_10.3.4\azcopy_windows_amd64_10.3.4\azcopy.log";
#$SOURCE="C:\TempAZ\Sombody.txt";
$SOURCE=Read-Host "Enter file you want to upload to Azure..."
if (!(Test-Path $SOURCE)) {
  Write-Warning "$userFile absent from locations"
  exit
}
$DMZ_LOCATION='https://xxxxxx.blob.core.windows.net/xxxxx-zone/';
$SAS="?sv=======xxxxxxxxxxxxxxxxxxx"
Write-Output "Current Directory..";
Write-Output $SOURCE;
Write-Output "1-datasource-1";
Write-Output "2-datasource-2";
Write-Output "3-datasource-3";
Write-Output "4-datasource-4";
$USER_CHOICE=Read-Host "Choose Datasource from Above..."




If($USER_CHOICE -eq 1 ){
	Write-Output "Your Choice is datasource-1";
	$FOLDER="datasource-1/";
	Write-Output "Zone Location Folder  ​= " $FOLDER
}


If($USER_CHOICE -eq 2){
	Write-Output "Your Choice is datasource-2";
	Write-Output "-------------------------------";
	Write-Output "1-entity1";
	Write-Output "2-entity2";
	Write-Output "-------------------------------";
	$ENTITY_CHOICE=Read-Host "Please Choose entity name..."
	If($ENTITY_CHOICE -eq 1){
		Write-Output "Your Choice is 1-entity1 Folder!";
		$FOLDER="datasource-2/1-entity1/";
	}
	ElseIf($ENTITY_CHOICE -eq 2){
		Write-Output "Your Choice is 2-entity2 Folder!";
		$FOLDER="datasource-2/2-entity2/";
	}
}


Write-Output "AZURE TARGET FOLDER..."
$TARGET = -join($DMZ_LOCATION,$FOLDER,$SAS)
Write-Output $TARGET
#uncomment below command once the $TARGET location found validated
.\azcopy cp $SOURCE $TARGET --log-level=INFO --recursive=TRUE --overwrite=TRUE --check-length=TRUE;

I am an Architect and I build things for earth’s most beautiful consciousness that evolved recently and rapidly, would say technology built by men.

Leave a Reply

Your email address will not be published. Required fields are marked *