In this blog, I will introduce how to create an automation script that can deploy the Data Factory’s resources to Azure with a key press 🆒. So, I will reuse the resources on Data Factory - 3 basic things post for demonstration.
All steps that script will run are
Since powershell script can load a json file quickly so we create a config.json file like this
{
"location": "EastUS",
"resourceGroup": "InDark",
"subscriptionName": "Developer Program Benefit",
"dataFactoryName": "indarkdatafactory",
"dataFactoryLinkedServices": [
"./ProdSQLLinkedService.json",
"./DevSQLLinkedService.json",
],
"dataFactoryDatasets": [
"./Input_ProdUsersTable.json",
"./Output_DevUsersTable.json",
],
"dataPipelines": [
"./CopyPipeline.json"
],
}
The resourceGroup and subscriptionName both are information requires to fill while datafactory is installing.
Scripts to read config file on powershell
$config = get-content config.json | ConvertFrom-Json;
$location = $config.location;
$subscriptionName = $config.subscriptionName;
$resourceGroupName = $config.resourceGroup;
$dataFactoryName = $config.dataFactoryName;
In the first step, script will ask user to login
### Authenticate ###
$user = Login-AzureRmAccount;
if ($user -eq $null) { Return; }
Scripts to setup Data Factory
### Create data factory ###
$dataFactory = Get-AzureRmDataFactory -ResourceGroupName $resourceGroupName -Name $dataFactoryName -ErrorAction SilentlyContinue;
if ($dataFactory -eq $null) {
"creating data factory $dataFactoryName...";
$dataFactory = New-AzureRmDataFactory -ResourceGroupName $resourceGroupName -Name $dataFactoryName -Location $location -Force;
"data factory $dataFactoryName has been created successfully.";
if ($dataFactory -eq $null)
{
Write-Error "Cannot create data factory $dataFactoryName";
Stop-Transcript;
return;
}
}
else {
"data factory $dataFactoryName found";
}
Scripts to install Linked services
$linkedServiceFiles = $config.dataFactoryLinkedServices;
foreach($fileName in $linkedServiceFiles) {
"installing linked services $fileName...";
$linkedService = New-AzureRmDataFactoryLinkedService -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -File $fileName -Force -ErrorAction SilentlyContinue;
if ($linkedService -eq $null)
{
Write-Error "Cannot create linked service $fileName";
Stop-Transcript;
return;
}
}
Scripts to install Datasets
$datasetFiles = $config.dataFactoryDatasets;
foreach($fileName in $datasetFiles) {
"installing dataset $fileName...";
$dataset = New-AzureRmDataFactoryDataset -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -File $fileName -Force -ErrorAction SilentlyContinue;
if ($dataset -eq $null)
{
Write-Error "Cannot create dataset $fileName";
Stop-Transcript;
return;
}
}
Scripts to install Pipelines
$pipelineFiles = $config.dataPipelines;
foreach($fileName in $pipelineFiles) {
"installing pipeline $fileName...";
$pipeline = New-AzureRmDataFactoryPipeline -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -File $fileName -Force -ErrorAction SilentlyContinue;
if ($pipeline -eq $null)
{
Write-Error "Cannot create pipeline $fileName";
Stop-Transcript;
return;
}
}
Would you like to work with the coolest IT guys in Vietnam? Apply to one of the following jobs:
Job requires a high level of technical skills and experience within Microsoft technologies and offers high salary, exciting projects, and constant challenges in terms of technology and design.
Tell me moreWant to have fun developing innovative Xamarin products? We are developing a number of exciting games and social applications of our own as well as supporting third party clients.
Tell me moreJob requires both good English as well as the ability to understand complex technical subjects and systems. You will mainly be writing SEO articles and guidelines for our many products.
Tell me moreWe are looking to fill Developer positions with a new team that uses JavaScript, TypeScript, HTML5, AngularJS.
Knowing Ionic framework or NodeJs is a plus, but is not mandatory.
2 Senior Developer positions in an Offshore Development Center team. You will work directly with a Danish Project Manager at our Vietnam office. The initial project is for a multinational French company.
Tell me more1 Senior QA/Test Engineer in an Offshore Development Center team. You will work directly with a Danish Project Manager at our Vietnam office. The initial project is for a multinational French company.
Tell me more