Azure Data Factory - How to deploy by PowerShell

Exploring how to use and deploy Azure Data Factory

Posted by RITVN on December 02, 2017

Azure Data Factory - How to deploy by PowerShell

In this blog, I will introduce how to create an automation script that can deploy the Data Factory’s resources to Azure with a key press πŸ†’. So, I will reuse the resources on Data Factory - 3 basic things post for demonstration.

All steps that script will run are

  1. Read config.json file to gather configuration information.
  2. Login user with Azure account.
  3. Setup datafactory.
  4. Install Linked services.
  5. Install Datasets.
  6. Install Pipelines.

Setup Config File

Since powershell script can load a json file quickly so we create a config.json file like this

{
    "location": "EastUS",
    "resourceGroup": "InDark",
    "subscriptionName": "Developer Program Benefit", 
    "dataFactoryName": "indarkdatafactory",
    "dataFactoryLinkedServices": [
        "./ProdSQLLinkedService.json",
        "./DevSQLLinkedService.json",
    ],
    "dataFactoryDatasets": [
        "./Input_ProdUsersTable.json",
        "./Output_DevUsersTable.json",
    ],
    "dataPipelines": [
        "./CopyPipeline.json"
    ],
}

The resourceGroup and subscriptionName both are information requires to fill while datafactory is installing.

Scripts to read config file on powershell

$config = get-content config.json | ConvertFrom-Json;
$location = $config.location;
$subscriptionName = $config.subscriptionName;
$resourceGroupName = $config.resourceGroup;
$dataFactoryName = $config.dataFactoryName;

Authentication

In the first step, script will ask user to login

### Authenticate ###
$user = Login-AzureRmAccount;
if ($user -eq $null) { Return; }

Create Data Factory

Scripts to setup Data Factory

### Create data factory ###
$dataFactory = Get-AzureRmDataFactory -ResourceGroupName $resourceGroupName -Name $dataFactoryName -ErrorAction SilentlyContinue;
if ($dataFactory -eq $null) {
    "creating data factory $dataFactoryName...";
    $dataFactory = New-AzureRmDataFactory -ResourceGroupName $resourceGroupName -Name $dataFactoryName -Location $location -Force;
    "data factory $dataFactoryName has been created successfully.";
    if ($dataFactory -eq $null)
    {
        Write-Error "Cannot create data factory $dataFactoryName";
        Stop-Transcript;
        return;
    }
}
else {
    "data factory $dataFactoryName found";
}

Install Linked Services

Scripts to install Linked services

$linkedServiceFiles = $config.dataFactoryLinkedServices;
foreach($fileName in $linkedServiceFiles) {
    "installing linked services $fileName...";
    $linkedService = New-AzureRmDataFactoryLinkedService -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -File $fileName -Force -ErrorAction SilentlyContinue;
    if ($linkedService -eq $null)
    {
        Write-Error "Cannot create linked service $fileName";
        Stop-Transcript;
        return;
    }
}

Install Datasets

Scripts to install Datasets

$datasetFiles = $config.dataFactoryDatasets;
foreach($fileName in $datasetFiles) {
    "installing dataset $fileName...";
    $dataset = New-AzureRmDataFactoryDataset -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -File $fileName -Force -ErrorAction SilentlyContinue;
    if ($dataset -eq $null)
    {
        Write-Error "Cannot create dataset $fileName";
        Stop-Transcript;
        return;
    }
}

Install Pipelines

Scripts to install Pipelines

$pipelineFiles = $config.dataPipelines;
foreach($fileName in $pipelineFiles) {
    "installing pipeline $fileName...";
    $pipeline = New-AzureRmDataFactoryPipeline -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -File $fileName -Force -ErrorAction SilentlyContinue;
    if ($pipeline -eq $null)
    {
        Write-Error "Cannot create pipeline $fileName";
        Stop-Transcript;
        return;
    }
}

We are hiring

Would you like to work with the coolest IT guys in Vietnam? Apply to one of the following jobs:

C# / .Net Developer
Senior / Specialist
1000 – 3000 USD

Job requires a high level of technical skills and experience within Microsoft technologies and offers high salary, exciting projects, and constant challenges in terms of technology and design.

Tell me more
Xamarin Developer
Experienced
800 – 1500 USD

Want to have fun developing innovative Xamarin products? We are developing a number of exciting games and social applications of our own as well as supporting third party clients.

Tell me more
Technical Writer
Advanced English
2000 – 4000 USD

Job requires both good English as well as the ability to understand complex technical subjects and systems. You will mainly be writing SEO articles and guidelines for our many products.

Tell me more
Front-End Developers
Experienced
1000 – 2500 USD

We are looking to fill Developer positions with a new team that uses JavaScript, TypeScript, HTML5, AngularJS.
Knowing Ionic framework or NodeJs is a plus, but is not mandatory.

Tell me more
.NET/WPF Developers
Advanced English
1000 – 3000 USD

2 Senior Developer positions in an Offshore Development Center team. You will work directly with a Danish Project Manager at our Vietnam office. The initial project is for a multinational French company.

Tell me more
Ruby/Cucumber QA/Test engineering
Advanced English
1000 – 2000 USD

1 Senior QA/Test Engineer in an Offshore Development Center team. You will work directly with a Danish Project Manager at our Vietnam office. The initial project is for a multinational French company.

Tell me more