The general availability of Azure Databricks, a fast, easy, and collaborative Apache Spark-based analytics platform optimized for Azure, was announced on Thursday.
Designed in collaboration with Databricks, Azure Databricks aims to accelerate the process of building Big Data & AI solutions by combining best of Databricks and Azure.
To meet this goal, Azure Databricks is built up on three principles:
- Enhance user productivity in developing Big Data applications and analytics pipelines using interactive notebooks that enable data science teams to collaborate using popular languages such as R, Python, Scala, and SQL and create powerful machine learning models by working on all their data.
- Enable customers to scale globally without limits by working on big data with a fully managed, cloud-native service that automatically scales to meet their needs, without high cost or complexity.
- Ensure to provide customers with enterprise security and compliance “with enterprise-grade SLAs, simplified security and identity, and role-based access controls with Azure Active Directory integration.”
You can get started to try Azure Databricks, head over the site here.
To know more about how to get started with Apache Spark on Azure Databricks, follow this tutorial, where you can perform an ETL (extract, transform, and load data) operation using Azure Databricks. You then extract data from Azure Data Lake Store into Azure Databricks, run transformations on the data in Azure Databricks, and then load the transformed data into Azure SQL Data Warehouse.
The illustration above shows the application flow, while the image below shows Azure Databricks workspace.
Also, tdoay Microsoft made the new “Terraform” solution available in the Azure Marketplace. This solution will enable teams to use shared identity, using Managed Service Identity (MSI), and shared state using Azure Storage.
The features will allow you to use a consistent hosted instance of Terraform for DevOps Automation and production scenarios.