Cloudbreak for Hortonworks Data Platform Now in Azure Marketplace

Azure Data Lake Store added as Event Hubs Capture provider, SQL TDE with BYOK support, preview of Power BI Report Server, and more now available.

Share online:

Cloudbreak, a cloud infrastructure tool, is now available for the Hortonworks Data Platform (HDP) in Azure Marketplace, and lets users easily provision, configure and scale HDP clusters in Microsoft Azure.

Hortonworks Data Platform is an enterprise-ready, open source Apache Hadoop distribution.

Cloudbreak is designed to support the following use cases:

  • Create clusters which you can fully control and customize to best fit your workload
  • Create on-demand clusters to run specific workloads, with data persisted in Azure Blob Storage or Azure Data Lake Store
  • Create, manage, and scale your clusters intuitively using Cloudbreak UI, or automate with Cloudbreak Shell or API
  • Automatically configure Kerberos and Apache Knox to secure your cluster
Cloudbreak for Hortonworks Data Platform in Azure Marketplace
Cloudbreak for Hortonworks Data Platform

Azure Event Hubs Capture, which enables you to send data to persistent storage without using code or configuring other compute services, on Monday, was added to Azure Data Lake Store as a new Capture provider.

With Capture now enabled on event hub, "it pulls data directly from Event Hubs to Azure Data Lake Store." Once enabled with Azure Data Lake Store as the provider, you can choose time and size window, to see events being captured in your chosen destination.

In addition, Capture will also manage all compute and downstream processing, writes program manager.

Event Hubs Capture provides the benefit of simple setup, reduced cost of ownership, no configuration overheads so you can focus on your apps while providing near-real time batch analytics.

Azure Toolkit for IntelliJ is now available for users running Spark to perform interactive remote debugging directly against code running in HDInsight.

Azure Toolkit for IntelliJ allows debugger to attach to Spark processes on HDInsight for direct remote debugging. Developers connect to HDInsight cluster at any time, leverage IntelliJ built-in debug capabilities, and automatically collect log files. Developers do not need to know the configurations of the cluster, nor understand the location of the logs.

Key benefits include:

  • Use IntelliJ to run and debug Spark application remotely on an HDInsight cluster anytime via "Run->Edit Configurations".
  • Use IntelliJ built-in debugging capabilities, such as conditional breakpoints, to quickly identify data-related errors. Developers can inspect variables, watch intermediate data, step through code, and finally edit the app and resume execution – all against Azure HDInsight clusters with production data.
  • Set a breakpoint for both driver and executor code. Debugging executor code lets developers detect data-related errors by viewing RDD intermediate values, tracking distributed task operations, and stepping through execution units.
  • Set a breakpoint in Spark external libraries allowing developers to step into Spark code and debug in the Spark framework.
  • View both driver and executor code execution logs in the console panel (see the "Driver Tab" and "Executor Tab").

Watch this demo of HDInsight Spark Remote Debugging:

Microsoft Azure will be changing the Azure SQL Database default compatibility level for newly created databases to 140, which is currently at 130.

As of today, the alignment of SQL versions to default compatibility levels are as follows: "100: in SQL Server 2008 and Azure SQL Database, 110: in SQL Server 2012 and Azure SQL Database, 120: in SQL Server 2014 and Azure SQL Database, 130: in SQL Server 2016 and Azure SQL Database, 140: in SQL Server 2017 and Azure SQL Database."

See this post, for more details on what compatibility level 140 specifically enables.

The preview of Transparent Data Encryption (TDE) with Bring Your Own Key (BYOK) support for Azure SQL Database and Azure SQL Data Warehouse is live as on Tuesday, enabling you to now have "control of the keys used for encryption at rest with TDE by storing these master keys in Azure Key Vault."

Enable TDE for Azure SQL Database and Warehouse
Enable TDE for Azure SQL Database and Warehouse

TDE with BYOK support gives you increased transparency and control over the TDE Protector, increased security with an HSM-backed external service, and promotion of separation of duties.

All features of Azure SQL Database and SQL Data Warehouse will work with TDE with BYOK support, and you can start enabling TDE with a key from Key Vault today using Azure Portal, PowerShell, and REST API, writes Azure team.

You can learn more about the feature including best practices, in this Transparent Data Encryption with Bring Your Own Key support, or watch this video:

The next version of Power BI Report Server in Preview now available includes: new self-service BI capabilities, including support for data sources other than SQL Server Analysis Services and viewing and interacting with Excel Workbooks.

With this August 2017 preview, users can create Power BI reports in Power BI Desktop that connect to any data source, and publish their reports to Power BI Report Server, writes BI team.

There's no special configuration required to enable this functionality, you can download /install and configure the August 2017 preview version of Power BI Report Server on your machine, and you're ready to go.

Update 08/30: Microsoft on Tuesday, released the public preview of Azure Active Directory (AD) Domain Services support for Azure Resource Manager virtual networks.

Azure Resource Manager lets users repeatedly deploy their app — and have confidence that their resources are deployed in a consistent state. Azure AD Domain Services helps users join Azure virtual machines to a domain without having to deploy domain controllers.

You can now create managed AD domains in virtual networks that were provisioned using Azure Resource Manager.

Azure Blob storage events, which reduce costs and complicated code, now in preview, allow applications to react to the creation and deletion of blobs without the need for complicated code and expensive polling services.

Azure Blob storage is a scalable, object-storage platform. It keeps costs down while storing hundreds to billions of objects in hot or cool tiers. It also supports any type of data, including images, videos, audio, documents and more.

This preview is currently available for Blob storage accounts in the US West Central location. Additional locations are coming soon.