Microsoft Integrate Big Data and AI, To Transform Apps and Business

Microsoft's data platform and cloud tools let developers and businesses create next gen intelligent apps, drive new efficiencies.

Share online:

At the Data Amp online event, Microsoft made several products and innovations announcement that integrate data and artificial intelligence (AI) to transform applications and business based on the following three main themes:

  • "close integration of AI functions into databases, data lakes, and the cloud, to simplify deployment of intelligent applications.
  • use of AI within Microsoft services, to enhance performance and data security.
  • flexibility Microsoft offers developers – to compose multiple cloud services into various design patterns for AI, to use Windows, Linux, Python, R, Spark, Hadoop, and other open source tools in building such systems."
SQL Server 2017 CTP 2.0 Preview

Among the annoucements, the first was the release of a relational database management system with built-in AI in a production-quality Community Technology Preview (CTP 2.0) of SQL Server 2017. "In this preview release, we are introducing in-database support for a rich library of machine learning functions, and now for the first time, Python support (in addition to R)," writes Microsoft.

Available on both Windows and Linux, this CTP brings a number of new capabilities, including: "ability to run advanced analytics using Python in a parallelized and highly scalable way, ability to store and analyze graph data, and capabilities to manage SQL Server for high performance and uptime, including Adaptive Query Processing family of intelligent database features and resumable online indexing."

This release also bring alonwith Python-based in-database analytics and machine learning in SQL Server with several advantages:

  • Elimination of data movement from the database to Python application or model.
  • Easy deployment as embedding it in a T-SQL script, and then any SQL client application can take advantage of Python-based models and intelligence by a simple stored procedure call.
  • Enterprise-grade performance and scale like in-memory table and column store indexes with high-performance scalable APIs in RevoScalePy package.
  • Rich extensibility with latest open source Python packages in SQL Server can be use to build deep learning and AI applications on huge amounts of data in SQL Server.
  • Python integration is available in all editions of SQL Server 2017, including the Express edition.
  • Data scientists can build models using full datasets on SQL Server instead of moving data to IDE or being forced to work with samples of data.
  • Application developers can take advantage of Python-based models by simply making a stored procedure call that has Python script embedded in it.
  • Database administrators can enable Python-based applications and set up policies to govern how Python runtime behaves on SQL Server.
  • Security is ensured by mechanisms like process isolation, limited system privileges for Python jobs, and firewall rules for network access.

The addition of Python builds on the foundation laid for R Services in SQL Server 2016 and extends that mechanism to include Python support for in-database analytics and machine learning. We are renaming R Services to Machine Learning Services, and R and Python are two options under this feature.

Public preview includes following enhancements for Analysis Services tabular.

  • Object-level security to secure model metadata in addition to data.
  • Transaction-performance improvements for a more responsive developer experience.
  • Dynamic Management View improvements for 1200 and 1400 models enabling dependency analysis and reporting.
  • Improvements to the authoring experience of detail rows expressions.
  • Hierarchy and column reuse to be surfaced in more helpful locations in the Power BI field list.
  • Date relationships to easily create relationships to date dimensions based on date columns.
  • Default installation option for Analysis Services is tabular, not multidimensional.

Other enhancements include:

  • New Power Query data sources. See this post for more info.
  • DAX Editor for SSDT. See this post for more info.
  • Existing Direct Query data sources support for M expressions. See this post for more info.
  • SSMS improvements, such as viewing, editing, and scripting support for structured data sources.

SQL Server Machine Learning Services with:

  • Real-time Scoring – Rx and MML
  • Improved Package install/uninstall; RxSync package (user-initiated restore)
  • Python Preview (subset of Rx algorithms, no MML)

SQL Server 2017 will also bring breakthrough performance, scale, and security features to data warehousing. "With up to 100x faster analytical queries using in-memory Columnstores, PolyBase for single T-SQL querying across relational and Hadoop systems, capability to scale to hundreds of terabytes of data, modern reporting, plus mobile BI and more," it provides a powerful integrated data platform for all your enterprise analytics needs.

For getting started with CTP 2.0, see these developer tutorials on how to install and use SQL Server 2017 on macOS, Docker, Windows, and Linux and quickly build an app in a programming language of choice.

Update 04/21: Resumable Online Index Rebuild is now available for public preview in the SQL Server vNext 2017 CTP 2.0 release.

With this release, a paused index can be resume from where the rebuild operation was paused rather than having to restart operation at the beginning.

In addition, this feature "rebuilds indexes using only a small amount of log space."

Graph extensions available in SQL Server 2017. A graph schema or database in SQL Server is a collection of node and edge tables.

The figure below shows the architecture of a graph database in SQL Server:

Graph Database in SQL Server 2017
Graph Database in SQL Server 2017
Microsoft R Server 9.1

Microsoft R Server 9.1, also released, "takes the concept of bringing intelligence to where your data lives to Hadoop and Spark, as well as SQL Server," Sirosh says. This release includes significant innovations such as:

  • New machine learning enhancements and inclusion of pre-trained cognitive models such as sentiment analysis & image featurizers
  • SQL Server Machine Learning Services with integrated Python in Preview
  • Enterprise grade operationalization with real-time scoring and dynamic scaling of VMs
  • Deep customer & ISV partnerships to deliver the right solutions to customers
  • A panoply of sources to help you get started with ease

R Server 9.1 can be downloaded from MSDN and Visual Studio Dev Essentials.

In the cloud, Microsoft Cognitive Services will enable you to infuse apps with cognitive intelligence. Now, the Face API, Computer Vision API, and Content Moderator are generally available in the Azure Portal.

  • Face API helps detect and compare human faces, organize faces into groups according to visual similarity, and identify previously tagged people in images.
  • Computer Vision API gives tools to understand the contents of any image: It creates tags that identify objects, beings like celebrities or actions in an image, and crafts coherent sentences to describe it. You can now detect landmarks and handwriting in images. Handwriting detection remains in preview.
  • Content Moderator provides machine-assisted moderation of text and images, augmented with human review tools.

Take a closer look at what these APIs can do for you in this video:

In other Microsoft Cloud new, also now general availability is Azure Analysis Services. Built on the proven business intelligence (BI) engine in Microsoft SQL Server Analysis Services, it delivers enterprise-grade BI semantic modeling capabilities with the scale, flexibility and management benefits of the cloud.

It integrates with many Azure data services enabling customers to build sophisticated analytics solutions.

  • Azure Analysis Services can consume data from Azure SQL Database and Azure SQL Data Warehouse. With customers are now adopting Azure Data Lake and HDInsight, Azure Analysis Services will soon offer ability to build BI models on top of these big data platforms, enabling a similar hub-and-spoke model as with Azure SQL Data Warehouse.
  • It can also consume data from on-premises data stores such as SQL Server, Oracle, and Teradata—-with support for several more data sources, both cloud and on-premises will add soon.
  • It can also be integrated into any Azure Data Factory pipeline by including an activity that loads data into the model. Azure Automation and Azure Functions can also be used for doing lightweight orchestration of models using custom code.
  • Power BI and Excel, both can connect to Azure Analysis Services models and offer a rich interactive experience. In addition, third party BI tools such as Tableau are also supported.
Microsoft Azure Analysis Services

H2O.ai, an AI platform is now available for users to use H2O.ai's open source solutions on Azure HDInsight Application Platform. By deploying H2O on HDInsight, "customers can easily build analytical solutions and run them at enterprise grade and scale."

Customer can install H2O during the creation of a new HDInsight cluster by simply selecting the customer applications when creating a cluster, selecting "H2O Artificial Intelligence for HDInsight", and agreeing to the license terms.

To learn more about H2O integration with HDInsight, see this how to H2O with Azure HDInsight tutorial.

H2O on Azure HDInsight

Microsoft also today introduces new monitoring and diagnostics capabilities in Azure, to help you discover and act on insights into your cloud and on-premises.

In addition to following capabilities, Microsoft also expanded support for Linux to monitoring tools and capabilities in Azure Automation & Control, including Linux patching and Linux file change tracking.

  • Map out process and server dependencies with Service Map, a new technology in Azure Insight & Analytics, to make it easier to troubleshoot and plan ahead for future changes or migrations.
  • Use DNS Analytics, a new solution in Azure Insight & Analytics, to help you visualize real-time security, performance, and operations-related data for your DNS servers.
  • Use Smart Diagnostics functionality in Azure Application Insights to diagnose sudden changes in the performance or usage of your web application.

You can also now ingest custom logs into Azure Application Insights for more powerful data correlations and analytics.

Azure Data Lake Analytics (ADLA), a breakthrough serverless analytics job service lets you develop and run massively parallel petabyte-scale data transformation programs that compose U-SQL, R, Python, and .NET.

"With no infrastructure to manage, you can process data on demand, scale instantly, and pay per job only," the team said.

Furthermore, the tech behind Cognitive Services is incorporated inside U-SQL directly as functions—so you can now process massive unstructured data, such as text/images, extract sentiment, age, and other cognitive features using Azure Data Lake, and query/analyze these by content.

Azure Data Lake Store (ADLS), a no-limit cloud HDFS storage system works with ADLA and other big data services for petabyte-scale data.

Both, Azure Data Lake Analytics and Azure Data Lake Store are now generally available in the Azure North Europe region.

Microsoft also seamlessly integrates data and AI with Azure DocumentDB is now integrated with Spark, to enable machine learning and advanced analytics on top of globally distributed data. "This service can significantly simplify the process of building distributed and intelligent applications at global scale."

"DocumentDB, a globally distributed, limitless NoSQL database service in Azure designed for mission-critical applications allows customers to distribute their data across any number of Azure regions worldwide, guarantees low read and write latencies, and offers comprehensive SLAs for data-loss, latency, availability, consistency, and throughput. "

"The Spark connector understands the physical structure of DocumentDB store (indexing and partitioning) and enables computation pushdown for efficient processing."

Microsoft Azure DocumentDB

In the cloud, Azure SQL Database, you can now turn on "Threat Detection" that using machine learning detect anomalous database activities indicating unusual and potentially harmful attempts to access or exploit databases around the clock.

Other features of Azure SQL Database such as "auto-performance tuning automatically implement, tune, and validate performance to guarantee the most optimal query performance."

Now also generally available is the 4TB Option for P11 and P15 databases with a 99.99 percent availability SLA.

As during preview, customers can use up to 4TB of included storage at no additional charge.

At Microsoft Data Amp, a limited previews for a new migration service and expanded SQL Database support for existing SQL Server databases is announced as well.

The new migration service helps moving existing SQL Server and Oracle databases to Azure SQL Database or SQL Server in an Azure virtual machine.

Additionally, SQL Database will now bring native VNET support and a large set of SQL server instance level features to help streamline lift and shift to Azure.

Apply to participate in the limited preview here.

Updates to Microsoft Intune on Azure now bring the following ability to help you manage mobility ecosystem from virtually any device and any browser, as well as:

  • managing increasingly larger numbers of devices and apps,
  • a modern micro-services cloud architecture,
  • enterprise-grade APIs,
  • reporting and automation support,
  • unified admins experience for all of Enterprise Mobility + Security (EMS),
  • and Role Based Access Controls (RBAC).
Microsoft Intue on Azure

Azure AD B2B Collaboration is now generally available and provides external user accounts with secure access to documents, resources, and applications—while maintaining control over internal data.

  • no need to add external users to directory, sync them, or worry about managing their lifecycle
  • IT can invite collaborators to use any email address—Office 365, on-premises Microsoft Exchange, or even a personal address (outlook.com, Gmail, Yahoo!, etc.)—and even set up Conditional Access Policies, including multi-factor authentication (MFA).
  • developers can use Azure AD B2B APIs to write applications that bring together different organizations in a secure way—and deliver a seamless and intuitive end user experience.

Azure Active Directory B2C, now generally available in Europe, is a highly available, global identity and access management service that enables organizations to securely connect with their customers at scale.

Azure Container Service (ACR) reaches the general availability now supporting a network-close, private registry for Linux and Windows container images.

"Azure Container Registry integrates well with orchestrators hosted in Azure Container Service, including Docker Swarm, Kubernetes and DC/OS as well as other Azure Services including Service Fabric and Azure App Services."

Customers can benefit from using familiar tooling capable of working with the open source Docker Registry v2.

Following features and capabilities are available now:

  • Availability in 23 regions, with a global footprint (with more coming)
  • Repositories, tag, and manifest listing in the Azure Portal
  • Dual-key password providing key rotation
  • Nested level repositories
  • Azure CLI 2.0 support

Watch this video to learn more about this Azure Container Registry GA:

Microsoft joining forces with Docker and Avanade, introducing a new program called "Modernize Traditional Applications," that will help enterprises embrace containerization and application modernization.

"It provides a low-cost engagement focused on containerizing an existing application with Docker EE in Azure in less than five days and with an overall engagement of four weeks," writes the team. "Customers can then choose to continue deploying additional containerized applications to Azure – in a public, hybrid or hosted environment."

Microsoft is also making container networking improvements to now support Docker swarm mode and Docker Enterprise Edition (Docker Datacenter).

For SQL Server 2017 and Azure SQL Database, Microsoft introduces a new set of adaptive query processing improvements that include: "batch mode memory grant feedback, batch mode adaptive joins, and interleaved execution. "

"The batch mode adaptive joins feature enables the choice of a hash join or nested loop join method to be deferred until the after the first input has been scanned."

The new Adaptive Join operator, defines a threshold that will be used to decide when we will switch to a nested loop plan.

Batch Mode Adaptive Joins in Microsoft Adaptive Query Processing

"Interleaved execution" changes the unidirectional boundary between the optimization and execution phases for a single-query execution and enables plans to adapt based on the revised estimates.

To simplify the development of AI systems, a set of new Cortana Intelligence solution templates that give customers means to rapidly conceive and implement big data, machine learning and analytics projects.

Cortana Intelligence on Azure

"Cortana Intelligence is a collection of fully managed big data and analytics services that can be composed together to build sophisticated enterprise-grade AI and analytics applications on Azure."

The first set of solution templates focus on following use cases:

  • Demand Forecasting & Price Optimization—template ensure product availability and maximize profits through intelligent demand forecasting. Customers can predict the volume of sales for a given time period and also model its sensitivity under various scenarios.
  • Personalized Offers —template lets you improve customer experience and maximize your customers' basket size through relevant targeted offers. Marketing professionals can predict and deliver the best performing offer based on customers' unique profiles.
  • Quality Assurance—template minimize wastage and improve quality by predicting production line failures before they happen.

You can visit Azure gallery for getting started with Cortana Intelligence solutions templates here.

AppSource, a single destination to discover and seamlessly try business apps built by partners and verified by Microsoft. Partners like KenSci have already begun to showcase their intelligent solutions targeting business decision-makers in AppSource.

Now partners can submit Cortana Intelligence apps at AppSource "List an app" page.

How you can visualize Azure Machine Learning models using MicroStrategy Desktop, Microsoft and MicroStrategy together helping users to create powerful, cloud-based machine learning (ML) applications through self-service analytics.

MicroStrategy Desktop, combined with Microsoft Azure ML, uses a drag-n-drop interface so users can "efficiently plan, create and glean insights from a predictive dashboard," Zeranski says.

Check out this PDF

This week at MicroStrategy World in Washington, D.C., Microsoft is conducting a hands-on workshop to show "how users can go from nothing to a fully-functional predictive data visualization built on machine learning within an hour," using Microsoft R Open, Azure Machine Learning and MicroStrategy 10 Desktop, Zeranski writes.

A Microsoft Azure Podcast featuring Azure Government, discuss latest and greatest of Azure Government including:

  • The two new DoD specific datacenters
  • How Azure Government is the only publicly available government cloud with DISA L5
  • Differences between Azure public and Azure Government
https://msdnshared.blob.core.windows.net/media/2017/04/Azure-Government-Podcast.mp3?_=1]

Power BI Desktop, releases an ability to connect to datasets in the Power BI service, and allows to create new reports off existing datasets already published to the cloud.

Following features are included:

  • Report view: Rename axis titles and experience new matrix visual enhancements including column sorting, column resizing, and word wrapping (preview).
  • Analytics: Quick measures and show value as (preview).
  • Modeling: Ability to set a model language to Spanish for the Q&A in Spanish (preview).
  • Data connectivity: Connect to datasets in the Power BI service (preview), Redshift Connector now includes beta support and publish to Power BI service, and SAP HANA & BW Connectors now provide enhancements to parameter input experience.
  • Query editing: Add column by example, split column (by delimiter or number of characters) into rows, group by-basic mode, and go to column.
  • Download the latest Power BI Desktop to experience the new features immediately. For more information on these new features and others, visit the Power BI blog.

Microsoft also made available some frequently requested Power BI features as part of April release:

  • Usage Metrics: Content owners can see detailed usage information for dashboard, reports, and datasets.
  • Related Content Pane: View and navigate between related dashboards, reports, and datasets with a single click.
  • Q&A in Spanish (Preview): Ask questions in Spanish for Spanish language data model.
  • Report Lifecycle: Download PBIX from content pack created reports.
  • Experience Improvements: improved experience when opening dashboards or reports which have not been accessed for long time.
  • New navigation UX becomes default (early May): The new (currently opt-in preview) navigation UX for the Power BI service becomes the default for all users. This new navigation UX makes it easier to discover and get to the content that matter most to users.

Microsoft Social Engagement Update 1.3, reached the general availability on April 5, introduces a new way to measure the success of social customer care activities: "the PowerBI Content Pack for Dynamics 365 – Microsoft Social Engagement."

This content pack is designed specifically for community managers, providing performance metrics for engagement actions taken from within Social Engagement. It also offers a new Conversation view, which shows you any conversation that involves private messages (Facebook or Twitter) and Twitter replies of one of your social profiles.

In-VM Scheduled Events on April 3, started in public preview, surfaces information regarding upcoming maintenance or user initiated events (for example, reboot) that may affect your Virtual Machine availability.

With this information, "you can prepare for potential availability issues and limit disruption."

The capability surfaces information about upcoming VM pauses, reboots, and redeploys, and helps you take actions such as reassigning VMs in a multi-instance workflow to avoid loss of replicas, completing (or canceling) in-flight transactions, reassigning tasks to other VMs in a cluster, or removing the Virtual Machine from a load balancer pool.

Scheduled Events is available for all Azure Virtual Machine types, including IaaS and PaaS.

Operations Management Suite | Log Analytics: Service Map GA, DNS Analytics

Map out process and server dependencies with Service Map, a new technology in Azure Insight & Analytics that makes it easier to troubleshoot and plan ahead for future changes or migrations.

Expand network visibility with DNS Analytics, a new solution in Azure Insight & Analytics that helps you visualize real-time security, performance and operations-related data for your DNS servers.

Also, now within the log analytics dashboard, you can remediate issues right away with a new menu option to Take Action and resolve an issue directly from a log search result.

Additionally, you can use the Smart Diagnostics functionality in Azure Application Insights to diagnose sudden changes in the performance or usage of your web application.

Azure management and security services help you to get greater visibility into your environment with advanced data analysis and visualization, and make it easy to turn insights into action. Learn more.

Microsoft introduces a public preview of Accelerated Networking for Azure Windows VMs with up to 25Gbps networking on a revised performance version!

Accelerated Networking enables network intensive workloads in Azure to achieve the speeds and consistency that are required from their applications while improving overall CPU utilization.

Azure Network Watcher now generally available helps you monitor and diagnose cloud Azure network with diagnostic and visualization tools that enable you to take remote packet captures on a VM, gain insights into your network traffic using flow logs, and diagnose VPN Gateway and Connections.