Azure Platform Solutions (BigData)

Azure have many platform solution related to Machine learning AI, BigData and IOT and solution Azure offers.


Devices and sensors that are connected to each other and to internet are called IOT(internet of things). Below are some example of IOT.


Issues with IOT


IOT Services in Azure

  • Azure IOT Central
  • Azure IOT Hub 
  • Azure Sphere
Azure IOT Hub is a platform service which provide the building blocks for connecting devices to cloud, managing devices and ingesting data in to service. You can program solution which can enable secure bidirectional communication with million of devices. 

You don't have to provision the object that represent the devices in Azure either There is service in IOT hub that can automatically provision devices when they connect to the services so you can deploy devices out in world with less effort. 

Example when temperature reaches threshold start the fan. Delivery truck share the traffic data.
IOT hub is a platform and it has a different API and SDK for different programming language. 
With this developer can develop custom solutions, you can simulate device too. IOT hub also support authentication capabilities for securing communication for example using X509 certificate. 


IOT Central platform is another planform solution , like IOT hubs is a planform service IOT Central is considered managed app platform. It has many solution template to speed up development process. 

There are build in device profile in IOT central which define char of well known IOT devices like  type of telemetry they can send and type of  action we can perform on them.  There are low level hardware solution like Raspberry pie. 


Azure Sphere solve two main issues of IOT ( Standard and Security) 
is an application platform that is made of microcontroller unit a custom based operating system and cloud based security service. 
 Devices build with Azure sphere are integrated into a IOT solution that do the usual data collection and device management. Device using Sphere service verify security of OS on start up and MS pushes patches and security updates. 


In general IOT, Big data and Machine learning work together but you can use them individually. 


Demo Azure IOT Central.

WE have devices which we can create, we have many device template which are readily available from which you can create devices. After that you can set Rules over those device. You can even capture the related data and see that under Analytics and use that data for prediction. You can use Job to manage multiple devices and it depend on capability of devices.


Big Data  

Big data refer to technology and strategies that refer to first gathering the data, organizing the data, process the data and then use this data insights. 

Previously we were using Excel but that's not scalable due to large data.

Some usage of Big data are in 
Sales and Marketing example Netflix shows what to see next based other user choice
Risk management 
Product Design 
Supply chain Management ( when to fill the shelves)
Planning and Safety. 

There are three V's of Big Data

Volume
Velocity ( as data can come very fast)
Variety (csv, excel, sql server oracle, streaming etc)


Big data don't expect data to be formatted or organized. It uses distributed computing and single engine manages it. So in such cases we can use cloud scalable feature as not always all nodes might be processing. 

Categories of data processing 

Ingest: That is usually ETL operation , modify , filter bad data, validate. , but we mostly save as raw for flexibility 
We can Ingest data as Azure data factory, IOT hub SSIS, Azure analytics. Kafka.
Persist: We persist data using data warehouses , distributed file system Hadoop, Blob storage.

Analyze: Batch processing Splitting mapping reducing, assembling 
Hadoop Map resource , Apache Spark

Real Time Processing : Apache Storm, Azure Stream Analytics. 

We can analyze with 'R' Python Scala, SQL  Java C#

Visualize: Power BI, Excel, Jupyter Notebook



Big Data Solutions: Platform Solution to Work with Big data.

There are three platform solution in Azure Big data. 

Azure HD insight is a managed platform from MS for running open source analytics tools like Hadoop , Spark Kafka. You get cluster of compute nodes that can scale up and down on demand. as well as using auto scale. and you get integration with other data services like data factory and Data lake storage. , Cosmos DB Blob. So tool are there for you to create analytics pipeline. 

Feature of HD Insight consist of (Analytics engine)
Hadoop distributed file system
MapReduce for batch processing. 
Apache Spark Support open source tools and development environment like Visual studio, Eclipse and language like python, Scala, R


Azure data bricks 
is company is famous tools for data analysis so MS  hosted that data brick platform.
Based on Apache Spark platform. With Azure data brick you get fully managed spark cluster and interactive workspace using which you can visualize and explore the data.  You can create new cluster in seconds and you have option (server less) to abstract infrastructure. It has dashboard for interactive report. It has access to other services and support new language.


Azure Synapse Analytics 
was previously called SQL data warehouse. Datawarehouse are large ordered repositories of data that can be used for analysis and reporting. Data lake consist of more raw data before it's prepared for  Big data analysis. 
Azure Synapse Analytics is consist of multiple components. 

1)SQL data ware house component (Storage component)
2) Azure Synapse Analytics  (Workspace) 
    SQL, Spark technology used in data analysis, pipeline for orchestration. 
With server less or provision option deployment of analytics node is flexible. 
ETL feature is there
Integration with Power BI and azure services. 


Demo Azure Synapse Analytics:

First a Azure Synapse Analytics workspace is created
Then Launch Synapse Studio 
Then Go to Manage you see SQL pools (Pre build pool) You can scale up/down  or Pause the pools. 
Under Linked services /

Develop Tab you can query SQL pool or even connect to Storage service and query data from there. You can even do analytics. 
Under Orchestrate tab you can crate pipeline to get data in to data ware house. In example a notebook was dragged into pipeline. 

Under Monitor review the JOB.


Comments

Post a Comment

Popular posts from this blog

Azure Platform Solutions (Machine Learning and Cognitive Services)

Machine Learning: Machine learning is about using existing data to forecast future behaviors outcome and  trends.  Here you without programming you are making computer to learn by training the model (using known data). Based on Model Model can take unknown data and makes prediction. For ML we have Azure Machine learning Studio  Demo Create a machine learning workspace. Launch ML Studio. There are three ways to learn experiment.  Notebook: are jupiter notebook where you can write Python code.  here you load data from data set, You  create experiment, train model and run experiment on computer code.  You see result of experiment on Experiments tab Automated ML :   This is another way to create experiments. This help you find best matching ml alogrithm without even programming it. It's using AI to do Machine learning. Example shown if client will opt for fixed term load. So you can load the data and based on data you get result for Model/Algori...
Read more

Azure Core Product: Data Storage

Image
 As companies are producing more data than before there is need to store this data and access it anytime from any location. Azure had storage services for various data types. There are many benefits of storing this data in Azure Automated backup and recovery  Replication across world  Encryption option  Security and platform integration  Development feature and support. Managed Relational database in Azure SQL Server With VM You can host a SQL server on a VM that gives you full control. Or you can provision from Azure marketplace by using existing image.  Flexible pricing options are there. You can configure automated backup and automated updates scheduling.  Azure SQL DB Then there is another version of this i.e. platform as a service where you always have a latest version of SQL server running. Flexible pricing model in based on Virtual core or Distributed transaction unit (DTU) i.e. based on usage.  For deployment you can either have single DB ...
Read more