Azure Core Product: Data Storage

 As companies are producing more data than before there is need to store this data and access it anytime from any location. Azure had storage services for various data types. There are many benefits of storing this data in Azure

• Automated backup and recovery 
• Replication across world 
• Encryption option 
• Security and platform integration 
• Development feature and support.

Managed Relational database in Azure

SQL Server With VM

You can host a SQL server on a VM that gives you full control. Or you can provision from Azure marketplace by using existing image. 

Flexible pricing options are there. You can configure automated backup and automated updates scheduling. 

Azure SQL DB

Then there is another version of this i.e. platform as a service where you always have a latest version of SQL server running. Flexible pricing model in based on Virtual core or Distributed transaction unit (DTU) i.e. based on usage. 

For deployment you can either have single DB or Elastic pool i.e. multiple DB using same underlying infrastructure. 

Automated Scaling

Service tier for different workload. 

Azure SQL Managed Instance. 

MY SQL

Demo:

In SQL database we have three main things SQL Database, SQL Server and storage account. For database you have following settings. Main are connection string, Geo Replication for disaster recover as a read only DB. Synching facility to another DB.

Under Security you have dynamic data masking to limit sensitive data exposure by masking data to non previliged users. IN overview you have Server name which you can use to connect from remote client. Now to connect to database you need to first whitelist you client. For that you need to go to Server/Firewall and virtual networks  here you can provide client IP address. At server level you can also choose how data should be encrypted using option Transparent data encryption. Options are

1) Service Managed key(MS) 2) Customer managed key

User can run query even in azure web portal using option Query editor, just connect to DB and run query 

COSMOS DB for semi structured Data.

Cosmos DB is multi model globally distributed database. This is a good choice for many uses cases with server less application which require fast response time and needs to have ability to scale rapidly and globally. 

Structure is there is container (Item) and some objects ( Person, work family etc)      Object have different properties using which we can query the data. 

You can have on property common which can be used as a index. Though portal you can create new database container or SP, UDF.

WE can even configure cosmos to work with some existing analytics services. 

There is concept of consistency for example you configure that to Strong or Eventual and based on that user is guaranteed to see across different regions latest version of record  or not respectively. 

For COSMOS service you can select any API ranging from SQL, Mongo or Cassandra. 

AZURE Storage Accounts:

Azure storage is set of services which provide storing of various data type.  First two type are similar are for files except for the fact that File Storage supports SMB protocol so it can be attached to VM like a network drive. So it's seamless integration with on premises VMs.

Disk storage stores the VM discs used by infrastructure as service VM and discs are stored in a type of blob called page blob  in the blob service. There is table storage service that lets you store structured in form of non SQL non relational data similar to data stored in Cassandra or mongo db.

Then there is Queue used for asynchronous reliable application.

Azure data is durable and highly available. Data is stored three times in primary data center by default an you an copy data in other availability zone in region that support that. Still you can define to copy your data data to different region  for disaster recovery and data gets automatically copied over without need for you to manage it. 

Data can be reached from internet over https and each storage service within azure storage account has it's own REST endpoint. You can also apply security control to prevent unauthorized access. 

 As data security is so important   you can provide access to data using

1) role based access control for users with identities stored in Azure active directory 

2) you can provide storage account key that give access to entire storage. 

3) Provide user with shared access signature. 

Shared access Signature

Using above security feature you can access the data using following ways:

There are programming APIs from .net, etc and Azcopy command line tool which can be used to add or update the data. You can access data using even SDK or PowerShell. 

Azure Files

There are many times application which rely on accessing files, configuration  from network storage (example N: drive etc) When moving such apps to cloud we can use Azure files service which work as network drive on cloud VM and makes moving to cloud seamless for such application as it uses SMB protocol. We can even use BLOB but in that we need to use write api to access that data. 

Advantage is that we can use it anywhere in the world with shared access signature appended at the end.

Azure file share can be mounted cocurrantly by cloud or by on premises deployment of windows, Linux or Macos.  For mounting data from on premises VM we need to open port 445 which is used by SMB protocol, but for security reason we orgnaization don't weant to do that they can use Azure VPN gateway or epxress route to tunnel the traffic. You first need to setup a private storage account in order to do that. 

Azure file share can be cached on to the Windows server (premises) with Azure file sync for fast access. It actually allows you to tier files based on how they are used.  You need to install a sync agent which can manage the caching of most used files. 

Blob: Binary Large object

Blob is used to store unstructured data. 

1) Block blob is used to store text and binary data it's called block as blob is consist of diffrent block which make data upload easy. 

2) Append Block is also a type of Block blog but they are designed for appending. for example logs.

3) page blob is used to store VM disks and database as it has max 8 TB max blob size 

Manage cost of Blob data:

Blob Features:

You can create Blob snapshot, leasing blobs to other people from modifying it. You can enable soft delete so it can be retrieved later. Static website hosting. Blob service integrate with other services like Content delivery network so you can optimize the delivery of blob all over the world. Azure search feature integration allows to search within html, json docs excel spreadsheet, powerpoint.

Demo:

Azure portal/ List of storage account/ Open storage account/Storeage explorer. 

Here you can see Blob/file/Queue/table information.

You can right click on blobs to share it with some one (access key appended at end)

Under Access key : you can regenerate keys which will revoke existing rights. Under configuration you can you can change type of replication from locally redundant storage to Geo redundant storage)

Access tier. 

Encryption allows you to assign your own encryption key which is stored in azure encryption key vault. ON share access signature vault. 

in firewall or Virtual network option you can give access to data to either All network or to specific virtual network. 

Under Data Protection you can select soft delete and choose how long it stays. You can also enable the audit to see made changes to blob. Though snapshot a blob can be reverted to a state in time. 

You can also do Life cycle management for blobs by defining some rule, such that blob move from hot to cool tier let's say based on .

You can download Microsoft storage Explorer where you have all option on your machine like you can even download disc or take snapshot of VM etc...

Data Migration Options:

Azure DMS (Data migration service) is managed service to migrate database data from multiple data sources to Azure data platform with minimal downtime. You can migrate from either on premises database or from database already on cloud like Azure and Amazon.  

 There is online a and offline migration. 

Azure Database Migration Service is designed to support different migration scenarios (source/target pairs) for both offline (one-time) and online (continuous sync) migrations. 

MSDN Page

Offline migration DB is taken down while in Online migration downtime is limited to time to cut over at the end of the migration. Using Data migration assistant (DMA) you can verify that there is no compatibly issue before you move you DB to cloud. Its optional step to access the migration but it is still good to do that to avoid any issues later. 

Use the Azure Migration service Create migration product, provide source and target. Through Data transfer option by passing information like network bandwidth, amount of data we can see list of options.