Posts

Azure Devops: Setting up pipeline to bake container images and push to container repositories

Data Science: Regression: Part1

Regression is one of the most widely used supervised learning model in the field of data science. It finds its use in the field of Sales forecasting, satisfaction analysis, price estimation, employment income etc. The types of Regression are as follows:- 1. Simple Regression     a. Simple Linear Regression     b. Simple Non-linear Regression 2. Multiple Regression     a. Multiple linear Regression     b. Multiple non-linear Regression Pros of using linear Regression model:- 1. Very Fast 2. No Parameter tuning required 3. Easy to understand and highly interpret-able In order to implement regression we need a data. Suppose our dataset is residing in sample.csv We need to construct dataframes out of this sample dataset to be able to implement regression. We would be using python libraries "pandas", "scikit-learn" and "numpy" for the purpose import pandas as pd import numpy as np from sklearn import linear_model # creating d...

Securing Azure storage account and accessing data

Azure Data analytics: Part2: Benfits of using azure data lake storage Gen2

Image
Azure has bee providing great storage capabilities using Azure storage accounts and Azure Blob storage. Then why go in for Azure Data Lake storage? Below are the key benefits of using Azure Data Lake storage Gen2 over a normal storage for your analytics needs:- 1. Has tiering and data life cycle management capabilities. 2. Provides High availability, security and durability. 3. Designed for handling exabyte scale of data providing a throughput of hundreds of Gigabytes hence can be used for both real-time and batch solutions 4. Hadoop Compatible Access: You can treat the data as if it is stored in a Hadoop Distributed File system hence can be used directly with Azure DataBricks, HDInsights, Azure Synapse Analytics without moving data between environments 5. Security: Supports Access Control Lists (ACL) and Portable Operating System Interface (POSIX) permissions. You can set permissions at Directory level and file level. Data stored at rest is encrpted using Microsoft or customer manage...

Azure Data Analytics: Part1: Hosting Data Lake storage: Gen1 and Gen2

In this article we will cover hosting Azure data Lake storage Gen1 and gen2 1. Go to Resource Groups -> create new -> give a name 2. Under resource group -> add resources -> Storage -> Storage Account give storage account name, location Click Next to go to advanced tab 3. Under Data Lake storage Gen2 enable Hierarchical namesapce 4. Click create for gen1 Data Lake storage 1. Under resource group -> add resources -> Storage -> Data Lake Storage Gen1 2. Enter name and click create

Determine if an email address is hosted on O365

Image
Time and again we do come across many email ID's . With cloud becoming a de facto of IT services offered increasingly organizations are trying to offload their mail service to vendors. Office 365 from Microsoft has emerged as a leader in providing email as a service. To determine if an organization is hosting their mail services on O365 we can use an API call as below:- https://login.microsoftonline.com/getuserrealm.srf?login=username@acmecomputercompany.com&xml=1 The output would be an xml as below If the "NameSpaceType" indicates "managed" then the organization is using O365 for their email services. Other values are "federated" for "Federated active directory" Easy isnt it! So go out there and try it out. Post your comments if you find any issues and I will look into it!

The Emerging world of Data Analytics

With enterprises around the world getting heavily dependent on data to gain insights into their customer preferences, it has become imperative for them to invest heavily into Data Analytics. As organizations look into generate faster and better insights from data, getting data into the hands of business users and analytics applications has become important. However the dizzying array of data sources, slow and complicated steps to prepare and transform data for analytics present a roadblock. Sometimes the data's path from source to use is slow, disconnected and complex. There is a massive disconnect between the expected value of data and what companies are actually getting out of it. The reason behind the data analysts are unable to keep up with the accelerating rate, volume and complexity of data flowing into their organization. A unified data analytics platform can be a solution to all the data woes. The developers of such a platform should focus on where the traditional tech...