Posts

Data Science Series: Data Analytics what is it and types

 Data analytics is concerned with examining, transforming and arranging data so that you can study it and extract useful information. There are five types of data analytics:- 1. Descriptive Analytics - Helps you answer questions about what has happened based on historical data Eg generating reports to view organizations sale and financial data. 2. Diagnostics Analytics- Helps you answer questions why things happened. They supplement descriptive analytics. Take findings from descriptive analytics and dig deeper to find cause. Three steps a. Identify anamolies in data  b. Collect data related to anamolies c. Use statistical technologies to discover relationships and trends to explain these anamolies 3. Predictive Analytics- Help answer questions what will happen in future. Use historical data to identify trends and determine if they are likely to re-occur. Include statistical and ML techniques such as Neural networks, decision trees, regression. 4. Prescriptive Analytics - Help ...

Azure Devops: Setting up pipeline to bake container images and push to container repositories

Data Science: Regression: Part1

Regression is one of the most widely used supervised learning model in the field of data science. It finds its use in the field of Sales forecasting, satisfaction analysis, price estimation, employment income etc. The types of Regression are as follows:- 1. Simple Regression     a. Simple Linear Regression     b. Simple Non-linear Regression 2. Multiple Regression     a. Multiple linear Regression     b. Multiple non-linear Regression Pros of using linear Regression model:- 1. Very Fast 2. No Parameter tuning required 3. Easy to understand and highly interpret-able In order to implement regression we need a data. Suppose our dataset is residing in sample.csv We need to construct dataframes out of this sample dataset to be able to implement regression. We would be using python libraries "pandas", "scikit-learn" and "numpy" for the purpose import pandas as pd import numpy as np from sklearn import linear_model # creating d...

Securing Azure storage account and accessing data

Azure Data analytics: Part2: Benfits of using azure data lake storage Gen2

Image
Azure has bee providing great storage capabilities using Azure storage accounts and Azure Blob storage. Then why go in for Azure Data Lake storage? Below are the key benefits of using Azure Data Lake storage Gen2 over a normal storage for your analytics needs:- 1. Has tiering and data life cycle management capabilities. 2. Provides High availability, security and durability. 3. Designed for handling exabyte scale of data providing a throughput of hundreds of Gigabytes hence can be used for both real-time and batch solutions 4. Hadoop Compatible Access: You can treat the data as if it is stored in a Hadoop Distributed File system hence can be used directly with Azure DataBricks, HDInsights, Azure Synapse Analytics without moving data between environments 5. Security: Supports Access Control Lists (ACL) and Portable Operating System Interface (POSIX) permissions. You can set permissions at Directory level and file level. Data stored at rest is encrpted using Microsoft or customer manage...

Azure Data Analytics: Part1: Hosting Data Lake storage: Gen1 and Gen2

In this article we will cover hosting Azure data Lake storage Gen1 and gen2 1. Go to Resource Groups -> create new -> give a name 2. Under resource group -> add resources -> Storage -> Storage Account give storage account name, location Click Next to go to advanced tab 3. Under Data Lake storage Gen2 enable Hierarchical namesapce 4. Click create for gen1 Data Lake storage 1. Under resource group -> add resources -> Storage -> Data Lake Storage Gen1 2. Enter name and click create

Determine if an email address is hosted on O365

Image
Time and again we do come across many email ID's . With cloud becoming a de facto of IT services offered increasingly organizations are trying to offload their mail service to vendors. Office 365 from Microsoft has emerged as a leader in providing email as a service. To determine if an organization is hosting their mail services on O365 we can use an API call as below:- https://login.microsoftonline.com/getuserrealm.srf?login=username@acmecomputercompany.com&xml=1 The output would be an xml as below If the "NameSpaceType" indicates "managed" then the organization is using O365 for their email services. Other values are "federated" for "Federated active directory" Easy isnt it! So go out there and try it out. Post your comments if you find any issues and I will look into it!