London cholera outbreak viz Tableau workbook

Data prep python code

The post Recreating John Snow’s Viz in Tableau first appeared on Perceptive Analytics.

Part I: To build the dashboards in Tableau, we built a WDC that can fetch the required data from FactSet’s servers. While developing the WDC, we used JavaScript extensively to get the ticker input by user, frame the appropriate URL, make the AJAX calls, receive information, parse the response & transform it into the requires shape so that it can be given to Tableau. HTML and CSS were used to build the user interface of the WDC where the Tableau user can input the ticker data. We dealt with Cross Origin Resource Sharing (CORS) limitations to fetch the required data. To overcome CORS, we had used a proxy server to route our request.

Part II: We built multiple dashboards to display the fetched data in a way useful for the analyst to understand the trading history of the stock/ticker, look into its financials, look into the broker estimates for future & make buy/hold/sell decision.

How to Get in touch
You can reach out to us by emailing cs AT perceptive-analytics.com

Tableau Expert

The post Tableau Web Data Connector for FACTSET first appeared on Perceptive Analytics.

Data in the right hands can be extremely powerful and can be a key element in decision making. American statistician, W. Edwards Deming quoted that, “In God we trust. Everyone else, bring data”. We can employ statistical measures to analyze data and make informed decisions. Tableau enables us to calculate multiple types of statistical measures like residuals, correlation, regression, covariance, trend lines and many more.

Today let’s discuss how people misunderstand causation and correlation using Tableau.

Correlation and Causation

Correlation is a statistical measure that describes the magnitude and direction of a relationship between two or more variables.

Causation shows that one event is a result of the occurrence of another event, which demonstrates a causalrelationship between the two events. This is also known as cause and effect.

Types of correlation:

1. 1 → Positive correlation.
2. -1 → Negative Correlation.
3. 0 → No correlation.

Why are correlation and causation important?

The objective of analysing data is to identify the extent by which a variable relates to another variable.

Examples of Correlation and Causation

1. Vending machines and obesity in schools: people gain weight due to junk food. One important source of junk food in schools is vending machines. So if we remove vending machines from schools obesity must reduce, right? But it isn’t true. Research shows that children who move from schools without vending machines to schools with vending machines don’t gain weight. Here we can find a correlation between children who were overweight and eating junk food from vending machines. In actuality, the “causal” point (which is the removal of vending machines from schools) has a negligible effect on obesity.
2. Ice cream sales and temperature: If we observe ice cream sales and temperature in the summer, we can determine that they are causally related; i.e. there is a strong correlation between them. As temperature increases, ice cream consumption also increases. Understanding correlation and causation allows people to understand data better.

Now let’s explore correlation using Tableau. We are going to use the orders table from the superstore dataset which comes default with Tableau.

Before going further let’s understand how to calculate the correlation coefficient ‘r’.

We can easily understand the above formula by breaking it into pieces.

In Tableau, we can represent the above formula as 1/SIZE() -1 where SIZE is function in Tableau.

We can use WINDOWSUM function for doing this summation in Tableau.

x_i is the sum of profit and x-bar is the mean of profit, which is window average of sum of profit, and s_x is standard deviation of profit. That means that we need to subtract mean from sum of profit and divide that by standard deviation.

(SUM([Profit])-WINDOW_AVG(SUM([Profit]))) / WINDOW_STDEV(SUM([Profit])))

This is similar to the formula above but we only need to swap profit with sales.

(SUM([Sales])-WINDOW_AVG(SUM([Sales]))) / WINDOW_STDEV(SUM([Sales])))

Now we have to join all these formulae to get the value of the correlation coefficient of r. Be careful while using parenthesis or you may face errors. Here is our final formula to calculate r.

1/(SIZE()-1) * WINDOW_SUM(( (SUM([Profit])-WINDOW_AVG(SUM([Profit]))) / WINDOW_STDEV(SUM([Profit]))) * (SUM([Sales])-WINDOW_AVG(SUM([Sales]))) / WINDOW_STDEV(SUM([Sales])))

Let’s implement this in Tableau to see how it works. Load superstore data into Tableau before getting started.

After loading the superstore excel file into Tableau, examine the data in the orders sheet. You can see that it contains store order details complete with sales and profits. We will use this data to find correlation between profit and sales.

Let’s get our hands dirty by making a visualization. Go to sheet1 to get started. I made a plot between profit and sales per category.

Now in order to find the correlation between profit and sales, we need to use our formula to make a calculated field which serves our purpose.

Now drag and drop our calculated field onto the colors card and make sure to compute using customer name as we are using it for detailing.

Here we can see the strength of the relationship between profits and sales of data per category; the darker the color, th he stronger the correlation.

Next we’ll add trend lines to determine the direction of forecasted sales.

These trend lines help demonstrate which type of correlation (positive, negative or zero correlation) there is in our data. You can explore some more and gain additional insights if you add different variables like region.

From this analysis we can understand how two or more variables are correlated with each other. We begin to understand how each region’s sales and profits are related.

Let’s see how a correlation matrix helps us represent the relationship between multiple variables.

A correlation matrix is used to understand the dependence between multiple variables at same time. Correlation matrices are very helpful in obtaining insights between the same variables or commodities. They are very useful in market basket analysis.

Let’s see how it works in Tableau. Download the “mtcars” dataset from this link. After downloading it, connect it to Tableau and explore the dataset.

The dataset has 35 variables where each row represents one model of car and each column represents an attribute of that car.

Variables present in dataset:

Mpg = Miles/gallon.

Cyl = Number of Cylinders.

Disp = Displacement (cubic inches)

Hp = Gross Horsepower

Drat = Rear axle ratio

Wt = Weight (lb/1000)

Qsec = ¼ mile time

Vs = V/Sec

Am = Transmission (0 = automatic, 1 = manual)

Gear = Number of forward gears

Carb =Number of Carburetors

Let’s use these variables to make our visualization. I made this amazing visualization showing correlation between models by referring to Bore Beran’s blog article, in which he explained how to make this visualization which helps us understand more about using Tableau to understand correlation.

Conclusion

We must keep in mind that if we want to measure the dependence between two variables, correlation is the best way to do it. A correlation value always lies between -1 and 1. The closer the value of the correlation coefficient is to 1, the stronger their relationship. We must remember that correlation is not causation and many people misunderstand this. There are many more relations and insights that can be unlocked from this dataset. Explore more by experimenting with this dataset using Tableau. Practice to be perfect.

The post Basic Statistics in Tableau: Correlation first appeared on Perceptive Analytics.

But do you think is it necessary to go to this level to segment the offerings?

I think the success of Netflix answers this question on its own. Netflix is considered to have one of the best recommendation engines. They even hosted a competition on Kaggle and offered a prize money of USD 1 million to the team beating their recommendation algorithm. This shows the sophistication and advanced capabilities developed by the company on its platform. This recommendation tool is nothing but a segmentation exercise to map the movies and users. Sounds easy, right?

Gone are the days when marketers used to identify their target customers based on their intuition and gut feelings. With the advent of big data tools and technologies, marketers are relying more and more on analytics software to identify the right customer with minimal spend. This is where segmentation comes into play and makes our lives easier. So, let’s first understand what is segmentation? and why do we need segmentation?

Segmentation, in very simple terms, is grouping of customers in such a way that that customers falling into one segment have similar traits and attributes. The attributes could be in terms of their likings, preference, demographic features or socio-economic behavior. Segmentation is mainly talked with respect to customers, but it can refer to products as well. We will explore few examples as we move ahead in the article.

With tighter marketing budgets, increasing consumer awareness, rising competition, easy availability of alternatives and substitutes, it is imperative to use marketing budgets to prudently to target the right customers, through the right channel, at the right time and offer them the right set of products. Let’s look at an example and understand why segmentation is important for marketers.

There is an e-commerce company which is launching a new service for a specific segment of customers who shop frequently and whose ticket size is also high. For this, the company wants to see which all customers to target for the service. Let’s first look at the data at an aggregate level and then further drill down to understand in detail. There are 5 customers for whom we want to evaluate the spend. The overall scenario is as follows:

Should the e-commerce company offer the service to all the five customers?

Who is the right customer to target for this service? Or which is the right customer segment to target?

We will see the details of each of the customers and see the distribution of data.

Looking at the data above, it looks like Customer 1 and Customer 2 would be the right target customers for company’s offering. If we were to segment these 5 customers into two segments, then Customer 1 and Customer 2 would fall in one segment because they have higher total spend and higher number of purchases than the other three customers. We can use Tableau to create clusters and verify our hypothesis. Using Tableau to create customer segments, the output would look like as below.

Customer 1 and customer 2 are part of cluster 1; while customer 3, customer 4 and customer 5 are part of cluster 2. So, the ecommerce company should focus on all the customers falling into cluster 1 for its service offering.

Let’s take another example and understand the concept further.

We will try to segment the countries in the world by their inbound tourism industry (using the sample dataset available in Tableau). Creating four segments we get the following output:

There are few countries which do not fall into any of the clusters because data for those countries is not available. Looking at clusters closely, we see that the United States of America falls in the cluster 4; while India, Russia, Canada, Australia, among others fall in the cluster 2. Countries in the Africa and South America fall in the cluster 1; while the remaining countries fall in the cluster 3. Thus, it makes it easier for us to segment countries based on certain macro-economic (or other) parameters and develop a similar strategy for countries in the same cluster.

Now, let’s go a step further and understand how Tableau can help us in segmentation.

Segmentation and Clustering in Tableau

Tableau is one of the most advanced visualization and business intelligence tool available in the market today. It provides a lot of interactive and user-friendly visualizations and can handle large amounts of data. It can handle millions of rows at once and provides connection support to almost all the major databases in the market.

With the launch of Tableau 10 in 2016, the company offered a new feature of clustering. Clustering was once considered a technique to be used only by statisticians and advanced data scientists, but with this new feature in Tableau it becomes as easy as simple drag and drop. This feature can provide a big support to marketers in segmenting their customers and products, and get better insights.

Steps to Becoming a Segmentation Sniper

Large number of sales channels, increase in product options and rise in advertisement cost has made it inevitable not only for marketers but for almost all the departments to analyze customer data and understand their behavior to maintain market position. We will now take a small example and analyze the data using Tableau to understand our customer base and zero-in on the target customer segment.

There is a market research done by a publishing company which is mainly into selling of business books. They want to further expand their product offerings to philosophy books, marketing, fiction and biographies. Their objective is to use customer responses and find out which age group like which category of books the most.

For an effective segmentation exercise, one should follow the below four steps.

1. Understand the objective
2. Identify the right data sources
3. Creating segments and micro-segments
4. Reiterate and refine

We will now understand each of the steps and use Tableau, along with, to see the findings at every step.

1. Understand the objective

Understanding the objective is the most important thing that you should do before starting the segmentation exercise. Having a clear objective is the most imperative thing because it will help you channelize your efforts towards the objective and prevent you from just spending endless hours in plain slicing and dicing. In our publishing company example, the objective is to find out the target age group which the company should focus on in each of the segments, namely philosophy, marketing, fiction and biography. This will help the publishing company in targeting its marketing campaign to specific set of customers for each of the genres. Also, it will help the company in identifying the target age group that like both business and philosophy or business and marketing, or similar other groups.

1. Identify the right data sources

In this digital age, data is spread across multiple platforms. Not using the right data sources could prove to be as disastrous as not using analytics at all. Customer data residing in CRM systems, operational data in SAP systems, demographic data, macro-economic data, financial data, social media footprint – there could be endless list of data sources which could prove to be useful in achieving our objective. Identifying right variables from each of the sources and then integrating them to form a data lake forms the basis of further analysis.

In our example, dataset is not as complex as it might be in real life scenarios. We are using a market survey data gathered by a publishing company. The data captures the age of customer and their liking/disliking for different genres of books, namely philosophy, marketing, fiction, business and biography.

1. Creating segments and micro-segments

At this stage, we have our base data ready in the analyzable format. We will start analyzing data and try to form segments. Generally, you should start by exploring relationships in the data that you are already aware of. Once you establish few relationships among different variables, keep on adding different layers to make it more granular and specific.

We will start by doing some exploratory analysis and then move on to add further layers. Let’s first see the results of the market survey at an aggregate level.

From the above analysis, it looks like fiction is the most preferred genre of books among the respondents. But before making any conclusions, let’s explore a little further and move closer to our objective.

If we split the results by age group and then analyze, results will look something like the below graph.

In the above graph, we get further clarity on the genre preferences by respondents. It gives us a good idea as to which age group prefers which genre. Fiction is most preferred by people under the age of 20; while for other age groups fiction is not among the top preference. If we had only taken the average score and went ahead with that, we would have got skewed results. Philosophy is preferred by people above the age of 40; while others prefer business books.

Now moving a step ahead, for each of the genre we want to find out the target age group.

The above graph gives us the target group for each of the genres. For biography and philosophy genres, people above the age of 40 are the right customers; while for business and marketing, age group 20-30 years should be the target segment. For fiction, customers under the age of 20 are the right target group.

Reiterate and refine

 In the previous section, we created different customer segments and identified the target segment for publishing company. Now, let’s say we need to move one more step ahead and identify only those age groups and genres which have overlap with business genres. To put it the other way, if the publishing company was to target only one new genre (remember, they already have customer base for business books) and one age group, which one should it be?

Using Tableau to develop a relation amongst the different variables, our chart should look like the one below.

Starting with the biography genre, age group 30-40 years comes closest to our objective, i.e., people in this age group like both biography and business genre (Biography score – 0.22, Business score – 0.31). Since, we have to find only one genre we will further explore the relationships.

For fiction, there is no clear overall with any of the age groups. For marketing, age group 20-30 year looks to be clear winner. The scores for the groups are – marketing – 0.32, business – 0.34. The relation between philosophy and business is not as strong as it is for business and marketing.

To sum it up, if the publishing company was to launch one more genre of books then it should be marketing and target customer group should be in the range of 20-30 years.

Such analysis can be refined further depending on the data we have. We can add gender, location, educational degree, etc. to the analysis and further refine our target segment to make our marketing efforts more focused.

I think after going through the examples in the article, you can truly appreciate the level of segmentation that Netflix has done and it clearly reflects the reason behind its success.

The post Tableau for Marketing: Become a Segmentation Sniper first appeared on Perceptive Analytics.

Business heads often use KPI tracking dashboards that provide a quick overview of their company’s performance and well-being. A KPI tracking dashboard collects, groups, organizes and visualizes the company’s important metrics either in a horizontal or vertical manner. The dashboard provides a quick overview of business performance and expected growth.

The post Tableau Sales Dashboard Performance first appeared on Perceptive Analytics.

TABLEAU FILTERING ACTIONS MADE EASY

The post Tableau Filtering Actions Made Easy first appeared on Perceptive Analytics.

Give me six hours to chop down a tree and I will spend the first four sharpening the axe.

—- Abraham Lincoln

The above Abraham Lincoln quote has a great influence in the machine learning too. When it comes to modeling different machine learning models most of the time need to spend on data preprocessing and feature engineering stages.

The general idea of feature engineering is to identify the influence features over all the available features. So the identified features used to train the model.

Identifying the influence features doesn’t mean picking the features in an analytical way.  Some times converting latent features into a meaningful feature. This is known as dimensionality reduction.

In this article, you will learn the basic concept to perform the dimensionality reduction with one famous approach know as Principal component analysis. In short PCA

Before we drive further let’s look at the table of content for this article.

Table of contents:

• Lifting the curse using principal component analysis
  • Curse of dimensionality in layman’s terms
• Shlen’s paper nuggets on Principal component analysis
• PCA conceptual background
• Principal component analysis implementation in R programming language
  • Loading Iris data set
  • Covariance matrix calculation
  • Eigen values and Eigen vectors calculation
  • PCA component calculation
  • Importance of the components
• Summary

The main intention of this article is to explain how to perform the principal component analysis in R.

So let’s begin.

Lifting the Curse using Principal Component Analysis

Many problems in Analytics are often visioned to have incomplete data with a few features. This led to the development of the common myth.

Having more features and more data will always improve the accuracy of solving the machine learning problem.

In reality, this is a curse more than a boon.

Sometimes, we face a situation. When there are a lot of features with few data points. Fitting a model in this scenario often leads to a low accuracy model even with many features. It has been encountered so many times that it is also called as the “Curse of dimensionality”.

Curse of dimensionality in layman’s terms

In layman’s language, the curse of dimensionality refers to the phenomenon when an increase in the number of features results in decrease the model accuracy.

We can understand the curse of dimensionality in another way also. Which is like increase in the number of features increase the model complexity (more precise the model complexity increase exponentially)

There are two ways to stay away from this curse of dimensionality.

1. Add more data to the problem.
2. Reduce the number of features in the data.

Adding data may not be possible in many scenarios as the data is often limited to what was collected. Reducing the number of features, a technique is known as

Adding data may not be possible in many scenarios as the data is often limited to what was collected. Reducing the number of features is more preferable. Such a technique is known as “Dimensionality reduction” is thus more preferable. PCA or Principal component analysis is a very popular dimensionality reduction technique.

Shlen’s paper nuggets on Principal component analysis

Principal component analysis aptly described in the famous Shlen’s paper.

Shlen’s Principal component analysis paper

The paper explains that even a simple problem such as recording the motion of a pendulum, which moves in only one direction. If one is unaware of the exact direction. The number of cameras required to record its movement will at least be three, given that we are able to place the cameras perpendicular to each other.

If we do not have the knowledge of keeping the cameras perpendicular to each other, then more cameras will be required. The problem then keeps on increasing and more and more cameras are required as available information keeps on decreasing.

The PCA technique transforms our features (or cameras) into a new dimensional space and represents it as a set of new orthogonal variables so that our problem is observed with a reduced set of features.

These orthogonal features are known as “Principal Components”.

In practice, our data is like the motion of the pendulum. If we had complete knowledge of the system, we will require a small number of features. Without the knowledge, we have to observe the system using a set of features which will convey maximum information if they are orthogonal. This is done using principal component analysis.

The new set of features which are produced after PCA transformation are linearly uncorrelated as they are orthogonal. Moreover, the features are arranged in decreasing order of their importance.

This means that the first principal component alone will explain a very large component of the data. The second principal component will explain less than the first major component but more than all other components. The last principal component will explain only a small change in the data. Typically, one can run PCA and take the top principal components such that they together explain most of the data. In most analytical problems, explaining 95-99% of the is considered very high.

PCA Conceptual Background

Before getting our hands dirty with PCA in R. Let’s understand the concept with a simple example.

Suppose we have a dataset with ‘m’ data points and ‘n’ features. We can call this dataset as a m*n matrix A

A₁₁ A₁₂ … A_1n

A= A₂₁ A₂₂ … A_2n

…………

A_m1 A_m2 … A_mn

We will transform this matrix A to A’ such that A’ is a m*k matrix and k<n. The number ‘k’ represents the subset of the number of transformed features.

How do we transform a given set of features into a new feature set such that they are orthogonal?

The answer is the eigenvectors of the matrix.

We know that eigenvectors are orthogonal to each other so transforming our features in the direction of the eigenvectors will also make them orthogonal. But wait! Before transforming a matrix, it is always recommended to normalize.

If the matrix is not normalized, our transformation will always be in favor of the feature with the largest scale of values. This is why PCA is sensitive to the relative scaling of the original variables.

We do that by observing the loadings of the PCA. Loadings are the factors which show what amount of load is shared by a particular feature. So, the component with maximum loading value can be considered as the most influential feature.

The loadings are arranged in such a manner that the higher significance factors are placed first, followed by subsequent factors. On the basis of our tolerance level or precision level required, we select the number of components to be considered. One important thing for PCA to get started is to get the same type of data sets, and therefore, we normalize the original data. In order to capture the variances, we need to identify the distribution of data, and therefore we transform the data in a normal distribution.

Variance is a nice measure of letting us know the kind of distribution. If we can maximize the variance, then our information is maximized. PCA can be done on any distribution, but as variance measure is the best parametric measure for normal distribution and also, data sets generally trend to follow the normal distribution or can be transformed easily to be made to follow the normal distribution, so we normally take normal distribution variance analysis in PCA.

To capture the variance of each feature with respect to other feature we try to get the variance – covariance matrix of the features, followed by finding out the eigenvalues of the matrix and then finding the eigenvectors of the matrix of the dataset which gives the various principal components.

We now construct the covariance matrix of A by multiplying A with its transpose A^t.  This will give us a m*m matrix Cov A

Cov A = A.A^t

We can now simply calculate the eigenvectors and eigenvalues of the Covariance matrix Cov A. The Eigenvalues will represent the relative variance of the data.

Principal component analysis implementation in R programming language

Now that we understand the concept of PCA. We can implement the same in R programming language.

The princomp() function in R calculates the principal components of any data. We will also compare our results by calculating eigenvectors and eigenvalues separately. Let’s use the IRIS dataset.

Let’s start by loading the dataset.

The iris dataset having 150 observations (rows) with 4 features.

Let’s use the cov() function to calculate the covariance matrix of the loaded iris data set.

The next step is to calculate the eigenvalues and eigenvectors.

We can use the eigen() function to do this automatically for us.

We have calculated the Eigen values from the data. We will now look at the PCA function princomp() which automatically calculates these values.

Let’s calculate the components and compare the values.

Let’s now compare the output variances

Output

Output

There is a slight difference due to squaring in PCA_data but the outputs are more or less similar. We can also compare the eigenvectors of both models.

Output

This time the eigenvectors calculated are same and there is no difference.

Let us now understand our model. We transformed our 4 features into 4 new orthogonal components. To know the importance of the first component, we can view the summary of the model.

Importance of components

From the Proportion of Variance, we see that the first component has an importance of 92.5% in predicting the class while the second principal component has an importance of 5.3% and so on. This means that using just the first component instead of all the 4 features will make our model accuracy to be about 92.5% while we use only one-fourth of the entire set of features.

If we want the higher accuracy, we can take the first two components together and obtain a cumulative accuracy of up to 97.7%. We can also understand how our features are transformed by using the biplot function on our model.

The X-Axis represents the first principal component. We see that the Petal Width and Petal Length vectors are parallel to the X-Axis. Hence they are combined and completely transformed into the first principal component. The first component also contains some part of Sepal Length and Sepal Width. The vertical part of the Sepal Length and Sepal Width Vectors are explained by the second principal component.

To determine what should be an ‘ideal’ set of features we should take after using PCA, we use a screeplot diagram. The screeplot() function in R plots the components joined by a line. We look at the plot and find the point of ‘arm-bend’. This is the point where the cumulative contribution starts decreasing and becomes parallel to the x-axis.

This plot shows the bend at the second principal component.

Let us now fit two naive Bayes models.

1. one over the entire data.
2. The second on the first principal component.

We will calculate the difference in accuracy between these two models.

Accuracy of the first model

Accuracy of the second model

We can see that there is a difference of 3 predictions between the two models. In return for reducing our data to one-fourth of the original, we lose on the accuracy only slightly. Hence it is a great tradeoff.

With this, we came to the end of the article. Let’s summarize key points.

Summary

PCA is a very popular method of dimensionality reduction because it provides a way to easily reduce the dimensions and is easy to understand. For this reason, PCA has been used in various applications from image compression to complex gene comparison. While using PCA, one should keep in mind its limitations well.

PCA is very sensitive to the scale of the data. It will create an initial basis in the direction of the largest variance in the data. Moreover, PCA applies a transformation over the data where all new components are orthogonal. The new features may not be interpretable in business.

Another limitation of PCA is the reliance on the mean and variance of the data. If the data has a relationship in higher dimensions such as kurtosis and skewness then PCA may not be the right technique to use on the data. In situations when the features are already orthogonal to each other and are uncorrelated, PCA will not produce any useful results except ordering the features in decreasing order of their variances.

PCA is very useful in situations when the data at hand is very large. Example, in case of image compression, PCA can be used to store the image in the first few hundred components and use less number of pixels.

The entire code used in the article is as follows:

You can also clone the complete code from DataAspirant GitHub.

The post How to Perform the Principal Component Analysis in R first appeared on Perceptive Analytics.

Histogram in R

How to create histograms in R

To start off with analysis on any data set, we plot histograms. Knowing the data set involves details about the distribution of the data and histogram is the most obvious way to understand it.

Besides being a visual representation in an intuitive manner. It gives an overview of how the values are spread.

We come across many depictions of data using histograms in our day to day life. For example, the distribution of marks in a class can be best represented using a histogram and so does the age distribution in an organization.

The good thing about histograms is that it can visualize a large amount of data in a single figure and convey lots of information.

It is quite easy to spot the median and mode by looking at histograms. A histogram can also indicate possible outliers and gaps in data. Thus a single figure can help know a lot about data.

So in this article, we are going implement different kinds of histograms. Starting with the basic histogram and to customize it to a great extended.

Before we drive further let’s look at the table of content for this article.

Table of contents:

• Basics of Histogram
• Implementing different kinds of Histograms

Basics of Histogram

A histogram consists of bars and is made for one variable at a time. That’s why knowledge of plotting a histogram is the foundation of univariate descriptive analytics.

To plot a histogram, we use one of the axis as the count or frequency of values and another axis as the range of values divided into buckets.Let’s jump to plotting a few histograms in R.

Implementing different kinds of Histograms

I will work on two different datasets and cite examples from them. The first data is the AirPassengers data.

This data is a time series denoting monthly totals of international airline passengers. There are 144 values from 1949 to 1960.

Let us see how the data looks like. The plot() function creates a plot of the time series.

Plot Air Passengers Data

Histogram for Air Passengers Data with Time

Some patterns are inherently visible in the time series. There are trends and seasonality component. The plot clearly shows how the values gradually increase from 100 to 600 due to increasing trend with a repeating seasonality pattern across years.

We can now use the built-in function hist() to plot histogram of the series in R

Histogram for Air Passengers Data with Frequency

Histogram for Air Passengers Data with Frequency

This plot is indicative of a histogram for time series data. The bars represent the range of values and their height indicates the frequency.

The data shows that most numbers of passengers per month have been between 100-150 and 150-200 followed by the second highest frequency in the range 200-250 and 300-350.

Since it is a time series with a gradual seasonality and trend, most of the values are towards the lower end of the spectrum.

That is why the histogram shows a decreasing trend as the values increases. Had it been a time series with a decreasing trend, the bars would have been in increasing order of the number of Air Passengers.

Let’s get into the game

The Air Passengers data was a single variable data. I will now use the iris dataset to help understand more about histograms.

The histogram can plot only one variable at a time. For plotting features of the iris dataset, the $ notation is used to specify the specific variable I start with plotting the petal length.

Petal Length in Distribution

Petal length is distributed

The data shows a clear demarcation of three clusters. The clusters have values from 1-2, 3-5 and 5-7 respectively. Let’s make the histogram to get additional insights

Histogram for iris petal length

Histogram for iris petal length

There are a lot of values in the range 1-1.5 and a small number of values between 1.5-2. This range corresponds to the first cluster. The second and third clusters somewhat overlap. This can be seen from the rest of the values.

There is no gap and the maximum number of values lie between 4-5. There are a few values in the range 2.5-3.5 and beyond 6 which can belong to cluster 2 and cluster 3 respectively.

The iris dataset also contains a non-numeric feature – Species. The plot function can give the count of each species. For this feature, the hist() function will give an error.

Distribution of Species in Iris data

Distribution or Species in iris data

Output:

As apparent as it is, the plot function provides a count of all the values and thus histogram is not used to show the distribution of non-numeric features.

However, the hist() function in R is very rich. You can specify a lot of parameters. The important ones are specifying the axis, title, and color of the histogram. You can also specify limits to the axis and change bin size

Adding cheery to the cake – parameters for hist() function

Before looking at the commonly used parameters for a histogram, we first use the help function.

Get the documentation for hist() function

Documentation Output

The hist function contains about 20+ parameters. We first look at the way to describe the title to the plot and the labels to x and y-axis. The parameter main=”” specifies the title and labels to the axis are plotted by xlab and ylab for x-axis and y-axis respectively. I will go step by step and set these values for iris petal length feature

Adding the labels

Histogram for petal length with labels

The next step is to add colors and border. The border parameter and col parameter can be used to set this.

Histogram for petal length with Labels and Colors

Now it looks a little lovelier. To make the y-axis indexes more readable, we can rotate it using the las function. las is 0 by default and is parallel to axis. It can be 1 for projecting horizontally. las=, 2 makes x-axis indexes perpendicular to axis along with y axis indexes 3 for y axis indexes parallel to axis while keeping x axis perpendicular.

Histogram with y-axis indexes horizontal

We now change the bars. hist() function allows setting the limits of the axis using the xlim and ylim parameters. We can also set the length of the bars using the breaks parameter

Histogram with axis limits

Keep in mind that setting the breaks can be done by keeping breaks as a vector as well. Another parameter include.lowest also goes with the breaks when it is a vector. When this parameter is set to true, setting right = FALSE or TRUE will keep all values equal to breaks values in the left bars or right bars respectively.

Additionally, there are two types are histograms. This one is the common one which plots the frequencies of values. A similar plot can be made using probability by setting the freq=FALSE or probability=TRUE parameters.

Probabilistic Plot

The hist() function also provides color shading. The density and angle parameters come in the picture for this. We can set the color density in lines per inch and the angle of the lines. Let’s try an example with density set to 50 and an angle of 60

Histogram with Color Density in Lines

Additionally, it may not even be necessary to get a plot. We can get the output in our console. The plot parameter, when set to FALSE, gives us the relevant. Let’s try it out

Console Output

The final parameter which I am going to explore is the labels parameter. To get more clarity on our plots, the labels parameter writes the exact value the bar is representing when set to TRUE

Drawing Labels on Bars

Isn’t this a lot of features a built-in histogram function can do? With histograms, one can know so much about the data and make the plots look good as well. For practice, here is the complete R code used in this article.

You can get this post complete code on our Github account.

The post How to Create Histograms in R first appeared on Perceptive Analytics.

As a result, each programmer has his/her own style in writing implementations to the same part of an algorithm.

Writing code can sometimes be the most difficult and time-consuming part of any project. If the code is written in such a way that it is hard to change or requires a lot of work for every small update, then the investments will keep on piling up and more and more issues will crop up as the project progresses.

A good and well-written code is reusable, efficient and written cleverly by a smart programmer. This is what differentiates programmers from each other.

So, here are some tips to becoming a SMART coder:

Table of contents:

• Writing codes for Programmer, Developer, and Even for A Layman
• Knowing how to improve the code
• Writing robust code
• When to use shortcuts and when not to use
• Reduce effort through code reuse
• Write  planned out code
• Active memory management
• Remove redundant tasks
• Learn to adapt
• Peer review

1. Writing Codes for Programmer, Developer, and Even for A Layman

Though codes are primarily written for the machine to understand. They should be structured and well organized for other developers or for any layman to understand. In reality, codes should be written for all the three.

Those who keep this fact in mind are one step ahead of other coders while those who are able to make sure everyone can understand their code are miles ahead than their struggling friends.

Good programmers always document their codes and make use of IDE. I will use R language to explain the concept. Using IDE such as Rstudio makes it easier to write code quickly.

The main advantage available in almost all IDE is the auto-completion feature which suggests the function or command when part of it is written.

IDE is also known to suggest the syntax of the selected functions which saves time. Rstudio IDE environment also displays environment variables alongside with some basic details of each variable.

Documentation is another ability which differentiates good programmers from the rest.

Let’s look at this viewpoint using an example. Say you read the following code:

Code snippet 1

Code snippet 2

Code snippet 3

The difference in documentation is highlighted in these three code snippets and this is just a simple demonstration of code understandability.

The first code is difficult to understand. It just sets the values of three variables. There are no comments and the variable names do not explain anything.

The second code snippet explains that ‘a’ is the maximum memory, ‘b’ is the minimum memory and ‘c’ is the mean of the two.

Without the comments in code snippet 2, no one can understand whether the calculation for ‘c’ is correct or not.

The third code is a step further with the variables representing what is stored in them.

The third code is the easiest to understand even though all the three codes perform similar tasks. Moreover, when the variables are used elsewhere, the variables used in the third snippet are self-explanatory and will not require a programmer to search in the code for what they store until an error occurs in the code.

2. Knowing how to Improve

R has multiple ways to achieve a task. Each of the possibilities comes from using more memory, faster execution or different algorithm/logic.

Whenever possible, good programmers make this choice wisely.

R has the feature to execute code in parallel. Lengthy tasks such as fitting models can be executed in parallel, resulting in time-saving. Other tasks can also be executed faster based on the logic and packages used.

As an illustration, the following code snippets reflects the same task, one with sqldf package and another with dplyr package.

Using sqldf version

Using dplyr version

I personally prefer the dplyr version whenever possible. However, there are some differences between the outputs.

The dplyr version will look at all variables with the same name and join using them. If there is more than one such variable, I need to use them by field. Moreover, left join using dplyr will not keep both copies of the variable used to join tables whereas sqldf does.

One advantage of sqldf is that sqldf is not case sensitive and can easily join tables even if the variable names in the two tables are completely different. However, it is slower than dplyr.

3. Writing Robust Code

While writing code, you can make the code simple but situation specific or write a generic code. One such way in which programmers write simple but situation-specific code is by ‘Hard Coding’.

It is the term given to fixing values of variables and is never recommended.

For example, dividing the sum of all salaries in a 50,000-row salary data by 50,000 rather than dividing the sum of that sum with the number of rows may seem to make the same sense but have a different meaning in programming.

If the data changes with the change in the number of rows, the number 50,000 needs to be searched and updated. If the programmer misses making the small change, all the work goes down the drain. On the other hand, the latter approach automatically does the task and is a robust method.

Another popular programming issue quite specific to languages such as R is Code Portability. Codes running on one computer may not work on another because the other computer does not have some packages installed or has outdated packages.

Such cases can be handled by checking for installed packages first and then installing them. These tasks can be collectively called as robust programming and make the code error free.

Using an illustration for checking and installing/updating h2o package.

4. When to Use Shortcuts and When Not to

Using shortcuts may be tempting in the pursuit of writing code swiftly but the right practice is to know when to use them.

For instance, shortcut keys are something which is really helpful and can always be used. Using Ctrl+L in windows clears the console output screen, Using Ctrl+Shift+C in windows comments and un-comments all selected lines of code in one go are my favorite shortcuts in Rstudio.

Another shortcut is writing code for fixing code temporarily or writing faulty fixes which are not desired.

Here are some of the examples of faulty fixes.

This code changes a particular column name without checking its existing name

This removes certain columns using a number. This may remove important ones and code may give the error if the number of columns less than 10 in this case.

This converts a value to numeric without checking if it actually has all numbers. If the value does not contain numbers, it may produce NAs by coercion

The following converts Num_val to 123 correctly

The following issues a warning and converts Num_val to NA as it is not a number

5. Reduce Effort Through Code Reuse

When you start writing a code, you don’t need to waste time if a particular piece of logic has already been written for you. Better known as “Code Re-use”, you can always use your own code you previously wrote or even google to reach out the large R community.

Don’t be afraid to search. Looking up for already implemented solutions online is very helpful in learning the methods prevalent for similar situations and the pros and cons associated with them.

Even when it becomes necessary to reinvent the wheel, the existing solutions can serve as a benchmark to test your new solution. An equally important part of writing code is to make your own code reusable.

Here are two snippets which highlight reusability.

Code which needs to be edited before resuing it

Code which can be reused with lesser editing

6. Write  Planned Out Code

Writing code on the fly may be a cool-to-have skill but not helpful for writing efficient codes. Coding is most efficient when you know what you are writing.

Always plan and write your logic on a piece of paper before implementing it. Inculcating the habit of adding tabs and spaces and basic formatting as you code is another time-saving skill for a good programmer.

For instance, every new ‘if’, ‘for’ or ‘while’ statement can be followed by tabs so that indentation is clearly visible. Although optional, such actions separate out blocks of code and helpful in identifying breakpoints as well as debugging.

A more rigorous but helpful approach is to write code using functions and modules and explaining every section with examples in comments or printing progress inside loops and conditions. Ultimately it all depends on the programmer how he/she chooses to document and log in the code.

7. Active Memory Management

Adding memory handling code is like handling a double-edged sword. It may not be useful for small-scale programs due to a slowdown in execution speed but nevertheless a great skill to have for writing scalable code.

In Rstudio, removing variables and frames when they are no longer required with the rm() function, garbage collection using gc() command and selecting the relevant features and data for proceeding are ways to manage memory.

Adjusting RAM usage with memory.limit() and setting parallel processing are also tasks for managing your memory usage. Remember! Memory management goes hand in hand with data backup.

It only takes a few seconds create and store copies of data. It should be done to ensure that data loss does not occur if backtracking is required.

Have a look at this example snippet which stores the master data and then frees up memory.

8. Remove Redundant Tasks

Sometimes programmers do some tasks repeatedly or forget to remove program code without knowing it.

Writing separate iterations for each data manipulation step, leaving libraries loaded even after they are no longer required, not removing features until the last moment, multiple joins and queries,etc. are some examples of redundancy lurking in your code.

While these happen somewhat naturally as more and more changes are made and new logic is added. It is a good practice to look at existing code and adjust your new lines to save runtime.

Redundancy can slow your code so much that removing it can do wonders in execution speed.

9. Learn to Adapt

No matter how good a programmer you are, you can always be better! This tip is not related to typical coding practices but teamwork. Sharing and understanding codes from peers, Reading codes online (such as from repositories).

setting yourself up to date with books and blogs and learning about new technologies and packages which are released for R are some ways to learn.

Being flexible and adaptive to new methods and keeping yourself up to date with what’s happening in the analytics industry today can help you in avoiding becoming obsolete with old practices.

10. Peer Review

The code you write may be straightforward for you but very complex for everyone else. How will you know that? The only way is to know what others think about it.

Code review is thus the last but not the least in terms of importance for better coding. Ask people to go through your code and be open to suggested edits. You may come across situations when some code you thought is written beautifully can be replaced with more efficient code.

Code review is a process which helps both the coder and reviewer as it is a way of helping each other to improve and move forward.

The Path is Not So Difficult: Conclusion

Becoming a good programmer is no easy feat but becoming better at programming as you progress is possible. Though it will take time, persevering to add strong programming habits will make you a strong member in every team’s arsenal.

These tips are just the beginning and there may be more ways to improve. The knowledge to always keep improving will take you forward and let you taste the sweet results of being a hi-tech programmer.

In the rapidly changing analytics world, staying with the latest tools and techniques is a priority and being good at R programming can be a prime factor towards your progress in your analytics career.

So go out there and make yourself acquainted with the techniques of becoming better at R programming.

The post 10 Smart R Programming Tips to become Better R Programmer first appeared on Perceptive Analytics.

If you were not aware of unsupervised learning algorithms, all the machine learning algorithms mainly classified into two main categories.

1. Supervised learning algorithms
2. Unsupervised learning algorithms

All the classification and regression algorithms belong to supervised learning algorithms. The other set of algorithms which fall under unsupervised learning algorithms are clustering algorithms.

In fact, the foremost algorithms to study in unsupervised learning algorithms is clustering analysis algorithms. Today we are going to learn an algorithm to perform the cluster analysis.

We have a decent number of algorithms to perform cluster analysis; In this article, we will be learning how to perform the clustering with the Hierarchical clustering algorithm.

Before we drive further. Let’s have a look at the table of contents.

Table of contents:

• What is clustering analysis?
• Clustering analysis example
• Hierarchical clustering
  • Dendrogram
  • Agglomerative clustering
  • Divisive clustering
• Clustering linkage comparison
• Implementing hierarchical clustering in R programming language
  • Data preparation
  • Packages need to perform hierarchical clustering
  • Visualizing clustering in 3d view
  • Complete code
• Summary
• Related courses
  • Exploratory data analysis in r
  • Clustering analysis in Python
  • Machine learning clustering and information retrieval

What is clustering analysis?

Clustering the name itself has a deep meaning about the ongoing process which happens in the cluster analysis. The fundamental problem clustering address is to divide the data into meaningful groups (clusters).

When we say, meaningful groups, the meaningfulness purely depends on the intention behind the purpose of the group’s formations.

Suppose if we intend to cluster the search results for a particular keyword. We Intended to find all the search results which were meaningfully similar to the search keyword.

If we intend to cluster the search results for a particular location, then we need to group the search results belongs to one specific place.

The identified cluster elements within the same cluster should be similar to each other when compared to the other cluster elements.

Suppose we have 100 articles and we want to group them into different categories. Let’s consider the below categories.

• Sports articles
• Business articles
• Entertainment articles

When we group all the 100 articles into the above 3 categories. All the articles belong to the sports category will be same, In the sense, the content in the sports articles belongs to sports category.

When you pick an article from sports category and the other article from business articles. Content-wise they will be completely different. This summarises the rule of thumb condition to form clusters.

All the elements in the same cluster should be similar and elements of the different cluster should not be similar.

Now let’s have a look at one clustering analysis example to understand it better.

Clustering analysis example

Clustering Example

Hierarchical clustering

    Hierarchical clustering is an alternative approach to k-means clustering for identifying groups in the dataset and does not require to pre-specify the number of clusters to generate.

It refers to a set of clustering algorithms that build tree-like clusters by successively splitting or merging them. This hierarchical structure is represented using a tree.

Hierarchical clustering methods use a distance similarity measure to combine or split clusters. The recursive process continues until there is only one cluster left or we cannot split more clusters. We can use a dendrogram to represent the hierarchy of clusters.

Dendrogram

A dendrogram is a tree-like structure frequently used to illustrate the arrangement of the clusters produced by hierarchical clustering.

Hierarchical classifications produced by either 

• Agglomerative
• Divisive

The agglomerative or divisive route may be represented by a two-dimensional diagram known as a dendrogram, which illustrates the fusions or divisions made at each stage of the analysis. Agglomerative clustering usually yields a higher number of clusters, with fewer leaf nodes in the cluster.

In a hierarchical classification, the data are not partitioned into a particular number of classes or clusters at a single step. Instead, the classification consists of a series of partitions, which may run from a single cluster containing all individuals to n clusters each containing a single individual.

Hierarchical clustering algorithms can be either bottom-up or top-down.

Hierarchical clustering agglomerative and divisive methods

Agglomerative clustering

Agglomerative clustering is Bottom-up technique start by considering each data point as its own cluster and merging them together into larger groups from the bottom up into a single giant cluster.

Divisive clustering

Divisive clustering is the opposite, it starts with one cluster, which is then divided in two as a function of the similarities or distances in the data. These new clusters are then divided, and so on until each case is a cluster.

Clustering linkage comparison

In this article, we describe the bottom-up approach in the detailed manner i.e. agglomerative algorithm 

The necessary steps of an agglomerative algorithm are (diagrammed visually in the figure below):

1. Start with each point in its own cluster.
2. Compare each pair of data points using a distance metric. This could be any of the methods discussed above.
3. Use a linkage criterion to merge data points (at the first stage) or clusters (in subsequent phases), where the linkage is represented by a function such as:
   • Maximum or complete linkage clustering: It computes all pairwise dissimilarities between the elements in cluster 1 and the elements in cluster 2, and considers the largest value (i.e., maximum value) of these dissimilarities as the distance between the two clusters. It tends to produce more compact clusters.
   • Minimum or single linkage clustering: It computes all pairwise dissimilarities between the elements in cluster 1 and the elements in cluster 2, and considers the smallest of these dissimilarities as a linkage criterion. It tends to produce long, “loose” clusters.
   • Mean or average linkage clustering: It computes all pairwise dissimilarities between the elements in cluster 1 and the elements in cluster 2, and considers the average of these dissimilarities as the distance between the two clusters.
   • Centroid linkage clustering: It computes the dissimilarity between the centroid for cluster 1 (a mean vector of length p variables) and the centroid for cluster 2.
   • Ward’s minimum variance method: It minimizes the total within-cluster variance. At each step, the pair of clusters with minimum between-cluster distance are merged.

Single-link | Complete-link | Average-link | Centroid distance Image Credit:: http://slideplayer.com/slide/9336538/

Implementing hierarchical clustering in R programming language

Data Preparation

To perform a cluster analysis in R, generally, the data should be prepared as follows:

1. Rows are observations (individuals) and columns are variables
2. Any missing value in the data must be removed or estimated.
3. The data must be standardized (i.e., scaled) to make variables comparable. Recall that, standardization consists of transforming the variables such that they have mean zero and standard deviation one.

Here, we’ll use the built-in R dataset iris, which contains 3 classes of 50 instances each, where each class refers to a type of iris plant.

To remove any missing value that might be present in the data, type this:

As we don’t want the clustering algorithm to depend to an arbitrary variable unit, we start by scaling/standardizing the data using the R function scale:

Packages need to perform hierarchical clustering

1. hclust [in stats package]
2. agnes [in cluster package]

We can perform agglomerative HC with hclust. First, we compute the dissimilarity values with dist and then feed these values into hclust and specify the agglomeration method to be used (i.e. “complete”, “average”, “single”, “ward.D”). We can plot the dendrogram after this.

Alternatively, we can use the agnes function. These functions behave very similarly; however, with the agnes function, we can also get the agglomerative coefficient, which measures the amount of clustering structure found (values closer to 1 suggest strong clustering structure).

This allows us to find certain hierarchical clustering methods that can identify stronger clustering structures. Here we see that Ward’s method identifies the strongest clustering structure of the four methods assessed.

Cluster dendrogram

Visualizing clustering in 3d view

Let’s examine, this time visually. How this algorithm proceeds using a simple dataset. 

As visual representations are limited to three dimensions, we will only use three attributes, but the computation is similar if we use more attributes. We will display these using the scatterplot3d() function of the plot3D package. 

which we will install and load after creating the attributes. We then examine the clustering solution provided by hclust() in order to assess whether it confirms the impressions we get from visual inspection.

Cluster dendrogram 3D view

In the left panel of the above plot, there are three groups of two points that are very close to each other. Another point is quite close to each of these groups of two. 

Consider that the groups of two constitute a group of three with the points that lie closest to them. Finally, the two groups on the left are closer to each other than they are to the group of three on the right. 

If we have a look at the dendrogram, we can see that the very same pattern is visible.

Complete Code

You can clone complete codes of dataaspirant from our GitHub account

Summary

Hierarchical methods form the backbone of cluster analysis. The need for hierarchical clustering naturally emerges in domains where it is not only required to discover similarity-based groups but also need to organize them. 

This is a valuable capability wherever the complexity of similarity patterns goes beyond the limited representation power of flat clustering models.

Hierarchical clustering tends to be presented as a form of descriptive rather than predictive modeling.

The post How to Perform Hierarchical Clustering in R first appeared on Perceptive Analytics.