Data Cleaning and Preprocessing

In data science there is a widely unappreciated and overlooked part of data analysis that could possibly be the most important: data cleaning and preprocessing. Data preprocessing—whether it being imaging, text, or numerical data—is the foundation to every single data project, and is a necessary step to have a functional and smooth analysis process. This week I wanted to explore the inner workings of data cleaning as it’s probably the most important step when handling new data. Furthermore, I feel like data cleaning is more of the “dirty work” of analysis: vital, yet always left unmentioned.

What is Data Cleaning and Preprocessing?

Data cleaning is almost exactly what it sounds like: cleaning data of anything you might not want, or arranging it in a form that is most efficient for your task. Imagine data as a room filled with dirty clothes, shoes, food, etc. Before you may add anything to the room, it’s essential to get the room tidy and organized. That is essentially data cleaning.

Specifically, data cleaning is identifying and correcting errors and missing values in your data. When cleaning data, there are some things to look out for: duplicates or unnecessary data that is not supposed to be there, missing values, and varying formats of data. When looking at a complete dataset, everything should have no duplicates unless necessary, and similar formats should be upheld for each datapoint. 

Data preprocessing, on the other hand, is about transforming your data into a format suitable for analysis. There are a couple of different forms preprocessing can take.

Scaling is ensuring all variables are on a similar scale. This could be converting all the values in your dataset of measurements into the same units to maintain consistency and easier analysis. For instance, trying to analyze data of skyscraper heights when half of the dataset is in meters and the other half in inches could be difficult.

Encoding refers to converting categorical data into numerical format. Especially for machine learning algorithms, it’s a lot easier for computers to handle numbers than letters. If you had data of three teams of basketball players, it may be easier for computer systems to handle a number to represent the team they were on instead of a string of the team name.

Handling outliers is similar to data cleaning, simply identifying and dealing with extreme values that can skew your analysis.

Why Does it Matter?

If your data is covered with errors, inconsistencies, or missing values, your analysis will also be filled with the same, and your models will produce inaccurate results. Data cleaning and preprocessing are essential for several reasons: they improve the accuracy of your analysis by eliminating biases caused by flawed data, they can reveal otherwise hidden trends that lead to valuable insights, and they assist machine learning models by providing them with well-structured data to learn from. Although this task sounds very tedious, it is very important, especially when dealing with larger datasets, as one mistake can deform analysis heavily.

While they might not be as exciting as building complex models or creating stunning visualizations, data cleaning and preprocessing are the essential foundation for any data science project. I’ve personally been through a project trying to use machine learning to analyze data, only to be stumped when the model gives me completely incorrect trends; all to find out that the data I was using was in different units of measurements. Data cleaning is very important and I am always trying to become more mindful of the data I am using and the most effective way to store it.