Basic Data Cleaning with Python and pandas

Before analysing data or training a model, real-world data almost always needs cleaning. Missing values, duplicate records, inconsistent spelling and wrong data types can all lead to wrong conclusions. Python's pandas library provides simple operations for these tasks. This guide assumes you have pandas installed and a table of student data with the columns ma_sv (student ID), ho_ten (full name), nganh (major), diem (score) and ngay_nhap_hoc (enrolment date).

Step 1: get an overview of the data

Read the data and look at its structure before changing anything.

import pandas as pd

df = pd.read_csv("sinhvien.csv")
print(df.head())        # first few rows
print(df.info())        # data types and non-null counts
print(df.describe())    # statistics for numeric columns

The output tells you which columns have unusual data types, which have many missing values, and whether numeric values lie within a sensible range.

Step 2: handle missing values

First, count the missing values in each column. Then decide how to handle them based on what the data means, rather than mechanically.

print(df.isna().sum())

df = df.dropna(subset=["ma_sv"])             # drop rows missing the key
df["diem"] = df["diem"].fillna(df["diem"].median())  # fill with the median

Filling with the median or mean is only one option. For some columns, keeping the missing value or adding an indicator column marking "was missing" is more appropriate. Record your decisions so others understand why the data changed.

Step 3: remove duplicate records

Duplicate records distort statistics. Decide which column identifies a record uniquely.

print(df.duplicated(subset=["ma_sv"]).sum())
df = df.drop_duplicates(subset=["ma_sv"], keep="first")

Step 4: standardise text

The same major may be written as "CNTT", "cntt " or "Cntt". Standardising case and whitespace makes grouping more accurate.

df["nganh"] = df["nganh"].str.strip().str.upper()
df["ho_ten"] = df["ho_ten"].str.strip().str.title()

If several spellings of the same value remain, you can use replace with a mapping dictionary, and check again with df["nganh"].value_counts().

Step 5: fix data types and check for abnormal values

Dates are often read as strings, and scores may be read as strings if they contain stray characters.

df["ngay_nhap_hoc"] = pd.to_datetime(df["ngay_nhap_hoc"], errors="coerce")
df["diem"] = pd.to_numeric(df["diem"], errors="coerce")

hop_le = df["diem"].between(0, 10)
print(df[~hop_le])      # show rows with scores outside 0-10

The parameter errors="coerce" turns values that cannot be converted into missing values for you to handle next, instead of raising an error. For values outside a sensible range, check their origin before deciding to correct or remove them.

Step 6: save the result and keep the original data

Save the cleaned data to a new file and do not overwrite the original, so you can re-check and reproduce the process.

df.to_csv("sinhvien_sach.csv", index=False)

Conclusion

Data cleaning is an iterative process: observe, find problems, fix them and check again. With pandas, basic operations such as isna, drop_duplicates, str.strip and to_numeric solve most common problems. More importantly, record your handling decisions and keep the original data intact, so that the analysis that follows can be verified.