← Guides

Clean a Messy Excel File in 15 Lines of Python with pandas

Stray spaces, duplicates, and wrong date formats: the minimal script to fix a customer export.

Python · 8 min read

Loading Without Type Errors

pd.read_excel(file, dtype=str) loads everything as text and prevents pandas from transforming a zip code like 01000 into 1000 or an order number into scientific notation. You can then explicitly convert the necessary columns.

Normalizing Text Columns

df['city'] = df['city'].str.strip().str.title() removes leading and trailing spaces and harmonizes casing. This is what reduces the number of "distinct" cities from 400 to 120 in a manually entered file.

Deleting Duplicates Intelligently

drop_duplicates(subset=['email'], keep='last') keeps the last entry, which is usually the most up-to-date. Without the subset parameter, pandas compares all columns and lets partial duplicates slip through.

Exporting Cleanly

df.to_excel('clean.xlsx', index=False) avoids the extra index column that everyone ends up deleting manually in Excel.