Fix UnicodeDecodeError in Python
Error Reference
UnicodeDecodeError: 'utf-8' codec can't decode byte
You're reading bytes with the wrong encoding — or reading binary data as text.
Declare the encoding
Text files are bytes plus an encoding. When the bytes don't match the assumed encoding (UTF-8 on most systems), decoding fails:
open('legacy.csv').read()
# UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 12
Files exported from Excel or older Windows tools are often cp1252 or latin-1:
text = open('legacy.csv', encoding='cp1252').read()
Always pass encoding= explicitly when opening text files — the default varies by platform and Python version.
Detecting the encoding
When you don't know the source encoding, sample the raw bytes:
from charset_normalizer import from_path # pip install charset-normalizer
best = from_path('legacy.csv').best()
print(best.encoding) # e.g. 'cp1252'
Byte 0xe9 at the error position is a hint by itself: in cp1252/latin-1 it's é — a strong sign of Western European legacy text.
Binary data is not text
Trying to read images, PDFs, or archives as text guarantees decode errors. Open them in binary mode:
raw = open('photo.jpg', 'rb').read() # bytes, no decoding
Last resort: error handlers
When some corruption is acceptable (log scraping, best-effort imports), tell the codec how to handle bad bytes:
text = open('mixed.log', encoding='utf-8', errors='replace').read()
# bad bytes become U+FFFD (�) instead of raising
errors='ignore' silently drops them — use it only when losing characters is truly fine, because it hides data corruption.
Frequently Asked Questions
How do I find out what encoding a file uses?
There is no marker inside most files, so you infer it: try utf-8 first, then cp1252/latin-1 for Western legacy data, or use the charset-normalizer / chardet libraries to guess statistically. On Linux, `file yourfile.txt` gives a quick hint.
What's the difference between UnicodeDecodeError and UnicodeEncodeError?
Decode errors happen going bytes → str (reading); encode errors happen going str → bytes (writing) when the target encoding cannot represent a character, like writing é to ASCII. The fix for encode errors is usually writing UTF-8 explicitly.
open('out.txt', 'w', encoding='utf-8').write('café')
Is errors='ignore' a safe fix?
It makes the error disappear, not the problem — invalid bytes are dropped silently, so names, accents, or entire fields can vanish from your data. Prefer finding the real encoding; use errors='replace' if you need visibility into where corruption happened.