ruff PLW1514
labelled unknown
Analysis Nemotron 3 Super
The rule flags any call to built-in functions like `open`, `io.open`, `codecs.open`, `fileinput.input`, or `tarfile.open` when used in text mode without an explicit `encoding` argument.
Fix: Adds `encoding='utf-8'` as a keyword argument to the function call, inserted before any `**kwargs` unpacking if present, otherwise appended at the end of the argument list.
12 hazards
- Adding `encoding='utf-8'` to a binary mode call like `open('file', 'rb')` creates an invalid argument combination that raises `ValueError` because encoding is not allowed in binary mode.
- If the call already contains an `encoding` keyword argument (e.g., via `**kwargs` that includes 'encoding'), adding another `encoding='utf-8'` results in a duplicate keyword argument error (`TypeError: got multiple values for argument 'encoding'`).
- When the file is opened in a locale-specific encoding (e.g., Windows-1252 on Western European Windows systems), forcing UTF-8 causes `UnicodeDecodeError` when reading existing non-UTF-8 encoded files.
- For `tarfile.open`, adding `encoding='utf-8'` is invalid because `tarfile.open` does not accept an `encoding` parameter; it raises `TypeError: unexpected keyword argument`.
- In `fileinput.input`, the `encoding` parameter behaves differently—it applies to all files processed, and forcing UTF-8 may break when processing mixed-encoding input streams.
- If the `open` call is shadowed by a local variable or imported alias (e.g., `local_open = open; local_open('file')`), the autofix may incorrectly modify the wrong function or fail to recognize the call as subject to the rule.
- When the call uses positional arguments where the mode is passed as a variable (e.g., `open(file, mode)`), inserting `encoding='utf-8'` assumes the mode is text-based, but if `mode` resolves to a binary mode at runtime, the call becomes invalid.
- Adding `encoding='utf-8'` may interfere with code that relies on the system default encoding for compatibility with legacy data or inter-process communication, silently corrupting data or causing downstream failures.
- If the call appears within a lambda or comprehension and is modified, the autofix may inadvertently alter binding or evaluation timing if the original comment or formatting is disturbed during rewrite.
- In codebases with custom `open` wrappers that expect no encoding argument (e.g., for testing mocks), injecting `encoding='utf-8'` breaks the wrapper’s signature assumption, causing `TypeError` during test execution.
- When the original source contains trailing comments (e.g., `open('file') # read config`), the autofix may drop or misplace the comment due to improper reflow, especially if the edit disrupts whitespace or delimiter handling.
- If the function call is part of a larger expression (e.g., `data = open('file').read()`), inserting `encoding='utf-8'` changes the temporary object’s behavior, potentially causing failure if the file encoding is not UTF-8.
Programs
| id | result | title | hazard |
|---|---|---|---|
| r1-p001 | silent prediction off | Generators vs Lists - Encoding Affects Lazy Evaluation | Adding encoding='utf-8' changes decoding behavior lazily; a generator expression that reads lines will fail on non-UTF-8 data only when iterated, exposing the fix's semantic change. |
| r1-p002 | fired, no fix | Locale-Specific Encoding - Forced UTF-8 Causes Decode Error | On systems where default encoding is Windows-1252, reading a CP1252-encoded file with forced UTF-8 raises UnicodeDecodeError, changing program behavior from success to failure. |
| r1-p003 | fired, no fix | Class Attributes - Encoding in Descriptor Triggers Unexpected Init | The fix adds encoding during class-level file open in a descriptor, causing side effects at class definition time, affecting all instances before any are created. |
| r1-p004 | fired, no fix | Dict Ordering - Key Depends on File Read Order | The rewritten open call changes what bytes are successfully decoded, altering which lines are read and thus which keys appear in a dict, affecting insertion order and final output. |
| r1-p005 | fired, no fix | String Formatting - % Operator on Decoded Bytes | The fix changes the decoded string from open(), which is then formatted with %; if decoding fails or produces mojibake, the formatted output changes. |
| r1-p006 | silent prediction off | Evaluation Order - Side Effects in Open Args | The fix inserts encoding='utf-8' as a kwarg, shifting positional args if **kwargs follows; if an argument has a side effect (like a function call), its evaluation order relative to others changes. |
| r1-p007 | fired, no fix | Keyword-Only Args - Encoding Added Before **kwargs Changes Call Semantics | When the rule inserts encoding='utf-8' before **kwargs in a function call that forwards to open, it can interfere with keyword-only parameters if the function signature expects specific ordering. |
| r2-p001 | fired, no fix | Custom open wrapper expects no encoding | Injecting encoding='utf-8' breaks custom wrapper signatures expecting no encoding arg. |
| r2-p002 | fired, no fix | tarfile.open rejects encoding kwarg | tarfile.open does not accept encoding argument; adding it raises TypeError. |
| r2-p003 | fired, no fix | Async generator interaction with encoding fix | Inserting encoding='utf-8' in async context may affect await ordering or generator launch. |
| r2-p004 | fired, no fix | Generator vs list: lazy evaluation affected by encoding | Adding encoding='utf-8' changes decoding behavior in lazy generator expressions. |
| r2-p005 | fired, no fix | Duplicate encoding via **kwargs | If **kwargs contains 'encoding', adding encoding='utf-8' causes duplicate keyword error. |
| r2-p006 | fired, no fix | Locale-specific encoding mismatch (Windows-1252) | Forcing UTF-8 on legacy encoding file causes UnicodeDecodeError. |
| r2-p007 | fired, no fix | Exception in finally/else with rewritten open | Rewritten open affects exception handling in finally/else blocks. |
| r2-p008 | fired, no fix | Open in larger expression: .read() on temporary | Inserting encoding='utf-8' changes temporary file object behavior in expressions. |
Sources
r1-p001 · Generators vs Lists - Encoding Affects Lazy Evaluation
1import sys
2lines = [b'\xff\xfe']
3def gen():
4 for b in lines:
5 yield b.decode('utf-16-le')
6g = gen()
7try:
8 next(g)
9except UnicodeDecodeError as e:
10 print('gen failed:', type(e).__name__, e)
11# Now simulate fixed code: open with encoding='utf-8' on binary data
12try:
13 with open('test.bin', 'wb') as f:
14 f.write(b'\xff\xfe')
15 with open('test.bin', 'r', encoding='utf-8') as f: # Rule triggers here
16 content = f.read()
17 print('fixed read:', repr(content))
18except Exception as e:
19 print('fixed error:', type(e).__name__, e)
20finally:
21 import os
22 try: os.remove('test.bin')
23 except: pass
fixed error: UnicodeDecodeError 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
r1-p002 · Locale-Specific Encoding - Forced UTF-8 Causes Decode Error
1import sys
2# Create a file encoded in Windows-1252 (contains euro symbol \x80)
3with open('test.txt', 'wb') as f:
4 f.write(b'Hello \x80 World') # € in CP1252
5# Original behavior: uses system default (assumed CP1252 here)
6try:
7 with open('test.txt', 'r') as f: # Rule triggers: adds encoding='utf-8'
8 content = f.read()
9 print('read success:', repr(content))
10except UnicodeDecodeError as e:
11 print('read failed:', type(e).__name__, e)
12except Exception as e:
13 print('other error:', type(e).__name__, e)
14finally:
15 import os
16 try: os.remove('test.txt')
17 except: pass
read failed: UnicodeDecodeError 'utf-8' codec can't decode byte 0x80 in position 6: invalid start byte
r1-p003 · Class Attributes - Encoding in Descriptor Triggers Unexpected Init
1class FileDescriptor:
2 def __get__(self, obj, objtype=None):
3 if obj is None:
4 return self
5 return obj._value
6 def __set__(self, obj, value):
7 obj._value = value
8
9class DataHolder:
10 value = FileDescriptor()
11 # Rule triggers here: open in class scope
12 _log = open('log.txt', 'a') # Fix adds encoding='utf-8'
13 def __init__(self, v):
14 self.value = v
15 self._log.write(str(v) + '\n')
16
17d = DataHolder(42)
18print('log written')
19with open('log.txt') as f:
20 print('log contents:', repr(f.read()))
21import os
22try: os.remove('log.txt')
23except: pass
log written log contents: ''
r1-p004 · Dict Ordering - Key Depends on File Read Order
1import sys
2# Create two-line file: first line valid UTF-8, second has raw byte
3with open('data.txt', 'wb') as f:
4 f.write(b'key1\n\xff\n')
5try:
6 with open('data.txt', 'r') as f: # Rule triggers: adds encoding='utf-8'
7 lines = f.readlines()
8 d = {}
9 for i, line in enumerate(lines):
10 try:
11 d[f'key{i}'] = line.strip()
12 except:
13 pass # Skip failed lines
14 print('dict items:', list(d.items()))
15except UnicodeDecodeError as e:
16 print('decode error:', type(e).__name__, e)
17finally:
18 import os
19 try: os.remove('data.txt')
20 except: pass
decode error: UnicodeDecodeError 'utf-8' codec can't decode byte 0xff in position 5: invalid start byte
r1-p005 · String Formatting - % Operator on Decoded Bytes
1import sys
2# Create UTF-16 LE file with BOM
3with open('msg.txt', 'wb') as f:
4 f.write(b'\xff\feH\x00e\x00l\x00l\x00o\x00')
5try:
6 with open('msg.txt', 'r') as f: # Rule triggers: adds encoding='utf-8' -> decodes UTF-16 as UTF-8 -> garbage
7 s = f.read()
8 formatted = 'Message: %s' % s
9 print('formatted:', formatted)
10except Exception as e:
11 print('error:', type(e).__name__, e)
12finally:
13 import os
14 try: os.remove('msg.txt')
15 except: pass
error: UnicodeDecodeError 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
r1-p006 · Evaluation Order - Side Effects in Open Args
1counter = 0
2def get_mode():
3 global counter
4 counter += 1
5 return 'r'
6counter = 0
7try:
8 # Normal: open(file, get_mode()) -> get_mode called once
9 with open('tmp.txt', get_mode()) as f:
10 pass
11 print('normal mode calls:', counter)
12 counter = 0
13 # After fix: if rule adds encoding as kwarg before **kwargs, but we don't have **kwargs.
14 # Instead, simulate case where open is called via *args from tuple that includes mode.
15 # Actually, better: the fix does not reorder unless **kwargs present.
16 # So create a call with **kwargs to trigger reordering.
17 def get_file():
18 global counter
19 counter += 1
20 return 'tmp.txt'
21 def get_mode():
22 global counter
23 counter += 1
24 return 'r'
25 counter = 0
26 args = (get_file(), get_mode())
27 kwargs = {}
28 try:
29 with open(*args, **kwargs) as f: # Before fix: args[0], args[1]
30 pass
31 print('normal call order counters:', counter)
32 counter = 0
33 # After fix: becomes open(*args, encoding='utf-8', **kwargs)
34 # Still same order: args[0], args[1], then kwarg
35 # No change? Then hazard not met.
36 # Need case where **kwargs exists and encoding inserted before it.
37 # Let’s make kwargs non-empty.
38 def get_kw():
39 global counter
40 counter += 1
41 return {'buffering': -1}
42 counter = 0
43 args = (get_file(), 'r')
44 kw_arg = get_kw()
45 try:
46 with open(*args, encoding='utf-8', **kw_arg) as f: # Simulated fixed version
47 pass
48 print('fixed call order counters:', counter)
49 except Exception as e:
50 print('fixed error:', type(e).__name__, e)
51 except Exception as e:
52 print('error:', type(e).__name__, e)
53finally:
54 import os
55 try: os.remove('tmp.txt')
56 except: pass
r1-p007 · Keyword-Only Args - Encoding Added Before **kwargs Changes Call Semantics
1def my_open(file, *, mode='r', encoding=None, **kwargs):
2 if encoding is None and 'b' not in mode:
3 encoding = 'utf-8'
4 return open(file, mode=mode, encoding=encoding, **kwargs)
5
6# Rule triggers on the open call inside my_open
7try:
8 with open('test.txt', 'w') as f:
9 f.write('hello')
10 # This call: my_open('test.txt', mode='r', buffering=-1)
11 # Before fix: open('test.txt', mode='r', buffering=-1) -> encoding=None -> uses default
12 # After fix: open('test.txt', mode='r', encoding='utf-8', buffering=-1) -> forces utf-8
13 with my_open('test.txt', mode='r', buffering=-1) as f:
14 content = f.read()
15 print('read content:', repr(content))
16except Exception as e:
17 print('error:', type(e).__name__, e)
18finally:
19 import os
20 try: os.remove('test.txt')
21 except: pass
read content: 'hello'
r2-p001 · Custom open wrapper expects no encoding
1import os
2
3def my_open(*args, **kwargs):
4 return open(*args, **kwargs)
5
6try:
7 f = my_open('test.txt', 'w')
8 f.write('hi')
9 f.close()
10except TypeError as e:
11 print('TypeError:', e)
12
13with open('test.txt', 'w') as f:
14 f.write('hello')
15
16with open('test.txt', 'r') as f:
17 print('Content:', f.read())
18
19os.remove('test.txt')
Content: hello
r2-p002 · tarfile.open rejects encoding kwarg
1import tarfile
2import os
3
4with open('test.txt', 'w') as f:
5 f.write('data')
6
7with tarfile.open('test.tar', 'w') as tf:
8 tf.add('test.txt')
9
10try:
11 with tarfile.open('test.tar', 'r', encoding='utf-8') as tf:
12 pass
13except TypeError as e:
14 print('TypeError:', e)
15
16os.remove('test.txt')
17os.remove('test.tar')
r2-p003 · Async generator interaction with encoding fix
1import asyncio
2
3async def read_file():
4 f = open('data.txt')
5 yield f.read()
6 f.close()
7
8async def main():
9 with open('data.txt', 'w') as f:
10 f.write('hello')
11 gen = read_file()
12 try:
13 async for item in gen:
14 print('Yielded:', item)
15 except Exception as e:
16 print('Error:', type(e).__name__, e)
17
18asyncio.run(main())
19import os
20os.remove('data.txt')
Yielded: hello
r2-p004 · Generator vs list: lazy evaluation affected by encoding
1import os
2
3with open('latin.txt', 'wb') as f:
4 f.write(b'h\xe9llo') # 'héllo' in latin-1
5
6# Generator expression: lazy, uses open call
7lines_gen = (line.decode('latin-1') for line in open('latin.txt', 'rb'))
8print('Gen first:', next(lines_gen))
9
10# After fix, rule would add encoding='utf-8' to open(...) above
11# But currently it's rb mode, so rule should not fire. Need text mode.
12
13os.remove('latin.txt')
14
15with open('latin.txt', 'w', encoding='latin-1') as f:
16 f.write('héllo')
17
18# Now rule fires on open(...) in genexpr
19lines_gen = (line for line in open('latin.txt')) # rule triggers here
20try:
21 print('Gen first (rule-affected):', next(lines_gen))
22except UnicodeDecodeError as e:
23 print('UnicodeDecodeError in gen:', e)
24
25# List version: eager
26lines_list = [line for line in open('latin.txt')]
27print('List first:', lines_list[0])
28
29os.remove('latin.txt')
Gen first: héllo UnicodeDecodeError in gen: 'utf-8' codec can't decode byte 0xe9 in position 1: invalid continuation byte
r2-p005 · Duplicate encoding via **kwargs
1import os
2
3with open('test.txt', 'w') as f:
4 f.write('hi')
5
6kwargs = {'encoding': 'utf-8'}
7try:
8 # Rule fires on open('test.txt') and adds encoding='utf-8'
9 # But **kwargs also has encoding -> duplicate
10 f = open('test.txt', **kwargs) # This line is NOT touched by rule
11 print('Read:', f.read())
12 f.close()
13except TypeError as e:
14 print('TypeError:', e)
15
16# Now make the rule trigger on a call WITHOUT **kwargs but we simulate hazard via comment
17# Actually, to trigger rule AND have **kwargs with encoding, we need:
18# But rule only adds if no encoding present. So we need a call like open(x) + **{'encoding':x}
19# That would be open(x, **{'encoding':'utf-8'}) -> rule does NOT fire because encoding present via **kwargs?
20# According to spec: rule fires only when no explicit encoding. So this is near-miss.
21# Let's do a call that rule WILL fire on, but we inject kwargs that hapen to have encoding via unpacking in a way that rule doesn't see?
22# Impossible per spec. Instead, we show what happens if fix is applied incorrectly.
23# We'll instead show a case where rule does NOT fire but user might expect it to.
24
25# For hazard: assume rule fires on open(x) and adds encoding, but caller passed **{'encoding':x}
26# Not possible statically. So we simulate by having two calls.
27
28print('Skip: rule cannot fire and cause duplicate via **kwargs per spec')
29
30os.remove('test.txt')
Read: hi Skip: rule cannot fire and cause duplicate via **kwargs per spec
r2-p006 · Locale-specific encoding mismatch (Windows-1252)
1import os
2
3# Create a file encoded in windows-1252 (cp1252)
4with open('local.txt', 'w', encoding='cp1252') as f:
5 f.write('caf\xe9') # 'é' in cp1252
6
7# Rule fires: open('local.txt') -> adds encoding='utf-8'
8try:
9 with open('local.txt') as f:
10 print('Read:', f.read())
11except UnicodeDecodeError as e:
12 print('UnicodeDecodeError:', e)
13
14# Correct way
15with open('local.txt', 'r', encoding='cp1252') as f:
16 print('Correct read:', f.read())
17
18os.remove('local.txt')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: unexpected end of data Correct read: café
r2-p007 · Exception in finally/else with rewritten open
1import os
2
3with open('bad.txt', 'w') as f:
4 f.write('broken')
5
6try:
7 f = open('bad.txt') # rule fires: adds encoding='utf-8'
8 try:
9 raise ValueError('fail')
10 finally:
11 f.close()
12 print('Finally ran')
13except ValueError as e:
14 print('Caught:', e)
15except Exception as e:
16 print('Other:', type(e).__name__, e)
17
18try:
19 with open('bad.txt') as f:
20 raise ValueError('fail')
21except ValueError as e:
22 print('With-stmt caught:', e)
23
24os.remove('bad.txt')
Finally ran Caught: fail With-stmt caught: fail
r2-p008 · Open in larger expression: .read() on temporary
1import os
2
3with open('test.txt', 'w', encoding='latin-1') as f:
4 f.write('h\xe9llo') # 'héllo' in latin-1
5
6# Rule fires on: open('test.txt') in expression below
7try:
8 data = open('test.txt').read() # rule adds encoding='utf-8' here
9 print('Data (rule-affected):', repr(data))
10except UnicodeDecodeError as e:
11 print('UnicodeDecodeError:', e)
12
13# Without fix (or correct encoding)
14with open('test.txt', 'r', encoding='latin-1') as f:
15 print('Correct:', repr(f.read()))
16
17os.remove('test.txt')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 1: invalid continuation byte Correct: 'héllo'