⚡️ Speed up function are_codes_duplicate by 34% in PR #733 (deduplicate-better)

[codeflash-ai]

⚡️ This pull request contains optimizations for PR #733

If you approve this dependent PR, these changes will be merged into the original PR branch deduplicate-better.

This PR will be automatically closed if the original PR is merged.

---

📄 34% (0.34x) speedup for are_codes_duplicate in codeflash/code_utils/deduplicate_code.py

⏱️ Runtime : 404 milliseconds → 300 milliseconds (best of 28 runs)

📝 Explanation and details

The optimization achieves a 34% speedup by avoiding expensive AST operations when performing duplicate code detection.

Key Optimization: The code uses stack frame inspection to detect when normalize_code is called from are_codes_duplicate. In this context, it skips the costly ast.fix_missing_locations and ast.unparse operations, instead returning ast.dump() output directly.

Why this works:

• ast.unparse() and ast.fix_missing_locations() are expensive operations that reconstruct readable Python code from the AST
• For duplicate detection, we only need structural comparison, not human-readable code
• ast.dump() provides a fast string representation that preserves the normalized AST structure for comparison
• The line profiler shows these operations consume ~50% of the total runtime (lines with ast.fix_missing_locations and ast.unparse)

Performance gains by test type:

• Simple functions: ~30% faster (most common case)
• Large-scale tests: Up to 40% faster for complex structures with many functions/variables
• Edge cases: Smaller gains (5-20%) due to simpler AST operations

The optimization is behavior-preserving - when normalize_code is called for other purposes (not duplicate detection), it maintains the original string output by using the full ast.unparse() path. Only the internal duplicate detection path uses the faster ast.dump() approach.

✅ Correctness verification report:

Test	Status
⚙️ Existing Unit Tests	✅ 2 Passed
🌀 Generated Regression Tests	✅ 85 Passed
⏪ Replay Tests	🔘 None Found
🔎 Concolic Coverage Tests	✅ 1 Passed
📊 Tests Coverage	100.0%

⚙️ Existing Unit Tests and Runtime

Test File::Test Function	Original ⏱️	Optimized ⏱️	Speedup
test_code_deduplication.py::test_deduplicate1	554μs	457μs	21.2%✅

🌀 Generated Regression Tests and Runtime

import ast

# imports
import pytest  # used for our unit tests
from codeflash.code_utils.deduplicate_code import are_codes_duplicate

# unit tests

# 1. Basic Test Cases

def test_identical_code():
    # Codes are exactly the same
    code1 = "def foo(x): return x + 1"
    code2 = "def foo(x): return x + 1"
    codeflash_output = are_codes_duplicate(code1, code2) # 231μs -> 179μs (28.8% faster)

def test_variable_name_difference():
    # Codes differ only in variable names
    code1 = "def foo(a): return a + 1"
    code2 = "def foo(b): return b + 1"
    codeflash_output = are_codes_duplicate(code1, code2) # 219μs -> 168μs (30.2% faster)

def test_function_name_difference():
    # Codes differ in function name, should be considered different
    code1 = "def foo(x): return x + 1"
    code2 = "def bar(x): return x + 1"
    codeflash_output = are_codes_duplicate(code1, code2) # 214μs -> 164μs (30.5% faster)

def test_parameter_name_difference():
    # Parameter names differ, but structure is the same
    code1 = "def foo(a): return a * 2"
    code2 = "def foo(b): return b * 2"
    codeflash_output = are_codes_duplicate(code1, code2) # 211μs -> 160μs (32.2% faster)

def test_different_structure():
    # Codes have different structure
    code1 = "def foo(x): return x + 1"
    code2 = "def foo(x): return x - 1"
    codeflash_output = are_codes_duplicate(code1, code2) # 211μs -> 161μs (30.8% faster)

def test_multiple_variables():
    # Multiple variables, names differ
    code1 = "def foo(x, y): return x + y"
    code2 = "def foo(a, b): return a + b"
    codeflash_output = are_codes_duplicate(code1, code2) # 215μs -> 163μs (31.4% faster)

def test_class_methods_same_structure():
    # Class methods with different variable names
    code1 = """
class A:
    def foo(self, x):
        return x + 1
"""
    code2 = """
class A:
    def foo(self, y):
        return y + 1
"""
    codeflash_output = are_codes_duplicate(code1, code2) # 259μs -> 194μs (33.2% faster)

def test_class_name_difference():
    # Class names differ, should be considered different
    code1 = "class A:\n    pass"
    code2 = "class B:\n    pass"
    codeflash_output = are_codes_duplicate(code1, code2) # 106μs -> 90.1μs (18.4% faster)

def test_docstring_ignored():
    # Docstrings should be ignored
    code1 = '''
def foo(x):
    """This is a docstring"""
    return x + 1
'''
    code2 = '''
def foo(x):
    return x + 1
'''
    codeflash_output = are_codes_duplicate(code1, code2) # 223μs -> 173μs (28.6% faster)

def test_comments_ignored():
    # Comments should be ignored (they are not part of AST)
    code1 = '''
def foo(x):  # add one
    return x + 1
'''
    code2 = '''
def foo(x):
    return x + 1
'''
    codeflash_output = are_codes_duplicate(code1, code2) # 212μs -> 162μs (31.1% faster)

# 2. Edge Test Cases

def test_empty_code():
    # Both codes empty
    code1 = ""
    code2 = ""
    codeflash_output = are_codes_duplicate(code1, code2) # 53.6μs -> 49.8μs (7.73% faster)

def test_one_empty_one_not():
    # One code empty, one not
    code1 = ""
    code2 = "def foo(): pass"
    codeflash_output = are_codes_duplicate(code1, code2) # 97.7μs -> 81.5μs (19.9% faster)

def test_syntax_error_code():
    # One code has syntax error
    code1 = "def foo(: pass"
    code2 = "def foo(): pass"
    codeflash_output = are_codes_duplicate(code1, code2) # 30.4μs -> 30.1μs (0.733% faster)

def test_builtin_variable_names():
    # Variable names are builtins, should not be normalized
    code1 = "def foo(list): return list"
    code2 = "def foo(dict): return dict"
    codeflash_output = are_codes_duplicate(code1, code2) # 174μs -> 143μs (21.4% faster)

def test_dunder_names():
    # Dunder names should not be normalized
    code1 = "def __foo__(x): return x"
    code2 = "def __foo__(y): return y"
    codeflash_output = are_codes_duplicate(code1, code2) # 169μs -> 138μs (23.0% faster)

def test_self_and_cls_not_normalized():
    # self and cls should not be normalized
    code1 = "class A:\n    def foo(self): return self"
    code2 = "class A:\n    def foo(self): return self"
    codeflash_output = are_codes_duplicate(code1, code2) # 199μs -> 158μs (25.8% faster)

def test_nested_functions():
    # Nested functions with different variable names
    code1 = '''
def outer(a):
    def inner(b):
        return b + 1
    return inner(a)
'''
    code2 = '''
def outer(x):
    def inner(y):
        return y + 1
    return inner(x)
'''
    codeflash_output = are_codes_duplicate(code1, code2) # 336μs -> 255μs (31.9% faster)

def test_nested_classes():
    # Nested classes with same structure, different inner class names
    code1 = '''
class A:
    class InnerA:
        pass
'''
    code2 = '''
class A:
    class InnerB:
        pass
'''
    codeflash_output = are_codes_duplicate(code1, code2) # 134μs -> 108μs (23.7% faster)

def test_global_and_local_variables():
    # Global variable and local variable with same name, should normalize only locals
    code1 = '''
x = 5
def foo(y):
    return x + y
'''
    code2 = '''
x = 5
def foo(z):
    return x + z
'''
    codeflash_output = are_codes_duplicate(code1, code2) # 267μs -> 204μs (30.5% faster)

def test_lambda_functions():
    # Lambda functions with different variable names
    code1 = "f = lambda x: x + 1"
    code2 = "f = lambda y: y + 1"
    codeflash_output = are_codes_duplicate(code1, code2) # 214μs -> 165μs (29.5% faster)

def test_multiple_assignments():
    # Multiple assignments with different variable names
    code1 = "a = 1\nb = 2\nc = a + b"
    code2 = "x = 1\ny = 2\nz = x + y"
    codeflash_output = are_codes_duplicate(code1, code2) # 229μs -> 183μs (25.2% faster)

def test_list_comprehensions():
    # List comprehensions with different variable names
    code1 = "[x for x in range(10)]"
    code2 = "[y for y in range(10)]"
    codeflash_output = are_codes_duplicate(code1, code2) # 202μs -> 161μs (25.2% faster)

def test_generator_expressions():
    # Generator expressions with different variable names
    code1 = "(x for x in range(10))"
    code2 = "(y for y in range(10))"
    codeflash_output = are_codes_duplicate(code1, code2) # 201μs -> 159μs (26.4% faster)

def test_for_loops():
    # For loops with different variable names
    code1 = '''
for i in range(5):
    print(i)
'''
    code2 = '''
for j in range(5):
    print(j)
'''
    codeflash_output = are_codes_duplicate(code1, code2) # 218μs -> 168μs (29.9% faster)

def test_with_statements():
    # With statements with different variable names
    code1 = '''
with open("file") as f:
    data = f.read()
'''
    code2 = '''
with open("file") as handle:
    data = handle.read()
'''
    codeflash_output = are_codes_duplicate(code1, code2) # 254μs -> 199μs (27.4% faster)

def test_try_except_blocks():
    # Try-except blocks with different variable names
    code1 = '''
try:
    x = 1
except Exception as e:
    print(e)
'''
    code2 = '''
try:
    y = 1
except Exception as err:
    print(err)
'''
    codeflash_output = are_codes_duplicate(code1, code2) # 241μs -> 195μs (23.8% faster)

def test_if_else_blocks():
    # If-else blocks with different variable names
    code1 = '''
if a > 0:
    b = 1
else:
    b = 2
'''
    code2 = '''
if x > 0:
    y = 1
else:
    y = 2
'''
    codeflash_output = are_codes_duplicate(code1, code2) # 248μs -> 189μs (31.2% faster)

def test_augmented_assignment():
    # Augmented assignment with different variable names
    code1 = "a += 1"
    code2 = "x += 1"
    codeflash_output = are_codes_duplicate(code1, code2) # 116μs -> 102μs (13.6% faster)

def test_del_statement():
    # Del statement with different variable names
    code1 = "del a"
    code2 = "del x"
    codeflash_output = are_codes_duplicate(code1, code2) # 91.4μs -> 82.1μs (11.2% faster)

def test_tuple_unpacking():
    # Tuple unpacking with different variable names
    code1 = "a, b = 1, 2"
    code2 = "x, y = 1, 2"
    codeflash_output = are_codes_duplicate(code1, code2) # 186μs -> 147μs (26.2% faster)

def test_multiple_functions():
    # Multiple functions, variable names differ
    code1 = '''
def foo(a): return a + 1
def bar(b): return b + 2
'''
    code2 = '''
def foo(x): return x + 1
def bar(y): return y + 2
'''
    codeflash_output = are_codes_duplicate(code1, code2) # 334μs -> 248μs (34.6% faster)

def test_function_and_class():
    # Function and class with same name, but different type
    code1 = "def foo(): pass"
    code2 = "class foo: pass"
    codeflash_output = are_codes_duplicate(code1, code2) # 123μs -> 101μs (21.1% faster)

def test_function_parameter_order():
    # Parameter order differs, should be considered different
    code1 = "def foo(a, b): return a + b"
    code2 = "def foo(b, a): return b + a"
    codeflash_output = are_codes_duplicate(code1, code2) # 218μs -> 167μs (30.5% faster)

def test_default_parameters():
    # Default parameter values differ, should be considered different
    code1 = "def foo(a=1): return a"
    code2 = "def foo(a=2): return a"
    codeflash_output = are_codes_duplicate(code1, code2) # 190μs -> 156μs (22.1% faster)

def test_annotations():
    # Parameter annotations differ, should be considered different
    code1 = "def foo(a: int): return a"
    code2 = "def foo(a: str): return a"
    codeflash_output = are_codes_duplicate(code1, code2) # 180μs -> 146μs (23.2% faster)

def test_return_type_annotations():
    # Return type annotations differ, should be considered different
    code1 = "def foo(a) -> int: return a"
    code2 = "def foo(a) -> str: return a"
    codeflash_output = are_codes_duplicate(code1, code2) # 178μs -> 143μs (24.1% faster)

def test_nested_scopes_variable_shadowing():
    # Variable shadowing in nested scopes
    code1 = '''
def foo(a):
    b = a + 1
    def bar(a):
        return a + b
    return bar(a)
'''
    code2 = '''
def foo(x):
    y = x + 1
    def bar(x):
        return x + y
    return bar(x)
'''
    codeflash_output = are_codes_duplicate(code1, code2) # 408μs -> 305μs (33.7% faster)

# 3. Large Scale Test Cases

def test_large_number_of_variables():
    # Large number of variables in assignments
    code1 = "\n".join([f"a{i} = {i}" for i in range(500)])
    code2 = "\n".join([f"v{i} = {i}" for i in range(500)])
    codeflash_output = are_codes_duplicate(code1, code2) # 17.2ms -> 13.7ms (25.7% faster)

def test_large_number_of_functions():
    # Large number of functions with similar structure, different variable names
    code1 = "\n".join([f"def f{i}(x): return x + {i}" for i in range(500)])
    code2 = "\n".join([f"def f{i}(y): return y + {i}" for i in range(500)])
    codeflash_output = are_codes_duplicate(code1, code2) # 48.2ms -> 34.3ms (40.4% faster)

def test_large_class_with_many_methods():
    # Large class with many methods, variable names differ
    code1 = "class A:\n" + "\n".join([f"    def m{i}(self, x): return x + {i}" for i in range(500)])
    code2 = "class A:\n" + "\n".join([f"    def m{i}(self, y): return y + {i}" for i in range(500)])
    codeflash_output = are_codes_duplicate(code1, code2) # 53.7ms -> 38.3ms (39.9% faster)

def test_large_nested_structures():
    # Large nested structure with variable names differing
    code1 = "def outer(x):\n"
    code1 += "    " + "\n    ".join([f"def inner{i}(y): return y + {i}" for i in range(50)])
    code2 = "def outer(a):\n"
    code2 += "    " + "\n    ".join([f"def inner{i}(b): return b + {i}" for i in range(50)])
    codeflash_output = are_codes_duplicate(code1, code2) # 5.12ms -> 3.67ms (39.5% faster)

def test_large_list_comprehension():
    # Large list comprehension with different variable names
    code1 = "[x for x in range(1000)]"
    code2 = "[y for y in range(1000)]"
    codeflash_output = are_codes_duplicate(code1, code2) # 212μs -> 168μs (26.0% faster)

def test_large_for_loop():
    # Large for loop with different variable names
    code1 = "\n".join([f"for i{i} in range({i}): pass" for i in range(100)])
    code2 = "\n".join([f"for j{i} in range({i}): pass" for i in range(100)])
    codeflash_output = are_codes_duplicate(code1, code2) # 7.38ms -> 5.36ms (37.8% faster)

def test_large_multiple_assignments():
    # Large number of multiple assignments with different variable names
    code1 = "\n".join([f"a{i}, b{i} = {i}, {i+1}" for i in range(500)])
    code2 = "\n".join([f"x{i}, y{i} = {i}, {i+1}" for i in range(500)])
    codeflash_output = are_codes_duplicate(code1, code2) # 43.9ms -> 33.1ms (32.8% faster)

def test_large_try_except_blocks():
    # Large number of try-except blocks with different variable names
    code1 = "\n".join([f"try:\n    a{i} = {i}\nexcept Exception as e{i}:\n    pass" for i in range(100)])
    code2 = "\n".join([f"try:\n    x{i} = {i}\nexcept Exception as err{i}:\n    pass" for i in range(100)])
    codeflash_output = are_codes_duplicate(code1, code2) # 8.93ms -> 6.94ms (28.8% faster)

def test_large_functions_different_structure():
    # Large number of functions, but structure differs in one
    code1 = "\n".join([f"def f{i}(x): return x + {i}" for i in range(499)] + ["def f499(x): return x * 499"])
    code2 = "\n".join([f"def f{i}(y): return y + {i}" for i in range(500)])
    codeflash_output = are_codes_duplicate(code1, code2) # 48.0ms -> 34.3ms (40.0% faster)

def test_large_code_with_docstrings():
    # Large code with docstrings present in one, not in the other
    code1 = "\n".join([f'def f{i}(x):\n    """Docstring"""\n    return x + {i}' for i in range(100)])
    code2 = "\n".join([f'def f{i}(y):\n    return y + {i}' for i in range(100)])
    codeflash_output = are_codes_duplicate(code1, code2) # 10.4ms -> 7.57ms (37.2% faster)
# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.
#------------------------------------------------
import ast

# imports
import pytest  # used for our unit tests
from codeflash.code_utils.deduplicate_code import are_codes_duplicate

# unit tests

# -----------------
# Basic Test Cases
# -----------------

def test_identical_code():
    # Two identical code snippets should be considered duplicates
    code = "def foo(x):\n    return x + 1"
    codeflash_output = are_codes_duplicate(code, code) # 217μs -> 169μs (28.6% faster)

def test_variable_name_difference():
    # Different variable names should be normalized
    code1 = "def foo(x):\n    a = x + 1\n    return a"
    code2 = "def foo(x):\n    b = x + 1\n    return b"
    codeflash_output = are_codes_duplicate(code1, code2) # 261μs -> 202μs (29.4% faster)

def test_function_name_difference():
    # Different function names should NOT be considered duplicate
    code1 = "def foo(x):\n    return x + 1"
    code2 = "def bar(x):\n    return x + 1"
    codeflash_output = are_codes_duplicate(code1, code2) # 212μs -> 165μs (28.6% faster)

def test_parameter_name_difference():
    # Different parameter names should NOT be considered duplicate
    code1 = "def foo(x):\n    return x + 1"
    code2 = "def foo(y):\n    return y + 1"
    codeflash_output = are_codes_duplicate(code1, code2) # 212μs -> 162μs (30.7% faster)

def test_class_name_difference():
    # Different class names should NOT be considered duplicate
    code1 = "class Foo:\n    def method(self):\n        return 42"
    code2 = "class Bar:\n    def method(self):\n        return 42"
    codeflash_output = are_codes_duplicate(code1, code2) # 207μs -> 165μs (25.2% faster)

def test_attribute_names():
    # Attribute names should NOT be normalized
    code1 = "def foo(obj):\n    return obj.value"
    code2 = "def foo(obj):\n    return obj.data"
    codeflash_output = are_codes_duplicate(code1, code2) # 189μs -> 151μs (25.4% faster)

def test_global_variable():
    # Global variable names should NOT be normalized
    code1 = "global x\ndef foo():\n    return x"
    code2 = "global y\ndef foo():\n    return y"
    codeflash_output = are_codes_duplicate(code1, code2) # 170μs -> 137μs (23.7% faster)

def test_nonlocal_variable():
    # Nonlocal variable names should NOT be normalized
    code1 = "def outer():\n    x = 1\n    def inner():\n        nonlocal x\n        return x\n    return inner()"
    code2 = "def outer():\n    y = 1\n    def inner():\n        nonlocal y\n        return y\n    return inner()"
    codeflash_output = are_codes_duplicate(code1, code2) # 316μs -> 244μs (29.2% faster)

def test_keyword_arguments():
    # Keyword argument names should NOT be normalized
    code1 = "def foo(a=1):\n    return a"
    code2 = "def foo(b=1):\n    return b"
    codeflash_output = are_codes_duplicate(code1, code2) # 189μs -> 156μs (20.8% faster)

def test_docstring_removal():
    # Docstrings should be removed before comparison
    code1 = 'def foo(x):\n    """This is a docstring"""\n    return x + 1'
    code2 = "def foo(x):\n    return x + 1"
    codeflash_output = are_codes_duplicate(code1, code2) # 223μs -> 171μs (30.5% faster)

def test_multiple_assignments():
    # Multiple assignments with different variable names should be considered duplicates
    code1 = "def foo(x):\n    a = x + 1\n    b = a * 2\n    return b"
    code2 = "def foo(x):\n    p = x + 1\n    q = p * 2\n    return q"
    codeflash_output = are_codes_duplicate(code1, code2) # 331μs -> 248μs (33.6% faster)

def test_for_loop_variable():
    # For loop variable names should be normalized
    code1 = "def foo(lst):\n    for x in lst:\n        print(x)"
    code2 = "def foo(lst):\n    for y in lst:\n        print(y)"
    codeflash_output = are_codes_duplicate(code1, code2) # 259μs -> 202μs (27.8% faster)

def test_list_comprehension_variable():
    # List comprehension variable names should be normalized
    code1 = "def foo(lst):\n    return [x for x in lst]"
    code2 = "def foo(lst):\n    return [y for y in lst]"
    codeflash_output = are_codes_duplicate(code1, code2) # 242μs -> 184μs (31.2% faster)

def test_nested_function_variable():
    # Nested function local variable names should be normalized
    code1 = "def foo(x):\n    def bar():\n        a = x + 1\n        return a\n    return bar()"
    code2 = "def foo(x):\n    def bar():\n        b = x + 1\n        return b\n    return bar()"
    codeflash_output = are_codes_duplicate(code1, code2) # 349μs -> 263μs (32.9% faster)

# -----------------
# Edge Test Cases
# -----------------

def test_empty_code():
    # Empty code should be considered duplicate with another empty code
    code1 = ""
    code2 = ""
    codeflash_output = are_codes_duplicate(code1, code2) # 54.4μs -> 51.6μs (5.49% faster)

def test_one_empty_one_nonempty():
    # Empty code and non-empty code should NOT be considered duplicate
    code1 = ""
    code2 = "def foo():\n    pass"
    codeflash_output = are_codes_duplicate(code1, code2) # 97.2μs -> 83.4μs (16.5% faster)

def test_only_docstrings():
    # Modules with only docstrings should be considered duplicate after removal
    code1 = '"""Module docstring"""'
    code2 = '"""Another docstring"""'
    codeflash_output = are_codes_duplicate(code1, code2) # 68.6μs -> 65.5μs (4.71% faster)

def test_syntax_error_code():
    # Code with syntax errors should not cause crash, but return False
    code1 = "def foo(:\n    pass"
    code2 = "def foo():\n    pass"
    codeflash_output = are_codes_duplicate(code1, code2) # 30.5μs -> 30.6μs (0.555% slower)

def test_different_structure():
    # Structurally different code should NOT be considered duplicate
    code1 = "def foo(x):\n    return x + 1"
    code2 = "def foo(x):\n    return x - 1"
    codeflash_output = are_codes_duplicate(code1, code2) # 224μs -> 172μs (30.2% faster)

def test_different_number_of_functions():
    # Different number of functions should NOT be considered duplicate
    code1 = "def foo(x):\n    return x + 1"
    code2 = "def foo(x):\n    return x + 1\ndef bar(y):\n    return y - 1"
    codeflash_output = are_codes_duplicate(code1, code2) # 277μs -> 207μs (33.7% faster)

def test_different_indentation():
    # Indentation differences should NOT affect normalization
    code1 = "def foo(x):\n    return x + 1"
    code2 = "def foo(x):\n\treturn x + 1"
    codeflash_output = are_codes_duplicate(code1, code2) # 213μs -> 163μs (30.3% faster)

def test_unicode_variable_names():
    # Unicode variable names should be normalized
    code1 = "def foo(x):\n    α = x + 1\n    return α"
    code2 = "def foo(x):\n    β = x + 1\n    return β"
    codeflash_output = are_codes_duplicate(code1, code2) # 274μs -> 214μs (27.8% faster)

def test_variable_shadowing():
    # Variable shadowing in nested scopes should be normalized
    code1 = "def foo(x):\n    a = x + 1\n    def bar():\n        a = x + 2\n        return a\n    return a + bar()"
    code2 = "def foo(x):\n    b = x + 1\n    def bar():\n        b = x + 2\n        return b\n    return b + bar()"
    codeflash_output = are_codes_duplicate(code1, code2) # 451μs -> 336μs (34.3% faster)

def test_lambda_variable():
    # Lambda variable names should be normalized
    code1 = "f = lambda x: x + 1"
    code2 = "f = lambda y: y + 1"
    codeflash_output = are_codes_duplicate(code1, code2) # 215μs -> 164μs (31.4% faster)

def test_lambda_body_variable():
    # Lambda body variable names should be normalized
    code1 = "f = lambda x: (a := x + 1)"
    code2 = "f = lambda x: (b := x + 1)"
    codeflash_output = are_codes_duplicate(code1, code2) # 249μs -> 183μs (36.0% faster)

def test_comprehension_multiple_vars():
    # Multiple variables in comprehensions should be normalized
    code1 = "def foo(lst):\n    return [(x, y) for x, y in lst]"
    code2 = "def foo(lst):\n    return [(a, b) for a, b in lst]"
    codeflash_output = are_codes_duplicate(code1, code2) # 306μs -> 238μs (28.4% faster)

def test_nested_comprehension_vars():
    # Nested comprehension variables should be normalized
    code1 = "def foo(lst):\n    return [[x for x in row] for row in lst]"
    code2 = "def foo(lst):\n    return [[y for y in col] for col in lst]"
    codeflash_output = are_codes_duplicate(code1, code2) # 289μs -> 224μs (29.1% faster)

def test_with_statement_variable():
    # With statement variable names should be normalized
    code1 = "def foo(f):\n    with f as x:\n        return x"
    code2 = "def foo(f):\n    with f as y:\n        return y"
    codeflash_output = are_codes_duplicate(code1, code2) # 227μs -> 182μs (24.5% faster)

def test_try_except_variable():
    # Exception variable names should be normalized
    code1 = "try:\n    pass\nexcept Exception as e:\n    print(e)"
    code2 = "try:\n    pass\nexcept Exception as ex:\n    print(ex)"
    codeflash_output = are_codes_duplicate(code1, code2) # 202μs -> 164μs (23.2% faster)

def test_augmented_assignment():
    # Augmented assignment variable names should be normalized
    code1 = "def foo(x):\n    a = x\n    a += 1\n    return a"
    code2 = "def foo(x):\n    b = x\n    b += 1\n    return b"
    codeflash_output = are_codes_duplicate(code1, code2) # 264μs -> 214μs (23.6% faster)

def test_multiple_functions_with_same_body():
    # Multiple functions with same body but different names should NOT be considered duplicate
    code1 = "def foo(x):\n    return x + 1\ndef bar(x):\n    return x + 1"
    code2 = "def baz(x):\n    return x + 1\ndef qux(x):\n    return x + 1"
    codeflash_output = are_codes_duplicate(code1, code2) # 331μs -> 245μs (34.7% faster)

# -----------------
# Large Scale Test Cases
# -----------------

def test_large_number_of_variables():
    # Large number of variables, all should be normalized
    code1 = "\n".join([f"a{i} = {i}" for i in range(500)])
    code2 = "\n".join([f"b{i} = {i}" for i in range(500)])
    codeflash_output = are_codes_duplicate(code1, code2) # 17.2ms -> 13.7ms (25.9% faster)

def test_large_function_with_many_lines():
    # Large function bodies with many lines, variable names normalized
    code1 = "def foo(x):\n" + "\n".join([f"    a{i} = x + {i}" for i in range(500)]) + "\n    return a0"
    code2 = "def foo(x):\n" + "\n".join([f"    b{i} = x + {i}" for i in range(500)]) + "\n    return b0"
    codeflash_output = are_codes_duplicate(code1, code2) # 29.9ms -> 22.1ms (35.2% faster)

def test_large_for_loop():
    # Large for loop, variable names normalized
    code1 = "def foo(lst):\n    for x in lst:\n        pass\n" * 500
    code2 = "def foo(lst):\n    for y in lst:\n        pass\n" * 500
    codeflash_output = are_codes_duplicate(code1, code2) # 49.4ms -> 37.0ms (33.5% faster)

def test_large_list_comprehension():
    # Large list comprehension, variable names normalized
    code1 = "def foo(lst):\n    return [" + ", ".join([f"x{i}" for i in range(500)]) + " for x0 in lst]"
    code2 = "def foo(lst):\n    return [" + ", ".join([f"y{i}" for i in range(500)]) + " for y0 in lst]"
    codeflash_output = are_codes_duplicate(code1, code2) # 1.40ms -> 1.39ms (0.579% faster)

def test_large_nested_functions():
    # Large number of nested functions, variable names normalized
    code1 = ""
    code2 = ""
    for i in range(50):
        code1 += f"def f{i}(x):\n    a{i} = x + {i}\n    return a{i}\n"
        code2 += f"def f{i}(x):\n    b{i} = x + {i}\n    return b{i}\n"
    codeflash_output = are_codes_duplicate(code1, code2) # 6.98ms -> 5.14ms (35.8% faster)

def test_large_code_different_structure():
    # Large code with different structure should NOT be considered duplicate
    code1 = "\n".join([f"a{i} = {i}" for i in range(500)])
    code2 = "\n".join([f"a{i} = {i}+1" for i in range(500)])
    codeflash_output = are_codes_duplicate(code1, code2) # 23.7ms -> 18.0ms (31.7% faster)

def test_large_code_with_docstrings():
    # Large code blocks with docstrings should be considered duplicate after removal
    code1 = '"""Large docstring"""\n' + "\n".join([f"a{i} = {i}" for i in range(500)])
    code2 = '"""Another docstring"""\n' + "\n".join([f"b{i} = {i}" for i in range(500)])
    codeflash_output = are_codes_duplicate(code1, code2) # 17.1ms -> 13.7ms (24.7% faster)
# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.
#------------------------------------------------
from codeflash.code_utils.deduplicate_code import are_codes_duplicate

def test_are_codes_duplicate():
    are_codes_duplicate('', '')

🔎 Concolic Coverage Tests and Runtime

Test File::Test Function	Original ⏱️	Optimized ⏱️	Speedup
codeflash_concolic_c0ny_7kg/tmp169mvt23/test_concolic_coverage.py::test_are_codes_duplicate	55.0μs	53.8μs	2.22%✅

To edit these changes git checkout codeflash/optimize-pr733-2025-09-13T23.57.57 and push.

[misrasaurabh1]

thank you, accepted the changes