Adding text with fpdf2 and saving it back generates PDF with corrupted text

[raffaem]

Opening a PDF with pypdf, adding text with fpdf2, and saving it back, generate a corrupted text

See here for the details (you can start from the third post, which is were the link will brings you)

[stefan6419846]

Thanks for the report. Even when cross-posting as in this case, please always use the provided issue templates and fill the accordingly to allow easier reproduction.

This sounds somehow similar to #3450, and similar to this issue, having a 1.5 MB reproducer with 123 pages increases complexity for review.

Are you able to reproduce this with plain pypdf to simplify analysis? Have you tried pinpointing this to a specific pypdf version?

[raffaem]

Thank you.

I'm unable to reproduce it with plain pypdf and a single page at a time.

For instance something like:

from pypdf import PdfReader, PdfWriter

# Get the data
reader_base = PdfReader("input.pdf")

reader = PdfReader("output_fpdf29.pdf")
page_box = reader.pages[0]

page_base = reader_base.pages[28]
page_base.merge_page(page_box)
writer = PdfWriter()
writer.add_page(page_base)
with open(f"output_2.pdf", "wb") as fp:
    writer.write(fp)

works as expected.

It looks like some kind of memory corruption when processing multiple pages.

[stefan6419846]

Sorry, but I am not able to reliably reproduce this on Python 3.11.13 - I had some issues first, but they are gone now. Without this, debugging/pinpointing this to a specific change is nearly impossible.

[raffaem]

Sorry, but I am not able to reliably reproduce this on Python 3.11.13 - I had some issues first, but they are gone now. Without this, debugging/pinpointing this to a specific change is nearly impossible.

What do you mean by "this"?

[raffaem]

The current MWE is this one

[stefan6419846]

As long as I am not able to reliably reproduce the reported issue, it is rather impossible for me to debug the problem and possibly identify related changes/versions.

Just did another quick check with Python 3.9.23, which does not allow me to reproduce this issue as well.

[raffaem]

As long as I am not able to reliably reproduce the reported issue, it is rather impossible for me to debug the problem and possibly identify related changes/versions.

Just did another quick check with Python 3.9.23, which does not allow me to reproduce this issue as well.

Can you please detail which code are you trying to run?

As stated, the current MWE (however long it may be) is this one, which was reproduced by at least one third-party.

Did you try with Python 3.13?

[stefan6419846]

Can you please detail which code are you trying to run?

I used py-pdf/fpdf2#1570 (comment). As mentioned, I initially saw some issues as well, but it "magically" went away at some point with the latest versions which would previously reproduce it.

Did you try with Python 3.13?

I have no direct access to Python 3.13 on my regular development machines.

The output of Python 3.12.3 on Ubuntu 24.04 looks clean for me as well.

[bellini666]

@raffaem does #3450 (comment) make any sense with this issue? i.e. if you try what I suggested there, does it fixes this issue here?

[raffaem]

@raffaem does #3450 (comment) make any sense with this issue? i.e. if you try what I suggested there, does it fixes this issue here?

I'm not 100% sure what's happening here, but I think you are onto something.

First off, I cannot reproduce it with 3.12, only with 3.13.

Second, if in 3.13 I follow your suggestion or I disable the garbage collector, the bug seems to disappear.

However, I have tried to log when the overlays get destroyed. While (without your fix) the overlays get in fact destroyed before the output file is written, this is still the case in 3.12, altough the garbage collector is much less aggressive (but still destroys some of them when exiting the context manager).

Moreover, altough they talked about changes in the garbage collector for 3.13 for a new "incremental" GC, it seems those changes got reverted, so the GC in 3.13 should be the same of 3.12.

Anyway, if we are fighting the garbage collector, maybe we should all go rusty instead.

from datetime import datetime
from contextlib import contextmanager
import pypdf
import fpdf
from fpdf import FPDF, get_scale_factor
from pypdf import PdfWriter, PdfReader
import io
import sys
from dataclasses import dataclass
from pathlib import Path
from tqdm import tqdm
import requests, zipfile, io

import gc
# !! FIX 2 !! #
gc.disable()

print("Using python version:", sys.version)
print("Using pypdf version:", pypdf.__version__)
print("Using fpdf version:", fpdf.__version__)

# Download Aptos font from Microsoft
zip_file_url = "https://download.microsoft.com/download/8/6/0/860a94fa-7feb-44ef-ac79-c072d9113d69/Microsoft%20Aptos%20Fonts.zip"
r = requests.get(zip_file_url)
z = zipfile.ZipFile(io.BytesIO(r.content))
z.extractall("./aptos")
font_path = Path("./aptos/Aptos.ttf")
assert font_path.is_file()

class MyPDFReader(PdfReader):
    def __init__(self, stream, pagei):
        super().__init__(stream)
        self.pagei = pagei

    def __del__(self):
        print(f"Destroyed overlay for page {self.pagei}")

overlays = list()

# Combine fpdf2 with pypdf, see https://py-pdf.github.io/fpdf2/CombineWithPypdf.html#combine-with-pypdf
@contextmanager
def add_to_page(reader_page, pagei, unit="mm"):
    k = get_scale_factor(unit)
    format = (reader_page.mediabox[2] / k, reader_page.mediabox[3] / k)
    pdf = FPDF(format=format, unit=unit)
    pdf.add_page()
    yield pdf
    page_overlay = MyPDFReader(stream=io.BytesIO(pdf.output()), pagei=pagei).pages[0]
    # !! FIX 1 !! #
    #overlays.append(page_overlay)
    print(f"Merging overlay for page {pagei}")
    reader_page.merge_page(page2=page_overlay)

# Text object dataclass
@dataclass
class TextObj:
    text: str
    coords: list
    font_family: str = "Aptos"
    font_size: int = 10
    font_path: Path = font_path

# Add text objects to a page
def add_text_objs(text_objs, writer, pagei):
    with add_to_page(writer.pages[pagei], pagei) as pdf:
        for text_obj in text_objs:
            pdf.add_font(family=text_obj.font_family, fname=text_obj.font_path)
            pdf.set_font(family=text_obj.font_family, size=text_obj.font_size)
            pdf.text(x=text_obj.coords[0], y=text_obj.coords[1], text=text_obj.text)

# Convert coordinates in mm
def pdfbox2mm(box):
    return [float(coord) / get_scale_factor("mm") for coord in box]

writer = PdfWriter()
reader = PdfReader("input.pdf")
pages = list(reader.pages)

for pagei, page in tqdm(enumerate(pages), total=len(pages)):

    writer.add_page(page)
    mediabox_mm = pdfbox2mm(page.mediabox)
    anns = list()

    # Add page numbers
    # In pypdf2, (0,0) is the bottom-left corner.
    # In fpdf2, (0,0) is the upper-left corner.
    coords = [mediabox_mm[2] * 0.9, mediabox_mm[3] * 0.96]
    text = f"{pagei + 1} / {len(reader.pages)}"
    ann = TextObj(text=text, coords=coords)
    anns.append(ann)

    # Add compilation time
    coords = [mediabox_mm[2] * 0.05, mediabox_mm[3] * 0.96]
    text = f"Snapshot of {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}"
    ann = TextObj(text=text, coords=coords)
    anns.append(ann)

    # Add disclaimer
    coords = [mediabox_mm[2] * 0.35, mediabox_mm[3] * 0.03]
    text = f"WORKING PAPER - PLEASE DO NOT DISTRIBUTE"
    ann = TextObj(text=text, coords=coords)
    anns.append(ann)

    add_text_objs(anns, writer, pagei)

writer.write("output.pdf")

# Use the overlays list to prevent garbage collection
print(len(overlays))
print("Output PDF created")