Opening large text files causes 100% CPU hang in difflib.SequenceMatcher (yunicode.py)

[ctcjab]

Problem

Opening a large text file (e.g. a 176MB log file) in JupyterLab with jupyter_server_ydoc installed causes the Jupyter server process to peg at 100% CPU indefinitely, making the server completely unresponsive. The server must be killed to recover, and the problem recurs on restart if the file tab is restored.

Root cause

YUnicode.set() in yunicode.py calls difflib.SequenceMatcher.ratio() to decide whether to perform a granular update or a full reload:

matcher = SequenceMatcher(a=before_bytes, b=after_bytes)

if (
    matcher.real_quick_ratio() >= SIMILARITY_THREESHOLD
    and matcher.ratio() >= SIMILARITY_THREESHOLD
):

SequenceMatcher.ratio() internally calls get_matching_blocks() → find_longest_match(), which is O(n * m) in the lengths of the two sequences. For a 176MB file this means diffing ~167M × 184M bytes, which takes hours or longer and blocks the main thread (with the GIL held) for the entire duration.

The real_quick_ratio() check is fast but doesn't help when the file is being loaded for the first time (old value is empty → ratio is 0.0 → but the file being opened as a new ydoc room goes through rooms.py:initialize → ybasedoc.py:aset → yunicode.py:set where the old value is already populated from ystore).

Stack trace

Captured with py-spy dump --pid <PID>:

Thread 7 (active+gil): "MainThread"
    find_longest_match (difflib.py:385)
        ahi: 167472901
        bhi: 184489591
    get_matching_blocks (difflib.py:454)
    ratio (difflib.py:619)
    set (jupyter_ydoc/yunicode.py:85)
    aset (jupyter_ydoc/ybasedoc.py:203)
    initialize (jupyter_server_ydoc/rooms.py:164)
    open (jupyter_server_ydoc/handlers.py:278)

Impact

• Server becomes completely unresponsive (all tabs, all kernels, all terminals)
• The .jupyter_ystore.db grows to match the file size (~177MB in our case)
• Restarting the server doesn't help if the browser restores the tab — the problem immediately recurs
• Recovery requires removing the file (or renaming it so the tab can't find it), deleting .jupyter_ystore.db, and restarting

Suggested fix

Add a file size threshold before attempting difflib-based granular updates. For files above a certain size (e.g. 5MB or 10MB), skip the SequenceMatcher entirely and fall back to the full-reload path which is already implemented in the else branch:

MAX_DIFF_SIZE = 5 * 1024 * 1024  # 5 MB

if len(before_bytes) > MAX_DIFF_SIZE or len(after_bytes) > MAX_DIFF_SIZE:
    self._ysource.clear()
    if value:
        self._ysource += value
else:
    matcher = SequenceMatcher(a=before_bytes, b=after_bytes)
    ...

Environment

• jupyter_ydoc 3.1.1
• jupyter_server_ydoc 2.3.0
• Python 3.13.12
• JupyterLab 4.5.6

[krassowski]

If the file is opened for the first time, is before_bytes equal to b''? Could we use that for defining a fast path?

[ctcjab]

Good question. In the case we hit, before_bytes is not empty — the flow is:

1. rooms.py:initialize loads Y updates from the ystore (apply_updates on line 123)
2. This populates the ydoc with the previous content, so self._ysource is already set
3. It then checks if the ystore content matches disk (line 148: if await self._document.aget() != model["content"])
4. If they differ (e.g. the file grew), it calls await self._document.aset(model["content"]) (line 160)
5. This hits yunicode.py:set() where old_value = content from ystore (large), value = content from disk (also large)

So both before_bytes and after_bytes are ~170MB in this case. The ystore had already ingested the file in a previous session, and on the next server start it's diffing the stored version against the current disk version.

That said, a fast path for before_bytes == b'' would still help for the true first-open case (no ystore history). But it wouldn't prevent the scenario we hit, where the ystore already has the file and the server restarts. A size threshold check would cover both cases.

[krassowski]

If the file grew by appending content at the end (regardless of whether it happened on runtime or in between restarts), we should handle that smoothly regardless of the size and avoid hard reload. I think we can do better than SequenceMatcher by introducing additional assumptions about what kind of modifications we expect.

In case of log files where content is ever appended at the end, we could just exclude the shared prefix from SeqeuenceMatcher analysis.

Would that help with your use case, or are the modifications on disk more substantial/random?

In case of random or random-like modifications, I wonder if we could add a limit on sequence matcher operations (rather than a fixed bytes threshold). We could easily implement this if get_matching_blocks was a generator; it should not be too difficult to reimplement it in a subclass by adding a bound on how many recursion steps it is allowed to execute.

[ctcjab]

Nice, thanks. Sounds like that would help with the case we hit today. I'm not sure if our users will also hit this with random modifications (and I already removed jupyter_ydoc from the environment they'll start picking up by next week at the latest, until this issue is fixed), but if they do, I can reply here with those details if that would be helpful.