Escape literal emphasis markers merged into a text token

[hdimer]

MarkdownRenderer re-emits a literal */_ bare when the paragraph also contains real emphasis, so reformatting turns plain text into emphasis.

import mistune
from mistune.renderers.markdown import MarkdownRenderer

reformat = mistune.create_markdown(renderer=MarkdownRenderer())
reformat(r"a *b* c \*d\* e" + "\n")
# 'a *b* c *d* e\n'   -> re-parses as <em>b</em> ... <em>d</em>

Root cause

text() escaped markers only when the whole token was made of them:

if raw and all(c in "*_" for c in raw):

That condition depends on inline-token fragmentation. With no emphasis in the paragraph, \*baz\* arrives as three tokens (*, baz, *) and each marker token is escaped. Once real emphasis is present the parser hands the renderer a single merged token (*bar* \*baz\* → emphasis + ' *baz*'), the condition is false, and the markers are re-emitted bare. So whether a literal marker survived a round-trip depended on unrelated content elsewhere in the same paragraph.

Fix

Escape per marker run rather than per token. A run needs a non-space character on one side to open or close emphasis, and a _ run also cannot do so from inside a word, so those two cases stay bare — 2 * 3 and snake_case are still emitted unescaped, as b042996 intended.

The two exemptions deliberately mirror _can_open_emphasis/_can_close_emphasis in _inline/emphasis.py and use the same str.isspace()/str.isalnum() predicates, so the renderer cannot disagree with the parser about what a delimiter is.

One follow-on: escaping the text of an autolink demoted it to a verbose inline link for any URL containing _ (<https://en.wikipedia.org/wiki/Foo_(bar)> → [https://en.wikipedia.org/wiki/Foo\_(bar)](...)). The <url> shortcut now matches on the raw text rather than the escaped render.

Known edge

A bare unmatched marker sitting directly against real emphasis markers, as in a***a*a*, is escaped where the token ends, which shortens the source's delimiter run and can drop the emphasis. Over an exhaustive sweep of {a,b,*,_} strings up to length 12 this affects 166 of 114,231 inputs against 4,521 fixed, and every affected input is an unescaped leftover marker adjacent to emphasis punctuation — never text that spells \* deliberately, which is what this is protecting. Happy to take a different trade-off if you would rather not escape at a token edge, though that reopens the case above.

Testing

Three cases added to TestMarkdownRendererRoundTrip; each fails without its source change (verified by reverting each hunk separately):

• test_escaped_marker_alongside_real_emphasis — the bug.
• test_literal_markers_escaped_only_where_they_could_delimit — asserts the emitted Markdown, not the HTML. assert_round_trip compares HTML and so cannot see a stray backslash; without this, the two exemptions could be deleted with the suite still green.
• test_autolink_url_containing_a_marker — the autolink shortcut.

Full suite passes (1160), ruff check and mypy clean. Reformatting all 652 bundled CommonMark spec examples and re-rendering goes from 85 meaning-changing round-trips to 82, with no new failures in any section.

---

Used AI assistance on this; I reviewed and tested the change myself.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.21%. Comparing base (75cab78) to head (6e9b133).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #480      +/-   ##
==========================================
+ Coverage   91.18%   91.21%   +0.02%     
==========================================
  Files          36       36              
  Lines        3631     3641      +10     
  Branches      677      677              
==========================================
+ Hits         3311     3321      +10     
  Misses        193      193              
  Partials      127      127              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[kbulygin]

I've encountered this too:

import mistune
from mistune.renderers.markdown import MarkdownRenderer

reformat = mistune.create_markdown(renderer=MarkdownRenderer())
print(reformat(r"(2) \\(2^n\\)"))

Output:

(2) \(2^n\)

(Ends up as (2) (2^n) when rendered.)

Expected (the same as input):

(2) \\(2^n\\)

(Renders to (2) \(2^n\) in HTML. \( and \) are used in KaTeX.)

It seems that a broader fix is needed. (Somewhat related, may help: xberg-io/html-to-markdown#458.)

[hdimer]

Thanks -- same root cause one character over. text() re-emits a literal backslash bare, so the re-parse consumes it as an escape for whatever follows and \( comes back as (.

Pushed a fix on this branch rather than opening a second PR, since it lands on the same three lines of the same method. The backslash is escaped first, ahead of the */_ and backtick escapes, which is what _escape_title() in this file already does for the same reason. The ordering is load-bearing rather than cosmetic: escaping it last would double the backslashes the method adds itself, and existing tests catch that.

>>> print(reformat(r"(2) \\(2^n\\)"))
(2) \\(2^n\\)

It also picks up a structural case I had missed: a literal backslash ending a line was re-emitted bare and came back as a hard line break, splitting a paragraph in two. That is in the test alongside your repro.

On over-escaping, since it is the obvious objection. The escape is unconditional, so a backslash that would not have been consumed gets doubled too and a Windows path comes back with doubled separators. Conditioning on the following character is harder than it looks, because parse_escape gives each escaped character its own text token:

>>> [t["raw"] for t in ast(r"(2) \\(2^n\\)")]
['(2) ', '\\', '(2^n', '\\', ')']

The backslash never sees the (, so a plain "escape only before escapable punctuation" lookahead never fires and your case survives unfixed. Adding "or on a token edge" does work -- I built it and it is correct -- but it costs a module-level regex plus a token-edge special case, and across the 652 bundled CommonMark examples it buys back exactly one example over the unconditional replace. Not worth it, and _escape_title already sets the doubling precedent in this file.

@lepture -- I rechecked the whole PR against current main, after the link/emphasis refactor in bde6ad7. Both bugs still reproduce there, the branch applies on top, suite green (1162). Over the bundled corpus, meaning-changing round-trips go 85 -> 79, nothing newly broken.

One divergence to disclose, since it is the only thing that sweep flagged:

src = "[\\`]`"
# before this branch: unchanged and stable
# after:              '[\\\\`]`', doubling on every further pass

That is a pre-existing parser bug rather than a renderer one. InlineParser.precedence_scan searches for codespan/link/prec_auto_link/prec_inline_html without skipping backslash-escaped positions, so an escaped backtick inside a bracket span opens a phantom code span and mistune mis-parses its own valid output:

[a\`b](u)\`   current  <p>[a\<code>b](u)\</code></p>
              expected <p><a href="u">a`b</a>`</p>

The literal backslash then lands in a text token, and the renderer now faithfully escapes it, so an input mistune already mis-parses grows a backslash per pass instead of resting on a broken fixed point. I tried a guard that skips a candidate preceded by an odd number of backslashes: it gives the CommonMark-correct output above and changes no example in the bundled corpus. That belongs in its own PR though -- happy to open one if you want it.

---

Used AI assistance on this; I reviewed and tested the change myself.