gh-137836: Support more RAWTEXT and PLAINTEXT elements in HTMLParser

[serhiy-storchaka]

• the "plaintext" element
• the RAWTEXT elements "xmp", "iframe", "noembed" and "noframes"
• optionally RAWTEXT (if scripting=True) element "noscript"

• Issue: Support other RAWTEXT and PLAINTEXT elements in HTMLParser #137836

---

📚 Documentation preview 📚: https://cpython-previews--137837.org.readthedocs.build/

[ezio-melotti]

This should be updated now that the list has been expanded.

It might be easier to have a short section about parsing modes, listing each mode, which elements trigger it, whether charrefs are converted or not, and when the state is terminated.

Here we could then say

Suggested change

	references (except the ones in ``script``/``style`` elements) are
	references (except the ones in RAWTEXT tags) are

with RAWTEXT linking to that section.

[serhiy-storchaka]

Do we need to document this here? This is a part of the HTML5 specification. What will the user get from this information?

[ezio-melotti]

I think it would be better to move this in set_cdata_mode by adding a third branch to the if/else that sets self.interesting.

[serhiy-storchaka]

I considered this option. But should we repeat condition tag == "plaintext" in two places or add "plaintext" to CDATA_CONTENT_ELEMENTS or RCDATA_CONTENT_ELEMENTS? In any case we will need to repeat "plaintext" twice. This can also create asymmetry with "noscript" if special cases will be handled in different places. So I came to the current code.

Other option is to use special value escapable=None to switch to the PLAINTEXT mode.

[ezio-melotti]

This PR seems to address 3 issues:

1. It adds a scripting arg, that is used to determine how <noscript>...</noscript> is handled. If the browser has JS enabled, the content of <noscript> is not parsed; if it's disabled, the content is parsed and displayed as a fallback. The former is emulated by passing scripting=True (which adds noscript to the list of RAWTEXT elements), the latter by passing scripting=False.
2. It adds 4 more elements that are (unconditionally) handled as RAWTEXT (in addition to the existing 2, script and style): xmp, iframe, noembed, and noframes.
3. It adds support for PLAINTEXT state, triggered by the plaintext element.

The difference between states is the following:

• Data state (the default): both tags (<element>) and charrefs (&name;) are parsed (and possibly converted if convert_charrefs=True);
• RCDATA state (triggered by title and textarea): tags are not parsed by charrefs are (and possibly converted if convert_charrefs=True). After a matching closing tag is met, the parser returns to data state. The list of elements is stored in the RCDATA_CONTENT_ELEMENTS constant;
• RAWTEXT state (triggered by script, style, xmp, iframe, noembed, noframes, and possibly noscript, if scripting=True): both tags and charrefs are not parsed/converted, and the content of those 6 elements is returned as is. After the matching closing tag is met, the parser returns to data state. Note that the list of elements is stored in the CDATA_CONTENT_ELEMENTS constant;
• PLAINTEXT state (triggered by plaintext): both tags and charrefs are not parsed/converted (like for RAWTEXT) until the EOF -- there is no matching closing tag and everything until the end is emitted as is.

[ezio-melotti]

I don't like too much (ab)using escapable=None for PLAINTEXT mode.

Currently the set_cdata_mode function does two things:

1. determines where the closing tag/end is, which depends on the value tag passed;
2. determines whether charrefs are converted, which depends on the value passed to escapable;

Even though there is some duplication, I would prefer something like this:

            if (tag in self.CDATA_CONTENT_ELEMENTS or
                (self.scripting and tag == "noscript") or
                tag == "plaintext"):
                self.set_cdata_mode(tag, escapable=False)
            elif tag in self.RCDATA_CONTENT_ELEMENTS:
                self.set_cdata_mode(tag, escapable=True)

This makes clear that all these cases are handled by set_cdata_mode, with the former ignoring charrefs and the latter converting them.

Then in set_cdata_mode we can set self.interesting based on the values of the args passed. This will also make it clearer what is considered interesting for each tag.

[serhiy-storchaka]

Thank you for your review @ezio-melotti.

[miss-islington-app]

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖

[bedevere-app]

GH-140841 is a backport of this pull request to the 3.14 branch.

[bedevere-app]

GH-140842 is a backport of this pull request to the 3.13 branch.

[serhiy-storchaka]

Backporting to older Python versions should be from 3.13.