My gut feeling is that we should perhaps re-allow devdocs/devel (or at least some subset of that) because that portion of the website which is targeted towards developers of Matplotlib is more useful as the devdocs than as the stable release docs. I agree that pointing to devdocs for examples/api for unreleased things is not ideal, just not sure a full ban on devdocs as a whole is warranted.

Agree with @ksunden on dev docs - almost wondering if it'd be worth it to hide the non-dev devel docs but that's probably overcomplicating things.

I added lines for each item in the 3.7.1 docs, except URLs that lead to "Page not found," such as matplotlib.org/devdocs/_static.

Its possible that something like

Disallow: /devdocs/
Allow: /devdocs/devel/

would accomplish the same thing, but I'm not sure.

It seems like the "Allow" tag is much newer than the "Disallow" tag, and may or may not be universally recognized by web crawlers. RFC 9309 from Sept. 2022, which includes the "Allow" tag, extends another RFC from 1994.

it shouldn't change that much, and its confusing to plop people in the middle of devdocs if they are not expecting it.

The ways it changes tend to be exactly the things you need to know to build the repo or docs.

However, I'm not quite sure what the justification for allowing the devdocs version of devel is - it shouldn't change that much, and its confusing to plop people in the middle of devdocs if they are not expecting it.

For me, one significant difference is that /stable/devel doesn't include the "make html-skip-subdirs" doc build yet. If someone searches "matplotlib build docs," I would hope they get a /devdoc/devel link just for that command.

IIRC there was the idea to take the development docs out of the library docs. They are logically independent and change at different rates / times.