I pushed the fix for the offending flake line to your fork @tacaswell (hoping that it helps more than it troubles).

In general, however, I have some problems with this nitpicky stuff.

1. It seems noone is currently able to understand the reasons for (most of) the missing references. They might even be caused by bugs upstream. It is hence likely that they won't be fixed soon. Also, if there are upstream fixes required, they might not help because currently the sphinx version used by matplotlib is pinned.
2. From now on, each time an edit of any of the files that contain missing references in docstrings will require to edit the line numbers, which is not only cumbersome, but clearly a burden we should not load on contributors' shoulders.
3. The race condition that led to this trouble here will likely reoccur in the future unless all missing references have been fixed at least once.

So is anyone actually optimistic this can be solved anytime soon? If not, I would propose to disable nitpicky for now.

@ImportanceOfBeingErnest I think you are right, but I would propose something slightly different:

The line number bit was suggested by @anntzer in the original PR. I propose we remove the warnings about unused nitpick ignores by line number – but otherwise leave nitpick turned on.

This will still prevent the addition of "new" incorrect references, but allow any existing missing references to persist. The disadvantage is that it will allow "old" missing reference ignores to persist as well.

Thanks @ImportanceOfBeingErnest definitely helps more than it hurts!

I'm going to merge this so that our docs build clean again and move discussion about this to a new issue.