I would be happy to try this, alone I cannot find any instructions how to get this change into the maplotlib git repo I just downloaded for 30 min. I am sure it is easy to do, but I cannot guess how to. There no git has provided, etc. If it is even in the git main repo ... sorry, I just have never used this.

I managed to get you changes on top of current master. No improvement. I firmly believe all "upgrades" on this should just be undone until this can get under control. Here the error message I get from my code which is too long to share:

In [3]: Traceback (most recent call last):
  File "/home/alex/matplotlib/lib/matplotlib/cbook/__init__.py", line 389, in process
    proxy(*args, **kwargs)
  File "/home/alex/matplotlib/lib/matplotlib/cbook/__init__.py", line 227, in __call__
    return mtd(*args, **kwargs)
  File "/home/alex/matplotlib/lib/matplotlib/backend_bases.py", line 3116, in release_zoom
    self._zoom_mode, twinx, twiny)
  File "/home/alex/matplotlib/lib/matplotlib/axes/_base.py", line 3650, in _set_view_from_bbox
    self.set_xlim((x0, x1))
  File "/home/alex/matplotlib/lib/matplotlib/axes/_base.py", line 2932, in set_xlim
    self.viewLim, "intervalx", emit, auto, kw)
  File "/home/alex/matplotlib/lib/matplotlib/axes/_base.py", line 2842, in _set_lim
    "Axis limits must be (or convert to) finite reals")
ValueError: Axis limits must be (or convert to) finite reals

The other problem is that I have a user-created axis that does not plot at all in mpl 2.1, and there is not even an error message. I believe this is due to some silencing of error messages that was also done in 2.1 for the sake of, aka matplotlib.cbook.CallbackRegistry.process() suppresses exceptions by default as listed on the API changes. It remains unmentioned how or where I can influence this as a user.

OK, I could find the reason for one of the issues, the behaviour of of ax.transData has changed from 2.0.2 to 2.1. If y-axis is log and a y-value of 0 is provided, it returns np.nan also for the x-value. This is a bug and should be fixed. Probably a separate bug report should be filed?

After identifying #9369 your patch seems to fix my problem. Thanks! When this is included, I assume you may close #9361.

The latest version actually works better than the pre-broken version: it doesn't even bother passing invalid values to set_xlim anymore.

I like the direction this is going 👍

Turns out the set_?lim unification is a mess, because custom axes typically don't define get_${axisname}lim so you need to chase them down to see who's using what name.
Stripped that part out for now, will probably be the object of a separate PR later.
Now we still get warnings about invalid values but at least it should behave similarly to 2.0.2.

Was the change to ban calling set_?lim with NaNs intentional? That's creating new challenges for us with real-world, real-time messy data. I've got a script that worked in 2.0 that now fails with 2.1 when we calculate limits from data that's all missing.

The issue was discussed (with a quick vote...) on #7460. I don't think the previous behavior was particularly helpful, but feel free to revisit it...

Thanks for pointing me to that...looks like I need to fix things on our end.

Independent of if we should treat np.nan as None in setting the limits, this avoids the whole problem when panning.