mpl 2.0b2

from matplotlib import pyplot as plt

plt.annotate("foo", (.5, .5),
             (15, 15), textcoords="offset points",
             arrowprops={"arrowstyle": "->"},
             bbox={"ec": "k"}).draggable()
plt.show()

Panning the axes shows that the annotation uses offset coordinates: it moves "with" the point it points to. However, after dragging the annotation somewhere else, it switches to figure coordinates: when panning the axes, the text box stops physically moving; instead it's the arrow that moves to follow the point.
(Try it yourself to see what I mean :-))

Edit: The culprits are here and here. Probably not too hard to fix; however can we just internally normalize anncoords to always be a pair? Perhaps related to the possible switch to traitlets (how's that going)?