Here: http://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.annotate it is written that xytext is the position of the label string. But while working I found that xytext coordinates are actually offset specified from xy coordinates.