matplotlib.github.com/3.4.0/devel/testing.html at main · matplotlib/matplotlib.github.com

History

392 lines (369 loc) · 28.2 KB

Raw

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"

"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<head>

<title>Testing — Matplotlib 3.4.0 documentation</title>

title="Search within Matplotlib 3.4.0 documentation"

href="../_static/opensearch.xml"/>

</head><body>

<div id="unreleased-message"> You are reading an old version of the documentation (v3.4.0). For the latest version see <a href="https://matplotlib.org/stable/devel/testing.html">https://matplotlib.org/stable/devel/testing.html</a></div>

<!--

</div>

-->

<div style="float: left; position: absolute; width: 496px; bottom: 0; padding-bottom: 24px"><span style="float: right; color: #789; background: white">Version 3.4.0</span></div>

<div id="forkongithub"><a href="https://github.com/matplotlib/matplotlib">Fork me on GitHub</a></div>

</div>

<ul>

<li><a href="../users/installing.html">Installation</a></li>

<li><a href="../contents.html">Documentation</a></li>

<li><a href="../gallery/index.html">Examples</a></li>

<li><a href="../tutorials/index.html">Tutorials</a></li>

<li><a href="index.html">Contributing</a></li>

</form>

</li>

</ul>

</nav>

<h3>Navigation</h3>

<ul>

<a href="../genindex.html" title="General Index"

accesskey="I">index</a></li>

<a href="../py-modindex.html" title="Python Module Index"

>modules</a> |</li>

<a href="documenting_mpl.html" title="Writing documentation"

accesskey="N">next</a> |</li>

<a href="development_setup.html" title="Setting up Matplotlib for development"

accesskey="P">previous</a> |</li>

<li><a href="../contents.html">contents</a> »</li>

<li class="nav-item nav-item-1"><a href="index.html" accesskey="U">The Matplotlib Developers' Guide</a> »</li>

<li class="nav-item nav-item-this"><a href="">Testing</a></li>

</ul>

</div>

<h3><a href="../contents.html">Table of Contents</a></h3>

<ul>

<li><a class="reference internal" href="#">Testing</a><ul>

<li><a class="reference internal" href="#requirements">Requirements</a></li>

<li><a class="reference internal" href="#running-the-tests">Running the tests</a></li>

<li><a class="reference internal" href="#writing-a-simple-test">Writing a simple test</a></li>

<li><a class="reference internal" href="#random-data-in-tests">Random data in tests</a></li>

<li><a class="reference internal" href="#writing-an-image-comparison-test">Writing an image comparison test</a></li>

<li><a class="reference internal" href="#creating-a-new-module-in-matplotlib-tests">Creating a new module in matplotlib.tests</a></li>

<li><a class="reference internal" href="#using-github-actions-for-ci">Using GitHub Actions for CI</a></li>

<li><a class="reference internal" href="#using-tox">Using tox</a></li>

<li><a class="reference internal" href="#building-old-versions-of-matplotlib">Building old versions of Matplotlib</a></li>

<li><a class="reference internal" href="#testing-released-versions-of-matplotlib">Testing released versions of Matplotlib</a><ul>

<li><a class="reference internal" href="#install-additional-dependencies">Install additional dependencies</a></li>

<li><a class="reference internal" href="#obtain-the-reference-images">Obtain the reference images</a></li>

<li><a class="reference internal" href="#run-the-tests">Run the tests</a></li>

</ul>

</li>

</ul>

</li>

</ul>

<div id="sidebar-pagesource" role="note" aria-label="source link"

style="margin-top: 1.5em; padding-top: 0.1em; border-top: 1px solid #86989b">

<a href="../_sources/devel/testing.rst.txt"

style="color: #c0c0c0" rel="nofollow">Show Page Source</a>

</div>

<span id="id1"></span><h1>Testing<a class="headerlink" href="#testing" title="Permalink to this headline">¶</a></h1>

<p>Matplotlib uses the <a class="reference external" href="http://doc.pytest.org/en/latest/">pytest</a> framework.</p>

<p>The tests are in <code class="file docutils literal notranslate"><span class="pre">lib/matplotlib/tests</span></code>, and customizations to the pytest

testing infrastructure are in <a class="reference internal" href="../api/testing_api.html#module-matplotlib.testing" title="matplotlib.testing"><code class="xref py py-mod docutils literal notranslate"><span class="pre">matplotlib.testing</span></code></a>.</p>

<span id="testing-requirements"></span><h2>Requirements<a class="headerlink" href="#requirements" title="Permalink to this headline">¶</a></h2>

<p>To run the tests you will need to

<a class="reference internal" href="development_setup.html#installing-for-devs"><span class="std std-ref">set up Matplotlib for development</span></a>. Note in

particular the <a class="reference internal" href="development_setup.html#test-dependencies"><span class="std std-ref">additional dependencies</span></a> for testing.</p>

<p>We will assume that you want to run the tests in a development setup.</p>

<p class="last">While you can run the tests against a regular installed version of

Matplotlib, this is a far less common use case. You still need the

<a class="reference internal" href="development_setup.html#test-dependencies"><span class="std std-ref">additional dependencies</span></a> for testing.

You have to additionally get the reference images from the repository,

because they are not distributed with pre-built Matplotlib packages.</p>

</div>

<h2>Running the tests<a class="headerlink" href="#running-the-tests" title="Permalink to this headline">¶</a></h2>

<p>In the root directory of your development repository run:</p>

<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">pytest</span>

</pre></div>

</div>

<p>pytest can be configured via a lot of <a class="reference external" href="http://doc.pytest.org/en/latest/usage.html">command-line parameters</a>. Some

particularly useful ones are:</p>

</colgroup>

<tr class="row-odd"><td><code class="docutils literal notranslate"><span class="pre">-v</span></code> or <code class="docutils literal notranslate"><span class="pre">--verbose</span></code></td>

<td>Be more verbose</td>

</tr>

<td>Run tests in parallel over NUM

processes (requires <a class="reference external" href="https://pypi.org/project/pytest-xdist/">pytest-xdist</a>)</td>

</tr>

<tr class="row-odd"><td><code class="docutils literal notranslate"><span class="pre">--capture=no</span></code> or <code class="docutils literal notranslate"><span class="pre">-s</span></code></td>

<td>Do not capture stdout</td>

</tr>

</tbody>

</table>

<p>To run a single test from the command line, you can provide a file path,

optionally followed by the function separated by two colons, e.g., (tests do

not need to be installed, but Matplotlib should be):</p>

<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">pytest</span> <span class="n">lib</span><span class="o">/</span><span class="n">matplotlib</span><span class="o">/</span><span class="n">tests</span><span class="o">/</span><span class="n">test_simplification</span><span class="o">.</span><span class="n">py</span><span class="p">::</span><span class="n">test_clipping</span>

</pre></div>

</div>

<p>An alternative implementation that does not look at command line arguments

and works from within Python is to run the tests from the Matplotlib library

function <a class="reference internal" href="../api/testing_api.html#matplotlib.test" title="matplotlib.test"><code class="xref py py-func docutils literal notranslate"><span class="pre">matplotlib.test()</span></code></a>:</p>

<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">matplotlib</span>

<span class="n">matplotlib</span><span class="o">.</span><span class="n">test</span><span class="p">()</span>

</pre></div>

</div>

<h2>Writing a simple test<a class="headerlink" href="#writing-a-simple-test" title="Permalink to this headline">¶</a></h2>

<p>Many elements of Matplotlib can be tested using standard tests. For

example, here is a test from <code class="file docutils literal notranslate"><span class="pre">matplotlib/tests/test_basic.py</span></code>:</p>

<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">test_simple</span><span class="p">():</span>

<span class="sd"> very simple example test</span>

<span class="k">assert</span> <span class="mi">1</span> <span class="o">+</span> <span class="mi">1</span> <span class="o">==</span> <span class="mi">2</span>

</pre></div>

</div>

<p>Pytest determines which functions are tests by searching for files whose names

begin with <code class="docutils literal notranslate"><span class="pre">"test_"</span></code> and then within those files for functions beginning with

<code class="docutils literal notranslate"><span class="pre">"test"</span></code> or classes beginning with <code class="docutils literal notranslate"><span class="pre">"Test"</span></code>.</p>

<p>Some tests have internal side effects that need to be cleaned up after their

execution (such as created figures or modified <a class="reference internal" href="../api/matplotlib_configuration_api.html#matplotlib.rcParams" title="matplotlib.rcParams"><code class="xref py py-obj docutils literal notranslate"><span class="pre">rcParams</span></code></a>). The pytest fixture

<code class="docutils literal notranslate"><span class="pre">matplotlib.testing.conftest.mpl_test_settings</span></code> will automatically clean

these up; there is no need to do anything further.</p>

</div>

<h2>Random data in tests<a class="headerlink" href="#random-data-in-tests" title="Permalink to this headline">¶</a></h2>

<p>Random data is a very convenient way to generate data for examples,

however the randomness is problematic for testing (as the tests

must be deterministic!). To work around this set the seed in each test.

For numpy use:</p>

<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>

<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">seed</span><span class="p">(</span><span class="mi">19680801</span><span class="p">)</span>

</pre></div>

</div>

<p>The seed is John Hunter's birthday.</p>

</div>

<h2>Writing an image comparison test<a class="headerlink" href="#writing-an-image-comparison-test" title="Permalink to this headline">¶</a></h2>

<p>Writing an image-based test is only slightly more difficult than a simple

test. The main consideration is that you must specify the "baseline", or

expected, images in the <a class="reference internal" href="../api/testing_api.html#matplotlib.testing.decorators.image_comparison" title="matplotlib.testing.decorators.image_comparison"><code class="xref py py-obj docutils literal notranslate"><span class="pre">image_comparison</span></code></a>

decorator. For example, this test generates a single image and automatically

tests it:</p>

<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">matplotlib.testing.decorators</span> <span class="kn">import</span> <span class="n">image_comparison</span>

<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>

<span class="nd">@image_comparison</span><span class="p">(</span><span class="n">baseline_images</span><span class="o">=</span><span class="p">[</span><span class="s1">'line_dashes'</span><span class="p">],</span> <span class="n">remove_text</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>

<span class="n">extensions</span><span class="o">=</span><span class="p">[</span><span class="s1">'png'</span><span class="p">])</span>

<span class="k">def</span> <span class="nf">test_line_dashes</span><span class="p">():</span>

<span class="n">fig</span><span class="p">,</span> <span class="n">ax</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">subplots</span><span class="p">()</span>

<span class="n">ax</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">),</span> <span class="n">linestyle</span><span class="o">=</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">)),</span> <span class="n">lw</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>

</pre></div>

</div>

<p>The first time this test is run, there will be no baseline image to compare

against, so the test will fail. Copy the output images (in this case

<code class="file docutils literal notranslate"><span class="pre">result_images/test_lines/test_line_dashes.png</span></code>) to the correct

subdirectory of <code class="file docutils literal notranslate"><span class="pre">baseline_images</span></code> tree in the source directory (in this

case <code class="file docutils literal notranslate"><span class="pre">lib/matplotlib/tests/baseline_images/test_lines</span></code>). Put this new

file under source code revision control (with <code class="docutils literal notranslate"><span class="pre">git</span> <span class="pre">add</span></code>). When rerunning

the tests, they should now pass.</p>

<p>Baseline images take a lot of space in the Matplotlib repository.

An alternative approach for image comparison tests is to use the

<a class="reference internal" href="../api/testing_api.html#matplotlib.testing.decorators.check_figures_equal" title="matplotlib.testing.decorators.check_figures_equal"><code class="xref py py-obj docutils literal notranslate"><span class="pre">check_figures_equal</span></code></a> decorator, which should be

used to decorate a function taking two <a class="reference internal" href="../api/figure_api.html#matplotlib.figure.Figure" title="matplotlib.figure.Figure"><code class="xref py py-obj docutils literal notranslate"><span class="pre">Figure</span></code></a> parameters and draws the same

images on the figures using two different methods (the tested method and the

baseline method). The decorator will arrange for setting up the figures and

then collect the drawn results and compare them.</p>

<p>See the documentation of <a class="reference internal" href="../api/testing_api.html#matplotlib.testing.decorators.image_comparison" title="matplotlib.testing.decorators.image_comparison"><code class="xref py py-obj docutils literal notranslate"><span class="pre">image_comparison</span></code></a> and

about their use.</p>

</div>

<h2>Creating a new module in matplotlib.tests<a class="headerlink" href="#creating-a-new-module-in-matplotlib-tests" title="Permalink to this headline">¶</a></h2>

<p>We try to keep the tests categorized by the primary module they are

testing. For example, the tests related to the <code class="docutils literal notranslate"><span class="pre">mathtext.py</span></code> module

are in <code class="docutils literal notranslate"><span class="pre">test_mathtext.py</span></code>.</p>

</div>

<h2>Using GitHub Actions for CI<a class="headerlink" href="#using-github-actions-for-ci" title="Permalink to this headline">¶</a></h2>

<p><a class="reference external" href="https://docs.github.com/en/actions">GitHub Actions</a> is a hosted CI system

"in the cloud".</p>

<p>GitHub Actions is configured to receive notifications of new commits to GitHub

repos and to run builds or tests when it sees these new commits. It looks for a

YAML files in <code class="docutils literal notranslate"><span class="pre">.github/workflows</span></code> to see how to test the project.</p>

<p>GitHub Actions is already enabled for the <a class="reference external" href="https://github.com/matplotlib/matplotlib/">main Matplotlib GitHub repository</a> -- for example, see <a class="reference external" href="https://github.com/matplotlib/matplotlib/actions?query=workflow%3ATests">the Tests

workflows</a>.</p>

<p>GitHub Actions should be automatically enabled for your personal Matplotlib

fork once the YAML workflow files are in it. It generally isn't necessary to

look at these workflows, since any pull request submitted against the main

Matplotlib repository will be tested.</p>

<p>You can see the GitHub Actions results at

<a class="reference external" href="https://github.com/your_GitHub_user_name/matplotlib/actions">https://github.com/your_GitHub_user_name/matplotlib/actions</a> -- here's <a class="reference external" href="https://github.com/QuLogic/matplotlib/actions">an

example</a>.</p>

</div>

<h2>Using tox<a class="headerlink" href="#using-tox" title="Permalink to this headline">¶</a></h2>

<p><a class="reference external" href="https://tox.readthedocs.io/en/latest/">Tox</a> is a tool for running tests

against multiple Python environments, including multiple versions of Python

(e.g., 3.6, 3.7) and even different Python implementations altogether

(e.g., CPython, PyPy, Jython, etc.), as long as all these versions are

available on your system's $PATH (consider using your system package manager,

e.g. apt-get, yum, or Homebrew, to install them).</p>

<p>tox makes it easy to determine if your working copy introduced any

regressions before submitting a pull request. Here's how to use it:</p>

<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ pip install tox

$ tox

</pre></div>

</div>

<p>You can also run tox on a subset of environments:</p>

<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ tox -e py37,py38

</pre></div>

</div>

<p>Tox processes everything serially so it can take a long time to test

several environments. To speed it up, you might try using a new,

parallelized version of tox called <code class="docutils literal notranslate"><span class="pre">detox</span></code>. Give this a try:</p>

<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ pip install -U -i http://pypi.testrun.org detox

$ detox

</pre></div>

</div>

<p>Tox is configured using a file called <code class="docutils literal notranslate"><span class="pre">tox.ini</span></code>. You may need to

edit this file if you want to add new environments to test (e.g.,

<code class="docutils literal notranslate"><span class="pre">py33</span></code>) or if you want to tweak the dependencies or the way the

tests are run. For more info on the <code class="docutils literal notranslate"><span class="pre">tox.ini</span></code> file, see the <a class="reference external" href="https://tox.readthedocs.io/en/latest/config.html">Tox

Configuration Specification</a>.</p>

</div>

<h2>Building old versions of Matplotlib<a class="headerlink" href="#building-old-versions-of-matplotlib" title="Permalink to this headline">¶</a></h2>

<p>When running a <code class="docutils literal notranslate"><span class="pre">git</span> <span class="pre">bisect</span></code> to see which commit introduced a certain bug,

you may (rarely) need to build very old versions of Matplotlib. The following

constraints need to be taken into account:</p>

<li>Matplotlib 1.3 (or earlier) requires numpy 1.8 (or earlier).</li>

</ul>

</div>

<h2>Testing released versions of Matplotlib<a class="headerlink" href="#testing-released-versions-of-matplotlib" title="Permalink to this headline">¶</a></h2>

<p>Running the tests on an installation of a released version (e.g. PyPI package

or conda package) also requires additional setup.</p>

<p class="last">For an end-user, there is usually no need to run the tests on released

versions of Matplotlib. Official releases are tested before publishing.</p>

</div>

<h3>Install additional dependencies<a class="headerlink" href="#install-additional-dependencies" title="Permalink to this headline">¶</a></h3>

<p>Install the <a class="reference internal" href="development_setup.html#test-dependencies"><span class="std std-ref">additional dependencies for testing</span></a>.</p>

</div>

<h3>Obtain the reference images<a class="headerlink" href="#obtain-the-reference-images" title="Permalink to this headline">¶</a></h3>

<p>Many tests compare the plot result against reference images. The reference

images are not part of the regular packaged versions (pip wheels or conda

packages). If you want to run tests with reference images, you need to obtain

the reference images matching the version of Matplotlib you want to test.</p>

<p>To do so, either download the matching source distribution

<code class="docutils literal notranslate"><span class="pre">matplotlib-X.Y.Z.tar.gz</span></code> from <a class="reference external" href="https://pypi.org/project/matplotlib/">PyPI</a>

or alternatively, clone the git repository and <code class="docutils literal notranslate"><span class="pre">git</span> <span class="pre">checkout</span> <span class="pre">vX.Y.Z</span></code>. Copy

the folder <code class="file docutils literal notranslate"><span class="pre">lib/matplotlib/tests/baseline_images</span></code> to the folder

<code class="file docutils literal notranslate"><span class="pre">matplotlib/tests</span></code> of your the matplotlib installation to test.

The correct target folder can be found using:</p>

<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">python</span> <span class="o">-</span><span class="n">c</span> <span class="s2">"import matplotlib.tests; print(matplotlib.tests.__file__.rsplit('/', 1)[0])"</span>

</pre></div>

</div>

<p>An analogous copying of <code class="file docutils literal notranslate"><span class="pre">lib/mpl_toolkits/tests/baseline_images</span></code>

is necessary for testing the <a class="reference internal" href="../api/toolkits/index.html#toolkits"><span class="std std-ref">Toolkits</span></a>.</p>

</div>

<h3>Run the tests<a class="headerlink" href="#run-the-tests" title="Permalink to this headline">¶</a></h3>

<p>To run the all the tests on your installed version of Matplotlib:</p>

</pre></div>

</div>

<p>The test discovery scope can be narrowed to single test modules or even single

functions:</p>

<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">pytest</span> <span class="o">--</span><span class="n">pyargs</span> <span class="n">matplotlib</span><span class="o">.</span><span class="n">tests</span><span class="o">.</span><span class="n">test_simplification</span><span class="o">.</span><span class="n">py</span><span class="p">::</span><span class="n">test_clipping</span>

</pre></div>

</div>

</div>

<br />

Last updated on Mar 26, 2021.

Created using

<a href="http://sphinx-doc.org/">Sphinx</a> 3.2.1.

Doc version v3.4.0-2-gcb03754703.

</div>

</footer>

</body>

</html>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FilesExpand file tree

testing.html

Latest commit

History

testing.html

File metadata and controls