Skip to content

Python Testing

The Python test type is used to test Python workspaces. It can be used to test vanilla Python or packages such as matplotlib.

Test Structure

Python test runners expect an explicit pass/fail function to be called: pass_tests() or fail_tests(message).

load_file_in_context(path) is used to load a learner file. This function also executes the learner's code and provides the test script context with all learner-defined top-level variables and functions. The test file is run in the learner's root workspace directory, so the path string does not need a relative filepath prefix.

The Python test runner provides an API with some helper functions that internally call the pass_tests() or fail_tests() functions and provide helpful error messages. Test authors can use these built-in helpers or write their own tests and manually call pass_tests() and fail_tests().

This document provides API documentation below, but the test API code itself can be viewed on GitHub.

Example Tests

Function Definition

Test to see if a function is defined in the learner's script.py file:

load_file_in_context('script.py')

function_defined('add_numbers')

Function Behavior

Test to see if a learner's function behaves as expected:

load_file_in_context('script.py')
function_defined('double_it')

doubled_four = double_it(4)
if double_four != 8:
  fail_tests(f"`double_it(4)` was {double_four} (expected 8)")

pass_tests()

In this test, the file is loaded, a check is run to see if the function is defined, and then called with an example input to see if it behaves as expected.

Jupyter Notebooks: testbook

Jupyter-enabled workspace types include the testbook package for accessing the contents of a notebook:

from testbook import testbook

with testbook('notebook.ipynb', execute=True) as tb:
  try:
    actual = tb.ref('company')
    expected = 'Codecademy'
    if actual != expected:
      fail_tests(f'Set the `company` variable to: {expected}.')
  except Exception as e:
      fail_tests(str(e))

  try:
    actual = tb.get('blabber')
    expected = 'blahblah'
    if actual() != expected:
      fail_tests(f'Expected `blabber()` to return: `{expected}`. Instead, it return `{actual}`')
  except Exception as e:
      fail_tests(str(e))

pass_tests()

In this test, the notebooks is loaded with a context manager, and the variables inside are available for assertions.

Jupyter Notebooks: nbformat

Jupyter-enabled workspace types also include the nbformat package for accessing the contents of a notebook:

import nbformat
import re

my_notebook = nbformat.read('./notebook.ipynb', as_version=4)
regexp = r'plt\.close\(.*\)'

for cell in my_notebook.cells:
  if re.search(regexp, cell.source) is None:
    continue
  else:
    pass_tests()
    break

fail_tests('Make sure to call `plt.close()` in one of your cells.')

In this test, the notebook is loaded as an object according to the nbformat format. This object can be traversed to access source code, execution count, and/or output.

Other Guidelines

Python Testing API

Quick Look

Refer to the Arguments section below for more information on how to use these fuctions.

full_equality

full_equality(name, expected_value, [error_text])

Description: Evaluates name and checks if it is equal to the expected value.

setwise_equality

setwise_equality(name, expected_value, [error_text])

Description: Evaluates name and checks if it and the expected value are setwise equal.

approximate_equality

approximate_equality(name, expected_value, threshold, [error_text])

Description: Evaluates name, checks if the difference between it and expected value are below threshold.

check_type

check_type(name, expected_type)

Description: Evaluates name and checks if its type is the same as the expected type.

module_imported

module_imported(module_name)

Description: Checks if module_name is on the sys.path.

function_defined

function_defined(name)

Description: Evaluates name and checks that it is type function (syntactic sugar for check_type() with expected_type of function).

Arguments

name

The first argument to any one of these functions is a string corresponding to the name of the token being tested. It may very well be that the trickiest part of using these helper functions is remembering to not call the function simply with the name of the variable. The reason for this is that all of the tests (with the technical exception of module_imported()) checks first to see if the variable is defined at all and will immediately fail the test with the error message "Did you remember to define {name}?" if there is no variable that corresponds with the given string.

expected_value

The second argument of the helper functions full_equality(), setwise_equality(), approximate_equality() is the expected value of the variable being referred to. After evaluating the name these functions will perform the following comparisons:

  • For full_equality(), this expected value will be compared using the == operator.
  • For setwise_equality(), both the expected value and the given variable will be cast to sets before performing an == comparison.
  • For approximate_equality(), which is intended to avoid floating-point or similar rounding concerns, a threshold is given as the third argument and the comparison made is given_value - expected_value < threshold.

expected_type

The second argument of check_type() is a Python type. There is a proper Python way to get primitive types, but a simple way to do this is to create an object of the type you want to check and calling the Python built-in type() on it. For example, instead of trying to find the integer type, you can use type(0) which will return <type 'int'>.

function_defined() is a specific case of check_type(), where the type is <type 'function'>.

module_name

module_imported() checks if the given string has been imported into the context.

error_text

A key to a Python dictionary of standardized error messages, defaulting to "simple" in most cases.

  • "simple"
"Value for `{name}` did not match the expected value."
  • "explicit"
"Value for `{name}` did not match `{expected_value}`."
  • "compare"
"Value for `{name}` did not match `{expected_value}`, (was `{value}`)."
  • "recall"
"Did you remember to set `{name}` to `{expected_value}`?"