Authoring Exercises

This guide explains how to write exercises and unit tests for notebook-ta.

Exercise Structure

Each exercise is defined in exercises.toml under [exercises.<id>]:

[exercises.ex1]
statement = "Write a function `add(a, b)` that returns the sum of two numbers."
additional_info = "No imports are needed."

The id (here ex1) is the stable identifier used in the %%notebook_ta cell magic line. The optional name is an editable display name used by the benchmarking interface. When omitted, it defaults to the subsection ID (ex1 here). Changing it does not break saved solutions or historical benchmark records.

statement is optional — see Embedding statements in the notebook for an alternative that avoids duplicating the exercise description.

Free-text exercises

Use answer_type = "free_text" when the cell body is an explanation, proof, reflection, or other prose answer rather than Python:

[exercises.explain_recursion]
answer_type = "free_text"
statement = "Explain recursion and the purpose of a base case."
evaluation_criteria = "A satisfactory answer explains self-invocation and termination."
prompt_on_free_text = "Evaluate the answer and identify one concrete improvement."

The student still submits through the same cell magic:

%%notebook_ta explain_recursion
Recursion occurs when a function calls itself. A base case stops the calls...

Notebook-ta treats that cell body as opaque text: it does not execute it, modify the notebook namespace, or run unit tests. A non-empty answer within max_student_answer_length is sent directly to the LLM for formative evaluation. No “Tests passed” indicator or hint button is displayed.

A free-text exercise must configure prompt_on_free_text or use global prompts.on_free_text. evaluation_criteria is optional but strongly recommended for consistent feedback. Tests and test-specific per-exercise limits are invalid for this answer type. LLM evaluation is non-deterministic and should not be treated as authoritative grading without human review and an appropriate audit process.

Writing Unit Tests

Tests are declared as TOML array tables: [[exercises.<id>.tests]]. The optional name defaults to Unit test 1, Unit test 2, and so on, according to the test’s position within the exercise. Numbering includes tests that have an explicit name.

Exercise tests are trusted Python. Namespace separation and test timeouts are not a security sandbox; see the trust and security model before running third-party material.

Inline Tests

Use the code field to write a Python function directly in the TOML file:

[[exercises.ex1.tests]]
name = "add(2, 3) == 5"
code = """
def test_add(add):
    result = add(2, 3)
    return result == 5, f"Expected 5, got {result}"
"""

The runner inspects the test function’s signature and resolves every parameter name in the student’s IPython namespace. In this example, the add parameter receives the object stored under the name add. Multiple parameters are resolved independently in the same way.

Resolution happens before the test is called. If any parameter name is absent from the student’s namespace, the test is not executed and is reported as failed with a message identifying the missing name. The reserved student_globals parameter follows the explicit export rules in Passing student symbols to tests instead.

It must return either:

A bool (True = pass, False = fail)
A tuple[bool, str] where the string is a human-readable message

Any text printed to stdout is also captured and included in the message. Common ANSI SGR color and text-style sequences (including standard and bright colors, bold, and underline) are rendered in the notebook output, so existing terminal-style custom test reports remain readable.

Tests from the Current Global Scope

If a test function is already defined in the notebook’s current global scope, specify only its name:

[[exercises.ex1.tests]]
name = "Custom add test"
function = "test_add"

The named object must exist and be callable when the exercise runs. Otherwise, the test is reported as failed with a message identifying the missing function. Its parameters are resolved from the same global scope in the usual way.

External Tests

For complex tests, reference a function in an importable Python module:

[[exercises.ex3.tests]]
name = "Performance test"
module = "course_tests.ex3"
function = "test_performance"

For example, course_tests/ex3.py could contain:

def test_performance(build_index):
    index = build_index(["alpha", "beta"])
    return len(index) == 2, "Expected an index containing both values."

The module must be importable from the notebook’s working directory or PYTHONPATH. After loading the configured function, the runner resolves its parameters exactly as it does for an inline test: each parameter name is looked up in the student’s IPython namespace. If any name is missing, the function is not called and the test is reported as failed with the missing name. The reserved student_globals parameter uses the shared rules below.

Passing student symbols to tests

The following rules apply to inline, global-scope, and external tests. Put student_symbols or export_student_globals in the test’s TOML table alongside code or function.

Named parameters

Named parameters are the preferred way to pass student definitions to a test. Only the requested objects are serialized for the isolated test process. For example, a function declared as def test_result(parse, render): ... receives the student’s parse and render objects. The test fails before invocation if either name is not defined.

Exporting multiple symbols as a dictionary

When a test needs a dictionary of several student definitions, declare student_symbols. The runner exports only those names and passes the resulting dictionary as student_globals:

[[exercises.ex2.tests]]
name = "Both functions are defined"
student_symbols = ["add", "mul"]
code = """
def test_both_defined(student_globals):
    has_add = "add" in student_globals and callable(student_globals["add"])
    has_mul = "mul" in student_globals and callable(student_globals["mul"])
    return has_add and has_mul, "Expected both add() and mul() to be defined."
"""

A missing selected symbol is reported as a failed test. A test that declares a student_globals parameter without either export option also fails with a configuration message.

The full IPython namespace can be exported explicitly with export_student_globals = true, but this is strongly discouraged. Notebook namespaces commonly contain large objects, open resources, module state, or objects that cloudpickle cannot serialize. Exporting all of them can therefore make tests slow or cause process preparation to fail even when the objects relevant to the test are valid. Use named parameters or student_symbols whenever possible.

Unit Test Timeouts

Each unit test is cancelled if it runs longer than the global unit_test_timeout from global_config.toml. The default is 5 seconds. A timeout is reported as a failed test in the notebook and in benchmark runs, and the timeout message is included in the LLM prompt.

Override the timeout for one exercise with unit_test_timeout:

[exercises.ex_slow]
statement = "Implement a function that handles a large input."
unit_test_timeout = 15.0

Prompt Size Limits

Two global settings keep student submissions and test diagnostics from producing unbounded LLM prompts:

max_student_answer_length = 10000
max_unit_test_output_length = 4000

Both limits count Unicode characters. Student answers longer than max_student_answer_length are still executed and tested, but notebook-ta emits an error and does not contact the LLM. max_unit_test_output_length is a shared budget across all unit test result messages in configured test order. Output beyond that budget is removed and a warning is displayed; test names and pass/fail statuses are retained.

Override either limit for one exercise:

[exercises.ex_verbose]
statement = "Implement the requested reporting function."
max_student_answer_length = 20000
max_unit_test_output_length = 8000

Exercise-Level Prompt Overrides

You can override the global prompts for a specific exercise. Overrides may reuse fragments from the global configuration’s [prompts.fragments] table:

[exercises.ex_hard]
statement = "Implement Dijkstra's algorithm."
prompt_on_success = """{{ prompt_base }}

Excellent! Analyse the time and space complexity of this graph algorithm.
"""
prompt_on_failure = """{{ prompt_base }}

Graph algorithms can be tricky. Think about the data structures you need. When the student asks
for hints, escalate guidance gradually.
"""

An unknown or malformed fragment reference in an exercise override raises ConfigurationError while notebook-ta loads, before the exercise can be used.

Embedding Statements in the Notebook

To avoid duplicating the exercise description in both the notebook and exercises.toml, you can omit statement from the TOML and embed it directly in a notebook markdown cell using a <div id="<exercise-id>"> block:

<div id="ex1">

## Exercise 1 — Add two numbers

Write a function `add(a, b)` that returns the sum of `a` and `b`.

Example:
```python
add(2, 3)  # → 5
```

</div>

The div renders normally in Jupyter and its inner content is used as the statement sent to the LLM. Multiple markdown cells with the same id are concatenated in document order.

To use this feature, pass notebook_path= to notebook_ta.load():

import notebook_ta
notebook_ta.load(
    "global_config.toml",
    "exercises.toml",
    notebook_path="my_notebook.ipynb",  # path to the current notebook
)

If notebook_path is not provided, notebook-ta will try to detect the notebook path automatically. Pass it explicitly when auto-detection fails or for reproducibility.

Error handling — if statement is absent from both the TOML and the notebook, load() raises a ConfigurationError identifying the affected exercise.

Complete Example

[exercises.reverse]
statement = """
Write a function `reverse_list(lst)` that returns a new list
with the elements in reverse order.
"""
additional_info = "Do not use lst[::-1] or lst.reverse()."

[[exercises.reverse.tests]]
name = "reverse_list([1,2,3]) == [3,2,1]"
code = """
def test_basic(reverse_list):
    return reverse_list([1,2,3]) == [3,2,1], "Basic reversal failed"
"""

[[exercises.reverse.tests]]
name = "Empty list returns empty list"
code = """
def test_empty(reverse_list):
    return reverse_list([]) == [], "Empty list should return []"
"""

[[exercises.reverse.tests]]
name = "Original list not modified"
code = """
def test_no_mutation(reverse_list):
    lst = [1, 2, 3]
    copy = list(lst)
    reverse_list(lst)
    return lst == copy, "The original list was mutated!"
"""

Tips

Keep test names short and descriptive — they appear in the notebook output.
Return a meaningful message from your tests to help students understand failures.
Use print() inside test functions to emit additional diagnostic information.
Prefer named test parameters; use student_symbols only when the test needs a dictionary.
External module tests are useful for complex or reusable test logic shared across exercises.