Notebook spec for authoring tests

.gitignore

@@ -99,3 +99,9 @@ ENV/
 
 # mypy
 .mypy_cache/
+
+# VS Code
+.vscode/
+
+.vs_code/
+.vscode

README.md

@@ -3,34 +3,40 @@
 [![CircleCI](https://circleci.com/gh/data-8/Gofer-Grader/tree/master.svg?style=shield)](https://circleci.com/gh/data-8/Gofer-Grader/tree/master)
 [![codecov](https://codecov.io/gh/data-8/Gofer-Grader/branch/master/graph/badge.svg)](https://codecov.io/gh/data-8/Gofer-Grader)
 
-Simple library for interactive autograding.
+A library for Jupyter notebook autograding.
 
-Previous names include `gradememaybe` and `okgrade`.
+(Previous project names include `gradememaybe` and `okgrade`.)
 
 [See the Gofer Grader documentation](http://okgrade.readthedocs.io/en/latest/) for more information.
 
 ## What?
 
-This library can be used to **autograde** Jupyter Notebooks and
-Python files.
+This library can be used to **autograde** Jupyter Notebooks.
 
-Instructors can write tests in a subset of the [okpy test format](docs/ok-test-format.md)
-(other formats coming soon), and students can dynamically check if their
-code is correct or not. These notebooks / .py files can later
-be collected and a grade assigned to them automatically.
+Instructors can embed tests into an assignment notebook in one of two formats:
 
-### Integrating Gofer into your course
+1. A subset of the [okpy test format](docs/ok-test-format.md)
+2. The [gofer format](docs/gofer-format.md)
 
-As an effort to help autograding with Berkeley's offering of Data 8 online, this
-repo also contains two components that could be useful to others for their own
-courses. The primary one is a tornado service that receives notebook submissions
-and runs them in docker containers. The second piece is a Jupyter notebook
-extension that submits the current notebook to the service. Though they could be
-modified to work on your own setup, these are meant to play particularly nicely
-with [Jupyterhub]().
+(other formats might follow). Students can dynamically run tests as they work.
+Notebooks can later be collected, and the Gofer tools will assign a score
+automatically based on the tests distributed to students and any hidden tests.
+Gofer also generates PDFs for manual scoring.
 
-Additional documentation in how to get them working is in the respective directories `gofer_service` and `submit_extension`.
+### Gofer components
 
+1. A Tornado service that receives notebook submissions and runs them in docker
+   containers.
+
+2. A Jupyter notebook extension that submits the current notebook to the service.
+
+Though they could be modified to work on another setup, these tools are designed
+to play particularly nicely with [Jupyterhub]().
+
+Additional documentation in how to get them working is in the respective
+directories `gofer_service` and `submit_extension`.
+
+Tools for authoring Jupyter notebooks with tests appear in `author_tools`.
 
 ## Why?
 

docs/gofer-format.md

@@ -0,0 +1,39 @@
+# Creating Notebooks with Embedded Tests
+
+The following specification for embedding tests into a Jupyter notebook has the
+following characteristics:
+
+1. Each test is a cell with the expressions or statements to be evaluated. The
+   expected output of the test is the output of the cell.
+2. Tests are grouped into questions, and each question has associated metadata
+   such as the question name and number of points.
+3. Metadata about questions and assignments are expressed within notebook cells
+   in YAML.
+
+This format is designed for easy assignment authoring. Typically, a notebook in this format will be converted to the [OK format](ok-test-format.md) before it is distributed to students.
+
+## Questions
+
+An example question within a notebook:
+
+<img src="example-question.png" />
+
+A question is a description *Markdown* cell, followed by a response cell, followed by zero or more test *Code* cells. The description cell must contain a code block (enclosed in triple backticks) that begins with a `BEGIN QUESTION` header line, followed by YAML that defines metadata associated with the question.
+
+The rest of the code block within the description cell must be YAML-formatted with the following fields (in any order):
+
+* name (required) - a string identifier that is a legal file name.
+* manual (optional) - a boolean (default False); whether to include the response cell in a PDF for manual grading.
+* points (optional) - a number (default 1); how many points the question is worth.
+
+The response cell must always appear directly below the description cell (where the question metadata is defined) and contain a correct response, along with annotations describing how the cell should be presented to students. **TODO(denero) specify solution stripping.**
+
+The test cells are any code cells following the response cell that begin with a comment containing either the capitalized word `TEST` or `HIDDEN TEST`. A `TEST` is distributed to students so that they can validate their work. A `HIDDEN TEST` is not distributed to students, but is used for scoring their work.
+
+## Assignment Metadata
+
+Metadata can be provided about the assignment as a whole in a code block (enclosed in triple backticks) that begins with a `BEGIN ASSIGNMENT` header line, followed by YAML that defines metadata. **TODO specify legal assignment metadata (e.g., what course this assignment is for)**
+
+## More info
+
+**TODO pointers to docs on how to distribute assignments to students, configure autograders, etc.**