# Creating a project

In this tutorial, you will learn how to create a project package that can be used for Autograde.

# Demo templates

Below you will find a repository with multiple project examples. We will focus on a python version, but the repository below also includes other language examples that can be used as reference.

The repository we will use is located HERE (opens new window).

The most up to date tester.py can be found HERE.

# Basic project package structure

.
├── starter
│   ├── Makefile
│   ├── p0.py
│   └── tester.py
└── test
    ├── public-add
    │   ├── answer
    │   ├── driver.py
    │   └── test.py
    ├── public-insecure
    │   ├── driver.py
    │   └── test.py
    └── public-union
        ├── answer
        ├── driver.py
        └── test.py

After cloning the repository, you'll find the python folder with all the files we will be working with. A project package will always contain a starter and test folder. Any other files are optional and ignored on upload.

# starter/

A starter folder contains any files that are sent to someone who initializes the project through Autograde. This is considered the "default state" and is what you want your users to have when they receive the project.

Any files you want the user to have can go in here, but make sure a tester.py is always present. The tester.py is a tool for running and testing provided tests in the test folder. Info on its usage can be found by running the tester.py with no params.

# test/

The test folder contains all tests associated with the project, where each sub folder is a test. More will be explained about this folder later, but for now it is important to know the following:

The folder name defines the properties of the test (for example: public-test1 defines a test as public).
Each test MUST contain a driver.py. This is the file that is executed first.
Each test MUST contain a test file that the driver.py runs (for example: test.py).

# Other files

In the example project, notice that the root folder contains a p0.py and a tester.py that is not within the starter subfolder. This essentially allows you, as an instructor, to test your solution and tests in a way that is convenient but does not disclose the solution to students via the starter.

# Test Properties

A test has many meta properties that can be defined by the folder name. These include the name, type, weight, and weight distribution type.

# Test Type

A test can be one of 4 categories that gives it unique properties when served through Autograde. In our example, you can see that one test is called public-add. This would create a public test called public-add. Changing this to hidden-add would make it a hidden test. Below are definitions of each test type:

# PUBLIC

Tests that are given out with the starter code. Every part of the test will be known to the user.

# SERVER

Only the name and debug output will be shown through Autograde. You can customize debug output through the driver.py.

# HIDDEN

Only the name will be shown. Students will not be able to view debug output even if the driver.py allows it.

# SECRET

Nothing about the test will be known (besides if it passed or not), and the test name will be obfuscated from the user. For example, secret-powerset0 becomes secret0.

# Additional properties

If you want to make uploading a project as seamless as possible, you can also customize every other part of a test as well. The basic format for this is the following:

<TEST TYPE>-w_<WEIGHT>-l_<LEVELS>-testname where

TEST TYPE = public, server, hidden, or secret.
WEIGHT = the weight of the test (any positive integer).
LEVELS = a list of weight types, per level, separated by an underscore. Can be r, b, or o (r = required, b = bonus, o = optional).

and each part is optional.

# Examples

public-w_15-add
- Name: public-add
- Type: public
- Weight: 15
- Weight Type: required (default)
hidden-w_5-l_b_r-add
- Name: hidden-add
- Type: hidden
- Weight: 10 (default)
- Weight Type
  - First Level: Bonus
  - Second Level: Required

# Test Structure

There are many ways to write a test, but the basis will always be the same. To understand the structure of a test, we can take first a look at public-add (opens new window).

# driver.py

This file is the entrypoint of a test and controls what exactly will happen. Starting at the top, we will see:

#!/usr/bin/python3
#####################################################
#############  LEAVE CODE BELOW ALONE #############
# Include base directory into path
import os, sys sys.path.append(os.path.abspath(os.path.join(os.path.dirname( __file__ ), '..', '..')))
# Import tester
from tester import failtest, passtest, assertequals, runcmd, preparefile, runcmdsafe
#############    END UNTOUCHABLE CODE #############
#####################################################

We do this in order to import the tester.py from the root directory of the project, and allow any project files to be included.

Next we will find preparefile() being used:

preparefile('./test.py')

On the server, this will perform chown and chmod routines in order to ensure a file can be accessed by the user's code. If a file is not prepared, the user in question will not be able to touch it. This is because driver.py is run by root, but the actual test that driver.py executes will be run by a less privileged user called autograde in a docker container.

WARNING

Note how the answer file is note prepared. This is because we do NOT want the student to be able to modify it.

Next we will see where driver.py runs the actual test code:

python_bin = sys.executable
b_stdout, b_stderr, b_exitcode = runcmdsafe(f'{python_bin} ./test.py')

runcmdsafe() wraps your command in runuser -l autograde in order to run code as the autograde user. Again, this is to ensure that student code has no way of snooping into root files to cheat the system. Since python is a scripting language, we can just create a test.py and run it as is. If you want to learn about test.py before moving on, see the section below.

Next we will get into the part of the driver.py that is accessed after the test is run.

# Convert stdout bytes to utf-8
stdout = "" 
stderr = "" 
try: 
	stdout = b_stdout.decode('utf-8') 
	stderr = b_stderr.decode('utf-8') 
except:
	pass

The output from runcmdsafe() returns raw byte streams from python's Popen, and must be decoded into utf-8 before returning. If you don't care about giving out raw stdout you don't need this, but it's useful if you want to return errors or printed statements from the test file if something goes wrong.

The last part of the file is the part where the driver.py determines if the test has passed or not.

# Comparison with answer and output here
try:
	with open('answer', 'r') as file1, open('output', 'r') as file2:
		answer = str(file1.read())
		output = str(file2.read())
		file1.close()
		file2.close()

		# Delete output 
		os.remove('output')

		# Built in tester.py function assertequals(expected, actual, info='')
		# If True, passes. If false, fails and gives expected != actual and and specified info.
		# parameters: 
		# - expected(required): the answer that was expected
		# - actual(required): the output from the user's code
		# - info(optional): any additional info you want printed if it fails
		assertequals(answer, output, stdout+"\n"+stderr)

except FileNotFoundError:
	failtest(stdout+"\n"+stderr)

In the case of public-add, the test.py creates an output file that is supposed to match the given answer file. If it does, the test passes. Otherwise, the test will fail.

The basic flow of this segment of code is:

Open both files, read them, and close.
Clean up any created files.
Check if answer and output are equal with assertequals().

WARNING

Cleaning up created files is a step that is needed clientside. It ensures that your test directories are cleaned of added files so that you don't accidentally upload them with your project package. For example, if you accidentally uploaded the output file, the docker container would fail to access it because it was never prepared.

assertequals() is a tester.py utility that checks equality between two values. If you don't want to use it, you could forgo it and just use passtest('') and failtest('') yourself. Anything that is passed into these functions will be considered debug output that is shown via public and server tests (or from the local tester.py with flag -v). Note that the File Exception at the bottom of this code passes in all of stdout that is printed from test.py.

# Test file (test.py)

The test file is the file that the driver.py executes. For this example we named it test.py for clarity, but it can be named whatever you want. For example, a java project could have test.java instead.

At the top of the file you'll notice a similar snippet of code that was also in the driver.py:

import os, sys
sys.path.append(os.path.abspath(os.path.join(os.path.dirname( __file__ ), '..', '..')))

Once again, this is used so that we can import files from an upper directory. For python projects, this will be in every single one of your tests.

Next is the actual test code:

from p0 import add

addresult = str(add(1, 1))

This is where you write what your test is actually testing. As you can see here, we imported the add() function and saved a result. After we get our result, we dump it to an output file:

ofile = open('output', 'w')
ofile.write(addresult)
ofile.close()

That output file then goes on to be compared to the answer file inside the driver.py. It is important you create your answer file ahead of time. You may notice that there is commented code here that does just that.

WARNING

An answer file is not neccessary, but having one allows you to streamline your driver.py and make almost zero changes to it between each test. The only part you would need to consider is what files to use preparefile() on.

# Using Pickle

What if you have an answer that is a little more complicated than 2? If you take a look at public-union (opens new window), you can find another more robust test example. The setup is almost the same, except that a serialized value is dumped into the answer and outputfile so that what is read into the driver.py is the unconverted value that came out of test.py. We never had to convert our value to a string. Note that some alterations were made to how these files were read in the driver.py.

# Insecure Tests

It is possible to write insecure tests. If you look in public-insecure (opens new window), you will find a test that checks for the right answer directly inside the test.py. This is a red flag in that now the underprivileged docker account has the ability to say what the answer is. So technically, the user could exit early and fabricate the answer! As long as you don't do this (and prepare the correct files), your tests will be secure.

# Finishing up

Once you've written all your tests and set up your starter folder, you're ready to submit to Autograde! Autograde takes .tar and .tar.gz submissions, where you tar from the top level directory. In the case of this example, we would tar from outside the python folder.

tar zcvf p0.tar.gz python

← Available Languages