61A Autograder

This is the communication layer between Okpy and the 61A Autograder Worker, which grades students assignments and sends score back to Okpy. The worker code is kept private for security reasons, but this only affects you if you choose to use our workers and don’t implement your own. If you wish to use ours, ask a 61A Infrastructure TA for guidance.

Setup

1. Clone the repo and switch into the ag_master directory.

2. Run sicp venv to create a virtual environment (python3 -m venv env) and install requirements (env/bin/pip install -r requirements.txt).

3. Run env/bin/python main.py to start the Autograder server. You may need to be logged into a gcloud account with access to the cs61a project in order to access cloud storage buckets, as local development is currently somewhat unsupported.

Recreating the Autograder

You can create your own version of this autograder to autograde assignments submitted through Okpy. Should you choose to do so, you’ll need to set up a server with some endpoints that Okpy can query. See API Model for more details.

Data Models

There are 2 data models used by this app: Assignment and Job.

docs.ag_master.models.create_models(app: flask.Flask)

Creates database models

Parameters

app (Flask) – the app to attach the database URL to

class docs.ag_master.models.Assignment(*args: Any, **kwargs: Any)

Represents an Assignment that can be autograded

Parameters

• assignment_secret (str) – a randomly-generated unique key to be used by Okpy to identify this assignment

• name (str) – the shortname of the assignment, conventionally the same as the Okpy shortname

• course (str) – the course code for the course whose assignment this is

• endpoint (str) – the Okpy endpoint for the relevant course (saved here as endpoints change across semesters)

• file (str) – the name of the grading zip file to use to grade the assignment

• command – the command to run on the worker to grade a backup

• last_modified (int) – the unix timestamp for the last modification of this assignment on the autograder

• batch_size (int) – the grading batch size for this assignment (one worker will grade this many items – must be set such that less than 50 total batches are created by an individual run)

• grading_base (str) – the server associated with this assignment (usually https://okpy.org)

• jobs (list[Job]) – all grading jobs associated with this assignment

class docs.ag_master.models.Job(*args: Any, **kwargs: Any)

Represents an autograding Job

Parameters

• job_secret (str) – the internal job identifier, used for communication between the host and the worker

• external_job_id (str) – the external job identifier, used for communication between the grading server and the host

• assignment_secret (str) – a foreign key to the assignment identifier

• assignment (Assignment) – the assignment that this job is associated with

• backup (str) – the grading server backup this job is associated with

• status (str) – whether this is queued, started, running, finished, etc.

• result (str) – the grading command’s output for this job

• access_token (str) – the grading server’s access token to get backup content and upload scores for this job

• queued_at (int) – the unix timestamp for when this job was queued – if multiple jobs are queued by the same request, they are automatically treated as a single trigger by the UI

• started_at (int) – the unix timestamp for when the worker began running this job

• finished_at (int) – the unix timestamp for when the worker returned the result for this job, whether it succeeded or errored

Utility Decorators

The autograder has two decorators that restrict access to various functions and endpoints.

docs.ag_master.utils.admin_only(func)

Require a user to either be logged into the UI, or so pass in an access token. Either way, the user must be an admin for the course they’re attempting to access.

Returns

a function that takes in an access_token and a course, along with the parameters passed into the original func, which is called with all of the available arguments except access_token

docs.ag_master.utils.super_admin_only(func)

Does almost the same thing as admin_only(), except the course must be cs61a.

Returns

a function that takes in an access_token and a course, along with the parameters passed into the original func, which is called without access_token or course

Course Admin Endpoints

docs.ag_master.admin.create_admin_endpoints(app)

Creates various RPC endpoints to manage assignment grading. See the following:

• upload_zip()

• create_assignment()

There is also a GUI available at autograder.cs61a.org for the following actions:

• Viewing all of a course’s assignments (/<course>)

• Viewing a particular assignment (/<course>/<assignment>)

• Viewing a particular job (/<course>/job/<id>)

• Forcing unfinished jobs for an assignment to fail (/<course>/<assignment>/fail_unfinished)

• Retriggering failed jobs for an assignment (/<course>/<assignment>/retrigger_unsuccessful)

Superadmin Endpoints

The following superadmin endpoint is available for viewing information about a course: /admin/<endpoint>.

Worker Communication

There are a few endpoints created solely to communicate grading jobs with the worker instances.

docs.ag_master.worker.job_transition(*, at: Union[str, List[str]], to: str)

Decorates a function by making sure that the relevant job transitions to a logically forward state when the function is called. This prevents exploiting job statuses, as it prevents the function from running if the job is in an invalid state.

Parameters

• at (str, list[str]) – the statuses the job must be in in order for the decorated function to run

• to (str) – the status the job will be set to once the decorated function runs

docs.ag_master.worker.create_worker_endpoints(app)

Creates various RPC endpoints to interface with the worker. See the following:

• get_submission()

• handle_output()

• set_failure()

docs.ag_master.worker.extract_scores(transcript)

Extract scores of different types from a grading job’s output, starting from the bottom of the output

Parameters

transcript (str) – the grading output

Returns

a list of (key, score) pairs

docs.ag_master.worker.parse_scores(output)

Parse a grading job’s output for scores or errors

Parameters

output (str) – the grading job’s output

Returns

a list of score dictionaries of the form {score: float, kind: str, message: str}

Okpy Communication

There are a few endpoints created to communicate job requests and outputs between Okpy and the autograder. There is one RPC endpoint, trigger_jobs(), and a handful of REST endpoints. For the REST endpoints, see the next section.

docs.ag_master.okpy.create_okpy_endpoints(app)

Creates various RPC endpoints to interface with Okpy. See the following:

• trigger_jobs()

API Model

This is the set of endpoints that will be used to communicate between Okpy (or other grading base) and the autograder host. If you wish to recreate the autograder, you will need to implement the following 3 endpoints. In addition, you will likely need to get the contents of each backup that is being graded, which can be done using the View a Backup target of the Okpy API. You will also need to POST the scores output to Okpy, which can be done using the Create a Score target of the Okpy API.

POST /api/ok/v3/grade/batch

Trigger grading for Okpy backup(s)

Receives a list of backups to grade, along with metadata and an access token. Returns a list of Job IDs.

Request JSON Object

• access_token (string) – the token to communicate with Okpy

• assignment (string) – assignment secret

• subm_ids[] (string) – Backup IDs

Status Codes

• 200 OK – successfully triggered jobs

• 404 Not Found – assignment not found

• 405 Method Not Allowed – more than 50 batches would be created

Response JSON Object

• jobs[] (string) – Job IDs

POST /results

Get results for multiple jobs

Receives a list of Job IDs. Queries and returns the status of each Job.

Request JSON Object

• [] (string) – Job IDs

Status Codes

• 200 OK – successfully returns the status of each Job

Response JSON Object

• id1.result (string) – autograder result

• id1.status (string) – job status

• id2.result (string) – autograder result

• id2.status (string) – job status

GET /results/{job_id}

Get results for a single job

Receives a Job ID. Returns the status/results of the job.

Parameters

• job_id (string) – the autograder’s external Job ID

Status Codes

• 200 OK – the autograder result of the job if it has finished

• 202 Accepted – "Nope!” if the job has not finished