In this homework, we will train a CNN to do vision-based driving in SuperTuxKart.
This assignment should be solved individually. No collaboration, sharing of solutions, or exchange of models is allowed. Please, do not directly copy existing code from anywhere other than your previous solutions, or the previous master solution. We will check assignments for duplicates. See below for more details.
We will design a simple low-level controller that acts as an auto-pilot to drive in supertuxkart. We then use this auto-pilot to train a vision based driving system. To get started, first download and install SuperTuxKart on your machine.
pip install -U PySuperTuxKart
If you encounter any issues installing this package, please post them in Piazza.
In the first part of this homework, you’ll write a low-level controller in
The controller function takes as input an aim-point and the current velocity of the car.
The aim-point is a point on the center of the track 15 meters away from the kart, as shown below.
In the first part of this assignment, we will use a ground truth aim-point from the simulator itself. In the second part, we remove this restriction and predict the aim-point directly from the image.
The goal of the low-level controller is to steer towards this point.
The output of the low-level controller is a
You can specify:
pystk.Action.steerthe steering angle of the kart normalized to -1 … 1
pystk.Action.accelerationthe acceleration of the kart normalized to 0 … 1
pystk.Action.brakeboolean indicator for braking
pystk.Action.drifta special action that makes the kart drift, useful for tight turns
pystk.Action.nitroburns nitro for fast acceleration
Implement your controller in the
control function in
You won’t need any deep learning to design this low-level controller.
You may use numpy instead of pytorch for this part.
Once you finish, you could test your controller using
python -m homework.controller [TRACK_NAME] -v
You should tune the hyper-parameters of your controller.
You might want to look into gradient-free optimization or exhaustive search.
The reference controller completes each level relatively efficiently:
lighthouse in under 50 sec,
snowtuxpeak in under 60 sec,
scotland in under 70 sec.
Grade your controller using
python -m grader homework
Hint: Skid if the steering angle is too large
Hint: Target a constant velocity
Hint: Use the aim-point to compute the absolute steering angle, learn or tune a scaling factor between absolute and normalized steering.
In the second part, you’ll train a planner to predict the aim-points. The planner takes as input an image and outputs the aim-point in the image coordinate. Your controller then maps those aim-points to actions.
Use your low-level controller to collect a training set for the planner.
python -m solution.utils zengarden lighthouse hacienda snowtuxpeak cornfield_crossing scotland
We highly recommend you limit yourself to the above training levels, adding additional training levels may create an unbalanced training set and lead to issues with the final test_grader.
This function creates a dataset of images and corresponding aim-points in
You can visualize the data using
python -m homework.visualize_data drive_data
Implement your planner model in
Planner class of
Your planner model is a
torch.nn.Module that takes as input an image tensor and outputs the aiming point in image coordinates (x:
We recommend using an encoder-decoder structure to predict a heatmap and extract the peak using a spatial argmax layer in
Complete the training code in
train.py and train your model using
python -m homework.train.
Once you completed everything, use
python -m homework.planner [TRACK_NAME] -v
to drive with your CNN planner and controller.
We will grade both your controller and planner in the following 6 tracks
Your controller/planner should complete each track within a certain amount of time. You receive 5% of your grade by completing each track with your low-level controller. You receive 10% of your grade by completing each track with your image-based agent. You may train on all the above testing track.
For the last 10%, you’ll need to complete an unseen test track. We chose a relatively easy test track. You can test your solution against the grader by
python -m grader homework
Extra credit (up to 10pt)
We will run a little tournament with all submissions, the top 9 submissions will receive 10, 9, 8, … extra credit respectively. The tournament uses several unreleased test tracks.
Once you finished the assignment, create a submission bundle using
python bundle.py [YOUR UT ID]
and submit the zip file online. If you want to double-check that your zip file was properly created, you can grade it again
python -m grader [YOUR UT ID].zip
Running your assignment on google colab
You might need a GPU to train your models. You can get a free one on google colab. We provide you with a ipython notebook that can get you started on colab for each homework. Follow the instructions below to use it.
- Go to http://colab.research.google.com/.
- Sign in to your Google account.
- Select the upload tab then select the
- Go to Runtime -> Change runtime type.
GPUas the hardware accelerator.
- Follow the instructions on the homework notebook to upload code and data.
This assignment should be solved individually.
What interaction with classmates is allowed?
- Talking about high-level concepts and class material
- Talking about the general structure of the solution (e.g. You should use convolutions and ReLU layers)
- Looking at online solutions, and pytorch samples without directly copying or transcribing those solutions (rule of thumb, do not have your coding window and the other solution open at the same time). Always cite your sources in the code (put the full URL)!
- Using any of your submissions to prior homework
- Using the master solution to prior homework
- Using ipython notebooks from class
What interaction is not allowed?
- Exchange of code
- Exchange of architecture details
- Exchange of hyperparameters
- Directly (or slightly) modified code from online sources
- Any collaboration
- Putting your solution on a public repo (e.g. github). You will fail the assignment if someone copies your code.
Ways students failed in past years (do not do this):
Student A has a GPU, student B does not. Student B sends his solution to Student A to train 3 days before the assignment is due. Student A promises not to copy it but fails to complete the homework in time. In a last-minute attempt, Student A submits a slightly modified version of Student B’s solution. Result: Both students fail the assignment.
Student A struggles in class. Student B helps Student A and shows him/her his/her solution. Student A promises to not copy the solution but does it anyway. Result: Both students fail the assignment.
Student A sits behind Student B in class. Student B works on his homework, instead of paying attention. Student A sees Student B’s solution and copies it. Result: Both students fail the assignment.
Student A and B do not read the honor code and submit identical solutions for all homework. Result: Both students fail the class.
Installation and setup
Installing python 3
Installing the dependencies
Install all dependencies using
pip install -r requirements.txt
Note: On some systems, you might be required to use
pip3 instead of
pip for python 3.
If you’re using conda use
conda env create environment.yml
Manual installation of pytorch
Go to https://pytorch.org/get-started/locally/ then select the stable Pytorch build, your OS, package (pip if you installed python 3 directly, conda if you installed Anaconda), python version, cuda version. Run the provided command. Note that cuda is not required, you can select cuda = None if you don’t have a GPU or don’t want to do GPU training locally. We will provide instruction for doing remote GPU training on Google Colab for free.
Manual installation of the Python Imaging Library (PIL)
The easiest way to install the PIL is through
pip install -U Pillow
There are a few important considerations when using PIL.
First, make sure that your OS uses
libjpeg-turbo and not the slower
libjpeg (all modern Ubuntu versions do by default).
Second, if you’re frustrated with slow image transformations in PIL use
CC="cc -mavx2" pip install -U --force-reinstall Pillow-SIMD
CC="cc -mavx2" is only needed if your CPU supports AVX2 instructions.
pip will most likely complain a bit about missing dependencies.
Install them, either through
conda, or your favorite package manager (