Admission requirements
Assumed prior knowledge
1. One or more of a Bachelor level course on Artificial Intelligence, Machine Learning, Data Science, or Data Mining.
2. Appropriate mathematical background for Master program computer science.
3. Bachelor level proficiency in the Python programming language.
4. Familiarity with deep learning is required.
Description
Deep reinforcement learning is a field of Artificial Intelligence that has attracted much attention since impressive achievements in Robotics, Atari, and most recently Go, where human world champions were defeated by computer players. These results build upon a combination of the rich history of reinforcement learning research and deep learning.
This course teaches the field of deep reinforcement learning: How does it work, why does it work, and what are the reinforcement learning methods on which Robotics and AlphaGo’s success are based? By the end of the course you should have acquired a good understanding of the field of deep reinforcement learning.
The defining characteristic of reinforcement learning is that agents learn through interaction with an environment, not unlike humans learn by doing. Instead of telling a learner which action to take, the agent analyzes which action to take so as to maximize a reward signal. Reinforcement learning is a powerful technique for solving sequential decision problems.
The defining characteristic of deep learning is that the model generalizes, it build a hierarchy of abstract features from its inputs.
Prominent reinforcement learning problems occur, amongst others, in games and robotics. In this course you will learn the necessary theory to apply reinforcement learning to realistic problems from the field of computer game playing.
The following topics and algorithms are planned to be discussed:
Tabular Value-based Reinforcement Learning, such as Q-learning
Deep Value-based Reinforcement Learning, such as DQN
Policy-based Reinforcement Learning, Robotics
Model-based Reinforcement Learning
Two-Agent Self-Play (AlphaGo)
Multi-Agent Reinforcement Learning (Poker, StarCraft)
Hierarchical Reinforcement Learning
Meta-Learning, such as MAML
In addition the role of reinforcement learning in artificial intelligence and the relation with psychology will be discussed (human learning).
This a hands-on course, in which you will be challenged to build working game playing programs with different reinforcement learning methods. This is a challenging course in which proficiency in Python and deep learning libraries (such as Keras and PyTorch) is important.
All assignments are made in Python.
Course objectives
After completing the reinforcement learning course, the students should be able to:
Create and apply Deep Reinforcement Learning to model sequential decision problems and have a deep understanding of the field. They know how reinforcement learning differs from supervised learning and unsupervised learning.
They should analyse and apply the Markov Decision Process formalism.
Students must evaluate and apply model-free, model-based, and value-based and policy-based approaches, for single agent, two-agent, and multi-agent problems. Students should be proficient in applying these approaches to these problems.
They should be able to relate their knowledge to applications in games and robotics, such as breakthrough applications in AlphaGo and robotics. Students will also be able to understand how reinforcement learning is used in recent applications such as large language models, and how metalearning helps generalization.
Timetable
The most recent timetable can be found at the Computer Science (MSc) student website.
In MyTimetable, you can find all course and programme schedules, allowing you to create your personal timetable. Activities for which you have enrolled via MyStudyMap will automatically appear in your timetable.
Additionally, you can easily link MyTimetable to a calendar app on your phone, and schedule changes will be automatically updated in your calendar. You can also choose to receive email notifications about schedule changes. You can enable notifications in Settings after logging in.
Questions? Watch the video, read the instructions, or contact the ISSC helpdesk.
Note: Joint Degree students from Leiden/Delft need to combine information from both the Leiden and Delft MyTimetables to see a complete schedule. This video explains how to do it.
Mode of instruction
Literature (see below). The relevant chapters should be read before the corresponding lecture.
Lectures
Computer lab
Course load
Hours of study: 168 hrs (= 6 EC)
Lectures: 26:00 hrs
Seminars: 26:00 hrs
Practical assignments: 70:00 hrs
Examination and preparation: 46:00 hrs
Assessment method
Assignments (2-4) to check ability to create and apply RL algorithms (formative and summative), and theory exam to check understanding and evaluation.
A grading rubric for the assignment is provided on Brightspace.
The final grade is a combination of grades for: (1) the written exam (50%, mandatory) and (2) the reports for the assignments (50%, mandatory).
Completed assignments are valid for one year. Failing the course means redoing all assignments again next year.
For assignments there is no re-take, for the exam there is. Exceptions to be discussed with the teacher.
No-show for an assignment yields a 0 for that assignment, and should be compensated with other assignments/exam.
Some assignments are in groups of three. All students should contribute equally.
Academic integrity is fundamental in this course, all sources should be academically referenced. Deviations may be reported to the board of examiners who may decide on a sanction.
The teacher will inform the students how the inspection of and follow-up discussion of the exams will take place.
Reading list
Mandatory:
- A. Plaat, Deep Reinforcement Learning, Springer 2022. Freely available here.
Optional:
- R. Sutton and A. Barto, Reinforcement Learning: an introduction, MIT Press, Second Edition, 2018. Freely available here.
Registration
As a student, you are responsible for enrolling on time through MyStudyMap.
In this short video, you can see step-by-step how to enrol for courses in MyStudyMap.
Extensive information about the operation of MyStudyMap can be found here.
There are two enrolment periods per year:
Enrolment for the fall opens in July
Enrolment for the spring opens in December
See this page for more information about deadlines and enrolling for courses and exams.
Note:
It is mandatory to enrol for all activities of a course that you are going to follow.
Your enrolment is only complete when you submit your course planning in the ‘Ready for enrolment’ tab by clicking ‘Send’.
Not being enrolled for an exam/resit means that you are not allowed to participate in the exam/resit.
Contact
Lecturer: prof. dr. Aske Plaat
Teaching assistants
Remarks
There is limited space for students who are not enrolled in the Computer Science programme or one of the Data Science specialisations (Data Science: Computer Science and Astronomy and Data Science). Please contact the programme coordinator/study advisor (mailto:mastercs@liacs.leideuniv.nl) if you are an external student.
Software
Starting from the 2024/2025 academic year, the Faculty of Science will use the software distribution platform Academic Software. Through this platform, you can access the software needed for specific courses in your studies. For some software, your laptop must meet certain system requirements, which will be specified with the software. It is important to install the software before the start of the course. More information about the laptop requirements can be found on the student website.