Admission requirements
The prerequisites include basic linear algebra (e.g., vector space and matrix operations), calculus (mainly differentiation), basic probability theory (probability distributions) and statistical inference, basic understanding of optimization techniques and machine learning fundamentals, and Python programming.
Description
This course covers the fundamentals of optimization, focusing on Bayesian optimization (BO) methods for optimizing expensive-to-evaluate functions in continuous search spaces. It introduces Gaussian processes as surrogate models and discusses various acquisition functions for balancing exploration and exploitation. The course includes practical implementation using Python libraries and real-world applications like applications in engineering and hyperparameter tuning. Advanced topics such as handling noisy observations and optimization on mixed-integer/discrete and/or constrained spaces are also explored.
This course is unique in its nature. While some universities around the world have offered seminars on BO (due to the increasing popularity of these algorithms for optimizing expensive-to-evaluate processes), we are the first to provide a comprehensive course on the topic. The course offers a rigorous overview of the entire field, starting with the fundamental principles of BO in continuous and unconstrained spaces, and then moving to more specific scenarios, always emphasizing the practical usefulness of these solvers. A concise list of topics that will be addressed includes:
- Fundamentals of black-box optimization
- Introduction to Bayesian optimization and sampling strategies
- Gaussian processes as surrogate models
- Acquisition functions for exploration and exploitation
- Handling noisy observations
- Optimization on mixed-integer/discrete spaces
- Optimization in constrained spaces
- High-dimensional BO
- Real-world applications in engineering and hyperparameter tuning in machine learning
Course objectives
The course offers a comprehensive overview of the field through a series of lectures and programming exercises.
By the end of the course, students should be able to:
Understand the fundamentals of black-box optimization by recognising the intrinsic differences between different algorithm families.
Explain the working principles of Bayesian optimization and compare different one-shot sampling strategies.
Explain Gaussian processes as surrogate models and illustrate the role of their hyperparameters implementing them in Python.
Define and distinguish between different acquisition functions based on their exploration and exploitation capabilities.
Examine more complex optimization scenarios handling noisy observations, mixed-integer/discrete variable, constrained search spaces and design effective optimization pipelines for these cases.
List the different categories of algorithms for high-dimensional BO and their working principles.
Present BO studies on real-world applications in engineering and hyperparameter tuning in machine learning.
Implement BO pipelines using Python libraries and use them to solve practical challenges.
Timetable
In MyTimetable, you can find all course and programme schedules, allowing you to create your personal timetable. Activities for which you have enrolled via MyStudyMap will automatically appear in your timetable.
Additionally, you can easily link MyTimetable to a calendar app on your phone, and schedule changes will be automatically updated in your calendar. You can also choose to receive email notifications about schedule changes. You can enable notifications in Settings after logging in.
Questions? Watch the video, read the instructions, or contact the ISSC helpdesk.
Note: Joint Degree students from Leiden/Delft need to combine information from both the Leiden and Delft MyTimetables to see a complete schedule. This video explains how to do it.
Mode of instruction
Lectures and practical sessions.
Hours of Study: 114 (= 6 EC)
Lectures: 14
Programming exercises: 3
Practical work: 48
Exam and preparation: 30
Assessment method
The final grade is a combination of grades for (1) the written exam (50%) and (2) the practical assignment (50%).
Both the written exam and the practical assignment are mandatory. The final grade for the course is determined by calculating the average of the two grades. Students have the opportunity to retake the exam but not the assignment.
Reading List
The following titles are recommended but not mandatory for the course:
Garnett, Roman. Bayesian Optimization. Cambridge University Press, 2023. https://bayesoptbook.com
Forrester, Alexander IJ, András Sóbester, and Andy J Keane. Engineering Design via Surrogate Modelling - A Practical Guide. John Wiley & Sons Ltd, 2008.
Frazier, PI. A Tutorial on Bayesian Optimization. arXiv:1807.028112018 http://arxiv.org/abs/1807.02811
Other resources will be communicated during the course.
Registration
As a student, you are responsible for enrolling on time through MyStudyMap.
In this short video, you can see step-by-step how to enrol for courses in MyStudyMap.
Extensive information about the operation of MyStudyMap can be found here.
There are two enrolment periods per year:
Enrolment for the fall opens in July
Enrolment for the spring opens in December
See this page for more information about deadlines and enrolling for courses and exams.
Note:
It is mandatory to enrol for all activities of a course that you are going to follow.
Your enrolment is only complete when you submit your course planning in the ‘Ready for enrolment’ tab by clicking ‘Send’.
Not being enrolled for an exam/resit means that you are not allowed to participate in the exam/resit.
Contact
Lecturers: Dr. E. Raponi
Website: See course page on Brightspace.
Remarks
Software
Starting from the 2024/2025 academic year, the Faculty of Science will use the software distribution platform Academic Software. Through this platform, you can access the software needed for specific courses in your studies. For some software, your laptop must meet certain system requirements, which will be specified with the software. It is important to install the software before the start of the course. More information about the laptop requirements can be found on the student website.