Admission requirements
The prerequisites include basic linear algebra (e.g., vector space and matrix operations), calculus (mainly differentiation), basic probability theory (probability distributions) and statistical inference, basic understanding of optimization techniques and machine learning fundamentals, and Python programming.
Description
This course covers the fundamentals of optimization, focusing on Bayesian optimization (BO) methods for optimizing expensive-to-evaluate functions in continuous search spaces. It introduces Gaussian processes as surrogate models and discusses various acquisition functions for balancing exploration and exploitation. The course includes practical implementation using Python libraries and real-world applications like applications in engineering and hyperparameter tuning. Advanced topics such as handling noisy observations and optimization on mixed-integer/discrete and/or constrained spaces are also explored.
This course is unique in its nature. While some universities around the world have offered seminars on BO (due to the increasing popularity of these algorithms for optimizing expensive-to-evaluate processes), we are the first to provide a comprehensive course on the topic. The course offers a rigorous overview of the entire field, starting with the fundamental principles of BO in continuous and unconstrained spaces, and then moving to more specific scenarios, always emphasizing the practical usefulness of these solvers. A concise list of topics that will be addressed includes:
- Fundamentals of black-box optimization
- Introduction to Bayesian optimization and sampling strategies
- Gaussian processes as surrogate models
- Acquisition functions for exploration and exploitation
- Handling noisy observations
- Optimization on mixed-integer/discrete spaces
- Optimization in constrained spaces
- High-dimensional BO
- Real-world applications in engineering and hyperparameter tuning in machine learning
Course objectives
The course offers a comprehensive overview of the field through a series of lectures and programming exercises.
By the end of the course, students should be able to:
Understand the fundamentals of black-box optimization by recognising the intrinsic differences between different algorithm families.
Explain the working principles of Bayesian optimization and compare different one-shot sampling strategies.
Explain Gaussian processes as surrogate models and illustrate the role of their hyperparameters implementing them in Python.
Define and distinguish between different acquisition functions based on their exploration and exploitation capabilities.
Examine more complex optimization scenarios handling noisy observations, mixed-integer/discrete variable, constrained search spaces and design effective optimization pipelines for these cases.
List the different categories of algorithms for high-dimensional BO and their working principles.
Present BO studies on real-world applications in engineering and hyperparameter tuning in machine learning.
Implement BO pipelines using Python libraries and use them to solve practical challenges.
Timetable
You will find the timetables for all courses and degree programmes of Leiden University in the tool MyTimetable (login). Any teaching activities that you have sucessfully registered for in MyStudymap will automatically be displayed in MyTimetable. Any timetables that you add manually, will be saved and automatically displayed the next time you sign in.
MyTimetable allows you to integrate your timetable with your calendar apps such as Outlook, Google Calendar, Apple Calendar and other calendar apps on your smartphone. Any timetable changes will be automatically synced with your calendar. If you wish, you can also receive an email notification of the change. You can turn notifications on in ‘Settings’ (after login).
For more information, watch the video or go the the 'help-page' in MyTimetable. Pleas note: Joint Degree students Leiden/Delft have to merge their two different timetables into one. This video explains how to do this.
Mode of instruction
Lectures and practical sessions.
Hours of Study: 114 (= 6 EC)
Lectures: 14
Programming exercises: 3
Practical work: 48
Exam and preparation: 30
Assessment method
The final grade is a combination of grades for (1) the written exam (50%) and (2) the practical assignment (50%).
Both the written exam and the practical assignment are mandatory. The final grade for the course is determined by calculating the average of the two grades. Students have the opportunity to retake the exam but not the assignment.
Reading List
The following titles are recommended but not mandatory for the course:
Garnett, Roman. Bayesian Optimization. Cambridge University Press, 2023. https://bayesoptbook.com
Forrester, Alexander IJ, András Sóbester, and Andy J Keane. Engineering Design via Surrogate Modelling - A Practical Guide. John Wiley & Sons Ltd, 2008.
Frazier, PI. A Tutorial on Bayesian Optimization. arXiv:1807.028112018 http://arxiv.org/abs/1807.02811
Other resources will be communicated during the course.
Registration
From the academic year 2022-2023 on, every student has to register for courses with the new enrollment tool MyStudymap. There are two registration periods per year: registration for the fall semester opens in July and registration for the spring semester opens in December. Please see this page for more information.
Please note that it is compulsory to register for every exam and retake. Not being registered for a course means that you are not allowed to participate in the final exam of the course.
Extensive FAQ on MyStudymap can be found here.
Contact
Lecturers: Dr. E. Raponi
Website: See course page on Brightspace.
Remarks
Not applicable.