Math 2805: Mathematical principles of machine learning #

Modern machine learning lies at the confluence statistics and probability, computer science, and mathematics, evolving through a rich interplay among the three fields. The main focus of the class it to explore the mathematical structures and techniques that inform this conversation, but we will not be able to insulate ourselves from the other two essential components: working with data with its inherent uncertainty and its analysis using algorithms from computer science.

Course description: An introduction to the mathematical theory and practice of machine learning. Supervised and unsupervised learning, with topics including regression, classification, clustering, dimension reduction, data visualization, denoising, norms and loss functions, neural networks, optimization, universal approximation theorems, and algorithmic fairness. Class will be lab and project-based, but no formal programming experience is necessary.

Learning goals: Upon completion of this course, a student should be able to:

• build mathematical models based on data while appreciating the shortcomings and limitations of these models in the decision making process,
• use a rich variety of tools to apply and analyze machine learning models focusing on techniques from linear algebra and real analysis,
• describe the interplay of theory and application in this branch of science,
• appreciate the formal guarantees provided by theoretical results in machine learning tasks,
• read recent research in the field and apply it to their own projects.

In one sentence, my goal for this class is to show how mathematical principles, structures, and techniques have informed the field of machine learning as it developed and continue to drive its further evolution.

Books and materials #

We will use a variety of sources during the course of the semester. The material for the lectures, activities, and the labs will be drawn from a variety of textbooks and research articles. While there are no required books for the course, the following may be useful during the semester. All will be available in the math department non-circulating library in Searles 214. Some of the following are also available for free from the authors; if so, they are linked directly below.

For topics in analysis:

• Principles of Mathematical Analysis by Walter Rudin,
• Introductory Real Analysis by A. Kolomogorov and S. Fomin, and
• Introduction to Analysis by Maxwell Rosenlicht

For topics in linear algebra:

• Introduction to Linear Algebra by Gil Strang,
• Linear Algebra and its Applications by D. Lay, S. Lay, and J. McDonald; and
• Matrix Computations by Gene H. Golub and Charles F. Van Loan.

For topics in machine learning:

• The Elements of Statistical Learning by T. Hastie, R. Tibshirani, and J. Friedman;
• Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville; and
• Machine Learning: A Probabilistic Perspective by Kevin Murphy.

Reading and watching mathematics: Learning mathematics is not a spectator sport. Reading mathematics is not like reading a novel; watching mathematics is not like watching an action thriller. Some paragraphs are easy to digest, but you may find yourself looking at one line of text for five or more minutes trying to understand what the author is trying to say.  Use the pause button with reckless abandon when watching a video. As you read or watch, take notes, just as you do in class. This is crucial! If questions arise, write them down and ask during office hours or class.

Course meetings and office hours #

We will meet in Searles 217 every Monday and Wednesday at 2:50pm. Whenever possible and appropriate, lectures will be recorded for your enjoyment and to help you review your notes. When necessary, we will meet over Zoom. The link is available on the main course website.

I will hold formal office hours on:

There may be times in the semester where I will have to reschedule office hours due to a conflict. I will let you know by email beforehand. I am available throughout the week for additional meetings. To schedule either individually or as a group, please send me an email with a couple of times that will work. And please don’t hesitate to set these up, I am very happy to see you.

Course assessment #

Your course grade will be based on homework and project grades. More specifically:

Homework: Homework problems will examine both the theoretical and computational aspects of the class. They will be assigned after every class and will be due once a week on Wednesdays at 9am. Each submission will consist of two parts:

• Cover sheet: In addition to your name, I will also ask you to recognize individuals you worked with and sources you used to complete your work.
• Homework problems: Problems will be assigned after every lecture.

Your homework will be graded using [GradeScope]. To begin, you will need to set up an account.  I will send you the code for our class by email.  Each homework assignment will need to be submitted as a .pdf file.  If you edit your homework electronically, make sure you can save or export your work in this format.  If you write-up your homework the old-fashioned way using pencil and paper, use a scanner or a phone scanner app.  See the GradeScope help document for a list of suggestions [pdf]. Please let me know if this does not work for you; I will come up with an alternative.

Projects: As part of your evaluation for the course, I would like you to complete a project that show your mastery of some of the concepts covered in the course. You will hand in a written report of your work, the goal of which will be to show;

• your mastery of knowledge of some component of the material that we have studied this semester, as well as
• your ability to build upon what we have learned, either by learning additional material beyond what we have covered or by applying techniques to pursue some further goal.

The course project will be due during finals week. More details and project suggestions will be announced.

Some axioms #

Federico Ardila enunciated the following axioms in his Todos Cuendan. They form a lens through with I view both teaching and doing mathematics.

• Mathematical potential is distributed equally among different groups, irrespective of geographic, demographic, and economic boundaries.
• Everyone can have joyful, meaningful, and empowering mathematical experiences.
• Mathematics is a powerful, malleable tool that can be shaped and used differently by various communities to serve their needs.
• Every student deserves to be treated with dignity and respect.

Outline of course topics #

Part I: Regression and classification

1. Approximating functions: introduction to supervised learning, metric spaces of functions, convolution product, feature detection, integral transforms, the Weierstrass, Stone-Weierstrass, and Universal Approximation Theorems.
2. Generalized regression: linear algebra review, orthonormal bases, linear projections, polynomial regression.
3. Empirical risk minimization: loss functions, gradient descent, multilinear calculus, the Hessian and curvature, logistic regression, information and cross entropy, Boston housing prices, classification of hand-written digits.
4. Bonus: the perceptron and the first AI winter, MNIST classification redux.

Part II: Neural networks

1. Neurons and networks: basic definitions, linear algebra implementation, introduction to Keras, expressiveness and the Universal Approximation Theorem.
2. Backpropagation: chain rule, Jacobian of linear transformations, adversarial images, MNIST classification redux –again.
3. Equivariance and invariance: feature detection redux and convolutional neural networks, another Universal Approximation Theorem.

Part III: Unsupervised learning

1. Clustering: \(k\)-means clustering, extraneous variables, Voronoi cells,
2. Ranking: Theorem, PageRank.
3. Dimension reduction: singular value decomposition and the PCA, UMAP, neural network embeddings, neural collapse, faces and dimension reduction, unfolding the data manifold