Math 398: Spring 2014: Independent Study in Operations Research / Linear Programming
Videos of lectures:
Lecture 1 (2/5/14): http://www.youtube.com/watch?v=4bimDLXu_I8&feature=youtu.be
Introduction to the course. Quick description of the various topics we'll study.
You should look at the two linear algebra review lectures from the MHC version of the class. These are not on YouTube and need to be downloaded.
First linear algebra review lecture: http://web.williams.edu/Mathematics/sjmiller/public_html/linprog/videos/LinProg_Lect01_LinAlgReview.MP4
Second linear algebra review lecture: http://web.williams.edu/Mathematics/sjmiller/public_html/linprog/videos/LinProg_Lect02_LinAlgReview.MP4
Lecture 2 (2/18/14): http://www.youtube.com/watch?v=LAl0kSJ1gBo&feature=youtu.be
In this lecture we cover Pascal's triangle modulo 2, and explain why the pattern persists and why its dimension is strictly between 1 and 2. We talk about dimensions and how one computes them. We discuss at length look-up tables and standardization (both for normal random variables and logarithms). We prove the change of base formula, which shows we only need to be able to compute logarithms in one base. We move on to talk about the advantages of different bases, and do a problem where we find base 3 is the best base for storage.
Homework problems: #1: Write a program to generate Pascal's triangle modulo 2. How far can you go? Can you use the symmetries to compute it quickly? #2: What is the dimension of the Cantor set? #3: There are many trig tables online (see for example http://www.sosmath.com/tables/trigtable/trigtable.html). Using the look-up table, discuss a simple way to estimate the value of the sine of any angle, then discuss a generalization that will be more accurate. For your better interpolation, provide a bound on your error as a function of the angle. #4: Redo the previous question, but now for interpolating values of the standard normal (http://www.sjsu.edu/faculty/gerstman/EpiInfo/z-table.htm). #5: Prove that if \(x_1 + \cdots + x_n = S\) with each \(x_i \ge 0\) then the product \(x_1 \cdots x_n\) is maximized when each \(x_i = S/n\).#6: One reason the Babylonians liked 60 is that it splits evenly many different ways. Look at the first 1,000 positive integers. Which integers have the most factors? Which have the least? What if you go up to 10,000? Which numbers do you think have a lot of factorizations relative to the numbers around them? In other words, which numbers have a lot of divisors relative to their neighbors? If you want to investigate this in Mathematica, for example, type For[n = 1, n £ 80, n++, Print[n, " ",Length[Divisors[n]]]] (you get the less than or equal to symbol by typing <=, which Mathematica then immediately replaces).
Lecture 3 (3/4/14):
We studied Horner's algorithm, fast exponentiation, Strassen's algorithm, and efficient ways to compute eigenvalues. The problem with the determinant method is it requires us to solve a polynomial of high degree, and there are real stability issues.
Homework problems: #1: Generalize Horner's algorithm to two variables: how would you evaluate \(\sum_{n=0}^N \sum_{m=0}^M a_{nm} x^n y^m\)? What if you had more variables? #2: Using fast exponentiation, how many operations does it take on average to evaluate \(x^n\), say for \(n\) ranging from \(1\) to \(N\)? We saw in the last lecture that base 3 is better than base 2 for storage; how many operations does it take on average to evaluate \(x^n\) in the same range if we now use ternary expansion instead of binary? #3: Write a program to implement Strassen's algorithm or write a program to numerically estimate the largest eigenvalue of a large matrix \(A\); do not do both. #4: Given a matrix \(A\), use knowledge of its coefficients to get some bounds or estimates on the eigenvalues. There is a lot of literature on the subject; you're welcome to search the literature after you try to see what you can figure out. #5: Watch the video on difference equations If the recurrence relation has constant coefficients, we can solve it exactly. Notice how useful linear algebra is in allowing us to jump to a solution. What happens if the coefficients are not constant? Can you solve for the terms in the sequence? Can you solve for them efficiently? By this I mean can you jump to the \(N\textsuperscript{th}\) coefficient from just knowing a few? Assume the recurrence relation has the form \(a_n = f_1(n) a_{n-1} + f_2(n) a_{n-2} + \cdots + f_k(n) a_{n-k}\) where \(f_1, \dots, f_k\) are fixed polynomials. The problem is deliberately open ended; you have freedom to choose the different polynomials. For example, take \(k=2\) and see what you can do. What if the functions aren't polynomials? For example, what if instead of being polynomials the functions are periodic, say \(k=2\) and the functions are periodic with period 2?
Lecture 4 (3/11/14): Canonical Linear Programming Problem, Diet Problem, Basic Solutions, Statement of Simplex Method.
Video: Part I: http://youtu.be/4cIpQNXz4I4 Video: Part II: http://youtu.be/aIeLw_zAxjo
Video on recurrence relations and roulette: https://www.youtube.com/watch?v=Esa2TYwDmwA
Video on duality: https://www.youtube.com/watch?v=aMorr1h4Egs
The day began with a discussion of canonical form, and the advantages of standardizing the representation of a problem. We then studied the Diet Problem, one of the most important applications of linear programming, and one of the earliest; see http://www.mpi-inf.mpg.de/departments/d1/teaching/ws12/ct/Dantzig-Diet.pdf for a humorous account by Dantzig on his work here. We talked a lot about how to find optimal solutions with constraints when calculus is available. We then moved to talk about Basic Solutions, and proved that there are only finitely many. We then stated the Simplex Method and the apparent circular argument of the proof, ending with a few words as to why all might be okay after all.
Homework:
Lecture 5 (3/18/14): Feasible and Basic Feasible Solutions, Proof of Simplex Method, Do Dogs Know Calculus, Euclidean algorithm.
Video: http://youtu.be/7ltj_O2wkR4
We showed how to pass from a feasible solution to a basic feasible solution. The idea is similar to the Method of Descent or the standard proof of the irrationality of \(\sqrt{2}\); we assume a solution exists with some minimal property, and then find a more minimal one. We then proved why the Simplex Method works. The argument is interesting, as we use Phase II to prove Phase I works, but Phase II assumes we can do Phase I! The reason it isn't a circular argument is that there is no problem provided we can input an initial basic feasible solution to Phase II. Instead of looking at the given problem, we look at a related problem where it's trivially easy to see a solution. The reason Phase II terminates is that it either constructs a sequence of feasible solutions with cost heading to \(-\infty\), or it passes from one basic feasible solution to another one with lower cost; as there are only finitely many basic feasible solutions the process must eventually end; note this is why we want to study basic feasible and not just feasible solutions. We left unsaid how quickly Phase II runs; it could take a long time. It's not always clear what's the best way to move towards an optimal solution; sometimes what is locally best is not globally best. We then talked about do dogs know calculus (see the comments below from one of my Calc classes) and the Euclidean Algorithm. There are many processes in mathematics that run in far less time than they appear. The Euclidean algorithm is a terrific example. It runs far faster than expected; it doesn't take min(x,y) steps but rather 2 log(min(x,y)). It's a beautiful example of a major theme of the course, the need to do things fast.
Homework:
#1: Write or use an existing program to solve a linear programming problem, say the Diet Problem with 4 products and three nutrients. Mathematica is an option, as is R or even Excel. I want you to have this skill. #2: We showed the Euclidean Algorithm runs faster than expected. What are the worst inputs? In other words, what two numbers \(x < y\) have the longest run time? Of course, this question is a bit vague. The larger \(x, y\) the longer the run-time, so maybe we should look at the run-time divided by \(\log_2 x\). #3: Choose a project for the class!
The purpose of the paragraphs below is to explain the rational for the course and the requirements. While the goal is to have this course cross listed across departments, due to the multidisciplinarity of the subject, it will be housed in Mathematics. I have chosen to make it a pre-core class as there is a curricular need for classes at this level, and the material can be covered in great depth without assuming more advanced courses. The latter allows a wider audience of students access to the material, and can serve as a motivation to explore these other subjects in greater detail. Many of the most important algorithms in CS and mathematics are very elementary to state and analyze (though discovering them was quite difficult). For example, Strassen’s algorithm revolutionized computations with matrices, yet if a professor felt so inclined could be included in a standard linear algebra class. Similarly the proof of the simplex method mostly requires simple results from linear algebra and only elementary convergence facts from real analysis; rather than requiring real as a pre-requisite and limiting the reach of the course, we have decided to open the course up to all and quickly teach the needed limiting results.
Units:
Ancient / Classical algorithms: These are warm-ups meant to show students how to do common items faster (i.e., things they’ve seen many times). Examples include the following.
Babylonian multiplication: base 60 is a pain; instead of learning the 60x60 table for xy, they noted xy = ((x+y)2 - x2 - y2) / 2, reducing to just needing to know squares, subtraction and division by 2. This leads to the concept of a look-up table and interpolation, and is a great opening to talk about classical tables (for special functions), and how modern appliances (such as cell phones) work with limiting processing power.
Horner’s algorithm: fast polynomial evaluation by cleverly grouping parentheses. We will discuss why such efficiencies are needed. One project will be for students to write code to iterate polynomials and create fractal sets, with and without Horner’s algorithm, noticing the sizable decrease in run-time (or at least I hope they will see this – I saw that in the 90s when I wrote such code!).
Fast exponentiation: The method of repeated squaring to evaluate multiplication quickly. This will lead into discussions of the advantages of different bases and applications in cryptography, and issues in increased storage in some implementations (and ways to minimize that). Student will write code to perform these computations.
Strassen algorithm: Will discuss how to obtain a power improvement and what this means, fits nicely with the Horner algorithm discussion. Will discuss real world implications of being able to do matrix multiplication faster. Will talk about how things are easier to program when the matrix size is a power of 2, and what that means in determining how data is collected. If time permits we will discuss algorithms involving sparse matrices.
Efficiency Revisited: These are some important problems where students may not have had as much experience. The point is to look at a variety of problems where standard definitions or brute force are too slow.
Finding eigenvalues: It should be a prosecutable crime to believe that one finds eigenvalues by looking at determinants! We will discuss a variety of methods (some involving the method of repeated squaring) to compute these efficiently. Students will write code to numerically approximate eigenvalues.
Solving difference equations: While in principle the definition and initial conditions determine all the terms, there is frequently a need to jump far down in the sequence or investigate limiting behavior. This leads to discussions on the Method of Divine Inspiration (to guess solutions to these, and related problems in differential equations), and generating functions. We will discuss recent work of myself and my students on Zeckendorf decompositions (writing any number as a sum of non-adjacent Fibonacci or, more generally, other recurrence relations with some constraints), and numerically investigate properties of the number of summands (especially the distribution of their gaps).
Video Resource: Fibonacci Numbers and Double Plus One: http://www.youtube.com/watch?v=Esa2TYwDmwA
Binomial Coefficients: Pascal’s triangle modulo 2 is a beautiful introduction to fractals. We will study this system and discuss the difficulties in efficiently computing it. We will analyze several approaches, and students will be required to write code to create. Issues will involve data storage, data access, and jumping to compute the triangle at a certain spot. We will also discuss issues involved in graphically presenting the results, and efficient ways to computing binomial coefficients and prove identities. For example, if we need all the coefficients in a given row, it is absurd to compute them individually, even though we have a closed form expression for each. It is far better to notice that (n choose k+1) = (n choose k) x ((n-k)/(k+1)). If time permits we will discuss computations related to graphing Ulam’s spiral.
Sorting algorithms: If time and interest permit, we’ll discuss various sorting algorithms as these are great examples to introduce run-time complexity calculations, and issues with how we score an algorithms (are we concerned about average run-time, worse case, …), and the advantages of switching off between two algorithms.
Factorization / primality testing: Interestingly it’s possible to efficiently determine if a number is composite without finding any factors! We’ll discuss some of the issues with factoring and primality testing, learning a small amount of group theory as needed. Students with stronger backgrounds will be encouraged to read the “Primes in P” paper. We will see applications of fast exponentiation with Fermat’s little theorem.
PRIMES is in P Pages 781-793 from Volume 160 (2004), Issue 2 by Manindra Agrawal, Neeraj Kayal, Nitin Saxena, Annals of Mathematics: http://annals.math.princeton.edu/wp-content/uploads/annals-v160-n2-p12.pdf
Linear Programming: The highlight of the course is a detailed analysis of linear programming. We will discuss the history of the subject, from original papers that claimed to be of theoretical interest but no practical value as there would never be any way to do so many computations, to its recent successes and challenges. Students will be required to write or use code to solve a linear programming problem. Notes of mine: http://web.williams.edu/Mathematics/sjmiller/public_html/416/currentnotes/AdvLinProgBook.pdf
Basics: basics of linear programming, duality, standard examples (diet problem).
Video resource: Duality and Chess: http://www.youtube.com/watch?v=aMorr1h4Egs
Simplex Method: We will prove the simplex method works, and analyze its run-time as much as possible. This will be the theoretical highpoint of the semester.
Binary Integer Programming: While we currently do not have an algorithm capable of resolving binary integer programming problems quickly, we can often get close to optimal very quickly, which suffices for many purposes. This is especially true as we frequently do not know what values the parameters take! This is the part of the course that will have the greatest stats component. We will discuss a paper I wrote with marketing colleagues which solved a binary integer programming problem for the movie industry, and discuss the issues with parameter estimation and model formation. This, as well as factorization, are excellent opportunities to discuss P – NP.
Silver Scheduler: Demand-Driven Scheduling of Movies in a Multiplex (with Jehoshua Eliashberg, Quintus Hegie, Jason Ho, Dennis Huisman, Sanjeev Swami, Charles B. Weinberg and Berend Wierenga), lead article International Journal of Market Research and won award for best paper of 2009 (doi:10.1016/j.ijresmar.2008.09.004) : http://web.williams.edu/Mathematics/sjmiller/public_html/math/papers/SilverScheduler_finalver_05012007.pdf
Baseball, Optimization and the World Wide Web, Ilan Adler, Alan L. Erera, Dorit S. Hochbaum and Eli V. Olinick. http://riot.ieor.berkeley.edu/~dorit/pub/baseball.html
Traveling salesman problem: http://xkcd.com/399/
Applications (Scheduling, Stochastic): We will study progress on the Traveling Salesman Problem and scheduling problems. We will explore how variations in parameter values effect the solutions. We will study stochastic programming problems, where the parameters themselves are independent random variables (which will require a quick crash course in probability, and hopefully motivate students to take that class, if they haven’t done so, in the spring). Another interesting application is using linear programming to compute what elimination numbers should be in baseball (MLB incorrectly defines the term and often, as in 2004, incorrectly calculates them!). Other applications include Hales’ proof of Kepler’s conjecture on sphere packing, Support Vector Learning, Quadratic Programming, ….