Meaning of Lagrange multiplier

Hey folks, in this video, I want to show you something pretty interesting about these Lagrange multipliers that we've been studying.

So the first portion, I'm just going to kind of get the setup, which is a lot of review from what we've seen already. But I think you're really going to like where this is going in the end.

One of the examples I showed, and I think this is a pretty nice prototypical example for constrained optimization problems, is that you're running a company and you have some kind of revenue function that's dependent on various choices you make in running the company. I think I said the number of hours of labor you employ and the number of tons of steel you use, you know, if you are manufacturing something metallic.

And you know, this might be modeled as some multivariable function of H and S. Right now, we don't really care about the specifics, and you're trying to maximize this. Right? That's kind of the whole point of this unit that we've been doing, is that you're trying to maximize some function, but you have a constraint. This is the real world; you can't just spend infinite money. You have some kind of budget, some sort of amount of money you spend as a function of those same choices you make: the hours of labor you employ, the tons of steel you use.

This again is going to equal some multivariable function that tells you how much money you spend for a given amount of hours and given number of tons of steel, and you set this equal to some constant. This tells you the amount of money you're willing to spend, and our goal has been to maximize some function subject to a constraint like this.

The mental model you have in mind is that you're looking in the input space inside the XY plane, or I guess really it's the HS plane in this case. Right? Your inputs are H and S, and points in this plane tell you possible choices you can make for hours of labor and tons of steel. You think of this budget as some kind of curve in that plane. Right? All the sets of H and S that equal $10,000 is going to give you some kind of curve.

The core value we care about is that when you maximize this revenue, you know, when you set it equal to a constant, I'm going to call M star, this like the maximum possible revenue, that's going to give you a contour that's just barely tangent to the constraint curve.

If that seems unfamiliar, definitely take a look at the videos preceding this one. But just to kind of continue the review, this gave us the really nice property that you look at the gradient vector for the thing you're trying to maximize, R, and that's going to be proportional to the gradient vector for the constraint function for this B. So, the gradient of B.

This is because gradients are perpendicular to contour lines. Again, this should feel mostly like review at this point.

So the core idea was that we take this gradient of R and then make it proportional with some kind of proportionality constant, Lambda, to the gradient of B, to the gradient of the constraint function. Up till this point, this value Lambda has been wholly uninteresting; it's just been a proportionality constant, right, because we couldn't guarantee that the gradient of R is equal to the gradient of B. All we care about is that they're pointing in the same direction.

So we just had this constant sitting here, and all we really said is make sure it's not zero. But here, we're going to get to where this little guy actually matters.

So if you'll remember, in the last video, I introduced this function called the Lagrangian. The Lagrangian takes in multiple inputs; they'll be the same inputs that you have for your budget function and your revenue function, or more generally, the constraint and the thing you're trying to maximize. It takes in those same variables, but also as another one of its inputs, it takes in Lambda, so it is a higher dimensional function than both of these two because we've got this extra Lambda.

The way it's defined looks a little strange at first. It just seems kind of like this random hodgepodge of functions that we're putting together. But last time, I kind of walked through why this makes sense. You take the thing you're trying to maximize, and you subtract off this Lambda multiplied by the constraint function, which is B of those inputs, minus whatever this constant is here, right? I'm going to give it a name; I'm going to call this constant lowercase b.

So maybe we're thinking of it as $10,000, but it's whatever your actual budget is. So we think of that, and I'm just going to emphasize here that that's a constant, right? That this B is being treated as a constant right now. You know, we're thinking of H and S and Lambda all as these variables, and this gives us some multivariable function.

If you'll remember from the last video, the reason for defining this function is it gives us a really nice compact way to solve the constrained optimization problem. You set the gradient of L equal to zero, or really the zero vector. Right? It'll be a vector with three components here, and when you do that, you'll find some solution, right? You'll find some solution which I'll call H star, S star, and Lambda.

You give it that green Lambda color, Lambda star. You'll find some value that when you input this into the function, the gradient will equal zero. Of course, you might find multiple of these, right? You might find multiple solutions to this problem.

But what you do is for each one of them, you're going to take a look at H star and S star. Then you're going to plug those into the revenue function or the thing that you're trying to maximize, and typically, you only get a handful; you get a number that you could actually plug each one of them into the revenue function and you'll just check which one of them makes this function the highest.

Whatever the highest value this function can achieve, that is M star; that is the maximum possible revenue subject to this budget. But it's interesting that when you solve this, you get some specific value of Lambda, right? There is a specific Lambda star that will be associated with this solution.

Like I said, this turns out not to just be some dummy variable; it's going to carry information about how much we can increase the revenue if we increase that budget. And here, let me show you what I mean.

So we've got this M star, and I'll just write it again, M star here. And what that equals, I'm saying that's the maximum possible revenue. So that's going to be the revenue when you evaluate it at H star and S star. H star and S star, they are whatever the solution to this gradient of the Lagrangian equals zero equation is, right? You set this multivariable function equal to the zero vector, you solve when each of its partial derivatives equal zero, and you'll get some kind of solution.

So when you plug that solution in the revenue, that gives you the maximum possible revenue. But what we could do is consider this as a function of the budget. Now, this is the kind of thing that looks a little bit wacky if you're just looking at the formulas. But if you actually think about what it means in this context of kind of a revenue and a budget, I think it's actually pretty sensible.

Where really if we consider this B no longer to be constant but something that you could change, right? You're wondering, well, what if I had a $20,000 budget? Or what if I had a $115,000 budget? You want to ask the question, what happens as you change this B? Well, the maximizing value H star and S star, each one of those guys is going to depend on B, right? As you change what this constant is, it's going to change the values at which the gradient of the Lagrangian equals zero.

So I'm going to rewrite this function as the revenue evaluated at H star and S star, but now I'm going to consider that H star and S star each as functions of B, right? Because they depend on it in some way. As you change B, it changes the solution to this problem. It's very implicit, and it's kind of hard to think about, right? It's hard to think, okay, as I change this B, how much does that change H star?

Well, that depends on what the definition of R was and everything there, but in principle, in this context, I think it's quite intuitive. You have a maximum possible revenue, and that depends on what your budget is.

So what turns out to be a beautiful, absolutely beautiful magical fact is that this Lambda star is equal to the derivative of M star, the derivative of this maximum possible revenue with respect to B, with respect to the budget.

Let me just show you what that actually means, right? So if, for example, let's say you did all of your calculations and it turned out that Lambda star was equal to 2.3. You know, previously that just seemed like some dummy number that you ignore, and you just look at whatever the associate values here are. But if you plug this in the computer and you see Lambda star equals 2.3, what that means is for a tiny change in budget, like let's say your budget increases from $10,000 to $10,000.01.

It goes up to $10,000.01; you increase your budget by just a little bit, a little dB. Then the ratio of the change in the maximizing revenue to that dB is about 2.3. So what that would mean is increasing your budget by a dollar is going to increase M star over here; it would mean that M star increases by about, you know, $2.30 for every dollar that you increase your budget.

That's information you'd want to know, right? If you see that this Lambda star is a number bigger than one, you'd say, hey, maybe we should increase our budget. We increase it from $110,000 to $110,001, and we're making more money, so maybe as long as Lambda star is greater than one, you should keep doing whatever it takes to increase that budget.

So this fact is quite surprising, I think, and it seems like it totally comes out of nowhere. So what I'm going to do in the next video is prove this to you, is prove why this is true, why this Lambda star value happens to be the rate of change for the maximum value of the thing we're trying to maximize with respect to this constant, with respect to whatever constant you set your constraint function equal to.

For right now though, I just want you to kind of try to sit back and digest what this means in the context of this specific economic example. And even if you never looked into the proof and never understood there, I think this is an interesting and even useful tidbit of knowledge to have about Lagrange multipliers.

So with that, I'll see you in the next video.