Notes on Diffy Qs: Differential Equations for Engineers

Given two (column) vectors in ${\mathbb{R}}^n\text{,}$ we define the (standard) inner product as the dot product:

Why do we seemingly give a new notation for the dot product ($⟨\overrightarrow{x},\overrightarrow{y}⟩$ instead of just $\overrightarrow{x}\cdot \overrightarrow{y}$)? Because there are other possible inner products, which are not the dot product, although we will not worry about others here. An inner product can even be defined on spaces of functions as we do in Chapter 4:

But we digress.

The standard inner product gives the euclidean length:

How does it give angles?

You may recall from multivariable calculus, that in two or three dimensions, the standard inner product (the dot product) gives you the angle between the vectors:

That is, $\theta$ is the angle that $\overrightarrow{x}$ and $\overrightarrow{y}$ make when they are based at the same point.

To compute the angle we compute

Our angles are always in radians. We are computing the cosine of the angle, which is really the best we can do. Given two vectors at an angle $\theta \text{,}$ we can give the angle as $-\theta \text{,}$ $2\pi -\theta \text{,}$ etc., see Figure A.5. Fortunately, $\cos \theta =\cos (-\theta )=\cos (2\pi -\theta )\text{.}$ If we solve for $\theta$ using the inverse cosine ${\cos }^{-1}\text{,}$ we can just decree that $0\le \theta \le \pi \text{.}$

As we said, the most important angle is the right angle. A right angle is ${}^{\pi }/{}_2$ radians, and $\cos ({}^{\pi }/{}_2)=0\text{,}$ so the formula is particularly easy in this case. We say vectors $\overrightarrow{x}$ and $\overrightarrow{y}$ are orthogonal if they are at right angles, that is if

The vectors $(1,0,0,1)$ and $(1,2,3,-1)$ are orthogonal. So are $(1,1)$ and $(1,-1)\text{.}$ However, $(1,1)$ and $(1,2)$ are not orthogonal as their inner product is $3$ and not 0.

To express one vector in terms of an orthogonal basis, we need to first project one vector onto another. Given a nonzero vector $\overrightarrow{v}\text{,}$ we define the orthogonal projection of $\overrightarrow{w}$ onto $\overrightarrow{v}$ as

For the geometric idea, see Figure A.6. That is, we find the “shadow of $\overrightarrow{w}$” on the line spanned by $\overrightarrow{v}$ if the direction of the sun’s rays were exactly perpendicular to the line. Another way of thinking about it is that the tip of the arrow of ${\mathrm{proj}}_{\overrightarrow{v}}(\overrightarrow{w})$ is the closest point on the line spanned by $\overrightarrow{v}$ to the tip of the arrow of $\overrightarrow{w}\text{.}$ In terms of euclidean distance, $\overrightarrow{u}={\mathrm{proj}}_{\overrightarrow{v}}(\overrightarrow{w})$ minimizes the distance $\Vert \overrightarrow{w}-\overrightarrow{u}\Vert$ among all vectors $\overrightarrow{u}$ that are multiples of $\overrightarrow{v}\text{.}$ Because of this, this projection comes up often in applied mathematics in all sorts of contexts we cannot solve a problem exactly: We cannot always solve “Find $\overrightarrow{w}$ as a multiple of $\overrightarrow{v}\text{,}$” but ${\mathrm{proj}}_{\overrightarrow{v}}(\overrightarrow{w})$ is the best “solution.”

The formula follows from basic trigonometry. The length of ${\mathrm{proj}}_{\overrightarrow{v}}(\overrightarrow{w})$ should be $\cos \theta$ times the length of $\overrightarrow{w}\text{,}$ that is $(\cos \theta )\Vert \overrightarrow{w}\Vert \text{.}$ We take the unit vector in the direction of $\overrightarrow{v}\text{,}$ that is, $\frac{\overrightarrow{v}}{\Vert \overrightarrow{v}\Vert }$ and we multiply it by the length of the projection. In other words,

As we said, a basis ${\overrightarrow{v}}_1,{\overrightarrow{v}}_2,\dots ,{\overrightarrow{v}}_n$ is an orthogonal basis if all vectors in the basis are orthogonal to each other, that is, if

for all choices of $j$ and $k$ where $j\ne k$ (a nonzero vector cannot be orthogonal to itself). A basis is furthermore called an orthonormal basis if all the vectors in a basis are also unit vectors, that is, if all the vectors have magnitude 1. For example, the standard basis $\{(1,0,0),(0,1,0),(0,0,1)\}$ is an orthonormal basis of ${\mathbb{R}}^3\text{:}$ Any pair is orthogonal, and each vector is of unit magnitude.

First suppose that $\overrightarrow{x}$ is in the span. Then it is the sum of the orthogonal projections:

In other words, if we want to write $\overrightarrow{x}=a_1{\overrightarrow{v}}_1+a_2{\overrightarrow{v}}_2+\cdots +a_n{\overrightarrow{v}}_n\text{,}$ then

Another way to derive this formula is to work in reverse. Suppose that $\overrightarrow{x}=a_1{\overrightarrow{v}}_1+a_2{\overrightarrow{v}}_2+\cdots +a_n{\overrightarrow{v}}_n\text{.}$ Take an inner product with ${\overrightarrow{v}}_j\text{,}$ and use the properties of the inner product:

As the basis is orthogonal, then $⟨{\overrightarrow{v}}_k,{\overrightarrow{v}}_j⟩=0$ whenever $k\ne j\text{.}$ That means that only one of the terms, the $j^{\text{th}}$ one, on the right-hand side is nonzero and we get

Solving for $a_j$ we find $a_j=\frac{⟨\overrightarrow{x},{\overrightarrow{v}}_j⟩}{⟨{\overrightarrow{v}}_j,{\overrightarrow{v}}_j⟩}$ as before.

If the basis is orthonormal rather than orthogonal, then all the denominators are one. It is easy to make a basis orthonormal—divide all the vectors by their size. If you want to decompose many vectors, it may be better to find an orthonormal basis. In the example above, the orthonormal basis we would thus create is

Then the computation would have been

As we said above, the formula still works even if $\overrightarrow{x}$ is not in the subspace, although then it does not get us the vector $\overrightarrow{x}$ but its projection. More concretely, suppose that $S$ is a subspace that is the span of ${\overrightarrow{v}}_1,{\overrightarrow{v}}_2,\dots ,{\overrightarrow{v}}_n$ and $\overrightarrow{x}$ is any vector. Let ${\mathrm{proj}}_S(\overrightarrow{x})$ be the vector in $S$ that is the closest to $\overrightarrow{x}\text{.}$ Then

We start with a basis of vectors ${\overrightarrow{v}}_1,{\overrightarrow{v}}_2,\dots ,{\overrightarrow{v}}_n\text{.}$ We construct an orthogonal basis ${\overrightarrow{w}}_1,{\overrightarrow{w}}_2,\dots ,{\overrightarrow{w}}_n$ as follows.

What we do is at the $k^{\text{th}}$ step, we take ${\overrightarrow{v}}_k$ and we subtract the projection of ${\overrightarrow{v}}_k$ to the subspace spanned by ${\overrightarrow{w}}_1,{\overrightarrow{w}}_2,\dots ,{\overrightarrow{w}}_{k-1}\text{.}$