The equation for least squares solution for a linear fit looks as follows.

\(\displaystyle y=\theta_1+\theta_2x_i\)

Recall the formula for method of least squares.

\(\displaystyle Ax=b\)

\(\displaystyle x=(A^T\cdot A)^{-1}A^T\cdot b\)

Remember when setting up the A matrix, that we have to fill one column full of ones.

\(\displaystyle A=\begin{bmatrix} 1 & 0 \\ 1 & 1\\ 1 & 2\\ 1 & 3\\ 1& 4\\ 1& 5 \end{bmatrix}\)

\(\displaystyle B=\begin{bmatrix} 1 \\ 2\\ 3\\ 4\\ 5\\10 \end{bmatrix}\)

To make things simpler, lets make \(\displaystyle C=A^T\cdot A\), and \(\displaystyle D=A^T\cdot b\)

\(\displaystyle C=\begin{bmatrix} 1 & 1&1 &1 &1 &1 \\ 0 & 1& 2& 3& 4& 5 \end{bmatrix}.\begin{bmatrix} 1&0 \\ 1& 1\\ 1& 2\\ 1& 3\\ 1& 4\\ 1& 5 \end{bmatrix}\)

\(\displaystyle C=\begin{bmatrix} 6 & 15 \\ 15 & 55 \end{bmatrix}\)

Now we need to solve for the inverse, we can do this simply by doing the following. We flip the sign on the off diagonal, and change the spots on the main diagonal, then we multiply by \(\displaystyle \frac{1}{\det(C)}\).

\(\displaystyle \det(C)=55(6)-(15)(15)=105\)

\(\displaystyle C^{-1}=\frac{1}{105}\begin{bmatrix}55 & -15 \\ -15 & 6 \end{bmatrix}\)

\(\displaystyle D=\begin{bmatrix} 1&1 &1 &1 &1 & 1\\ 0&1 &2 &3 &4 &5 \end{bmatrix} . \begin{bmatrix} 1\\ 2\\ 3\\ 4\\ 5\\ 10\\ \end{bmatrix}\)

\(\displaystyle D=\begin{bmatrix} 25\\ 90 \end{bmatrix}\)

\(\displaystyle x=C^{-1}\cdot D=\frac{1}{105}\begin{bmatrix} 55& -15\\ -15& 6 \end{bmatrix} \cdot \begin{bmatrix} 25\\ 90 \end{bmatrix}=\begin{bmatrix} \frac{5}{21}\\ \\ \frac{11}{7} \end{bmatrix}\)

\(\displaystyle \begin{align*}&\text{When the rows of a matrix outnumber the columns in an equation,}\\&\text{there are a greater number of equations than variables. In order}\\&\text{to approximate the values of these variables, a least-squares}\\&\text{approach is appropriate. This is done by multiplying each side}\\&\text{of the equation by the transpose of the equation matrix, and}\\&\text{solving the resulting equation:}\\&Ax=B\\&A^{T}Ax=A^{T}B\\&(A^{T}A)^{-1}(A^{T}A)x=(A^{T}A)^{-1}A^{T}B\\&x=(A^{T}A)^{-1}A^{T}B\\&\text{Using this, we can approximate our values:}\\&\begin{bmatrix}-12&13&-5\\9&14&-16\end{bmatrix}\begin{bmatrix}-12&9\\13&14\\-5&-16\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}-12&13&-5\\9&14&-16\end{bmatrix}\begin{bmatrix}-100\\-5\\14\end{bmatrix}\\&\begin{bmatrix}338&154\\154&533\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}1065\\-1194\end{bmatrix}\\&\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}4.80\\-3.63\end{bmatrix}\end{align*}\)

\(\displaystyle \begin{align*}&\text{When the rows of a matrix outnumber the columns in an equation,}\\&\text{there are a greater number of equations than variables. In order}\\&\text{to approximate the values of these variables, a least-squares}\\&\text{approach is appropriate. This is done by multiplying each side}\\&\text{of the equation by the transpose of the equation matrix, and}\\&\text{solving the resulting equation:}\\&Ax=B\\&A^{T}Ax=A^{T}B\\&(A^{T}A)^{-1}(A^{T}A)x=(A^{T}A)^{-1}A^{T}B\\&x=(A^{T}A)^{-1}A^{T}B\\&\text{Using this, we can approximate our values:}\\&\begin{bmatrix}15&-7&-3\\15&-12&9\end{bmatrix}\begin{bmatrix}15&15\\-7&-12\\-3&9\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}15&-7&-3\\15&-12&9\end{bmatrix}\begin{bmatrix}-120\\-4\\-180\end{bmatrix}\\&\begin{bmatrix}283&282\\282&450\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}-1232\\-3372\end{bmatrix}\\&\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}8.29\\-12.69\end{bmatrix}\end{align*}\)

\(\displaystyle \begin{align*}&\text{When the rows of a matrix outnumber the columns in an equation,}\\&\text{there are a greater number of equations than variables. In order}\\&\text{to approximate the values of these variables, a least-squares}\\&\text{approach is appropriate. This is done by multiplying each side}\\&\text{of the equation by the transpose of the equation matrix, and}\\&\text{solving the resulting equation:}\\&Ax=B\\&A^{T}Ax=A^{T}B\\&(A^{T}A)^{-1}(A^{T}A)x=(A^{T}A)^{-1}A^{T}B\\&x=(A^{T}A)^{-1}A^{T}B\\&\text{Using this, we can approximate our values:}\\&\begin{bmatrix}-13&9&11\\13&9&-11\end{bmatrix}\begin{bmatrix}-13&13\\9&9\\11&-11\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}-13&9&11\\13&9&-11\end{bmatrix}\begin{bmatrix}-109\\-168\\46\end{bmatrix}\\&\begin{bmatrix}371&-209\\-209&371\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}411\\-3435\end{bmatrix}\\&\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}-6.02\\-12.65\end{bmatrix}\end{align*}\)

\(\displaystyle \begin{align*}&\text{When the rows of a matrix outnumber the columns in an equation,}\\&\text{there are a greater number of equations than variables. In order}\\&\text{to approximate the values of these variables, a least-squares}\\&\text{approach is appropriate. This is done by multiplying each side}\\&\text{of the equation by the transpose of the equation matrix, and}\\&\text{solving the resulting equation:}\\&Ax=B\\&A^{T}Ax=A^{T}B\\&(A^{T}A)^{-1}(A^{T}A)x=(A^{T}A)^{-1}A^{T}B\\&x=(A^{T}A)^{-1}A^{T}B\\&\text{Using this, we can approximate our values:}\\&\begin{bmatrix}20&-9&14\\-19&-18&-10\end{bmatrix}\begin{bmatrix}20&-19\\-9&-18\\14&-10\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}20&-9&14\\-19&-18&-10\end{bmatrix}\begin{bmatrix}-8\\11\\37\end{bmatrix}\\&\begin{bmatrix}677&-358\\-358&785\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}259\\-416\end{bmatrix}\\&\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}0.13\\-0.47\end{bmatrix}\end{align*}\)

\(\displaystyle \begin{align*}&\text{When the rows of a matrix outnumber the columns in an equation,}\\&\text{there are a greater number of equations than variables. In order}\\&\text{to approximate the values of these variables, a least-squares}\\&\text{approach is appropriate. This is done by multiplying each side}\\&\text{of the equation by the transpose of the equation matrix, and}\\&\text{solving the resulting equation:}\\&Ax=B\\&A^{T}Ax=A^{T}B\\&(A^{T}A)^{-1}(A^{T}A)x=(A^{T}A)^{-1}A^{T}B\\&x=(A^{T}A)^{-1}A^{T}B\\&\text{Using this, we can approximate our values:}\\&\begin{bmatrix}14&5&5\\-16&-18&-14\end{bmatrix}\begin{bmatrix}14&-16\\5&-18\\5&-14\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}14&5&5\\-16&-18&-14\end{bmatrix}\begin{bmatrix}-92\\-165\\-168\end{bmatrix}\\&\begin{bmatrix}246&-384\\-384&776\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}-2953\\6794\end{bmatrix}\\&\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}7.31\\12.37\end{bmatrix}\end{align*}\)

\(\displaystyle \begin{align*}&\text{When the rows of a matrix outnumber the columns in an equation,}\\&\text{there are a greater number of equations than variables. In order}\\&\text{to approximate the values of these variables, a least-squares}\\&\text{approach is appropriate. This is done by multiplying each side}\\&\text{of the equation by the transpose of the equation matrix, and}\\&\text{solving the resulting equation:}\\&Ax=B\\&A^{T}Ax=A^{T}B\\&(A^{T}A)^{-1}(A^{T}A)x=(A^{T}A)^{-1}A^{T}B\\&x=(A^{T}A)^{-1}A^{T}B\\&\text{Using this, we can approximate our values:}\\&\begin{bmatrix}0&-2&3\\16&-18&5\end{bmatrix}\begin{bmatrix}0&16\\-2&-18\\3&5\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}0&-2&3\\16&-18&5\end{bmatrix}\begin{bmatrix}-73\\158\\-35\end{bmatrix}\\&\begin{bmatrix}13&51\\51&605\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}-421\\-4187\end{bmatrix}\\&\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}-7.82\\-6.26\end{bmatrix}\end{align*}\)

\(\displaystyle \begin{align*}&\text{When the rows of a matrix outnumber the columns in an equation,}\\&\text{there are a greater number of equations than variables. In order}\\&\text{to approximate the values of these variables, a least-squares}\\&\text{approach is appropriate. This is done by multiplying each side}\\&\text{of the equation by the transpose of the equation matrix, and}\\&\text{solving the resulting equation:}\\&Ax=B\\&A^{T}Ax=A^{T}B\\&(A^{T}A)^{-1}(A^{T}A)x=(A^{T}A)^{-1}A^{T}B\\&x=(A^{T}A)^{-1}A^{T}B\\&\text{Using this, we can approximate our values:}\\&\begin{bmatrix}1&-6&1\\-17&1&-13\end{bmatrix}\begin{bmatrix}1&-17\\-6&1\\1&-13\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}1&-6&1\\-17&1&-13\end{bmatrix}\begin{bmatrix}188\\149\\19\end{bmatrix}\\&\begin{bmatrix}38&-36\\-36&459\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}-687\\-3294\end{bmatrix}\\&\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}-26.87\\-9.28\end{bmatrix}\end{align*}\)

\(\displaystyle \begin{align*}&\text{When the rows of a matrix outnumber the columns in an equation,}\\&\text{there are a greater number of equations than variables. In order}\\&\text{to approximate the values of these variables, a least-squares}\\&\text{approach is appropriate. This is done by multiplying each side}\\&\text{of the equation by the transpose of the equation matrix, and}\\&\text{solving the resulting equation:}\\&Ax=B\\&A^{T}Ax=A^{T}B\\&(A^{T}A)^{-1}(A^{T}A)x=(A^{T}A)^{-1}A^{T}B\\&x=(A^{T}A)^{-1}A^{T}B\\&\text{Using this, we can approximate our values:}\\&\begin{bmatrix}14&-4&-9\\-8&-5&-2\end{bmatrix}\begin{bmatrix}14&-8\\-4&-5\\-9&-2\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}14&-4&-9\\-8&-5&-2\end{bmatrix}\begin{bmatrix}-161\\188\\30\end{bmatrix}\\&\begin{bmatrix}293&-74\\-74&93\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}-3276\\288\end{bmatrix}\\&\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}-13.01\\-7.26\end{bmatrix}\end{align*}\)

\(\displaystyle \begin{align*}&\text{When the rows of a matrix outnumber the columns in an equation,}\\&\text{there are a greater number of equations than variables. In order}\\&\text{to approximate the values of these variables, a least-squares}\\&\text{approach is appropriate. This is done by multiplying each side}\\&\text{of the equation by the transpose of the equation matrix, and}\\&\text{solving the resulting equation:}\\&Ax=B\\&A^{T}Ax=A^{T}B\\&(A^{T}A)^{-1}(A^{T}A)x=(A^{T}A)^{-1}A^{T}B\\&x=(A^{T}A)^{-1}A^{T}B\\&\text{Using this, we can approximate our values:}\\&\begin{bmatrix}-13&-12&-2\\12&1&19\end{bmatrix}\begin{bmatrix}-13&12\\-12&1\\-2&19\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}-13&-12&-2\\12&1&19\end{bmatrix}\begin{bmatrix}-103\\-178\\-16\end{bmatrix}\\&\begin{bmatrix}317&-206\\-206&506\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}3507\\-1718\end{bmatrix}\\&\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}12.04\\1.51\end{bmatrix}\end{align*}\)