Homework Assignment 1

Deadline: March 2, 2025. Your answer to the assignment must be written by hand and uploaded as a pdf to “Assignments” on Learn. Read the rules here: Homework Assignments.

Remember, all answers must be justified and well-reasoned.

Exercise 1: Tangent Vector to a Level Set is Orthogonal to the Gradient

Consider a differentiable function

\[\begin{equation*} g: \mathbb{R}^n \to \mathbb{R}, \end{equation*}\]

and let

\[\begin{equation*} \pmb{r}: [a,b] \,\to\, \mathbb{R}^n \end{equation*}\]

be a differentiable vector function, whose image space is a subset of the level set:

\[\begin{equation*} \{(x_1, x_2, \ldots, x_n)\in\mathbb{R}^n \,\,|\,\, g(x_1,x_2,\ldots,x_n) = c\}, \end{equation*}\]

for a constant \(c\). In other words, we have:

\[\begin{equation*} g\bigl(\pmb{r}(t)\bigr) \;=\; c \quad \text{for all } t \in [a,b]. \end{equation*}\]

We consider \(\pmb{r}\) as a parametrization of a curve.

Question a

Explain why

\[\begin{equation*} \frac{d}{dt}\,g\bigl(\pmb{r}(t)\bigr) \;=\; 0 \quad \text{for all } t \in ]a,b[. \end{equation*}\]

Question b

Show using the chain rule that

\[\begin{equation*} \langle \nabla g\bigl(\pmb{r}(t)\bigr), \pmb{r}'(t) \rangle = 0 \quad \text{for all } t \in ]a,b[. \end{equation*}\]

Question c

Explain how the result from the previous question implies that the gradient \(\nabla g\) is perpendicular (orthogonal) to the level surface

\[\begin{equation*} g(x_1,x_2,\ldots,x_n) = c \end{equation*}\]

at the point \(\pmb{r}(t)\) for every \(t \in ]a,b[\).

Exercise 2: The Chain Rule on a Function with no a Functional Expression

We consider two differentiable vector functions:

\[\begin{equation*} \pmb{f} : \mathbb{R}^4 \to \mathbb{R}^3 \quad \text{and} \quad \pmb{g} : \mathbb{R}^3 \to \mathbb{R}^2. \end{equation*}\]

We are informed that \(\pmb{f}\) has the functional expression:

\[\begin{equation*} \pmb{f}(x_1,x_2,x_3,x_4)=\Bigl(x_1^2 - x_2,\; \sin(x_3) + x_4,\; x_1x_4 - x_2x_3\Bigr). \end{equation*}\]

Furthermore, we are informed that the Jacobian matrix \(\pmb{J}_{\pmb{g}}(\pmb{y}) \in \mathbb{R}^{2 \times 3}\) of \(\pmb{g}\) is given by:

\[\begin{equation*} \pmb{J}_{\pmb{g}} (y_1,y_2,y_3)= \begin{bmatrix} 2y_1 - y_2y_3 & 3y_2 - y_1y_3 & -y_1y_2 + 4y_3 \\[1mm] 1+\mathrm e^{y_1} & y_2^2-2 & \sin(y_3) \end{bmatrix}. \end{equation*}\]

We wish to determine the Jacobian matrix of the composite function \(\pmb{g}\circ \pmb{f}\) at the point

\[\begin{equation*} (x_1,x_2,x_3,x_4)=(1,-1,\tfrac{\pi}{2},0). \end{equation*}\]

Question a

Determine \(\pmb{f}(1,-1,\tfrac{\pi}{2},0)\).

Question b

Determine te Jacobian matrix \(\pmb{J}_{\pmb{f}}(1,-1,\tfrac{\pi}{2},0)\) of the function \(\pmb{f}\) at the point \((1,-1,\tfrac{\pi}{2},0)\).

Question c

Argue that the composite function \(\pmb{g} \circ \pmb{f}\) is differentiable on \(\mathbb{R}^4\). Use the chain rule to determine the Jacobian matrix of the composite function \(\pmb{g} \circ \pmb{f}\) at the point \((1,-1,\tfrac{\pi}{2},0)\).

Exercise 3: Differentiability of a Piecewise Polynomial and of Softplus

Consider the functions \(f:\mathbb{R} \to \mathbb{R}\) and \(g:\mathbb{R} \to \mathbb{R}\) given by the functional expressions:

\[\begin{equation*} f(x)= \begin{cases} x & \text{for } x<0 \\ x^2 & \text{for } x \ge 0 \end{cases} \end{equation*}\]

and

\[\begin{equation*} g(x)=\ln(1+\mathrm e^x). \end{equation*}\]

The function \(g\) is the so-called Softplus function, which is an approximation of ReLU. It can, for example, be used as an activation function in neural networks to achieve differentiability, thereby facilitating the optimization process.

Question a

Explain why \(f\) is differentiable on \(\mathbb{R} \setminus \{0\}\). Provide a functional expression for the differential quotient \(f'\).

Question b

Explain why \(f\) is continuous but not differentiable at the point \(0\).

Question c

Argue that \(g\) is differentiable on the entire real line and find a functional expression for \(g'\).

Question d

In machine learning, ReLU and Softplus are typically applied to vectors. These vector functions are defined by applying the given scalar functions coordinate-wise. For example, the function \(\pmb{g}: \mathbb{R}^n \to \mathbb{R}^n\) is given by \(\pmb{g}(\pmb{x}) = [g(x_1), g(x_2), \dots, g(x_n)]\). Compute the Jacobian matrix \(\pmb{J}_{\pmb{g}}(\pmb{x})\) of the Softplus vector function.

Exercise 4: A new Inner Product

Consider \(\mathbb{R}^n\) with the usual inner product \(\langle \cdot, \cdot \rangle\). Let

• \(U \in \mathbb{C}^{n \times n}\) be a unitary matrix (meaning \(U^*U=I\)), and let

• \(\Lambda \in \mathbb{R}^{n \times n}\) be a diagonal matrix with strictly positive diagonal element, meaning \(\lambda_i>0\) for \(i=1,\ldots,n\).

Define

\[\begin{equation*} B = U\,\Lambda\,U^*. \end{equation*}\]

Furthermore define

\[\begin{equation*} A = U\,D\,U^*, \end{equation*}\]

where \(D\) is a diagonal matrix with the elements

\[\begin{equation*} d_i = \sqrt{\lambda_i}, \quad i=1,\ldots,n. \end{equation*}\]

Question a

Show that

\[\begin{equation*} A^*A = B , \quad B^* = B \quad \text{and} \quad A^* = A. \end{equation*}\]

Question b

Determine the inverse matrix \(A^{-1}\).

Question c

Define for the column vectors \(\pmb{x}, \pmb{y} \in \mathbb{R}^n\) the inner product

\[\begin{equation*} \langle \pmb{x}, \pmb{y} \rangle_B = \langle B\,\pmb{x}, \pmb{y} \rangle. \end{equation*}\]

Show that

\[\begin{equation*} \langle \pmb{x}, \pmb{y} \rangle_B = \langle A\,\pmb{x}, A\,\pmb{y} \rangle. \end{equation*}\]

Question d

Show that \(\langle \pmb{x}, \pmb{y} \rangle_B\) actually is an inner product on \(\mathbb{R}^n\). You may use, without proof, that the usual inner product \(\langle \pmb{x}, \pmb{y} \rangle\) is an inner product on \(\mathbb{R}^n\).

Question e

Let \(B = \operatorname{diag}(2,4)\). Provide two vectors \(\pmb{x}, \pmb{y} \in \mathbb{R}^2\) that are orthogonal to each other with respect to \(\langle \cdot, \cdot \rangle_B\). None of the vectors may be the zero vector in \(\mathbb{R}^2\).