2.5

chapter_02.tex

@@ -159,55 +159,104 @@ The minimization of the error signal $e[n]$ can by achieved by applying differen
 \end{itemize}
 As computaional efficiency is a key requirement for the implementation of real-time ANR on a low-power digital signal processor, the Least Mean Squares algorithm is chosen for the minimization of the error signal and therefore will be further explained in the following subchapter.
 
-\subsubsection{Use of Least Mean Squares algorithm in adaptive filtering}
+\subsubsection{The Wiener filter and Gradient Descent}
-Before the Least Mean Squares algorithm can be explained in detail, the Wiener filter and the concept of gradient descent have to be introduced.
+Before the Least Mean Squares algorithm can be explained in detail, the Wiener filter and the concept of gradient descent have to be introduced. \\ \\
 \begin{figure}[H]
     \centering
     \includegraphics[width=0.7\linewidth]{Bilder/fig_wien.jpg}
     \caption{Simple implementation of a Wien filter.}
     \label{fig:fig_wien}
 \end{figure}
-\noindent The Wiener filter, the base of many adaptive filter designs, is a statistical filter used to minimize the mean square error between a desired signal and the output of a linear filter. The output $y[n]$ of the Wiener filter is the sum of the weighted input samples, where the weights are represented by the filter coefficients.
+\noindent The Wiener filter, the base of many adaptive filter designs, is a statistical filter used to minimize the Mean Square Error between a target signal and the output of a linear filter. The output $y[n]$ of the Wiener filter is the sum of the weighted input samples, where the weights are represented by the filter coefficients.
 \begin{equation}
 \label{equation_wien}
  y[n] = w_0x[n] + w_1x[n-1] + ... + w_Mx[n-M] = \sum_{k=0}^{M} w_kx[n-k]
 \end{equation}
-The Wiener filter aims to adjust it´s coefficients to generate a filter output, which resembles the corruption-noise $n[n]$ contained in the target signal $d[n]$ as close as possible. After the filter output is substracted from the target signal, we recvieve the error signal $e[n]$, which represents the cleaned signal $š[n]$ after the noise-component has been removed.
+The Wiener filter aims to adjust it´s coefficients to generate a filter output, which resembles the corruption noise signal $n[n]$ contained in the target signal $d[n]$ as close as possible. After the filter output is substracted from the target signal, we recvieve the error signal $e[n]$, which represents the cleaned signal $š[n]$ after the noise component has been removed. For better unsderstanding, a simple Wiener filter with one coefficient shall be illustrated in the following mathematical approach, before the generalization to an n-dimensional filter is made.
 \begin{equation}
 \label{equation_wien_error}
  e[n] = d[n] - y[n] = d[n] - wx[n]
 \end{equation}
 If we square the error signal and calculate the expected value, we receive the Mean Squared Error $J$, mentioned in the previous chapter, which is the metric the Wiener filter aims to minimize by adjusting it´s coefficients $w$.
 \begin{equation}
-\label{equation_wien_error}
+\label{equation_j}
  J = E(e[n]^2) = E(d^2[n])-2wE(d[n]x[n])+w^2E(x^2[n]) = MSE 
 \end{equation}
-The termns contained in Equation \ref{equation_wien_error} can be further be defined as:
+The terms contained in Equation \ref{equation_j} can be further be defined as:
 \begin{itemize}
 \item $\sigma^2$ = $E(d^2[n])$: The expected value of the squared corrupted target signal - a constant term independent of the filter coefficients $w$.
 \item P = $E(d[n]x[n])$: The cross-correlation between the corrupted target signal and the noise reference signal - a measure of how similar these two signals are.
-\item R = $E(x^2[n])$: The auto-correlation of the noise reference signal - a measure of the signal's spectral power.
+\item R = $E(x^2[n])$: The auto-correlation (or serial-correlation) of the noise reference signal - a measure of the similarity of a signal with it´s delayed copy and therefore of the signal's spectral power.
 \end{itemize}
-For a large number of samples, Equation {\ref{equation_wien_error}} can therefore be further simplified and written as:
+Equation {\ref{equation_j}} can therefore be further simplified and written as:
 \begin{equation}
-\label{equation_wien_error_final}
+\label{equation_j_simple}
  J = \sigma^2 - 2wP + w^2R
 \end{equation}
-As every part of Equation \ref{equation_wien_error_final} beside $w^2$ is constant,  the MSE is a quadratic function of the filter coefficients $w$, offering a calculatable minimum. To find this minimum, we can calculate the derivative of $J$ with respect to $w$ and set it to zero:
+As every part of Equation \ref{equation_j_simple} beside $w^2$ is constant, $j$ is a quadratic function of the filter coefficients $w$, offering a calculatable minimum. To find this minimum, the derivative of $J$ with respect to $w$ can be calculated and set to zero:
 \begin{equation}
-\label{equation_gradient_j}
+\label{equation_j_gradient}
  \frac{dJ}{dw} = -2P + 2wR = 0
 \end{equation}
-Solving Equation \ref{equation_gradient_j} for $w$ delivers the equation to calculate the optimal coefficients for the Wiener filter::
+Solving Equation \ref{equation_j_gradient} for $w$ delivers the equation to calculate the optimal coefficients for the Wiener filter:
 \begin{equation}
-\label{equation_wien_optimal}
+\label{equation_w_optimal}
- w_{opt} = \frac{P}{R}
+ w_{opt} = {P}R^{-1}
 \end{equation}
-To find the optimal set of coefficients $w$ minimizing the Mean Squared Error $J$, we can apply the concept of gradient descent. Gradient descent is an iterative optimization algorithm used to find the minimum of a function by moving in the direction of the steepest descent, which is determined by the negative gradient of the function. In our case, we want to minimize the MSE $J$ by adjusting the filter coefficients $w$. The update rule for the coefficients using gradient descent can be expressed as:
+\begin{figure}[H]
+    \centering
+    \includegraphics[width=0.7\linewidth]{Bilder/fig_w_opt.jpg}
+    \caption{Minimum of the Mean Square Error J located at the optimcal coefficient w* \cite{source_dsp_ch9}}
+    \label{fig:fig_mse}
+\end{figure}
+\noindent If the Wiener filter now consists not out of one coefficient, but out of several coefficients, Equation \ref{equation_wien} can be written in a matrix form as
+\begin{equation}
+\label{equation_wien_matrix}
+ y[n] = \sum_{k=0}^{M} w_kx[n-k] = \textbf{W}^T\textbf{X}[n]
+\end{equation}
+where \textbf{X} is the input signal matrix and \textbf{W} the filter coefficient matrix. 
+\begin{align}
+\label{equation_input_vector}
+ \textbf{X}[n] = [x[n],x[n-1],...,x[n-M]]^T \\
+ \label{equation_coefficient_vector}
+ \textbf{W}[n] = [w_0,w_1,...,w_M]^T
+\end{align}
+Equation \ref{equation_j} can therefore also be rewritten in matrix form to:
+\begin{equation}
+\label{equation_j_matrix}
+ J = \sigma^2 - 2\textbf{W}^TP + \textbf{W}^TR\textbf{W}
+\end{equation}
+After settings the derivative of Equation \ref{equation_j_matrix} to zero and solving for $W$, we receive the optimal filter coefficient matrix:
+\begin{equation}
+\label{equation_w_optimal_matrix}
+ \textbf{W}_{opt} = PR^{-1}
+\end{equation}
+\noindent For a large filter, the numerical solution of Equation \ref{equation_w_optimal_matrix} can be computational expensive, as it involves the inversion of potential large matrix. Therefore, to find the optimal set of coefficients $w$, the concept of gradient descent, introduced by Widrow\&Stearns in 1985, can be applied. The gradient decent algortihm aims to to minimize the MSE $J$ iteratively sample by sample by adjusting the filter coefficients $w$ in small steps towards the direction of the steepest descent to find the optimal coefficients. The update rule for the coefficients using gradient descent can be expressed as
 \begin{equation}  
 \label{equation_gradient}
- w(n+1) = w(n) - \mu \nabla J(w(n)) 
+ w(n+1) = w(n) - \mu \frac{dJ}{dw} 
 \end{equation}
-\subsection{Signal flow diagram of an implanted cochlear implant system}
+where $\mu$ is the constant step size determining the rate of convergence. Figure \ref{fig:fig_w_opt} visualizes the concept of stepwise minimization of the MSE $J$ using gradient descent. After the derivative of $J$ with respect to $w$ r4aches zero, the optimal coefficients $w_{opt}$ are found and the coefficients are no longer updated.
+\begin{figure}[H]
+    \centering
+    \includegraphics[width=0.9\linewidth]{Bilder/fig_gradient.jpg}
+    \caption{Vizualization of the steepest decent alorithm used on the Mean Squared Error. \cite{source_dsp_ch9}}
+    \label{fig:fig_w_opt}
+\end{figure}
+\subsubsection{The Least Mean Squares algorithm}
+The given approach of the steepest decent algorithm in the subchapter above still involves the calculation of the derivative of the MSE $\frac{dJ}{dw}$, which is also a compuational expensive operation to calulate, as it requires knowledge of the statistical properties of the input signals (cross-correlation P and auto-correlation R). Therefore, in energy critical real-time applications, like the implementation of ANR on a low-power DSP, a sample-based aproxmation in form of a Least Mean Squares (LMS) algorithm is used instead. The LMS algorithm approximates the gradient of the MSE by using the instantaneous estimates of the cross-correlation and auto-correlation. To achieve this, we remove the statistical expectation out of the MSE $J$ and take the derivative to obtain a samplewise approximate of $\frac{dJ}{dw[n]}$. 
+\begin{align}
+\label{equation_j_lms}
+ J = e[n]^2 = (d[n]-wx[n])^2 \\
+ \label{equation_j_lms_final}
+ \frac{dJ}{dw[n]} = 2(d[n]-w[n]x[n])\frac{d(d[n])-w[n]x[n]}{dw[n]} = -2e[n]x[n]
+\end{align}
+The result of Equation \ref{equation_j_lms_final} can now be inserted into Equation \ref{equation_gradient} to receive the LMS update rule for the filter coefficients:
+\begin{equation}
+\label{equation_lms}
+ w[n+1] = w[n] + 2\mu e[n]x[n]
+\end{equation}
+The LMS algorithm therefore updates the filter coefficients $w[n]$ after every sample by adding a correction term, which is is calculated by the error signal $e[n]$ and the reference noise signal $x[n]$, scaled by the constant step size $\mu$. By iteratively applying the LMS algorithm, the filter coefficients converge towards the optimal values that minimize the mean squared error between the target signal and the filter output. When a predefined acceptable error level is reached, the adaptation process can be stopped to save computing power.\\ \\
+ \subsection{Signal flow diagram of an implanted cochlear implant system}
 \subsection{Derivation of the system’s transfer function based on the problem setup}
 \subsection{Example applications and high-level simulations using Python}

literature.bib

@@ -23,21 +23,21 @@
 
 @misc{source_dsp1,
   author = {Li Tan, Jean Jiang},
-  title = {Digital Signal Processing Fundamentals and Applications 2nd Ed},
+  title = {Digital Signal Processing Fundamentals and Applications 3rd Ed},
   howpublished = {Elsevier Inc.},
   year = {2013},
   note = {ISBN: 978-0-12-415893-1}
 }
 @misc{source_dsp_ch1,
   author = {Li Tan, Jean Jiang},
-  title = {Digital Signal Processing Fundamentals and Applications 2nd Ed},
+  title = {Digital Signal Processing Fundamentals and Applications 3rd Ed},
   howpublished = {Elsevier Inc.},
   year = {2013},
   note = {Chapter 1}
 }
 @misc{source_dsp_ch2,
   author = {Li Tan, Jean Jiang},
-  title = {Digital Signal Processing Fundamentals and Applications 2nd Ed},
+  title = {Digital Signal Processing Fundamentals and Applications 3rd Ed},
   howpublished = {Elsevier Inc.},
   year = {2013},
   note = {Chapter 2}
@@ -49,3 +49,10 @@
   year = {1960},
   note = {Pat. Nr. 2966549}
 }
+@misc{source_dsp_ch9,
+  author = {Li Tan, Jean Jiang},
+  title = {Digital Signal Processing Fundamentals and Applications 3rd Ed},
+  howpublished = {Elsevier Inc.},
+  year = {2013},
+  note = {Chapter 9}
+}