ADVANCED DIGITAL SIGNAL PROCESSING EENG 413

1. ADVANCED DIGITAL SIGNAL PROCESSING EENG 413

2. Contents • Unit I Parametric Methods for Power Spectrum Estimation • Unit II Non-Parametric Methods for Power Spectrum Estimation • Unit III Adaptive Signal Processing • Unit IV Multirate Signal Processing • Unit V Discrete Transforms

3. Reference Books • John G.Proakis, Dimitris G.Manobakis, “Digital Signal Processing, Principles, Algorithms and Applications”, Third edition, PHI, 2001. • Monson H.Hayes, “Statistical Digital Signal Processing and Modeling”, Wiley, 2002. • Roberto Crist, “Modern Digital Signal Processing”, Thomson Brooks/Cole, 2004. • Raghuveer. M. Rao, Ajit S.Bopardikar, “Wavelet Transforms, Introduction to Theory and applications”, Pearson Education, Asia, 2000. • K.P Soman, K.I Ramachnadran and N.G Reshmi, “Insights into wavelets: From theory to Practice”, 3rd Edition,PHI,2010.

4. Unit I Parametric Methods for Power Spectrum Estimation • Relationship between the auto correlation and the model parameters • Yule Walker method for the AR Model Parameters • Burg method for the AR model parameters • Unconstrained least square method for the AR model parameters • Sequential estimation methods for the AR model parameters

5. Introduction • What is Spectral Estimation? From a finite record of a stationary data sequence, estimate how the total power is distributed over frequencies , or more practically, over narrow spectral bands (frequency bins)

6. Spectral Estimation Methods Spectral Estimation Parametric Non Parametric Ex: Periodogram and Welch method AR, ARMA based Subspace Based (high-resolution) Model fitting based Ex: MUSIC and ESPRIT Ex: Least Squares AR: Autoregressive (all-pole IIR) ARMA: Autoregressive Moving Average (IIR) MUSIC: MUltiple SIgnal Classification ESPRIT: Estimation of Signal Parameters using Rotational Invariance Techniques

7. Spectral Estimation Methods • Classical (Nonparametric) Methods Ex. Pass the data through a set of band-pass filters and measure the filter output powers. • Parametric (Modern) Approaches Ex. Model the data as a sum of a few damped sinusoids and estimate their parameters. Trade-Offs: (Robustness vs. Accuracy) • Parametric Methods may offer better estimates if data closely agrees with assumed model • Otherwise, Nonparametric Methods may be better

8. Few Applications of Spectral Estimation • Speech • Formant estimation (for speech recognition) • Speech coding or compression • Radar and Sonar • Source localization with sensor arrays • Synthetic aperture radar imaging and feature extraction • Electromagnetics • Resonant frequencies of a cavity • Communications • Code-timing estimation in DS-CDMA systems

9. Power Spectrum • Deterministic signal x(t) • Assume Fourier transformX(f)exists • Power spectrum is square of absolute value of magnitude response (phase is ignored) • Multiplication in Fourier domain is convolution in time domain • Conjugation in Fourier domain is reversal and conjugation in time autocorrelation

10. x(t) 1 0 Ts t rx(t) Ts -Ts Ts t Autocorrelation • Autocorrelation ofx(t): • Slide x(t)against x*(t)instead of flip-and-slide • Maximum value at rx(0) if rx(0) is finite • Even symmetric, i.e. rx(t) = rx(-t) • Discrete-time: • Alternate definition:

11. Power Spectrum • Estimate spectrum if signal known at all time • Compute autocorrelation • Compute Fourier transform of autocorrelation • Autocorrelation of random signal n(t) • For zero-mean Gaussian random processn(t)with variances2

12. ESTIMATION • Method of Moment Estimation (MME) • Ordinary Least Squares (OLS) Estimation • Maximum Likelihood Estimation (MLE) • Least Squares Estimation • Conditional • Unconditional

13. THE METHOD OF MOMENT ESTIMATION • It is also known as Yule-Walker estimation. Easy but not efficient estimation method. Works for only AR models for large n. • BASIC IDEA: Equating sample moment(s) to population moment(s), and solve these equation(s) to obtain the estimator(s) of unknown parameter(s).

14. THE METHOD OF MOMENT ESTIMATION • Let nis the variance/covariance matrix of X with the given parameter values. • Yule-Walker for AR(p): Regress Xtonto Xt−1, . . ., Xt−p. • Durbin-Levinson algorithm with replaced by . • Yule-Walker for ARMA(p,q): Method of moments. Not efficient.

15. THE YULE-WALKER ESTIMATION • For a stationary (causal) AR(p)

16. THE YULE-WALKER ESTIMATION • To find the Yule-Walker estimators, we are using, • These are forecasting equations. • We can use Durbin-Levinson algorithm.

17. THE YULE-WALKER ESTIMATION • If • If {Xt} is an AR(p) process, Hence, we can use the sample PACF to test for AR order, and we can calculate approximate confidence intervals for the parameters.

18. THE YULE-WALKER ESTIMATION • If Xt is an AR(p) process, and n is large, • 100(1)% approximate confidence interval for j is

19. THE YULE-WALKER ESTIMATION • AR(1) Find the MME of . It is known that 1 = .

20. THE YULE-WALKER ESTIMATION • So, the MME of  is • Also, is unknown. • Therefore, using the variance of the process, we can obtain MME of .

21. THE YULE-WALKER ESTIMATION

22. THE YULE-WALKER ESTIMATION • AR(2) Find the MME of all unknown parameters. • Using the Yule-Walker Equations

23. THE YULE-WALKER ESTIMATION • So, equate population autocorrelations to sample autocorrelations, solve for 1 and 2.

24. THE YULE-WALKER ESTIMATION Using these we can obtain the MME of To obtain MME of , use the process variance formula.

25. THE YULE-WALKER ESTIMATION • AR(1) • AR(2)

26. THE YULE-WALKER ESTIMATION • MA(1) • Again using the autocorrelation of the series at lag 1, Choose the root so that the root satisfying the invertibility condition

27. THE YULE-WALKER ESTIMATION • For real roots, If , unique real roots but non-invertible. If , no real roots exists and MME fails. If , unique real roots and invertible.

28. THE YULE-WALKER ESTIMATION • This example shows that the MMEs for MA and ARMA models are complicated. • More generally, regardless of AR, MA or ARMA models, the MMEs are sensitive to rounding errors. They are usually used to provide initial estimates needed for a more efficient nonlinear estimation method. • The moment estimators are not recommended for final estimation results and should not be used if the process is close to being nonstationary or noninvertible.

29. THE MAXIMUM LIKELIHOOD ESTIMATION • Assume that • By this assumption we can use the joint pdf instead of which cannot be written as multiplication of marginal pdfs because of the dependency between time series observations.

30. MLE METHOD • For the general stationary ARMA(p,q) model or

31. MLE • The joint pdf of (a1,a2,…, an) is given by • Let Y=(Y1,…,Yn) and assume that initial conditions Y*=(Y1-p,…,Y0)’and a*=(a1-q,…,a0)’ are known.

32. MLE • The conditional log-likelihood function is given by Initial Conditions:

33. MLE • Then, we can find the estimators of =(1,…,p), =(1,…, q) and  such that the conditional likelihood function is maximized. Usually, numerical nonlinear optimization techniques are required. After obtaining all the estimators, where d.f.=  of terms used in SS   of parameters = (np)  (p+q+1) = n  (2p+q+1).

34. MLE • AR(1)

35. MLE The Jacobian will be

36. MLE • Then, the likelihood function can be written as

37. MLE • Hence, • The log-likelihood function:

38. MLE • Here, S*() is the conditional sum of squares and S() is the unconditional sum of squares. • To find the value of  where the likelihood function is maximized, • Then,

39. MLE • If we neglect ln(12), then MLE=conditional LSE. • If we neglect both ln(12) and , then

40. MLE • Asymptotically unbiased, efficient, consistent, sufficient for large sample sizes but hard to deal with joint pdf.

41. CONDITIONAL LEST SQUARES ESTIMATION • AR(1)

42. CONDITIONAL LSE • If the process mean is different than zero

43. CONDITIONAL LSE • MA(1) • Non-linear in terms of parameters • LS problem • S*() cannot be minimized analytically • Numerical nonlinear optimization methods like Newton-Raphson or Gauss-Newton,... *There are similar problem is ARMA case.

44. UNCONDITIONAL LSE • This nonlinear in . • We need nonlinear optimization techniques.

45. BURG METHOD • An order-recursive least-squares lattice method , based on the minimization of the forward and backward errors in linear predictors, with the constraint that the AR parameters satisfy the Levinson – Durbin recursion.

46. BURG METHOD • To derive the estimator, let the given data be x(n), n = 0, 1,………N-1 and let the forward and backward linear prediction estimates of order ‘m’ , be :-

47. BURG METHOD Forward error, Backward error, The least squares error is :-

48. BURG METHOD • This error is to be minimized by selecting the prediction coefficients , subject to the constraint that they satisfy the Levinson- Durbin recursion given by :- where is the mth reflection coefficient in the lattice filter realization.

49. BURG METHOD • The forward and backward prediction errors in terms of • reflection coefficients is given by : By substituting above equation into Levinson – Durbin Recursion and performing minimization w.r.t. reflection Coefficient ,we get :

50. BURG METHOD • is an estimate of the cross correlation between the forward and backward prediction errors. As the denominator term is simply the least- squares estimate of the forward and backward errors, , so • is an estimate of the total squared error .