Introduction to Radial Basis Function

1. Introduction to Radial Basis Function Mark J. L. Orr

2. Radial Basis Function Networks • Linear model

3. Radial functions • Gassian RBF:c : center, r : radius • monotonically decreases with distance from center • Multiquadric RBF • monotonically increases with distance from center

4. Gaussian RBF multiqradric RBF

5. Least Squares • model • training data : {(x1, y1), (x2, y2), …, (xp, yp)} • minimize the sum-squared-error

6. Example • Sample points (noisy) from the curve y = x : {(1, 1.1), (2, 1.8), (3, 3.1)} • linear model : f(x) = w1h1(x) + w2h2(x),where h1(x) = 1, h2(x) = x • estimate the coefficient w1, w2

7. f(x) = x

8. New model : f(x) = w1h1(x) + w2h2(x) + w3h3(x)where h1(x) = 1, h2(x) = x, h3(x) = x2

9. absorb all the noise : overfit • If the model is too flexible, it will fit the noise • If it is too inflexible, it will miss the target

10. The optimal weight vector • model • sum-squared-error • cost function : weight penalty term is added

15. Example • Sample points (noisy) from the curve y = x : {(1, 1.1), (2, 1.8), (3, 3.1)} • linear model : f(x) = w1h1(x) + w2h2(x),where h1(x) = 1, h2(x) = x • estimate the coefficient w1, w2

17. The projection matrix • At the optimal weight:the value of cost function C = yTPythe sum-squared-error S = yTP2y

18. Model selection criteria • estimates of how well the trained model will perform on future input • standard tool : cross validation • error variance

19. Cross validation • leave-one-out (LOO) cross-validation • generalized cross-validation

20. Ridge regression • mean-squared-error

21. Global ridge regression • Use GCV • re-estimation formula • initialize  • re-estimate , until convergence

22. Local ridge regression • research problem

23. Example

24. Selection the RBF • forward selection • starts with an empty subset • added one basis function at a time • most reduces the sum-squared-error • until some chosen criterion stops • backward elimination • starts with the full subset • removed one basis function at a time • least increases the sum-squared-error • until the chosen criterion stops decreasing