Frame-based Kernel Learning Methods: Approximating Complex Functions with Regularity
This document presents a comprehensive framework for kernel learning utilizing frames in Hilbert spaces. It explores wavelet-based approximation, multiresolution schemes, and the construction of reproducing kernel Hilbert spaces (RKHS) adapted to specific function structures. The paper provides insights into semiparametric estimation approaches and multiscale approximation techniques that ensure stability and regularity in learning highly oscillating functions. The findings highlight practical applications and demonstrate the efficiency of frame-based methods in kernel learning.
Frame-based Kernel Learning Methods: Approximating Complex Functions with Regularity
E N D
Presentation Transcript
Frame, Reproducing Kernel and Learning Alain Rakotomamonjy Stéphane Canu Perception, Systèmes et Information Insa de Rouen, 76801 St Etienne du Rouvray France Alain.Rakoto,Stephane.Canu@insa-rouen.fr http://asi.insa-rouen.fr/~arakotom NIPS 2000 Workshop on Kernel methods
Motivations • Wavelet-based approximation (wavelet or ridgelet networks) are regularization networks? • Construction of multiresolution scheme of approximation • kernel adapted to the structures of function to be learned NIPS 2000 Workshop on Kernel methods
Motivations Ctd. • Frame based framework for learning Approximating highly oscillating structure Without losing regularity in smooth region NIPS 2000 Workshop on Kernel methods
Road Map • Introduction on Frame • From Frame to Kernels • From Frame kernels to learning • Conclusions and perspectives NIPS 2000 Workshop on Kernel methods
Frame : A definition • H : Hilbert Space dot product A sequence of elements of H is a frame of H if there exists A,B > O s.t A,B are the frame bounds NIPS 2000 Workshop on Kernel methods
Frame : definition Ctd. • Frame intepretation • Frame allows stable representation • as for all f in H Frame = "Basis" + linear dependency + redundancy being a dual frame of Fn in H NIPS 2000 Workshop on Kernel methods
Particular cases of Frame • Tight Frame • Frame with bounds s.t A=B • Orthonormal Basis • A=B=1 • Riesz Basis • Frame elements are linearly independent NIPS 2000 Workshop on Kernel methods
Examples of Frame • Tight Frame of IR2 • Frame of L2(IR) F2 F1 F3 Y is an admissible wavelet NIPS 2000 Workshop on Kernel methods
Road Map • Introduction on Frame • From Frame to Kernels • From Frame kernels to learning • Conclusions and perspectives NIPS 2000 Workshop on Kernel methods
Frameable RKHS • Condition for having a RKHS Suppose H is a Hilbert space of function and a frame of H H is a RKHS if On a frameable Hilbert Space, this is equivalent to The Reproducing Kernel is NIPS 2000 Workshop on Kernel methods
Construction of Frameable RKHS • A Practical way to build a RKHS • F is a Hilbert Space of function A finite set of F elements such that is a RKHS with {Fn} as frame elements NIPS 2000 Workshop on Kernel methods
Example of Frameable RKHS • frameable RKHS included in L2(IR) Fi : L2 function (e.g Fi is a wavelet) span {Fi}i=1…N is a RKHS Example 3 wavelets at same scale j span a RKHS with kernel NIPS 2000 Workshop on Kernel methods
Road Map • Introduction on Frame • From Frame to Kernels • From Frame kernels to learning • Conclusions and perspectives NIPS 2000 Workshop on Kernel methods
Semiparametric Estimation • Context Learning from training set (xi,yi)i=1..N Semiparametric framework One looks for the minimizer of the risk functional in a space H + span{Yi}i=1…m H being a RKHS Under general conditions, span{Yi}i=1…m : parametric hypothesis space NIPS 2000 Workshop on Kernel methods
Semiparametric Estimation • Parametric hyp. space is a frameable RKHS P is a frameable RKHS spanned by {Fn}, with P H, H RHKS Semiparametric estimation on H with P as a parametric hyp. space One looks for the minimizer in H of As spaces are orthogonal, backfitting is sufficient for estimating f* NIPS 2000 Workshop on Kernel methods
Semiparametric Estimation • Frame view point • H frameable • H defined by kernel K H= P + N P : Frameable RKHS, N : Frameable RKHS H N: "unknown component" to be regularized P N : due to linear dependency of frame P : "known component" not to regularized KN=KH-KP P : Frameable RKHS NIPS 2000 Workshop on Kernel methods
Multiscale approximation • H a frameable RKHS H is splitted in different spaces {Fi}i=1…m-1 and H0 And any space Hi or Fi is a RKHS Hi : Trend Spaces Fi : Details Spaces NIPS 2000 Workshop on Kernel methods
Resid. Resid. Resid. f* Multiscale Approximation Ctd. At each step j, trend obtained at step j-1 is decomposed in trend and details H H2 F2 H1 F1 H0 F0 NIPS 2000 Workshop on Kernel methods
Multiscale Approximation Ctd. • Validity • At each step, representer Theorem Hypothesis must be verified • Solution NIPS 2000 Workshop on Kernel methods
Illustration on toy problem Function to be learned Data xi : N points from the random sampling of [0, 10] Algorithm - SVM Regression - Multiscale Regularisation networks on Frameable RKHS Sin/Sinc based kernel Wavelet based kernel NIPS 2000 Workshop on Kernel methods
Results • N=902 • Results are averagerad over 300 experiments and normalized with regards to SVM performance Wavelet Kernel Sinc Kernel SVM 1 ± 0.096 0.9297 ± 0.312 0.5115 ± 0.098 L2 error 0.7252± 8.022 0.8280± 0.025 1 ± 0.028 NIPS 2000 Workshop on Kernel methods
Plots of typical results NIPS 2000 Workshop on Kernel methods
Road Map • Introduction on Frame • From Frame to Kernels • From Frame kernels to learning • Conclusions and perspectives NIPS 2000 Workshop on Kernel methods
Summary • new design of kernel based on frame elements • algorithm for multiscale learning • But • no explicit definition of kernel • Time-consuming NIPS 2000 Workshop on Kernel methods
Future work • Multidimensional extension • Tight Frame of multidimensional wavelet • Using a priori knowledge on the learning problem • How to choose the frame elements? • Theoretical justification and analysis of multiscale approximation NIPS 2000 Workshop on Kernel methods
