1. Inductive Logic Programming

2. Content • Introduction to ILP • Basic ILP techniques • An overview of the different ILP systems • The application field of ILP • Summary

3. Introduction to ILP • Inductive logic programming (ILP) =Inductive concept learning (I) Logic Programming (LP) • Goal: • Develop a theoretical framework for induction • Build practical algorithms for inductive learning of relational concepts described in the form of logic programs • Background: • ILP theory based on the theory of LP • ILP algorithms based on experimental and application oriented ML research • Motivation: • Use of an expressive representational formalism as proportional logic • Use background knowledge in learning (in AI the use of domain knowledge is essential for achieving intelligent behaviour)

4. Introduction to ILP 2 • Inductive learning with background knowledge:Given a set of training examples E and background knowledge B find a hypothesis H, expressed in some concept description language L, such that H is complete and consistent with respect to the background knowledge B and the examples E • A hypothesis H is complete with regard to the background knowledge B and examples E if it covers all the positive examples i.e., if • A hypothesis H is consistent with respect to the background knowledge B and examples E if it covers none of the negative examples i.e.,

5. Introduction to ILP 3

6. Introduction to ILP 4 • Example: The task is to define the target relation daughter(X,Y) • Background knowledge consists of ground facts about the predicates female(X) and parent(Y,X):parent(ann, mary) female(ann)parent(ann, tom) female(marry)parent(tom, eve) female(eve)parent(tom, ian) • Training examples:+: daughter(marry, ann) daughter(eve, tom)-: daughter(tom, ann) daughter(eve, ann) • Possible target relation: • Here the target relation is:

7. Introduction to ILP 5 • Dimension of ILP • Learning either a single concept or multiple concepts • Requires all the training examples to be given before the learning process (batch learners) or accepts examples one by one (incremental learners) • The learner may rely on an oracle to verify the validity of generalisation and/or classify examples generated by the learner (interactive; non interactive) • The learner may try to learn a concept from scratch or can accept an initial hypothesis (theory) which is then revised in the learning process. The latter system is called theory revision. • Existing ILP systems • Empirical ILP system: Batch non-interactive system that learns single predicates from scratch • Interactive ILP system: Interactive and incremental theory revision system that learns multiple predicates

8. Content • Introduction to ILP • Basic ILP techniques • Generalisation techniques • Specialisation techniques • An overview of the different ILP systems • The application field of ILP • Summary

9. -subsumption • STRUCTURING THE HYPOTHESIS SPACE: Introducing partial ordering into a set of clauses based on the -subsumption • Def: A substitution is a function from variables to terms. The application of a substitution to a W is obtained by replacing all occurences of each variable in W by the same term • Def:Let c and c' be two program clauses. Clause c -subsumes c' if there exits a substitution , such that • Def: Two clauses c and d are -subsumption equivalent if c -subsumes d and d -subsumes c. • Def: A clause is reduced if it is not -subsumption equivalent to any proper subset of itself.

10. -subsumption (2) • Example1: Let c be the clause: c = daughter(X,Y) parent(Y,X).A substitution applied to clause c is obtained by applying to each of its literal: c = daughter(mary, ann) parent(ann, mary). • Example2: Clause c -subsumes the clause c' = daughter(X,Y) female(X), parent(Y,X)under the empty substitution • Example3: Clause c -subsumes the clause c' = daughter(mary,ann) female(mary),parent(ann,mary),parent(ann,tom)under the substitution

11. -subsumption (3) • -subsumption introduces the syntactic notation of generality: • Clause c is at least as general as clause c' ( ), if c -subsumes c' • Clause c is more general than clause c' ( ), if holds and does not hold • c' is a refinement (specialisation) of c • c is a generalisationof c'

12. -subsumption (4) • -subsumption is important for learning: • It provides a generality order for hypotheses, thus structuring the hypothesis space • It can be used to prune large parts of the search space: • If generalising c to c' all the examples covered by cwill be also covered by c'This property is used to prune the search of more general clauses when e is a negative example: if c is inconsistent then all its generalisations will also be inconsistent. Hence, the generalisation of c do not need to be considered. • When specialising c to c' an example not covered by c will not be covered by any of its specialisations either.This property is used to prune the search of more specific clauses:if c does not cover a positive example none of its specialisation will do. Hence, the specialisations of c do not need to be considered.

13. -subsumption (5) • .Techniques based on the -subsumption: • Bottom-up: creating the least general generalisation from the training examples, relative to the background knowledge • Top-down searching of refinement graphs

14. Least General Generalisation • Properties of -subsumption: • If c -subsumes c' then c logically entails c', the reverse is not always true • The relation defines a lattice over the set of reduced clauses.This means that any two clauses have a least upper bound (lub) and a greatest lower bound (glb). • Def: The least general generalisation(lgg) of two reduced clauses c and c', denoted by lgg(c,c'), is the least upper bound of c and c' in the -subsumption lattice. • Example: • Let c and c' be two clauses:c= daughter(mary,ann) female(mary), parent(ann,mary).c'= daughter(eve,tom) female(eve), parent(tom,eve). • lgg of c and c': daughter(X,Y) female(X), parent(Y,X).

15. Least General Generalisation 2 • Computation of lgg with -subsumption: • lgg of terms • and V is a variable which represents • and at least one of s and t is a variable in this case, V is a variable which represents • Example: where V stands for lgg(a,b)

16. Least General Generalisation 3 • lgg of atoms • If atoms have the same predicate symbol p • lgg of literals • If and are atoms, then is computed as defined above • If both and are negative literals and then • If is a positive and is a negative literal, or vice versa,is undefined • Example:

17. Least General Generalisation 4 • lgg of clauseLet and . Then • Examples:If and thenwhere X stands for lgg(mary,eve) and Y stands for lgg(ann,tom)

18. Least General Generalisation 5 • Search lgg( , ) Input: and are atoms or literalsOutput: lgg( , ) = function lgg( , ): beginif head( ) head( ) then := ; return( )else := ; := whiledo find_position( , , , ); generate_variable ; substitute( , , , , )endwhile := ; return( )endifend

19. Least General Generalisation 6 Input: and are terms or literals Output: and terms procedure find_position( , , , ) begin ifthenreturn( , ) endif if ( is atomic or is atomic) then return( , ) endif ifthenreturn( , ) endif i:=1 while do find_position( , , , ) if thenreturn( , ) endif i:= i+1 endwhile end

20. Least General Generalisation 7 Input: and are terms or literals, and terms, X variable Output: and with substitution X for and procedure substitute( , , , , X) begin ifandthenreturn(X,X) endif ifthenreturn( , ) endif if ( is atomic or is atomic) thenreturn( , ) endif ; ; if thenreturn( , ) endif i:=1 whiledo substitute( , , , , X); i:= i+1 endwhile end

21. Generalisation techniques • Start from the most specific clause that covers a given positive example and then generalise the clause until it cannot be further generalised without covering negative examples. • Generalisation operator: Let L be a language bias, a generalisation operator maps a clause c to a set of clauses which are generalisations of c: • Generalisation operators perform two basic syntactic operations on a clause: • Apply an inverse substitution to the clause • Remove a literal from the body of the clause • Basic generalisation techniques: • Relative least generalisation (rlgg) • Inverse resolution

22. Relative Least Generalisation 1 • Relative least generalisation: The relative least generalisationrelativeto background knowledgeB K is the conjunction of ground facts and are positive example • Example: • Positive example: and and B as before where K is a conjunctionresult:

23. Relative Least Generalisation 2 Search for rlgg( , ) Input: and are two clauses in the form = { } and = { } Output: rlgg( , ) = function rlgg( , ): begin k := 1; l := 1; while k < n do while l < m do lgg( , , L); ; l := l+1 endwhile k := k+1 endwhile ; return( ) end

24. Inverse resolution • Basic idea: invert the resolution rule of deductive inference (Robinson) e.g., invert the SLD-Resolution proof procedure for definite programs • Example: Given the theory suppose we want to derive uproposition w resolves with to give v which is then resolved with derive u.

25. Inverse resolution 2 • Example: first order derivation tree for family example and andLet Suppose we want to derive the fact from

26. Inverse resolution 3 • Inverse resolution inverts the resolution process using the generalisation operator based on the inverting substitution • Given a W, an inverse substitution of a substitution is a function that maps terms in to variable such that • Example: • Take and the substitution : • Applying the inverse substitution the original clause is obtained • Example: Inverse substitution with places • Let and • Applying to W: • The specifies that the first occurrence of the subterm ann in the term is replaced by variable Xand the second occurrence is replaced by Y. • The use of places ensures that

27. Inverse resolution 4 • Example: Inverse Resolution • B consists of two clauses and • Let • The learner encounters the positive example: • The inverse resolution processes as follows: • It attempts to find which will together with entail and can be added to the current hypothesis Choosing the inverse resolution step generates becomes the current hypothesis H such that • It takes andBy computing using it generalise with respect to B, yieldingIn the H can be replaced by which together b entailsThe induced hypothesis is

28. Inverse resolution 4

29. Specialisation techniques • They search the hypothesis space in top-down manner, from general to specific hypotheses using -subsumption based on a specialisation operator (refinement operator) • Refinement operator: Given a language bias L, a refinement operator maps a clause c to a set of clause which are specialisations (refinements) of c • This operator typically computes only the set of minimum (most general) specialisations of a clause under -subsumptionIt employs two basic syntactic operations on a clause: • Apply a substitution to the clause • Add a literal to the body of the clause

30. Specialisation techniques (2) • Basic specialisation technique is top-down search of the refinement graph • Top-down learners start from the most general clauses and repeatedly refine them until they no longer cover negative examples

31. Specialisation techniques 2 • For a selected L and a given B the hypothesis space of program clauses is a lattice structured by the -subsumption generality ordering • In this lattice a refinement graph can be defined and used to direct the search from general to specific hypotheses • The refinement graph is a directed, acyclic graph in which nodes are program clauses and edges correspond to the basic refinement operators: substituting a variable with a term, adding a literal to the body of the clause. • First used Model Inference System (MIS, Shapiro 1983)

32. Specialisation techniques 3

33. Specialisation techniques 4 • Generic Top-Down-Algorithm:Input:E the set of positive examples, B the background knowledgeL the description languageOutput: Hypothesis Hprocedure top_down_ILP( E, B, H) begin ; repeat choose repeat find the best refinement ; untilC is acceptable ; until hypothesis H satisfies stopping criterionreturn( H)end

34. Content • Introduction to ILP • Basic ILP techniques • An overview of the different ILP systems • The application field of ILP • Summary

35. An overview of the different ILP systems • Prehistory(1970) • Plotkin lgg, rlgg 1970-1971 • Early enthousiasm (1975-1983) • Vere 1975,1977, Summers 1977, Kodratoff and Jouanaud 1979 • Shapiro 1981, 1983 • Dark ages (1983-87)

36. An overview of the different ILP systems 2 • Renaissance (1987- ...) • MARVIN - Sammut & Banereji 1986, DUCE Muggleton 1987 • Helft 1987 • QuMAS - Mozetic 1987 Linus Lavrac et. al. 1991 • CIGOL - Muggleton &Buntine 1988 • ML-SMART Bergadano Giordana & Saitta 1988 • CLINT - De Raedt & Bruynooghe 1988 • BLIP, MOBAL Morik, Wrobel et. al. 1988 1989 • GOLEM - Muggleton &Feng 1990 • FOIL - quinland 1990 mFOIL - Dzerowski 1991 FOCL - Brunk & Pazzani FFOIL - Quinland 1996 • CLAUDIEN De Raedt & Bruynooghe 1993 • PROGOL - Muggleton et al 1995 • FORS- Karalic & Bratko 1996 • TILDE - Blockeel & De Raedt 1997 • .... • Studies Comparing the inductive learning system Foil with its versions FOCL, mFOIL, FFOIL, FOIL-I, FOSSIL, FOIDL and IFOIL

37. Content • Introduction to ILP • Basic ILP techniques • An overview of the different ILP systems • The application field of ILP • Summary

38. The application field of ILP • Application areas: • Knowledge acquisition for expert systems • Knowledge discovery in databases • Scientific knowledge discovery • Logic programs synthesis and verification • Inductive data engineering • Successful application • Finite element mesh design • Structure-activity prediction for drug design • Protein secondary-structure prediction • Predicting mutagenicity of chemical compounds

39. Summary • Inductive logic programming (ILP) =Inductive concept learning (I) Logic Programming (LP) • Empirical vs. Interactive ILP systems • Generalisation vs. specialisation techniques