Monte Carlo Semantics McPIET at RTE-4: Robust Inference and Logical - PowerPoint PPT Presentation

Monte Carlo Semantics McPIET at RTE-4: Robust Inference and Logical Pattern Processing Based on Integrated Deep and Shallow Semantics Richard Bergmair University of Cambridge Computer Laboratory Natural Language Information Processing Text Analysis Conference, Nov-17 2008

Desiderata for a Theory of RTE ◮ Does it describe the relevant aspects of the systems we have now ? ◮ Does it suggest ways of building better systems in the future ?

A System for RTE ◮ informativity : Can it take into account all available relevant information? ◮ robustness : Can it proceed on reasonable assumptions, where it is missing relevant information.

Current RTE Systems A spectrum between ◮ shallow inference (e.g. bag-of-words) ◮ deep inference (e.g. FOPC theorem proving, see Bos & Markert)

The Informativity/Robustness Tradeoff informativity robustness

The Informativity/Robustness Tradeoff informativity deep robustness

The Informativity/Robustness Tradeoff informativity deep shallow robustness

The Informativity/Robustness Tradeoff informativity deep intermediate shallow robustness

The Informativity/Robustness Tradeoff informativity ? robustness

Outline Informativity, Robustness & Graded Validity Propositional Model Theory & Graded Validity Shallow Inference: Bag-of-Words Encoding Deep Inference: Syllogistic Encoding Computation via the Monte Carlo Method

Outline Informativity, Robustness & Graded Validity Propositional Model Theory & Graded Validity Shallow Inference: Bag-of-Words Encoding Deep Inference: Syllogistic Encoding Computation via the Monte Carlo Method

Informative Inference. predicate/argument structures The cat chased the dog. ⊤ > → The dog chased the cat. monotonicity properties, upwards entailing Some ( grey X ) are Y ≥ ⊤ → Some X are Y Some X are Y ⊤ > → Some ( grey X ) are Y

Robust Inference. monotonicity properties, upwards entailing Some X are Y Some X are Y > → Some ( grey X ) are Y → Some ( clean ( grey X )) are Y graded standards of proof Socrates is a man Socrates is a man > → → Socrates is a man Socrates is mortal Socrates is a man Socrates is a man > → → Socrates is mortal Socrates is not a man

. . . classically (i) T ∪ { ϕ } | = ψ and T ∪ { ϕ } �| = ¬ ψ ; ENTAILED / valid (ii) T ∪ { ϕ } �| = ψ and T ∪ { ϕ } | = ¬ ψ ; CONTRADICTION / unsatisfiable (iii) T ∪ { ϕ } | = ψ and T ∪ { ϕ } | = ¬ ψ ; UNKNOWN / possible (iv) T ∪ { ϕ } �| = ψ and T ∪ { ϕ } �| = ¬ ψ . UNKNOWN / possible

. . . classically (i) T ∪ { ϕ } | = ψ and T ∪ { ϕ } �| = ¬ ψ ; ENTAILED / valid (ii) T ∪ { ϕ } �| = ψ and T ∪ { ϕ } | = ¬ ψ ; CONTRADICTION / unsatisfiable (iii) T ∪ { ϕ } | = ψ and T ∪ { ϕ } | = ¬ ψ ; UNKNOWN / possible (iv) T ∪ { ϕ } �| = ψ and T ∪ { ϕ } �| = ¬ ψ . UNKNOWN / possible

. . . classically (i) T ∪ { ϕ } | = ψ and T ∪ { ϕ } �| = ¬ ψ ; ENTAILED / valid (ii) T ∪ { ϕ } �| = ψ and T ∪ { ϕ } | = ¬ ψ ; CONTRADICTION / unsatisfiable (iii) T ∪ { ϕ } | = ψ and T ∪ { ϕ } | = ¬ ψ ; UNKNOWN / possible (iv) T ∪ { ϕ } �| = ψ and T ∪ { ϕ } �| = ¬ ψ . UNKNOWN / possible

. . . classically (i) T ∪ { ϕ } | = ψ and T ∪ { ϕ } �| = ¬ ψ ; ENTAILED / valid (ii) T ∪ { ϕ } �| = ψ and T ∪ { ϕ } | = ¬ ψ ; CONTRADICTION / unsatisfiable (iii) T ∪ { ϕ } | = ψ and T ∪ { ϕ } | = ¬ ψ ; UNKNOWN / possible (iv) T ∪ { ϕ } �| = ψ and T ∪ { ϕ } �| = ¬ ψ . UNKNOWN / possible

. . . classically (i) T ∪ { ϕ } | = ψ and T ∪ { ϕ } �| = ¬ ψ ; ENTAILED / valid (ii) T ∪ { ϕ } �| = ψ and T ∪ { ϕ } | = ¬ ψ ; CONTRADICTION / unsatisfiable (iii) T ∪ { ϕ } | = ψ and T ∪ { ϕ } | = ¬ ψ ; UNKNOWN / possible (iv) T ∪ { ϕ } �| = ψ and T ∪ { ϕ } �| = ¬ ψ . UNKNOWN / possible

. . . classically (i) T ∪ { ϕ } | = ψ and T ∪ { ϕ } �| = ¬ ψ ; ENTAILED / valid (ii) T ∪ { ϕ } �| = ψ and T ∪ { ϕ } | = ¬ ψ ; CONTRADICTION / unsatisfiable (iii) T ∪ { ϕ } | = ψ and T ∪ { ϕ } | = ¬ ψ ; UNKNOWN / possible (consistency) (iv) T ∪ { ϕ } �| = ψ and T ∪ { ϕ } �| = ¬ ψ . UNKNOWN / possible

. . . classically (i) T ∪ { ϕ } | = ψ and T ∪ { ϕ } �| = ¬ ψ ; ENTAILED / valid (ii) T ∪ { ϕ } �| = ψ and T ∪ { ϕ } | = ¬ ψ ; CONTRADICTION / unsatisfiable (iii) T ∪ { ϕ } | = ψ and T ∪ { ϕ } | = ¬ ψ ; UNKNOWN / possible (consistency) (iv) T ∪ { ϕ } �| = ψ and T ∪ { ϕ } �| = ¬ ψ . UNKNOWN / possible (completeness)

. . . instead (i) T ∪ { ϕ } | = 1 . 0 ψ and T ∪ { ϕ } | = 0 . 0 ¬ ψ ; (ii) T ∪ { ϕ } | = 0 . 0 ψ and = t ′ ¬ ψ , for 0 < t , t ′ < 1 . 0. (iii) T ∪ { ϕ } | = t ψ and T ∪ { ϕ } | (a) t > t ′ (b) t < t ′ More generally, for any two candidate entailments ◮ T ∪ { ϕ i } | = t i ¬ ψ i , ◮ T ∪ { ϕ j } | = t j ¬ ψ j , decide whether t i > t j , or t i < t j .

. . . instead (i) T ∪ { ϕ } | = 1 . 0 ψ and T ∪ { ϕ } | = 0 . 0 ¬ ψ ; (ii) T ∪ { ϕ } | = 0 . 0 ψ and = t ′ ¬ ψ , for 0 < t , t ′ < 1 . 0. (iii) T ∪ { ϕ } | = t ψ and T ∪ { ϕ } | (a) t > t ′ (b) t < t ′ More generally, for any two candidate entailments ◮ T ∪ { ϕ i } | = t i ¬ ψ i , ◮ T ∪ { ϕ j } | = t j ¬ ψ j , decide whether t i > t j , or t i < t j .

Outline Informativity, Robustness & Graded Validity Propositional Model Theory & Graded Validity Shallow Inference: Bag-of-Words Encoding Deep Inference: Syllogistic Encoding Computation via the Monte Carlo Method

Model Theory: Classical Bivalent Logic Definition ◮ Let Λ = � p 1 , p 2 , . . . , p N � be a propositional language. ◮ Let w = [ w 1 , w 2 , . . . , w N ] be a model. The truth value � · � Λ w is: � ⊥ � Λ w = 0 ; � p i � Λ w = w i for all i ;  if � ϕ � Λ w = 1 and � ψ � Λ 1 w = 1 ,    if � ϕ � Λ w = 1 and � ψ � Λ  0 w = 0 ,  � ϕ → ψ � Λ w = if � ϕ � Λ w = 0 and � ψ � Λ w = 1 , 1    if � ϕ � Λ w = 0 and � ψ � Λ  1 w = 0 ;  for all formulae ϕ and ψ over Λ .

Model Theory: Satisfiability, Validity Definition ◮ ϕ is valid iff � ϕ � w = 1 for all w ∈ W . ◮ ϕ is satisfiable iff � ϕ � w = 1 for some w ∈ W . Definition 1 � � ϕ � W = � ϕ � w . |W| w ∈W Corollary ◮ ϕ is valid iff � ϕ � W = 1 . ◮ ϕ is satisfiable iff � ϕ � W > 0 .

Outline Informativity, Robustness & Graded Validity Propositional Model Theory & Graded Validity Shallow Inference: Bag-of-Words Encoding Deep Inference: Syllogistic Encoding Computation via the Monte Carlo Method

Bag-of-Words Inference (1) assume strictly bivalent valuations; |W| = 2 6 ; Λ = { socrates , is , a , man , so , every } , ( T ) socrates ∧ is ∧ a ∧ man so ∧ every ∧ man ∧ is ∧ socrates ; ∴ ( H ) |W T | = 2 1 ; Λ T = { a } , |W O | = 2 3 ; Λ O = { socrates , is , man } , |W H | = 2 2 ; Λ H = { so , every } , 2 1 ∗ 2 3 ∗ 2 2 = 2 6 ;

Bag-of-Words Inference (2) How to make this implication false ? ◮ Choose the 1 out of 2 4 = 16 valuations from W T × W O which makes the antecedent true. ◮ Choose any of the 2 2 − 1 = 3 valuations from W H which make the consequent false. ...now compute an expected value. Count zero for the 1 ∗ ( 2 2 − 1 ) = 3 valuations that make this implication false. Count one, for the other 2 6 − 3. Now � T → H � W = 2 6 − 3 = 0 . 95312 , 2 6 or, more generally, 2 | Λ H | − 1 � T → H � W = 1 − 2 | Λ T | + | Λ H | + | Λ O | .

Outline Informativity, Robustness & Graded Validity Propositional Model Theory & Graded Validity Shallow Inference: Bag-of-Words Encoding Deep Inference: Syllogistic Encoding Computation via the Monte Carlo Method

Language: Syllogistic Syntax Let Λ = { x 1 , x 2 , x 3 , y 1 , y 2 , y 3 } ; All X are Y =( x 1 → y 1 ) ∧ ( x 2 → y 2 ) ∧ ( x 3 → y 3 ) Some X are Y =( x 1 ∧ y 1 ) ∨ ( x 2 ∧ y 2 ) ∨ ( x 3 ∧ y 3 ) All X are not Y = ¬ Some X are Y , Some X are not Y = ¬ All X are Y ,

Proof theory: A Modern Syllogism Some X are Y ( S 1 ) , ( S 2 ) , All X are X Some X are X ∴ ∴ All Y are Z All Y are Z All X are Y ( S 3 ) , Some Y are X ( S 4 ) , All X are Z Some X are Z ∴ ∴ Some X are Y ( S 5 ); Some Y are X ∴

Proof theory: “Natural Logic” ( NL 1 ) , All cats are animals ( NL 2 ) , All ( red X ) are X ∴ ∴ Some X are ( red Y ) Some X are cats , Some X are animals , Some X are Y ∴ ∴ Some ( red X ) are Y Some cats are Y , Some animals are Y , Some X are Y ∴ ∴ All X are ( red Y ) All X are cats , All X are animals , All X are Y ∴ ∴ All X are Y All animals are Y All ( red X ) are Y , ; All cats are Y ∴ ∴

Natural Logic Robustness Properties Some X are Y Some X are Y > Some X are ( big ( red Y )) , Some X are ( red Y ) ∴ ∴ Some X are Y Some X are Y > Some ( big ( red X )) are Y , Some ( red X ) are Y ∴ ∴ All X are Y All X are Y > All X are ( big ( red Y )) , All X are ( red Y ) ∴ ∴ All ( red X ) are Y All ( big ( red X )) are Y > . All X are Y All X are Y ∴ ∴