[PPT] - Joint prediction in MST style discourse parsing for argumentation PowerPoint Presentation

SLIDE 1

Joint prediction in MST style discourse parsing for argumentation mining

Andreas Peldszus Manfred Stede

Applied Computational Linguistics, University of Potsdam

Dagstuhl 16161 - 18.04.2016

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 1 / 29

SLIDE 2

Outline

1 Argumentation Mining 2 Dataset & Scheme 3 Joint prediction in argumentation mining 4 Multi-layer discourse annotation

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 2 / 29

SLIDE 3

Outline

1 Argumentation Mining 2 Dataset & Scheme 3 Joint prediction in argumentation mining 4 Multi-layer discourse annotation

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 3 / 29

SLIDE 4

What is argumentation mining?

Health insurance companies should naturally cover alternative medical

treatments. Not all practices

and approaches that are lumped together under this term may have been proven in clinical trials, yet it's precisely their positive effect when accompanying conventional 'western' medical therapies that's been demonstrated as

beneficial. Besides many

general practitioners offer such counselling and treatments in parallel anyway - and who would want to question their broad expertise? Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 4 / 29

SLIDE 5

What is argumentation mining?

[e1] Health insurance companies should naturally cover alternative medical treatments. [e2] Not all practices and approaches that are lumped together under this term may have been proven in clinical trials,

1

[e3] yet it's precisely their positive effect when accompanying conventional 'western' medical therapies that's been demonstrated as beneficial.

2

[e4] Besides many general practitioners offer such counselling and treatments in parallel anyway -

3

[e5] and who would want to question their broad expertise?

4 5

c3 c4 c2

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 4 / 29

SLIDE 6

What is argumentation mining?

[e1] Health insurance companies should naturally cover alternative medical treatments. [e2] Not all practices and approaches that are lumped together under this term may have been proven in clinical trials,

1

[e3] yet it's precisely their positive effect when accompanying conventional 'western' medical therapies that's been demonstrated as beneficial.

2

[e4] Besides many general practitioners offer such counselling and treatments in parallel anyway -

3

[e5] and who would want to question their broad expertise?

4 5

c3 c4 c2

Tasks:

EDU segmentation
ADU segmentation
resp. argumentative relevance
ADU type classification
Relation identification
Relation type classification

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 4 / 29

SLIDE 7

Outline

1 Argumentation Mining 2 Dataset & Scheme 3 Joint prediction in argumentation mining 4 Multi-layer discourse annotation

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 5 / 29

SLIDE 8

Text genres: pro & contra commentaries

Source:

pro & contra newspaper commentaries
in Potsdam Commentary Corpus

[Stede, 2004] [Stede and Neumann, 2014]

Properties:

lots of arguments
rather explicitly marked argumentation
special background knowledge required
main claim may be implicit
full range of persuasive ’tricks’ professional

writers can offer

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 6 / 29

SLIDE 9

Dataset: argumentative microtexts

Properties:

about 5 segments long
each segment is arg. relevant
explicit main claim
at least one possible objection considered

Texts:

23 texts: hand-crafted, covering different arg. configurations
92 texts: collected in a controlled text generation experiment
with professional parallel translation to English
see [Peldszus and Stede, 2015b]
freely available, CC-by-nc-sa license
https://github.com/peldszus/arg-microtexts

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 7 / 29

SLIDE 10

Dataset: argumentative microtexts

Properties:

about 5 segments long
each segment is arg. relevant
explicit main claim
at least one possible objection considered

Texts:

23 texts: hand-crafted, covering different arg. configurations
92 texts: collected in a controlled text generation experiment
with professional parallel translation to English
see [Peldszus and Stede, 2015b]
freely available, CC-by-nc-sa license
https://github.com/peldszus/arg-microtexts

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 7 / 29

SLIDE 11

Scheme

[e1] Health insurance companies should naturally cover alternative medical treatments. [e2] Not all practices and approaches that are lumped together under this term may have been proven in clinical trials,

1

[e3] yet it's precisely their positive effect when accompanying conventional 'western' medical therapies that's been demonstrated as beneficial.

2

[e4] Besides many general practitioners offer such counselling and treatments in parallel anyway -

3

[e5] and who would want to question their broad expertise?

4 5

c3 c4 c2

Freeman’s theory, revised & slightly generalized:

[Freeman, 1991, 2011] [Peldszus and Stede, 2013]

node types = argumentative role

proponent (presents and defends claims)

pponent (critically questions)
link types = argumentative function

support own claims (normally, by example) attack other’s claims (rebut, undercut) IAA: 3 expert annotators κ = 0.83

[Peldszus, 2014]

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 8 / 29

SLIDE 12

Scheme

[e1] Health insurance companies should naturally cover alternative medical treatments. [e2] Not all practices and approaches that are lumped together under this term may have been proven in clinical trials,

1

[e3] yet it's precisely their positive effect when accompanying conventional 'western' medical therapies that's been demonstrated as beneficial.

2

[e4] Besides many general practitioners offer such counselling and treatments in parallel anyway -

3

[e5] and who would want to question their broad expertise?

4 5

c3 c4 c2

Freeman’s theory, revised & slightly generalized:

[Freeman, 1991, 2011] [Peldszus and Stede, 2013]

node types = argumentative role

proponent (presents and defends claims)

pponent (critically questions)
link types = argumentative function

support own claims (normally, by example) attack other’s claims (rebut, undercut) IAA: 3 expert annotators κ = 0.83

[Peldszus, 2014]

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 8 / 29

SLIDE 13

Outline

1 Argumentation Mining 2 Dataset & Scheme 3 Joint prediction in argumentation mining 4 Multi-layer discourse annotation

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 9 / 29

SLIDE 14

Tasks tackled:

[e1] Health insurance companies should naturally cover alternative medical treatments. [e2] Not all practices and approaches that are lumped together under this term may have been proven in clinical trials,

1

[e3] yet it's precisely their positive effect when accompanying conventional 'western' medical therapies that's been demonstrated as beneficial.

2

[e4] Besides many general practitioners offer such counselling and treatments in parallel anyway -

3

[e5] and who would want to question their broad expertise?

4 5

Trained 4 base classifiers:

attachment (at)

464 pairs yes, 2000 pairs no

central claim (cc)

112 yes, 451 no

role (ro)

451 proponent, 125 opponent

function (fu)

290 support, 174 attacks

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 10 / 29

SLIDE 15

Joint prediction in argumentation mining

Key features:

MST decoding:

Valid global structures from (possibly incompatible) local predictions

Joint prediction:

Combine predictions of different base classifiers in the graph

[Peldszus and Stede, 2015a]

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 11 / 29

SLIDE 16

Joint prediction in argumentation mining

Key features:

MST decoding:

Valid global structures from (possibly incompatible) local predictions

Joint prediction:

Combine predictions of different base classifiers in the graph

[Peldszus and Stede, 2015a]

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 11 / 29

SLIDE 17

Key feature 1: MST decoding

Procedure:

predict edge score
apply classification threshold

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 12 / 29

SLIDE 18

Key feature 1: MST decoding

Procedure:

predict edge score
apply classification threshold

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 12 / 29

SLIDE 19

Key feature 1: MST decoding

Procedure:

predict edge score
apply classification threshold

Not a tree for 85% of the texts!

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 12 / 29

SLIDE 20

Key feature 1: MST decoding

Procedure:

predict edge score
apply minimum spanning tree algorithm

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 13 / 29

SLIDE 21

Key feature 1: MST decoding

Procedure:

predict edge score
apply minimum spanning tree algorithm

[Chu and Liu, 1965, Edmonds, 1967] [McDonald et al., 2005] [Baldridge et al., 2007, Muller et al., 2012]

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 13 / 29

SLIDE 22

Key feature 1: MST decoding

Procedure:

predict edge score
apply minimum spanning tree algorithm

[Chu and Liu, 1965, Edmonds, 1967] [McDonald et al., 2005] [Baldridge et al., 2007, Muller et al., 2012]

Always a tree!

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 13 / 29

SLIDE 23

Key feature 2: Joint prediction

[e1] Health insurance companies should naturally cover alternative medical treatments. [e2] Not all practices and approaches that are lumped together under this term may have been proven in clinical trials,

1

[e3] yet it's precisely their positive effect when accompanying conventional 'western' medical therapies that's been demonstrated as beneficial.

2

[e4] Besides many general practitioners offer such counselling and treatments in parallel anyway -

3

[e5] and who would want to question their broad expertise?

4 5

Why restrict to only one feature of the graph?

attachment (at)
central claim (cc)
role (ro)
function (fu)

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 14 / 29

SLIDE 24

Key feature 2: Joint prediction

Procedure:

predict attachment probability
predict role, function, cc probability
combine predictions in one score
apply MST algorithm
identify central claim
derive final role class for each

segment

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 15 / 29

SLIDE 25

Key feature 2: Joint prediction

Procedure:

predict attachment probability
predict role, function, cc probability
combine predictions in one score
apply MST algorithm
identify central claim
derive final role class for each

segment

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 15 / 29

SLIDE 26

Key feature 2: Joint prediction

Procedure:

predict attachment probability
predict role, function, cc probability
combine predictions in one score
apply MST algorithm
identify central claim
derive final role class for each

segment

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 15 / 29

SLIDE 27

Key feature 2: Joint prediction

Procedure:

predict attachment probability
predict role, function, cc probability
combine predictions in one score
apply MST algorithm
identify central claim
derive final role class for each

segment

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 15 / 29

SLIDE 28

Key feature 2: Joint prediction

Procedure:

predict attachment probability
predict role, function, cc probability
combine predictions in one score
apply MST algorithm
identify central claim
derive final role class for each

segment

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 15 / 29

SLIDE 29

Key feature 2: Joint prediction

Procedure:

predict attachment probability
predict role, function, cc probability
combine predictions in one score
apply MST algorithm
identify central claim
derive final role class for each

segment

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 15 / 29

SLIDE 30

Key feature 2: Joint prediction

Combines four probabilities in one edge score:

Probability of attachment from source to target
Probability of proper function per edge type
Probability of not being the central claim
Probability of preserved/switched role for sup/att

edges

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 16 / 29

SLIDE 31

Key feature 2: Joint prediction

Combines four probabilities in one edge score:

Probability of attachment from source to target
Probability of proper function per edge type
Probability of not being the central claim
Probability of preserved/switched role for sup/att

edges

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 16 / 29

SLIDE 32

Results: German

simple EG equal EG best at maF1 .679 .712 .710 κ .365 .424 .421

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 17 / 29

SLIDE 33

Results: German

simple EG equal EG best at maF1 .679 .712 .710 κ .365 .424 .421 cc maF1 .849 .879 .890 κ .698 .759 .780

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 17 / 29

SLIDE 34

Results: German

simple EG equal EG best at maF1 .679 .712 .710 κ .365 .424 .421 cc maF1 .849 .879 .890 κ .698 .759 .780 ro maF1 .755 .737 .734 κ .511 .477 .472

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 17 / 29

SLIDE 35

Results: German

simple EG equal EG best at maF1 .679 .712 .710 κ .365 .424 .421 cc maF1 .849 .879 .890 κ .698 .759 .780 ro maF1 .755 .737 .734 κ .511 .477 .472 fu maF1 .703 .735 .736 κ .528 .573 .570

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 17 / 29

SLIDE 36

Results: German

simple EG equal EG best at maF1 .679 .712 .710 κ .365 .424 .421 cc maF1 .849 .879 .890 κ .698 .759 .780 ro maF1 .755 .737 .734 κ .511 .477 .472 fu maF1 .703 .735 .736 κ .528 .573 .570 avg maF1 .747 .766 .768 κ .526 .558 .561

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 17 / 29

SLIDE 37

Results: German

simple EG equal EG best MP MP+p MP+p+r at maF1 .679 .712 .710 .724 .728 .728 κ .365 .424 .421 .449 .456 .456 cc maF1 .849 .879 .890 .825 .855 .855 κ .698 .759 .780 .650 .710 .710 ro maF1 .755 .737 .734 .464 .477 .669 κ .511 .477 .472 .014 .022 .340 fu maF1 .703 .735 .736 .499 .527 .723 κ .528 .573 .570 .293 .326 .557 avg maF1 .747 .766 .768 .628 .647 .744 κ .526 .558 .561 .352 .379 .516

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 18 / 29

SLIDE 38

Results: English

simple EG equal EG best MP MP+p MP+p+r at maF1 .663 .692 .693 .707 .720 .720 κ .333 .384 .386 .414 .440 .440 cc maF1 .817 .860 .869 .780 .831 .831 κ .634 .720 .737 .559 .661 .661 ro maF1 .750 .721 .720 .482 .475 .638 κ .502 .445 .442 .024 .015 .280 fu maF1 .671 .707 .710 .489 .514 .681 κ .475 .529 .530 .254 .296 .491 avg maF1 .725 .745 .748 .615 .635 .718 κ .486 .520 .524 .313 .353 .468

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 19 / 29

SLIDE 39

Outlook

Outlook:

more powerful decoding: ILP

, second order features

end-to-end: tackle edu to adu segmentation
joint learning instead of joint prediction
more fine-grained relations types
more complex text: pro-contra, news-editorials etc.

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 20 / 29

SLIDE 40

Outline

1 Argumentation Mining 2 Dataset & Scheme 3 Joint prediction in argumentation mining 4 Multi-layer discourse annotation

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 21 / 29

SLIDE 41

Multi-layer discourse annotation

How does argumentation structure relate to other discourse structures?

Rhetorical Structure Theory (RST)

[Mann and Thompson, 1988]

Segmented Discourse Structure Theory (SDRT)

[Asher and Lascarides, 2003]

Joint work with Jérémy Perret, Stergos Afantenos, Nicholas Asher

[Stede et al., 2016]

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 22 / 29

SLIDE 42

Multi-layer discourse annotation: Harmonize segmentation

[e1] Supermarket employees and people who work in shopping centres also have the right to a Sunday off work. [e2] Likewise public holidays should remain what they are:

1

[e3] for some a day

f introspection,

for others a paid day off that is not taken away from the annual paid leave proper.

2+3

[e4] Hence it is good when shops are not open on Sundays and public holidays. [e5] People, however, who work during the week and

n Saturdays then

have a problem:

4

[e6] everyone else can shop weekdays,

5+6+7

[e7] but they can't. [e8] For those people the late

pening hours, which

meanwhile already extend to 12:00 midnight, present a good alternative.

8 2+3 5+6+7

c12 c11 c14 c13

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 23 / 29

SLIDE 43

Multi-layer discourse annotation: Add RST & SDRT

[e1] Health insurance companies should naturally cover alternative medical treatments. [e2] Not all practices and approaches that are lumped together under this term may have been proven in clinical trials,

1

[e3] yet it's precisely their positive effect when accompanying conventional 'western' medical therapies that's been demonstrated as beneficial.

2

[e4] Besides many general practitioners

ffer such

counselling and treatments in parallel anyway

3

[e5] and who would want to question their broad expertise?

4 5

c9 c7 c6

(a) ARG (b) RST

1 π1 2 3 π2 4 5 Elaboration Contrast Background Comment

(c) SDRT

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 24 / 29

SLIDE 44

Multi-layer discourse annotation: Common dependency format

ARG RST SDRT 1 2 3 4 5

rebut undercut support link

1 2 3 4 5

concession reason reason joint

1 2 3 4 5

elaboration contrast background comment

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 25 / 29

SLIDE 45

Multi-layer discourse annotation: Correlations

e x a m p l e j

i

n l i n k r e b u t s u p p

r

t u n d e r c u t N O N E antithesis . 3 . 9 1 6 7 background . 1 2 . 4 . 8 cause . 4 1 . 11 . 2 circumstance . 4 . . . . 1 concession . . . 6 1 32 18 condition . 13 . 1 1 . . conjunction . 10 6 . . 2 23 contrast . . . 1 . . 3 disjunction . 2 . . . . 2 e-elaboration 2 5 . . . . 1 elaboration 4 7 . 2 3 . 11 evaluation-s . 2 . . . . . evidence . . . . 8 . 2 interpretation . . . . . . 2 joint . 2 5 1 4 1 8 justify . . . . 4 . 3 list . 1 . 1 2 . 53 means . 1 . . . . . motivation . . . 1 2 . . preparation . 3 . . . . . purpose . 3 . . . . . reason . 6 . 3 99 . 55 restatement . . . . 2 . 2 result . 1 . . 1 . . sameunit . 1 . 1 . . . solutionhood . . . . . . 1 unless . . . 1 . . 1 NONE 2 10 7 72 92 20 . Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 26 / 29

SLIDE 46

Multi-layer discourse annotation: Outlook

Research questions for argumentation:

Is it possible to map RST and/or SDRT to argumentation structures?
Is it possible to learn this mapping?
Discourse structure features for argumentation parsing?
Argumentation structure features for discourse parsing?
Joint learning of multiple discourse structures?

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 27 / 29

SLIDE 47

Literatur I

Nicholas Asher and Alex Lascarides. Logics of Conversation. Cambridge University Press, Cambridge, 2003. Jason Baldridge, Nicholas Asher, and Julie Hunter. Annotation for and robust parsing of discourse structure on unrestricted texts. Zeitschrift für Sprachwissenschaft, 26:213–239, 2007.

Y. J. Chu and T. H. Liu. On the shortest arborescence of a directed graph. Science Sinica, 14:1396–1400, 1965.

Jack Edmonds. Optimum Branchings. Journal of Research of the National Bureau of Standards, 71B:233–240, 1967. James B. Freeman. Dialectics and the Macrostructure of Argument. Foris, Berlin, 1991. James B. Freeman. Argument Structure: Representation and Theory. Argumentation Library (18). Springer, 2011. William Mann and Sandra Thompson. Rhetorical structure theory: Towards a functional theory of text organization. TEXT, 8:243–281, 1988. Ryan McDonald, Fernando Pereira, Kiril Ribarov, and Jan Hajic. Non-projective dependency parsing using spanning tree algorithms. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 523–530, Vancouver, British Columbia, Canada, October 2005. Association for Computational Linguistics. Philippe Muller, Stergos Afantenos, Pascal Denis, and Nicholas Asher. Constrained decoding for text-level discourse parsing. In Proceedings

f COLING 2012, pages 1883–1900, Mumbai, India, December 2012. The COLING 2012 Organizing Committee. URL

http://www.aclweb.org/anthology/C12-1115. Andreas Peldszus. Towards segment-based recognition of argumentation structure in short texts. In Proceedings of the First Workshop on Argumentation Mining, Baltimore, U.S., June 2014. Association for Computational Linguistics. Andreas Peldszus and Manfred Stede. From argument diagrams to automatic argument mining: A survey. International Journal of Cognitive Informatics and Natural Intelligence (IJCINI), 7(1):1–31, 2013. Andreas Peldszus and Manfred Stede. Joint prediction in MST-style discourse parsing for argumentation mining. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 938–948, Lisbon, Portugal, September 2015a. Association for Computational Linguistics. URL https://aclweb.org/anthology/D/D15/D15-1110.pdf.

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 28 / 29

SLIDE 48

Literatur II

Andreas Peldszus and Manfred Stede. An annotated corpus of argumentative microtexts. In Proceedings of the First European Conference

n Argumentation: Argumentation and Reasoned Action, Lisbon, Portugal, June 2015b.

Manfred Stede. The potsdam commentary corpus. In Proceedings of the 2004 ACL Workshop on Discourse Annotation, DiscAnnotation ’04, pages 96–102, Stroudsburg, PA, USA, 2004. Association for Computational Linguistics. Manfred Stede and Arne Neumann. Potsdam commentary corpus 2.0: Annotation for discourse research. In Proc. of the International Conference on Language Resources and Evaluation (LREC), Reykjavik, 2014. Manfred Stede, Stergos Afantenos, Andreas Peldszus, Nicholas Asher, and Jérémy Perret. Parallel discourse annotations on a corpus of short texts. In Proc. of the International Conference on Language Resources and Evaluation (LREC), Portorož, Slovenia, 2016.

Peldszus, Stede (Uni Potsdam) Joint prediction for argumentation mining Dagstuhl 16161 29 / 29