Frame Semantic Structure Extraction


	Frame Semantic Structure Extraction Task #19
	Organizing team:
	Collin Baker (International Computer Science Institute, Berkeley)
	Michael Ellsworth (International Computer Science Institute, Berkeley)
	Katrin Erk (U Texas, Austin)
	Summary of Task \| Datasets \| Evaluation \| Resources \| Mailing list and forum \| Systems and Results \| Bibliography

News

Full-text DTD added to "Task-specific Software"
Deadline extended to Thursday, April 5th! ! !
We have been asked if we could extend the deadline for submission of participants' results on Task 19, Frame Semantic Structure Extraction. We believe that we can do a fast turnaround, and get the evaluations out quickly, so we are extending the deadline for submission of results to
Friday, April 6th, at 06:59 GMT. Note that this is 7 hours later than the extension announced previously.
We should still be able to give back evaluations to participants by April 10th, leaving a week to write up the papers on each system.
We recognize that this is a difficult task, and we hope that this will make it possible for participants to produce more complete results. Please note that this does not override the 21/7 day limits for the training/testing data; it just means that participants will have until the 21/7 day limit or April 6th at 06:59, whichever is earlier.
If there are any questions about the format for the submission of results, the nature of the evaluation, etc., please feel free to contact us, either by email or by posting a question on the SemEval section of the FrameNet forum on the public website. (Note that in order to post to the forum, you will need to create a login for yourself on the public website--this is separate from anything else, but only takes a few minutes.)
The scoring script that implements the flat, labels-only evaluation option has been updated to fix a divide-by-zero bug. The new version is under "Task-specific Software".
The semantic dependency scoring software has also been updated on this website, under Task-specific Software, including a flag for ignoring named entities. A divide-by-zero error in this program has also been fixed.
We are working on new versions of both the flat labels-only scoring and the semantic dependency scoring that will give partial credit for cases in which the label boundaries are correct but the frames is wrongly assigned. If the gold-standard frame is connected to the frame in the results by frame-to-frame relations, and the number of links in the chain is low, you may get partial credit for the near-miss on the frame and FEs (assuming that the FEs are also linked).
A corrected version of the training dataset has been uploaded to the SemEval website. This data should now work correctly with the fttosem program; it should not crash, and should produce correct dependency trees in a higher proportion of cases.
The SemEval Forum is now on-line. See Mailing list and forum.

Summary of Task

This task consists of recognizing words and phrases that evoke semantic frames of the sort defined in the FrameNet project (http://framenet.icsi.berkeley.edu), and their semantic dependents, which are usually, but not always, their syntactic dependents (including subjects). For example, in the sentence

Matilde said, "I rarely eat rutabaga",
said evokes the Statement frame, and eat evokes the Ingestion frame. The role of Speaker in the Statement frame is filled by Matilda, and the role of Message, by the whole quotation. In the Ingestion frame, I is the Ingestor and rutabaga fills the Ingestibles role. Since the Ingestion event is contained within the Message of the Statement event, we can represent the fact that the Message conveyed was about Ingestion, just by annotating the sentence with respect to these two frames. Note that these role names (called Frame Elements, or FEs) are quite frame-specific; generalizations over them, relating most of them to a small set of thematic roles are expressed via frame-to-frame and FE-FE relations.

A number of automatic systems have been built to do this sort of annotation, which can be divided into three parts:

frame recognition: identify words or phrases that evoke frames and assigning the correct frame to them (similar to the WSD task)
for frames that occur in the test data but not the training data, determine the closest known frame
Semantic role labeling (SRL): for each instance of a frame that has been found, find and label the parts of the sentence that fill their semantic roles (FEs)

The evaluation will measure precision and recall for frames and frame elements, with partial credit for incorrect but closely related frames. This is a more advanced version of the Automatic Semantic Role Labeling task of Senseval-3 (Litkowski 2004).

Training and Testing Data

The training data consists of

FrameNet data release 1.3, freely available for non-commercial use directly from FrameNet (http://framenet.icsi.berkeley.edu), containing more than 150,000 manually annotated instances of frames and
a supplemental set of documents with full-text annotation, which participants in this task can download from the SemEval website. Some of the documents were not released previously, and some are updated versions of documents in Release 1.3. Two of the ANC documents from the Berlitz travel guides, HistoryOf Greece.xml and HistoryOfLasVegas.xml do not contain any manual annotation, only two types of automatic labeling: named entities, plus labels indicating which words are potentially frame evoking and which are not likely to need FN annotation.

The testing data, also available for participants to download from the SemEval website, consists of three previously unannotated texts, one from the American National Corpus (http://www.americannationalcorpus.org) and two from the website of the Nuclear Threat Initiative (http://www.nti.org). The gold standard will be the manual annotation by the FrameNet team of these texts for all frame evoking expressions and the fillers of the associated frame elements.

Evaluation

A perl script for converting nested frame annotations into hierarchical XML representations, called fttosem.pl has been developed and is available under Resources. (1) Frame Semantic Dependency Structure Evaluation:
Participants' results will be evaluated by comparing the trees output from fttosem.pl on their annotation XML to the trees output by from fttosem.pl on the gold standard annotation XML. The evaluation will measure precision and recall for frames and frame elements, with partial credit for incorrect but closely related frames. A more detailed specification of the process is included with the training dataset, and can also be viewed here. A perl script implementing this algorithm is also listed under Resources. (2) Flat FN Labels-only Evaluation:
Participants' results will be evaluated by comparing their annotation XML directly with the gold standard annotation XML. A Perl script for this purpose is provided in the Task-specific software section.

Resources

Semantic Parsing

(a.k.a. Automatic Semantic Role Labeling (ASRL)) A number of people have built ASRL systems There are at least two freely available semantic role labeling systems which can be trained on the FN data: (1) Shalmaneser, developed at the SALSA Project at Univ. des Saarlandes, Saarbrücken, by Sebastian Padó and Katrin Erk, and (2) ASSERT, developed at Univ. of Colorado, written by Sameer Pradhan and associates, has been used in a number of automatic labeling tasks, using the PropBank tags..

Shalmaneser comes with pre-trained parameter sets based on the FrameNet Release 1.3 data, and has recently been modified to read FN full-text annotation as well. Downloads of Shalmaneser are available at the SALSA website:
http://www.coli.uni-saarland.de/projects/salsa/shal/
Note that the output from Shalmaneser is in SALSA/TIGER XML, rather than FrameNet full-text XML.

Task-specific Software

N.B. The versions of the software on this website are more recent than those on the SemEval website. Please see the SemEval forum for a discussion of the minimal file format required.
Here is the DTD that is needed for validation; you will need to save it to a file and place this file in a directory called "docs", one level up from the directory containing the full-text annotation files:
corpusV1_2.dtd

fttosem.pl Reads the FrameNet full-text annotation XML and converts it to the dependency XML that is the basis of the semantic dependency evaluation.
SynSemTreeMaker.pm Required by fttosem.pl. Last revised 23 Mar, 2007
SynSemDocumentation.txt (documentation for the above)

There is a separate script,
semDep2dot,
that reads the semantic dependency XML and converts it to the "dot" format used by Graphviz (www.graphviz.org), making it possible to produce graphics that are easier for humans to read than the underlying XML.

We have written a script,
fssScore.pl,
for scoring your results against the training data as described in the FS dependency evaluation specification. It calculates precision, recall, and F1 score, per file (or optionally per sentence).

There is now also a script which calculates precision, recall, and F1 score for the flat, labels-only evaluation:
srlScore.pl. This implementation is quite simple, and requires exact matches for both boundaries and labels of the target itself (i.e. the frame label) and the FEs (i.e. the FE labels). We are in the process of implementing a new algorithm that gives partial credit for cases in which the boundaries are correct, but the frame in question is not the gold-standard frame, but is close to it, as defined by the frame-to-frame relations defined in frRelation.xml. The amount of partial credit will depend on the number of links between the proposed frame and the gold-standard one.
Note that both fssScore and srlScore require a copy of frames.xml, part of the regular FN data release. Both scoring scripts include a flag (-n) causing them to ignore named entities in scoring precision and recall. This has been implemented with minimal change to the script, that is, by simply setting the value of NE matches to 0; if the "verbose" option is used, named entities will be listed as a category, but their total will always be zero.

Mailing list and Forum

A Mailman mailing list has been set up at ICSI for those interested in this task; posting to the list is limited to those who have subscribed. Please go to http://mailman.icsi.berkeley.edu/mailman/listinfo/fn-semeval to subscribe.

A forum has been set up within the regular FrameNet website for SemEval questions and answers and comments. Please ask questions there if possible; this will help us to get the answers out both to registered participants and to others who are interested, but not actually participating.

Systems and Results

Summary information about systems and results will be filled in here after the testing period.

Bibliography

Coming soon.

For more information, visit the SemEval-2007 home page.

Last modified: Tue Apr 3 15:28:55 PDT 2007