lumberjack
Read and process constituency trees in various formats.
Install:
- From crates.io:
cargo install lumberjack-utils
- From GitHub:
cargo install --git https://github.com/sebpuetz/lumberjack
Usage as standalone:
- Convert treebank in NEGRA export 4 format to bracketed TueBa V2 format
lumberjack-conversion --input_file treebank.negra --input_format negra \
--output_format tueba --output_file treebank.tueba --projectivize
- Retain only root node,
NPs andPPs and print to simple bracketed format:
echo "NP PP" > filter_set.txt
lumberjack-conversion --input_file treebank.simple --input_format simple \
--output_format tueba --output_file treebank.filtered \
--filter filter_set.txt
- Convert from treebank in simple bracketed to CONLLX format and annotate parent tags of terminals as features.
lumberjack-conversion --input_file treebank.simple --input_format simple\
--output_format conllx --output_file treebank.conll --parent
- Modifications in the following order:
- Reattach all terminals with part-of-speech starting with
$to the root node - Remove all nonterminals except the root,
Ss,NPs,PPs andVPs - Assign unique identifiers based on the closest
Sto terminals - Insert nodes with label
labelabove terminals that aren't dominated byNPorPP - Annotate label of parent node on terminals.
- Print to CONLLX format with annotations.
echo "S VP NP PP" > filter_set.txt
echo "NP PP" > insert_set.txt
echo "S" > id_set.txt
lumberjack-conversion --input_file treebank.simple --input_format simple\
--output_format conllx --insertion_set insert_set.txt \
--insertion_label label --id_set id_set.txt --reattach $\
--parent parent --output_file treebank.conllx
Usage as rust library:
- read and projectivize trees from NEGRA format and print to simple bracketed format
use std::io::{BufReader, File};
use lumberjack::io::{NegraReader, PTBFormat};
use lumberjack::Projectivize;
fn print_negra(path: &str) {
let file = File::open(path).unwrap();
let reader = NegraReader::new(BufReader::new(file));
for tree in reader {
let mut tree = tree.unwrap();
tree.projectivize();
println!("{}", PTBFormat::Simple.tree_to_string(&tree).unwrap());
}
}
- filter non-terminal nodes from trees in a treebank and print to simple bracketed format:
use lumberjack::{io::PTBFormat, Tree, TreeOps, util::LabelSet};
fn filter_nodes(iter: impl Iterator<Item=Tree>, set: LabelSet) {
for mut tree in iter {
tree.filter_nonterminals(|tree, nt| set.matches(tree[nt].label())).unwrap();
println!("{}", PTBFormat::Simple.tree_to_string(&tree).unwrap());
}
}
- convert treebank in simple bracketed format to CONLLX with constituency structure encoded in the features field
use conllx::graph::Sentence;
use lumberjack::io::Encode;
use lumberjack::{Tree, TreeOps, UnaryChains};
fn to_conllx(iter: impl Iterator<Item=Tree>) {
for mut tree in iter {
tree.collaps_unary_chains().unwrap();
tree.annotate_absolute().unwrap();
println!("{}", Sentence::from(&tree));
}
}
569 Jan 3, 2023
496 Jan 8, 2023
165 Jan 1, 2023
19 Dec 24, 2022
52 Jan 7, 2023
7 Mar 3, 2023
82 Jan 4, 2023
331 Dec 28, 2022
53 Sep 24, 2022
26 Dec 30, 2022
2 Jul 5, 2018
48 Nov 11, 2022
274 Nov 5, 2022
953 Jan 3, 2023
322 Dec 26, 2022
2.6k Jan 8, 2023
75 Jan 8, 2023
1.3k Jan 8, 2023