TRACER - Preprocessing Marco Büchler, Emily Franzini, Greta Franzini, Maria Moritz etrap Research Group Göttingen Centre for Digital Humanities Institute of Computer Science Georg August University Göttingen, Germany
Hacking Installation & configuration guide for TRACER 1) Copy Tracer from http://etrap.gcdh.de/dh-estonia/tracer your storage folder such as /roedel/mbuechler 2) Change to your storage folder with cd command 3) Unzip archive: gunzip tracer.tar.gz 4) Untar archive: tar -xvf tracer.tar 5) Change to tracer folder: cd Tracer 6) Open the config file with vim conf/tracer_config.xml 7) Configure your input file:
Hacking - Starting TRACER 1) Start the tool with the command: java -Xmx600m -Dde.gcdh.medusa.config.ClassConfig=conf/tracer_config.xml -jar tracer.jar Explanation: -Xmx600m (up to 600 MB memory), -Dfile.encoding sets the encoding of your input file (optionally), -Dde.gcdh.medusa.config.ClassConfig (configuration file)
Overview What is preprocessing? Overview of preprocessing techniques Hacking Conclusion with some test questions
Reminder: Current approach
Pre-step: Segmentation - an example
Pre-step: Segmentation
Question What do you associated with preprocessing?
Foundations for preprocessing Zipfian Law
Implications of the Zipfian Law Approx. 50% of all words occur only once Approx. 16% of all words occur only twice Approx. 8% of all words occur three times... Approx. 90% of all words in a corpus occur 10 times or less The top 300 700 most frequent words cover already about 50% of all tokens (depending language)
Question What does lemmatisation mean for this plot?
Preprocessing
Preprocessing: Directed Graph Normalisation e.g. lemmatisation
Preprocessing: Indirected Graph Normalisation e.g. synonyms, string similarity
Hacking Tasks: Run on your texts... 1)... without preprocessing 2)... 1) + lemmatisation 3)... 2) + synonym replacement
Hacking Questions: Compare the input file with the *.prep file for all preprocessing techniques. Which methods seems to work best for you? Which does make no sense for the dataset? Compare all *.meta files containing some numbers! How many words have changed and by which method? (optional and advanced) what is the number of word types for each preprocessing (can be derived from *.prep.inv first column)
Preprocessing 1) without preprocessing Hint: Configuration file can be found in ${TRACER_HOME}/conf/tracer_conf.xml All values show false
Preprocessing 2) Removing diachritics Hint: BoolRemoveDiachritics is switched on by value true
Preprocessing 3) Lower case Hint: boolmakealllowercase is switched on by value true
Preprocessing 4) Lemmatising text Hint: boollemmatisation is switched on by value true Lemmatisation can be configured by <property name="baseform_file_name" value="data/corpora/bible/bible.lemma" />
Preprocessing 5) Synonym handling Hint: boolreplacesynonyms is switched on by value true Synonyms can be configured by <property name="synonyms_file_name" value="data/corpora/bible/bible.syns" />
Preprocessing 6) String similarity for normalising variants Hint: boolreplacestringsimilarwords is switched on by value true Thresholds:
Open issue: Fragmentary words
Open issue: Fragmentary words dealing with gaps and Leiden Convention Οὐιβίῳ Ἀλεξάά [ν]δρῳ τῷ κρατίστῳ ἐπιστρατήγῳ παρὰ Ἀντ[ωνίου Δ]όμνά ου τοῦ καὶ Φιλαντι[νό]οά υά Ἀντωνίοά [υ Ῥωμανο]ῦά Τραιανείου τοῦ καά [ὶ Στρα]τά είου Ἀντινοέως. [οὐκ ἂν] εἰς τοῦτο προήχθά [η]νά, ἐά πιτρόπων [μέγιστ]εά, μέ[τριος] καὶ ἀπράά γά μων ὢνά ἄνθρά [ωπος,] εά ἰ μὴ [ὓβρι]ν τὴν μά [εγ]ίστηνά ἐπά επόνθ[ειν ὑπὸ] Ὡρίωνο[ς κ]ωά μογρα[μ]μά ατέως Φ[ι]λαδελφείά [ας τῆ]ς Ἡρακλεά ίά δου μερίδοά [ς] τά οῦ Ἀρά σινοίτου. [οὗ χά]ριν μην[ύ]ω παρὰ τ[ὰ ἀ]πειρημένα ἑαά [υτὸ]νά ἐνσείσανά τα εἰς τὴν κωμογραμματείανά [μ]ήτε σιτολογήσαντα μήτε πρά [α]κτορεύσαντά α παντελῶς ἄπορον ὄν[τ]αά. δι ἣά ν αἰτίαν κά αὶ πρότερον οὐ διέλιπον ἐντυγχάά νων καὶ νῦά ν ἀξιῶ, ἐάν σου τῇ τύχῃ δόξ[ῃ], ἀκοῦσά αί μου π[ρ]ὸς αὐτὸν πρὸς τὸ τυχεῖν με τά ῆά ςά ἀπὸ σοῦ [μι]σοπονήρου ἐγδ[ι]κίας, ἵν ὦ ὑπὸά [σ]οά ῦά κατὰά πά άντα βά εά βοηθ(ημένος). διευτύχει Ἀντώνιος Δόμνά οά ς ἐπιδέδωκα.
Gap between knowledge and experience
Test questions Statement: My lemmatisation tool <XYZ> is able to compute the baseforms of 80% of all tokens in a corpus. Good or bad???
Test questions Fact file: Language variants Different writing styles (some) Dialects Diachritics OCR errors Question: What is the difference for you?
Test questions Fact file: Language variants Different writing styles (some) Dialects Diachritics OCR errors Question: What do you think is the difference for the computer?
Importance of preprocessing Cleaning and harmonising the data When working with a new corpus (not only language but also same language in a different epoch or geographical region can take up to 70% of the overall time. Preprocessing mantra: Garbage in, garbage out.
Thank you! "Stealing from one is plagiarism, stealing from many is research" (Wilson Mitzner, 1876-1933) Visit us at http://etrap.gcdh.de