Dynamic sign synthesis from a virtual character to support language teaching to deaf students Eleni Efthimiou eleni_e@ilsp.gr Supporting Technologies Group Institute of Language and Speech Processing
Language Technology Integrates human knowledge on languages with IT systems Allows for and aims to human-computer interaction in human-human terms
Natural Language Processing Grammar approach Statistical approach Combination of both methods
precondition: Language resources: > Syntactic and semantic dictionaries > Electronic corpus
Supporting Technologies in the framework of Language Technology Research and applications in Speech Technology -Speech synthesis -Speech recognition Research and applications in Natural Language Analysis and Processing -Sign language applications
Sign language modeling requires: Analysis of sign language data Creation of digital language resources Production of representations of signed syntactic patterns Resource coding wrt the phonological characteristics of sign languages
e-learning for all Demand for universal access to Information Society technologies Traditional activities in HCI should be redesigned to allow access from all users
Designing accessible educational platforms An accessible platform must provide HCI means such as: Speech synthesis and recognition Virtual signers Procedures for simplified keyboards and switchbased navigation to support all possible users Our implementation: develop technologies to allow a virtual signer to mediate between a teacher and a student to present GSL educational material
e-learning in GSL Avatar animation technologies with GSL language resources to develop and refurbish educational material Application: teach GSL grammar to young elementary school students The same platform allows deaf students to access written language material in the form of signed summaries Harmonized with Universal Access and Design for All requirements, in the framework of the Information Society program
Development characteristics Scripting: STEP (Scripting Technology for Embodied Persona) Advantage: separate gesture and sign definition from avatar geometry and hierarchy definition Result: sign definitions can be modified without the need to re-model the virtual character h-anim compliant Avatars Web page deployment maximizes accessibility The web page includes embedded VRML objects (avatar), STEP engine references and JavaScript function calls
Input: written Greek text converted to GSL signs and displayed on screen
Extensions include Mouthed phonemes Facial expressions Body movement which usually declare non-verbal information in SLs Multi-sing units may be composed of a) free signs b) combinations of free and bound signs to create sign sequences, according to the rules of either inflectional or derivational morphology, or to construct sign phrases
Language resources: - vocabulary (HamNoSys notation) - computational grammar (ELAN notation) Combination of resources representation of any given signed phrases
Greek-to-GSL conversion: an expandable tool Future work: Library extension to cater for non-manual structural components that will support signing performance of the avatar closer to natural signing Enrichment of the system s linguistic knowledge will also allow for improvement of its generative capacity New written texts can be launched, so the e-learning platform may receive unlimited educational content besides primary school grammar units From a socio-economic point of view, the existence of such a tool will contribute towards ensuring equal opportunities for the deaf population in Greece
Related international projects: VISICAST, SignSynth, esign, Thetos, http://www.leidenuniv.nl/hil/sign-lang/sl-sites.html#technical http://www.sign-lang.uni-hamburg.de/quellen/default.html http://www.fhs-hagenberg.ac.at/mtd/projekte/fff/3dsign