Electro-Magnetic Articulography (EMA) has been a rapidly growing technology for accurate measurement of articulatory kinematics. This technology is based on measuring the induced current caused by motion of encapsulated miniature toroid coils in a system of electromagnetic fields.
The Marquette Speech and Swallowing Lab, directed by Professor Jeffrey Berry, has an EMA Wave System manufactured by Northern Digital, Inc. Our NDI Wave system captures both position and orientation at a sampling rate of up to 400 Hz, with position error on the scale of +/- 0.5mm. A single sensor captures 5 Degree of Freedom (DOF) information, including 3 dimensional position information plus the 2-dimensional orientation of the sensor plane. A 6 DOF sensor can be constructed using dual non-planar coils to capture full orientation information.
The simultaneous acoustic and articulatory kinematic data collected via this system is used to pursue research on a range of topics, including:
- Acoustic to Articulatory inversion with applications to Computer Aided Language Learning and Pronunciation Assessment
- Articulatory to Acoustic Synthesis mapping
- Kinematics-driven voice modification for real-time acoustic feedback
We have several National Science Foundation funded projects related to this EMA work. These projects include:
-
EAGER: Acoustic-Articulator Modeling for Pronunciation Analysis, NSF IIS-1142826 (Dataset web page: EMA-MAE: EMA database of Mandarin-Accented English)
In order to support effective learning and provide specific, useful pronunciation feedback to users, Computer Aided Language Learning (CALL) systems for pronunciation correction must be able to capture pronunciation errors and accurately identify and describe errors in articulation. To do this, it is necessary to estimate articulator trajectory patterns from users’ acoustic data. Due to the difficulty of acoustic-articulator inversion and the complexities of inter-speaker differences in articulator patterns, this capacity is not yet well developed. Current systems are limited in the specificity of the corrective feedback that is provided, often only providing a “good versus bad” pronunciation match to the target and even at best only providing the general category of pronunciation error. This project, funded by the NSF through the EAGER program, aims to address these key limitations through collection of a matched acoustic and five degree of freedom electromagnetic articulograph (EMA) data corpus for both native American English (L1) speakers and native Mandarin Chinese (L2) speakers who speak English as a second language. This has potential to be used for a variety of research efforts, including areas such as pronunciation variation modeling, acoustic-articulator inversion, L2-L1 speaker comparisons, pronunciation error detection, and corrective feedback for accent modification.
-
RI: Small: Speaker Independent Acoustic-Articulator Inversion for Pronunciation Assessment, NSF IIS-1320892
This project addresses the problem of robust speaker-independent acoustic-to-articulator inversion, with a focus on pronunciation assessment applications. Acoustic-to-articulator inversion, the estimation of articulatory trajectories from an acoustic signal, is a challenging problem due to the complexity of articulation patterns and significant inter-speaker differences, and is even more so when applied to non-native speakers without any kinematic training data. We propose to address this problem through development of a robust normalized working space for articulatory representation and use of a novel speaker-independent inversion approach called Parallel Reference Speaker Weighting (PRSW), which uses parallel acoustic-articulator adaptation to create speaker-specific models for new speakers without any kinematic training data. The approach will be evaluated on our newly developed acoustic / 3-D electromagnetic articulography (EMA) dataset of native American English (L1) speakers and native Mandarin Chinese (L2) speakers who speak English as a second language.