| 1 |
Can you ‘read’ tongue movements? Evaluation of the contribution of tongue display to speech understanding”
|
|
|
|
In: http://www.gipsa-lab.fr/%7Epierre.badin/PublisPDF_Badin_Endnote/Tarabalka_badin_Elisei_Bailly_TongueReading_ASSISTH_2007.pdf (2010)
|
|
Abstract:
ASBTRACT A large number of studies has established that the vision of typically visible articulators (lips, jaw, face, tongue tip, teeth) eases speech understanding by humans, and significantly increases the detection and identification performance of words in noise. However, everything cannot necessarily be "read" without ambiguity from the mere vision of face. "Cued Speech", a language used by an increasing number of hearing impaired speakers, aims precisely to complement the lip information by means of a set of hand shapes and positions in relation to the face that provides most of the missing phonetic information, in particular related to tongue articulation. This coding system, although efficient in terms of information theory, is arbitrary and is not directly related to tongue movements. Therefore, we have attempted to determine if direct and full vision of the tongueinformation presumably more intuitive -can be used. We have therefore exploited the virtual audiovisual talking head available at the laboratory that can display all speech articulators, including the tongue. The augmented reality condition chosen is a cutaway profile view. We have elaborated a set of audiovisual VCV stimuli by determining the talking head control parameters though inversion from the positions of the coils of an electromagnetic articulograph glued on the tongue, the jaw and the lips of the subject from which the talking head had been modelled. These stimuli have been played in a audiovisual perception test in four different conditions: audio signal alone (AU), audio signal + cutaway view of the virtual head along the sagittal plane without tongue (AVJ), audio signal + cutaway view with Tongue (AVT), audio signal + complete Face with skin texture (AVF). Each condition was played at four different Signal to Noise Ratios (SNRs) of white noise added to the sound: −∞ (i.e. no audio), −9 dB, +3 dB, +∞ (i.e. no noise). For each stimulus, the subject was forced to make a choice between eight consonants. In order to assess the learning effects, 12 subjects (group I) have transcribed the stimuli presented with decreasing SNRs for each condition, while 11 other subjects (group II) have identified the stimuli with increasing SNRs (which opened the possibility of learning the relation between audio and video when the audio signal was clear at the beginning of the test for a given condition). Another set of VCV stimuli (AVT condition, SNR = -9 dB) has finally been used at the end of the test in order to assess the generalisation abilities of the subjects in both groups. A series of analyses lead to the following results. The identification scores of the group II are significantly higher than those of the group I, which supports the idea that group II has benefited from a stronger implicit learning. All the video presentation conditions give better scores than the audio alone. The scores for all SNRs rank, for each group, with statistically significant differences, in the decreasing order: AVF, AVT, AVJ, AU. For each SNR, AVF is significantly better decoded than AVJ: the subjects prefer an ecological rendering of the movements to a cutaway view. The AVT condition is not significantly better perceived than the AVJ condition except when the audio signal is absent, for the group II, who has benefited from a stronger implicit learning: in this case the AVT score is higher by 18% than the AVJ score. This result suggests that "tongue reading" can take over for the audio information when this latter is not sufficient any longer to complement lip reading. Moreover, the fairly high identification score of a generalisation test proposed at the end of the session with different VCV stimuli, and the global difference of performance between the two groups seem to demonstrate that fast learning can be achieved. These very preliminary results need to be complemented by more systematic tests implying notably visual attention measurements, in order to confirm that natural human tongue reading abilities are weak, or that they are simply dominated by the lip reading ones. However, we envisage elaborating protocols to show that tongue reading learning is fast and easy. Our future goal is thus to use the augmented speech abilities of our virtual talking head for applications in the domain of speech therapy for speech retarded children, perception and production rehabilitation of hearing impaired children, and pronunciation training for second language learners.
|
|
URL: http://www.gipsa-lab.fr/%7Epierre.badin/PublisPDF_Badin_Endnote/Tarabalka_badin_Elisei_Bailly_TongueReading_ASSISTH_2007.pdf
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.1069.3177
|
|
BASE
|
|
Hide details
|
|
|
| 2 |
Author manuscript, published in "Interspeech, Brisbane: Australie Australia (2008)" Can you “read tongue movements”?
|
|
|
|
In: http://hal.archives-ouvertes.fr/docs/00/33/36/88/PDF/pb_IS08.pdf (2008)
|
|
BASE
|
|
Show details
|
|
|
| 3 |
Can you ”read tongue movements
|
|
|
|
In: http://www.gipsa-lab.fr/%7Epierre.badin/PublisPDF_Badin_Endnote/Badin_Tarabalka_Elisei_Bailly_TongueReading_Interspeech_2008.pdf (2008)
|
|
BASE
|
|
Show details
|
|
|
| 4 |
Can you ”read tongue movements
|
|
|
|
In: http://www.gipsa-lab.grenoble-inp.fr/%7Egerard.bailly/publis/synthese/_pbadin/pb_IS08.pdf (2008)
|
|
BASE
|
|
Show details
|
|
|
| 5 |
Author manuscript, published in "Auditory-Visual Speech Processing (AVSP), Moreton Island: Australia (2008)" Speaking with smile or disgust: data and models
|
|
|
|
In: http://hal.archives-ouvertes.fr/docs/00/33/36/73/PDF/gb_AVSP08.pdf (2008)
|
|
BASE
|
|
Show details
|
|
|
| 6 |
Degrees of freedom of facial movements in face-to-face conversational speech
|
|
|
|
In: http://www.gipsa-lab.fr/%7Epierre.badin/PublisPDF_Badin_Endnote/Bailly_Elisei_Badin_Savariaux_DoFFacialMovements_MMC_2006.pdf (2006)
|
|
BASE
|
|
Show details
|
|
|
| 7 |
Degrees of freedom of facial movements in face-to-face conversational speech
|
|
|
|
In: http://www.gipsa-lab.grenoble-inp.fr/%7Echristophe.savariaux/PDF/LREC_2006.pdf (2006)
|
|
BASE
|
|
Show details
|
|
|
| 8 |
Towards a 3D articulatory model of velum based on MRI and CT images [Online resource]
|
|
|
|
In: Speech production and perception : experimental analyses and models / editors Susanne Fuchs, Pascal Perrier and Bernd Pompino-Marschall, Zentrum für Allgemeine Sprachwissenschaft, Sprachtypologie und Universalienforschung (Berlin): ZAS papers in linguistics ; Vol. 40 (2005) 40 (2005), 195-211
|
|
Linguistik-Repository
|
|
Show details
|
|
|
| 9 |
The weight of phonetic substance in the structure of sound inventories
|
|
|
|
In: http://www.zas.gwz-berlin.de/papers/zaspil/articles/zp28/28-vallee.pdf (2002)
|
|
BASE
|
|
Show details
|
|
|
| 10 |
The weight of phonetic substance in the structure of sound inventories [Online resource]
|
|
|
|
In: Papers on phonetics and phonology : The articulation, acoustics and perception of consonants, Zentrum für Allgemeine Sprachwissenschaft, Sprachtypologie und Universalienforschung, Berlin; ZAS papers in linguistics ; Vol. 28 28 (2002), 145-168
|
|
Linguistik-Repository
|
|
Show details
|
|
|
| 11 |
Mother: A new generation of talking heads providing a flexible articulatory control for video-realistic speech animation
|
|
|
|
In: http://hal.inria.fr/docs/00/38/93/62/PDF/icslp00.pdf (2000)
|
|
BASE
|
|
Show details
|
|
|
| 12 |
Mother : A New Generation Of Talking Heads Providing A Flexible Articulatory Control For Video-Realistic Speech Animation
|
|
|
|
In: http://www.icp.grenet.fr/~bailly/publis/synthese/_lr/mother_lr_ICSLP00.ps (2000)
|
|
BASE
|
|
Show details
|
|
|
| 13 |
Towards the Use of a Virtual Talking Head and of Speech Mapping tools for pronunciation training
|
|
|
|
In: http://www.icp.grenet.fr/ICP/publis/acoustique/_pb/STiLL98.ps (1998)
|
|
BASE
|
|
Show details
|
|
|
| 14 |
Articulatory Synthesis From X-Rays And Inversion For An Adaptive Speech Robot
|
|
|
|
In: http://www.asel.udel.edu/icslp/cdrom/vol2/783/a783.pdf (1996)
|
|
BASE
|
|
Show details
|
|
|
| 15 |
you ”read tongue movements
|
|
|
|
In: http://halshs.archives-ouvertes.fr/docs/00/33/36/88/PDF/pb_IS08.pdf
|
|
BASE
|
|
Show details
|
|
|
| 16 |
Three-dimensional modeling of speech organs: Articulatory data and models
|
|
|
|
In: http://hal.archives-ouvertes.fr/docs/00/16/73/78/PDF/IEICE_2006_Badin_Serrurier.pdf
|
|
BASE
|
|
Show details
|
|
|
| 17 |
Virtual Talking Heads and audiovisual articulatory synthesis
|
|
|
|
In: http://www.icp.inpg.fr/ICP/publis/synthese/_autres/vth_pb_03.pdf
|
|
BASE
|
|
Show details
|
|
|
| 18 |
Visual articulatory feedback for phonetic correction in second language learning
|
|
|
|
In: http://www.gavo.t.u-tokyo.ac.jp/L2WS2010/papers/L2WS2010_P1-10.pdf
|
|
BASE
|
|
Show details
|
|
|
| 19 |
Cross-speaker Acoustic-to-Articulatory Inversion using Phone-based Trajectory HMM for Pronunciation Training
|
|
|
|
In: http://www.cstr.ed.ac.uk/downloads/publications/2012/Hueber_etal_IS2012.pdf
|
|
BASE
|
|
Show details
|
|
|
| 20 |
INTERSPEECH 2011 Toward a multi-speaker visual articulatory feedback system
|
|
|
|
In: http://www.cstr.ed.ac.uk/downloads/publications/2011/BenYoussef-etal_IS11.pdf
|
|
BASE
|
|
Show details
|
|
|
|
|
