Discriminative Language Modeling Based on Risk Minimization Training

Μέγεθος: px

Εμφάνιση ξεκινά από τη σελίδα:

Download "Discriminative Language Modeling Based on Risk Minimization Training"

Ἠώς Καλύβας
5 χρόνια πριν
Προβολές:

1 1,a) Bayes Dscrmnatve Language Modelng Based on Rsk Mnmzaton Tranng Kobayash Ako 1,a) Oku Takahro 1 Fujta Yuya 1 Sato Shoe 1 Nakagawa Sech 2 Abstract: Ths paper descrbes dscrmnatve language models (LMs) that reflect nformaton about word errors n automatc speech recognton (ASR). The dscrmnatve LMs are mplemented as a set of penalty scores employng lngustc features and ther weghtng factors. The models are estmated n the bass of mnmzaton of expected rsks that are closely assocated wth word errors. In transcrbng Japanese broadcast programs, the sem-supervsed dscrmnatve LM acheved the best results n word error rates compared wth the supervsed and unsupervsed LMs and conventonal dscrmnatve LMs based on maxmzaton of condtonal log-lkelhoods. Keywords: Bayes rsk, dscrmnatve tranng, mult objectve programmng, sem-supervsed tranng 1. [1] 1 NHK NHK Scence and Technology Research Laboratores, Setagaya, Tokyo , Japan 2 Toyohash Unversty of Technology, Toyohash, Ach, , Japan a) kobayash.a-fs@nhk.or.jp [2] c 2012 Informaton Processng Socety of Japan 1

2 / [3], [4] [3] [4] [5] [6], [7] Bayes [8] (Mult-objectve Optmzaton Programmng, MOP)[9] [6], [7] ( ) λ Λ 2.2 Bayes Bayes N-best [8] ŵ = arg mn R(w, w )P (w x) (2) w w P (w x) x N-best w R(w, w ) w w 2 Levenshten ( ) Bayes (1) Λ 2.3 [4], [5] x (u) m (m =1,...,M) k w m,k () U(Λ) = 1 M P (w m,k x (u) m ; Λ)χ(w m,k ) (3) m k P (w m,k x (u) m ; Λ) w m,k (1) (3) χ(w m,k ) 2. χ(w m,k )= k R(w m,k, w m,k )P (w m,k x (u) m ; Λ) (4) 2.1 x w { } P (w x; Λ) exp f am (x w)+λ lm f lm (w)+ λ f (w) (1) f am (x w) f lm (w) f (w) x (l) n L(Λ) = 1 N P (w n,k x (l) n ; Λ)R(w ref n, w n,k ) (5) n k (4) (5) c 2012 Informaton Processng Socety of Japan 2

3 (Mnmum Phone Error, MPE) [10] Bayes Levenshten x m L m 2 e e l(e, e ) [5] l(e, e 0 label(e) = label(e ) ) 1 label e ζ(e) e overlap(e) l(e, e )p(e ) (6) overlap e p(e ) p(e) = 1 {α(σ(e)) s(e) β(τ(e))} (7) ᾱ σ(e) e τ(e) α(σ(e)) σ(e) ᾱ β(τ(e)) τ(e) s(e) φ am (e) φ lm (e) { } s(e) = exp λ am φ am (e)+λ lm φ lm (e)+ λ φ (e) (8) λ am λ lm φ (e) f e 1 0 ζ(e) p(e) γ m m γ m/m Λ [4] e δ m,e δ m,e = p(e)(γ(e) γ m )φ (e) (9) γ(e) e m λ e L m δ,e m (u) (u) = 1 δ,e m (10) M m e L m (6) (5) (l) 2.4 [3] L(Λ) = 1 log P (w ref n x (l) n ; Λ) (11) N n [11] m x (u) m U(Λ) = 1 P (w m,k x (u) m ; Λ) log P (w m,k x (u) m ; Λ) M m k (12) (12) [9] 2 c 2012 Informaton Processng Socety of Japan 3

4 [6], [7] ( ) Λ ε [12] 2 ε [9] L(Λ) U(Λ) Λ = arg mn L(Λ) subject to U(Λ) Ū (13) Λ Ū Ū=αU(0) (14) α(< 1.0) Λ=0 5% 20 % L(Λ) [13] 2 κ F(Λ) = L(Λ) + ρ 2ρ + U(Λ) Ū (15) κ ρ x max {x, 0}. (13) F ( ) F(Λ) = (l) κ +2ρ λ 2ρ + U(Λ) Ū (u) (16) (15) κ ρ [13] 2 α f =h1(u 1,u 2,u 3)(w) =c u1,u 2,u 3 (w) (17) w 3 (u 1,u 2,u 3 ) c u1,u 2,u 3 h 1 (u 1,u 2,u 3 ) 3 [4] [14] w q () MFCC HMM bgram 200-best trgram 650 MPE trgram ( 239M ) 100k NHK 3 ( 1) ( ) 1 2 (PP) (WER) (OOV) trgram c 2012 Informaton Processng Socety of Japan 4

5 Table 1 1 Evaluaton data for dscrmnatve language modelng PP OOV(%) WER(%) k k Table 2 Tranng data for dscrmnatve language modelng k 697.5k k 2.84M 3 Table 3 Perplextes and word error rates for tranng data Table 4 4 Feature functons for dscrmnatve language modelng 2 1.3k k k k PP OOV(%) WER(%) GER(%) [2] (4.5 ) 2.17 k (47.2 k ) 5 3 (GER) L-BFGS [15] 1 α ( 4) 3.2 ( ) 5 [7] 21.5 % 3.6 % % 6.3 % () 2.8 % ( 5%) ( 5 %) % c 2012 Informaton Processng Socety of Japan 5

6 5 (%) Table 5 Expermental results for dscrmnatve language modelng (WER,%) () ( ) ( ) ( ) (, %) Table 6 Sem-supervsed dscrmnatve language modelng wth varous amounts of unlabeled tranng data (WER, %) + 10 % % % % ( %) Table 7 Sem-supervsed dscrmnatve language modelng wth varous amounts of unlabeled tranng data (rsk mnmzaton, %) VTR 10 % % % % (100 %) (3.8 k ) VTR(, 3.3 k ) 2 ( 7) VTR VTR VTR 29.2 % 27.7 % 5.1 % VTR 4.3 %(16.4 % 15.7 %) VTR c 2012 Informaton Processng Socety of Japan 6

7 8 (%) Table 8 Comparson of feature functons (%) %(29.2 % 27.0 %) 4.3 ( 8) 21.5 % 22.4 % (22.3 %) 5. [16] 1 [1],,,,, :,, Vol. 63, No. 3, pp (2008). [2],,,, :,, Vol. J93-D, No. 10, pp (2010). [3] Roark, B., Saraclar, M. and Collns, M.: Dscrmnatve n-gram language modelng, Computer Speech and Language, Vol. 21, pp (2007). [4],,,,, :,, Vol. J93-D, No. 5, pp (2010). [5] Kobayash, A., Oku, T., Homma, S., Ima, T. and Nakagawa, S.: Lattce-based rsk mnmzaton tranng for unsupervsed language model adaptaton, Proc. Interspeech, pp (2011). [6] Kobayash, A., Oku, T., Ima, T. and Nakagawa, S.: Mult-objectve optmzaton for sem-supervsed dscrmnatve language modelng, Proc IEEE ICASSP, pp , (2012). [7] Kobayash, A., Oku, T., Ima, T. and Nakagawa, S.: Rsk-based sem-supervsed dscrmnatve language modelng for broadcast transcrpton, IEICE Trans. Inf. & Syst., Vol.E95-D, No.11 (2012, n press). [8] Goel, V. and Byrne, W.: Mnmum Bayes-rsk automatc speech recognton, Computer Speech and Language, Vol. 14, pp (2000). [9] Marler, R. T. and Arora, J. S.: Survey of multobjectve optmzaton methods for engneerng, Structural and multdscplnary optmzaton, Vol. 26, pp (2004). [10] Povey, D. and Woodland, P. C.: Mnmum phone error and I-smoothng for mproved dscrmnatve tranng, Proc. ICASSP, pp. I (2002). [11] Grandvalet, Y. and Bengo, Y.: Sem-supervsed learnng by entropy mnmzaton, Advances n neural nformaton processng systems, pp (2005). [12] Mettnen, K.: Nonlnear multobjectve optmzaton, Sprnger, [13] Snyman, J.: Practcal mathematcal optmzaton, Sprnger (2005). [14],,,, :, ( ), No. 2-P-35(a) (2011). [15] Lu, D. and Nocedal, J.: On the lmted memory BFGS method for large scale optmzaton, Mathematcal Programmng, Vol. 45, No. 3, pp (1989). [16] Lehr, M. and Shafran, I.: Dscrmnatvely estmated jont acoustc, duraton and language model for speech recognton, Proc. ICASSP, pp , (2010). c 2012 Informaton Processng Socety of Japan 7

Σχετικά έγγραφα

A Sequential Experimental Design based on Bayesian Statistics for Online Automatic Tuning. Reiji SUDA,

A Sequential Experimental Design based on Bayesian Statistics for Online Automatic Tuning. Reiji SUDA, Bayes, Bayes mult-armed bandt problem Bayes A Sequental Expermental Desgn based on Bayesan Statstcs for Onlne Automatc Tunng Re SUDA, Ths paper proposes to use Bayesan statstcs for software automatc tunng