Parameter Estimation of Stochastic Grammars with Probabilistic Logic Programs

Transcript

1 人工知能学会研究会資料 SIG-FPAI-B Parameter Estimation of Stochastic Grammars with Probabilistic Logic Programs Satoru Yamaguchi 1 Ryo Yoshinaka 1 Akihiro Yamamoto Graduate School of Informatics, Kyoto University Abstract: We propose a parameter estimation method of stochastic grammars by utilizing elementary formal systems and probabilistic logic progarams. A stochastic grammar is a formal grammar where probabilisties are assigned to production rules. Stochastic context-free grammars are well-known because they are used for statistical parsing and predicting secondaty structures of RNA sequences. However, some natural language sentences and RNA sequences have more complicated structure than that expressed with context-free grammars. Our method can be applied to the class of phrase structure grammars. The advantage of our method is from using a kind of logic programs called elementary formal systems (EFSs) to express grammars. We extend them into probabilistic EFSs by assigning probabilities to each clause. We estimate those probabilities by utilizing a parameter estimation method of a kind of logic programs called probabilistic logic programs (PLPs). We convert a probabilistic EFS into an extended PLP and apply the parameter estimation method to the PLP. 1,,,. (Elementary Formal System, EFS) EFS, EFS (Probabilistic Logic Program, PLP) EFS EFS EFS, EFS,,. Cuturi yamaguchi.satoru@iip.ist.i.kyoto-u.ac.jp, RNA [2][4]., Inside-Outside EM,,, (Multiple Context-Free Grammar), RNA.., RNA [5].,,, {a 2n n 1} {a n b n c n n 1},,., (Elementary Formal System, EFS)

2 [6]. EFS EFS, EFS EFS, EFS [3]., EFS,, EFS, EFS Σ Π Σ, Π Σ Π X. X. X Σ, 1. n p t 1,..., t n, p(t 1,..., t n ),,. h 1,..., h m, b 1,..., b n (m, n 0) h 1,..., h m : b 1,..., b n., h 1,..., h m, b 1,..., b n. : b 1,..., b n. h : b 1,..., b n.,.. Γ. (Σ, Π, Γ) (Elementary Formal System, EFS) Γ, ({a, b, c}, {p, q}, Γ) EFS. p(xy Z) : q(x, Y, Z) (1) q(ax, by, cz) : q(x, Y, Z) (2) q(a, b, c) : (3) v 1,..., v n (n 0), t 1,..., t n, {v 1 /t 1,..., v n /t n }. t, θ, tθ t v i (i = 1,..., n) t i. a = p(t 1,..., t n ), aθ = p(t 1 θ,..., t n θ). c = h : b 1,..., b n, cθ = hθ : b 1 θ,..., b n θ. cθ, cθ c. Γ c, Γ c. 1. c Γ Γ c. 2. θ, Γ c Γ cθ. 3. Γ h : b 1,..., b n Γ b n : Γ h : b 1,..., b n 1. Γ c c Γ( EFS(Σ, Π, Γ)) EFS E = (Σ, Π, Γ) 1 p Π, L(E, p) = {t Σ + Γ p(t) : } Σ. L L(E, p) = L EFS E p, L EFS EFS E, L(E, p) = { a n b n c n n 1}. e, V (e) e h : b 1,..., b n V (h) V (b i ) (i = 1,..., n),, EFS (Σ, Π, Γ), Γ, EFS., V (h) n i=1 V (b i), Γ, EFS EFS EFS, EFS 1. EFS. a 1, a 2, a 1 θ = a 2 θ θ a 1, a 2. a 1, a 2, a 1, a 2 EFS E = (Σ, Π, Γ) g, E g, g i, c i θ (g i, c i, θ i ). 1. g 0 = g. 2. g i =: h 1,..., h k, i h 1 h:- b 1,..., b n,c i = h : b 1,..., b n θ i a 1 a, g i+1 = (b 1,..., b n, h 2,..., h k )θ i. ii, c i =, θ i = {} (g i, c i, θ i ). 3 g i =, c i =, θ i = {} (g i, c i, θ i )

3 (,, {}) 2.2 EFS, EFS C, Π Ψ. C, Π, Ψ.. X C, Π Ψ X.,.,, Xyz X y,z. [t 1,..., t n ] (n 0) [t 1,..., t n v] (n 1), t 1,..., t n, v s 1,..., s m, [t 1,..., t n [s 1,..., s n v]] [t 1,..., t n, s 1,..., s n v], [t 1,..., t n [s 1,..., s n ]] [t 1,..., t n, s 1,..., s n ].,. p(t 1,..., t n ). p n, t 1,..., t n.,, a, a \+a. a, \ + a. a,. h b 1,..., b n, h : b 1,..., b n h, b 1,..., b n.,., h.,. (Probabilistic Logic Program, PLP) D L. c L 0 1 w c, c = h : b 1,..., b n, w c :: h b 1,..., b n append([],x,x). append([h X],Y,[H Z]):-append(X,Y,Z). p(xyz):-append(x,yz,xyz), append(y,z,yz), q(x,y,z). 0.8::q([a X],[b Y],[c Z]):-q(X,Y,Z). 0.2::q([a,b,c]). v 1,..., v n (n 0), t 1,..., t n, {v 1 /t 1,..., v n /t n }. t, θ, tθ t v i (i = 1,..., n) t i. a = p(t 1,..., t n ), aθ aθ = p(t 1 θ,..., t n θ). a, aθ = a. l l = a, lθ = aθ, l = \ + a lθ = \ + aθ. c = h : b 1,..., b n, cθ cθ = hθ : b 1 θ,..., b n θ. cθ cθ c. Γ c, Γ c. 1. c Γ Γ c. 2. θ, Γ c Γ cθ. 3. Γ h : b 1,..., b n,b 1 Γ b n : Γ h : b 1,..., b n Γ h : b 1,..., b n,b 1 Γ b n : Γ h : b 1,..., b n 1. Γ c c Γ. PLP T = (D, L), L G T, G G T. T G { }{ } P (G T ) = w c (1 w c ). (4) cθ G cθ / G PLP T l. P (l T ) = P (G T ). (5) G G T,G D l PLP T, P (l). 2.3 PLP. PLP, PLP T = (D, L), p(i), L PLP, Deterministic Decompos- I = {i j j 1}, n j=1-83 -

4 able Negation Normal Form (d-dnnf) Knowledge Compilation [3]. d-dnnf, PLP T (Conjunctive Normal Form, CNF) d-dnnf, d-dnnf [3]., d-dnnf, d-dnnf, d-dnnf 3 Elementary Formal System 4 EFS EFS PLP, PLP EFS. EFS PLP., PLP EFS PLP, EFS PLP ( ), EFS PLP EFS EFS. EFS EFS E = (Σ, Π, Γ), (Σ, Π, Γ, Ω, p 0 ) EFS. Ω 0 1, p 0 Γ 1. c Γ w c Ω. p Π Γ p, c Γ p w c = 1. w c c = h : b 1,..., b n w c :: h : b 1,..., b n 3.1. Γ,, ({a, b, c}, {p, q}, Γ, {1.0, 0.8, 0.2}, p) EFS. 1.0 :: p(xy Z) : q(x, Y, Z) (6) 0.8 :: q(ax, by, cz) : q(x, Y, Z) (7) 0.2 :: q(a, b, c) : (8) EFS E = (Σ, Π, Γ, Ω, p 0 ) Σ. EFS EFS. d = (g 1, c 1, θ 1 ),..., (g n, c n, θ n ) g E., d P (d) = { wc1 P ((g 2, c 2, θ 2 ),..., (g n, c n, θ n )) (if c 1 ) 1 (otherwise) g S g. g P (g) = d S g P (d) (9). Σ t P (t) = P (: p 0 (t)). 4.1 EFS, PLP, EFS aabbcc a PLP [a, a, b, b, c, c] [a]., EFS,, PLP, EFS PLP,,. append append([], Xs, Xs). (10) append([x Xs], Ys, [X, Zs]) : append(xs, Ys, Zs). (11) append(xs, Ys, Zs) Zs Xs Ys PEFS 0.7 :: p(xy Z) : q(x, Y ), r(z) (12) PLP, 0.7 :: p(xyz) : append(x, Yz, Xyz), append(y, Z, Yz),. q(x, Y), r(z) (13)

5 4.2 EFS PLP. PLP 4.3. EFS p, p Γ p w c1 :: h 1 :. (14) w c2 :: h 2 : b 1, b 2. (15) w c3 :: h 3 : b 3. (16) c 1, c 2 c 3. PLP h 1 : choose p 1. (17) h 2 : b 1, b 2, choose p 2. (18) h 3 : b 3, choose p 3. (19) w c1 :: choose p 1. (20) w c2 /(1 w c1 ) :: choose p 2 : \ + choosep 1. (21) choose p 3 : \ + choosep 1, \ + choosep 2. (22), choose p i, p i,,. (20) (22), c i. (20), choose p 1 w c 1., (21) choose p 2, choose p 1. c 1 c 2. (22), choose p 3 choosep i,,. 4.4, p(xy) : q(x), r(y) EFS, p(abc) p(abc) : q(ab), r(c) p(abc) : q(a), r(bc), PLP, PLP. excl., PLP c, V (c) c v 1,..., v m, c h : b 1,..., b n, excl([v 1,..., v k ], [v k+1,..., v m ]) (23). {v 1,..., v k } V (c) (24) {v k+1,..., v m } V (c)\{v 1,..., v k } (25). excl c,. PLP CNF,, 4.5 EFS PLP, EFS,,,,, PLP,,.. PLP,.,,, EFS, c Γ p i

6 ,, append j [p, i, j]., (13) Γ p,. 0.7 :: p(xyz, C) : append(x, Y, Xy), append(xy, Z, Xyz), 4.6 q(x, Y, [[p, 3, 1] C]), r(z, [[p, 3, 2] C]), choose p 3 (C), excl([x, Y, Xy, Z], [Xyz, C]).. EFS c, clause P LP (c, C) C PLP,. 1., i,. ii, 2. append 3. choose p i , excl(l 1, l 2 )., l 1, l 2 EFS E = (Σ, Π, Γ, Ω, p 0 ) PLP T = (D, L) 1. D, L. 2. D (10) (11). 3. p Π. i i = 1,..., n, c p i Γ p i a L w c p i :: choose p i (C) : \+choosep 1 (C),..., \+choose p i 1 (C).. b D clause P LP (c p i, C)., w = w c p c p /(1 w i i c p w 1 c p ). i 1 PLP p 0 p 0, EFS t t p 0 (t) = p 0(t, []). PLP, EFS 5 (EFS) EFS, EFS EFS (PLP), PLP. EFS PLP., PLP [1] Adnan Darwiche: On the tractability of counting theory models and its application to belief revision and truth maintenance, Journal of Applied Non-Classical Logics 11(1-2): 11-34, (2001). [2] Rogin D. Dowell and Sean R. Eddy: Evaluation of several lightweight stochastic context-free grammars for RNA secondary structure prediction, BMC Bioinformatics, (2004). [3] Daan Fierens, Guy Van den Broeck, Joris Renkens, Dimitar Shterionov, Bernd Gutmann, Ingo Thon, Gerda Janssens and Luc De Raedt: Inference and learning in probabilistic logic programs using weighted Boolean formulas, Theory and Practice of Logic Programming, (2013). [4] Daniel Jurafsky, Chuck Wooters, Jonathan Segal, Andreas Stolcke, Eriv Fosler, Gary Tajchman and Nelson Morgan: Using a stochastic contextfree grammar as a language model for speech recognition, Acoustics, Speech, and Signal Processing, (1995). [5] Yuki Kato, Hiroyuki Seki, and Tadao Kasami: RNA Structure Prediction Including Pseudoknots Based on Stochastic Multiple Context-Free Grammar, Workshop on Probabilistic Modeling and Machine Learning in Structural and Systems Biology, (2006). [6],, :,, pp , (1999)