The State of the Art and Difficulties in Automatic Chinese Word Segmentation

HTML
DOWNLOAD

Μέγεθος: px

Εμφάνιση ξεκινά από τη σελίδα:

Download "The State of the Art and Difficulties in Automatic Chinese Word Segmentation"

Μυρίνη Παπακωνσταντίνου
6 χρόνια πριν
Προβολές:

1 138 Vol. 17 No. 1 JOURNAL OF SYSTEM SIMULATION 1 2 1, ( ) X (2005) TP391 A The State of the Art and Difficulties in Automatic Chinese Word Segmentation ZHANG Chun-xia 1 2, HAO Tian-yong 1 ( 1 Institute of Computing Technology, Chinese of Academy Sciences, Beijing, , China; 2 Graduate School of the Chinese of Academy Science, Beijing , China) Abstract: Automatic Chinese word segmentation is a basic research issue on Chinese information processing tasks such as information extraction, information retrieval, machine translation, text classification, automatic text summarization, speech recognition, text-to-speech, natural language understanding, and so on. Though it has been investigated for more than twenty years, it is still a bottleneck for Chinese information processing. We give a detailed analysis of the state of the art in automatic Chinese word segmentation, build a formal model of word segmentation, discuss factors affecting word segmentation and the two great difficulties in word segmentation and their resolutions, and finally, point out the existing problems, especially those on the word segmentation evaluation, as well as the research problems to be resolved. Keywords: automatic Chinese word segmentation; formal model; unknown words; word segmentation evaluation 1 (1) [9] (1) (2) 70% [10] ( ) (3) ( ) (4) [9, 11] (2) (3) [1] [6] [2 5, 6-7] [13] [6] [1-2,9,11] [9,11] (# # ) (1) (#2001CCA03000 #2002DEA30036) (2) (1974 ),,,, (1981 ),,,, (3)

2 , 139 (3) (4) (1) (2) 1 (5) (3) (4) [20] [12] N Markov (1) (2) (1) (2) (1) m (3) m (a) (2) x y p ( x, y) MI(x, y)= log 2 p ( x ) p ( y ) p(x,y) x y p(x) p(y) x y MI(x, y) [21] (1) MI(x, y)>>0 x, y [13] MI(x, y) (2) MI(x, y) 0 x, y Hash [14] (3) MI(x, y)<<0 x, y MI(x, y) Hash (b) N I/O 1.56 [15] [19] Sproat [4] N [16] [17] (c) TRIE [22] [8] (1) [18] (2) (BP PNN) 2.89 [11,15,19] (1) (1) (2) (2) (3) [20]

3 140 Vol. 17 No. 1 (1) 94.3 (2) (3) (1) (2) (3) (4) 1.3 (d) Markov [9] [11] Markov ( ) Masao Utiyama [23] [27] (1) (2) (3) (1) (2) (4) (5) (6) [11] Markov Palmer [2] (e) EM [24] EM EM 80% 20% Kok [28] (f) [25] N-gram [29] M N 1 M N N-gram N N T={S 1, S 2, S 3,, S m } N-gram m N N S i (1) (2) C={C 1, C 2, C 3,, C n } n N C i (i=1, 2,,n) 1 (g) [26] x 2-2 S=S 1 S 2 S 3 S p p(p N ) S i T (i=1, 2,, p) [20]

4 , ACWSS(Automatic Chinese Word Segmentation System) (St, Lk, Si, A, Ewsr, Gwsr) (1) St (Source Texts) (2) Lk (Linguistic Knowledge) (3) Si (Statistic Information) (4) A (Algorithm) (5) Ewsr (Experimental Word Segmentation Result) Result) (6) Gwsr (Goal Word Segmentation ACWSS S 1, S 1 S 2, S 2 S 3, S 3 Mrd(Machine Readable Dictionary) Si S=S 1 S 2 S 10 S 1 ',S 2 ' 4 S S' S S 2 '=S 1 S 2 /S 3 S 4 /S 5 S 6 /S 7 S 8 S 9 S 10 S'=W 1 /W 2 /W 3 / /W m / / W i Mrd (i=1, 2,,m) W i 5 S S' S S'=W 1 '/W 2 '/W 3 '/ /W m '/ S / S' W i ' 9 S k S 1 '=S 1 S i1 /S i1+1 S i2 /S i2+1 S ij /S ij+1 S ik /S ik+1 S m S 2 '= S 1 S i1 /S i1+1 S i2 /S i2+1 S ij /S ij+1 S ik /S ik+1 S m S S 6 S 1 S S 1 ', S 2 ', S n ' S 1 ' S 2 ' S 1 ' S 2 ' SSN(S') S' n i, j= 1 (Si'- Sj') S SSN(T S1' ) SSN(T S2' ) T S 1 ' S 2 ' S S S=S 1 S 2 S 10 S 1 ', S 2 ', S 3 ' S 1 '=S 1 S 2 /S 3 S 4 /S 5 S 6 S 7 /S 8 S 9 S 10 S 2 '=S 1 /S 2 S 3 S 4 /S 5 S 6 S 7 /S 8 S 9 /S 10 S 3 '=S 1 S 2 /S 3 S 4 /S 5 S 6 S 7 /S 8 /S 9 S 10 S S 1 S 2 S 3 S 4 S 8 S 9 S 10 7 S S 1 S 2 S 1 S 2 S 1 /S 2 / S 1 S 2 / S 1 S 2 S 1, S 2, S 1 S 2 8 S S 1 S 2 S 3 S 1 S 2 S 3 S 1 S 2 /S 3 / S 1 /S 2 S 3 / S 1 '=S 1 S 2 S 3 /S 4 S 5 S 6 /S 7 S 8 S 9 S 10 S 1 S 2 S 3 S 1 S 2 S 3 S 4 S 5 S 6 S S 1 ' S 2 ' S 1 S i1 S i1+1 S i2 S ij+1 S ik S ik+1 S m S / T S SSN(T S1' ) = SSN(T S2' ) T 3 [30]

5 142 Vol. 17 No. 1 E-step [30] M-setp EM [27] 3.1 [11] [33] (1) (2) (3) [31] 3.2 [32,33] T-Test Markov SVM k-nn [34] [35] EM [32] T- m (1) n (2) n/m [33] T [37] (1) (1) (2) (3) (2) ( (1) ) (1) (2) (3) (2) // (1) [34] Bigram (2) Markov Keh-Jiann Chen [38] [35] SVM K-NN (1) (2) (3) SVM SVM k-nn [36] EM (1) (2) (1) (2) Chiang [39] 4 ME (3)

6 , 143 (a) (b) [1] 01 (GB ) [M]., [40] [2] Robert Dale, Herman Moisl, Harold Somers. Handbook of Natural Language Processing [M]. New York: Marcel Dekker, Inc, [33] [3] David D. Palmer. A Trainable Rule-based Algorithm for Word Segmentation [A]. Proceedings of the 35 th Annual Meeting of the Association for Computational Linguistics [C] [4] Richard Sproat, Chilin Shih, William Gale, and Nancy Chang. A Stochastic Finite-State Word-Segmentation Algorithm for Chinese (c) [J]. Computing Linguist, 1996, 22(3): [5]. [J]., 1998, 4: [6],. 10 [M] ACWSS 1 =(St 1, Lk 1, Si 1, A 1, Ewsr 1,, 2001, 15(2): 1-8. Gwsr 1 ) ACWSS 2 = (St 2, Lk 2, Si 2, A 2, Ewsr 2, Gwsr 2 ) [8]. [J]., 1999, 16(10): 8-9. (1) St 1 =St 2 [9]. [J]. (2) Gwsr 1 =Gwsr 2 11 ACWSS 1= (St 1, Lk 1, Si 1, A 1, Ewsr 1, Gwsr 1 ) ACWSS 2 =(St 2, Lk 2, Si 2, A 2, Ewsr 2, Gwsr 2 ) [7]. [J]., 1992, 29(11): [10],,. [C] [11]. [J]., 2002, 14(5): [12],. [J]., 1999, 36(9): (1) Gwsr 1 Gwsr 2 [13]. [J]. (2) St 1 St 2 [J], 2002, 18(3): [14]. [J]., 2002, 38(11): [15]. [J]., F-measure F=(1+β)PR/(βP+R) [3] 2000, 23(2): [16]. [J]., 1997, 34(7): St 1 St 2 [17]. [J]., 2000, 14(1):1-6. Gwsr 1 Gwsr 2 [18]. [J]., 1996, 33(4): [19]. [J]., 1997, (d) 20(4): [20]. [J]., 2001, 15(6): [21] Yubin Dai, Teck Ee Loh, Christopher Khoo. A New Statistical Formula for Chinese Text Segmentation Incorporating Contextual Information [C]. Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval [22]. [J]. 5, 1998, 17(1): [23] Masao Utiyama, & Hitoshi Isahara. A Statistical Model for Domain-Independent Text Segmentation[C]. The 39th Annual Meeting of the Association for Computational Linguistics and 10th Conference of the European Chapter of the Association for Computational Linguistics

7 , RBF : [6] Cirrincione M Miceli R, Galluzzo. G R and Trapanese M. Preisach function identification by neural networks[j]. Magnetics, IEEE Transactions on, 2002,38(5): [7] Cincotti S, Marchesi M, and Serri A. A neural network model of RBF parametric nonlinear hysteretic inductors [J]. Magnetics, IEEE Transactions on 1998, 34(5): [8],,. [J]., 2000, 22(4): [9],. [J]. 2002, 13(5): [1] Mayergoyz I D, Friedman G, and Salling C. Comparison of the classical and generalized preisach hysteresis models with experiments[a]. Magnetics Conference, Digests of NTERMAG '89. International[C]. 1989, HC2 -HC2. [2] Jiles D C, Atherton D L. Theory of ferromagnetic hyseteresis[j]. J. Magn. MagnMater, 1986, 61: [3] Serpico C, Vison C. Magnetic Hysteresis Modeling via Feed-Forward Neural Networks [J]. IEEE Transactions on Magnetics, 1998,34(3): [4] Cincotti S, Daneri I. Neural network identification of a nonlinear circuit model of hyseteresis[j]. Electronics Letters, 1997, 33(13): [5] Cruz-Hernández J M, Hayward V. On the linear compensation of hyseteresis[a]. in 36th IEEE Conf. Decision Contr.[C]. 1997, 143 [24]. EM [J]. [J]., 2000, 15(1): , (3): [25]. Chinese by a Corpus-based Learning Method [J]. International [J]., 2001, 15(6): [26]. [J]. Processing. 1998, 3(1): , 1996, 9(4): [27],,. Statistical Models for Word Segmentation and Unknown Word [J]., 2001, 15(1): [28] Kok Wee Gan. Integrating Word Boundary Identification with Sentence Understanding [C]. The 31 st Association for Computational Linguistics Annual Meeting of the [29],,. [41]. [J]., 1989, 3(1): [A]. 9 [C] [30].., 2001, [43]. [J]., 15(5): [32]. [J]., 2002, 38(11): [33]. 6(1): [J]., 1997, 34(5): [34]., 1995, 4(4): [J]., 1999, 39(5): [35]. SVM K-NN 1997, 6(1): [J]., 2001, 15(6): [36]. EM, (5): [J] , 15(2): [37],,,,. [38] Keh-Jiann Chen, Ming-Hong Bai. Unknown Word Detection for Journal of Computational Linguistics & Chinese Language [39] Tung-Hui Chiang, Jing-Shin Chang, Ming-Yu Lin, and Keh-Yih Su. Resolution [C]. Proceedings of ROCLING-V, ROC Computational Linguistics Conference [40]. [J]., 1999, 8(2): [42]. [M]., , 22(2): [31]. [M] [44]. N [J]., 1998, 12(1): [45]. [J]., 1997, [46],. [J]. [47]. [J].. [48],,. [J].

Σχετικά έγγραφα

1530 ( ) 2014,54(12),, E (, 1, X ) [4],,, α, T α, β,, T β, c, P(T β 1 T α,α, β,c) 1 1,,X X F, X E F X E X F X F E X E 1 [1-2] , 2 : X X 1 X 2 ;

1530 ( ) 2014,54(12),, E (, 1, X ) [4],,, α, T α, β,, T β, c, P(T β 1 T α,α, β,c) 1 1,,X X F, X E F X E X F X F E X E 1 [1-2] , 2 : X X 1 X 2 ; ISSN1000-0054 CN11-2223/N ( ) 2014 54 12 JTsinghuaUniv(Sci& Technol), 2014,Vol.54, No.12 4/20 1529-1533,, (,, (), 100084) [1-2] :,,,,,,,, :, 0.3~ [3] 0.8BLEU,, : ; ; [4], ; :TP391.2 :A, :1000-0054(2014)12-1529-05,