Exhaustive Topic Detection and Query Expansion Support Based on Substance-Oriented Term Clustering

Μέγεθος: px

Εμφάνιση ξεκινά από τη σελίδα:

Download "Exhaustive Topic Detection and Query Expansion Support Based on Substance-Oriented Term Clustering"

Σίλας Βασιλικός
7 χρόνια πριν
Προβολές:

1 DEWS2006 2C-i {hiromi,masada,takasu,adachi}@nii.ac.jp Tangibility Tangibility Tangibility WWW Exhaustive Topic Detection and Query Expansion Support Based on Substance-Oriented Term Clustering Hiromi WAKAKI, Tomonari MASADA, Atsuhiro TAKASU, and Jun ADACHI Graduate School of Information Science and Technology, The University of Tokyo Hitotsubashi, Chiyoda-ku, Tokyo, Japan National Institute of Informatics Hitotsubashi, Chiyoda-ku, Tokyo, Japan {hiromi,masada,takasu,adachi}@nii.ac.jp Abstract Conventional search engines are designed mainly for general keyword search. Users who are unfamiliar with the field they seek cannot find an appropriate combination of query terms. We aim at a query disambiguation method which discovers various topics buried in a search result by making clusters with the terms appearing in the retrieved documents. At that time, it is desirable that the terms should be key for users to discover new knowledge with term clusters, e.g. proper nouns. In this paper, we propose a new measure, called Tangibility, to gather terms matching the above purpose. With existing term weighting methods, most of the high scored terms are generally used regardless of topics. On the other hand, with our Tangibility term weighting method, we can obtain distinctive terms each of which corresponds to a certain topic and all of which cover various topics implied by the query. We also conduct an experiment and compare our formula with the existing term weighting formulas. Key words Information Retreival, Data Mining, Clustering WWW

jp Tangibility Tangibility Tangibility WWW Exhaustive Topic Detection and Query Expansion Support Based on Substance-Oriented Term Clustering Hiromi WAKAKI, Tomonari MASADA, Atsuhiro TAKASU, and Jun

2 Tangibility Tangibility Tangibility Tangibility 2. (representativeness) [8] χ 2 [12] Web KeyGraph [6] [14] DualNAVI [7] 3. Tangibility 3. 1 t i t j t i t j t i P (t i) t i t i t j P (t j t i) t i t j t i t i document frequency( DF ) DF(t i) 3. 2 Tangibility Tangibility Tangibility Tangibility TNG3 t j t i ti (t j)=p (t j t i) log P (tj ti) (1) P (t j) F i = {t j ti (t j) > 0} Tangibility TNG3 ( t TNG3(t k F ti (t k ) DF(t k )) i i)= F i [10] [11] [10] [11] 3. 3 Tangibility KL t i t j KL KL(t j; t i)= P (t j t i) log P (t j t i ) P (t j ) (2) +P ( t j t i) log P ( t j t i ) P ( t j ) (3)

2 Tangibility Tangibility Tangibility Tangibility TNG3 t j t i ti (t j)=p (t j t i) log P (tj ti) (1) P (t j) F i = {t j ti (t j) > 0} Tangibility

3 TNG3 (1) (3) P (t j t i) log P (t j t i ) P (t j (1) t ) j t i t j 0 TNG3 F i = {t j ti (t j) > 0} (1) (2) F i ti (t j) 0 t j ti (t j) 0 t j TNG3 t i t i Tangibility Tangibility (1) (2) 3. 4 t i t j Sim(t i,t j)= t i t j t i t j t i t j t i t j 2 t i t j t i C 1 t j C 2 s(c 1,C 2) C 1 C 2 s(c 1,C 1) C 1 s(c 1,C 2)= Sim(t i,t j) t i C 1 t j C 2 s(c 1,C 1)= Sim(t i,t j) t j C 1 t i C 1 s(c 1,C 2) (s(c 1,C 1)) (s(c 2,C 2)) > = τ (5) C 1 C 2 τ (4) 1 t i t j t j 1 t i t j Sim(t j,t j) 1 [4] Ding [2] Ding Tangibility (2) (Mutual Information: MI) [9] Robertson s Selection Value(RSV) [5] document frequency(df) [13] DF (DFL) DF (DFS) DFS DF 1 DF 1 DFS DF 10 DF (w(t t j i,t j) DF(t j)) W (t i)= (6) {t j w(t i,t j) > 0} w(t i,t j)= t i t j w(t i,t j) (TNG3 MI) RSV RSV

4 1 t i t j t i t j 2 t i t j t i C 1 t j C 2 s(c 1,C 2) C 1 C 2 s(c 1,C 1) C 1 s(c 1,C 2)= Sim(t i,t j) t i C 1 t j C 2 s(c 1,C 1)= Sim(t i,t j) t j C 1 t i C 1 s(c 1,C 2) (s(c 1,C 1)) (s(c 2,C

4 DFL DFS DF MI t i t j t i t j t j t i MI(t i,t j)= P (t i){p (t j t i) log P (t j t i ) P (t j ) +P ( t j t i) log P ( t j t i ) } P ( t j ) +P ( t i){p (t j t i) log P (t j t i ) P (t j ) +P ( t j t i) log P ( t j t i ) P ( t j ) } MI W (t i)= j (MI(t i,t j ) DF (t j )) {t j w(t i,t j )>0} (7) RSV RSV (Robertson s Selection Value) [5] RSV RSV =( rdf R df N ) { α log N rdf+0.5 +(1 α) log df } R rdf+0.5 df rdf+0.5 N df R+rdf+0.5 (8) rdf : S R df N α t : S : S U t : U : α 0.5 U Web RSV NTCIR3 NTCIR4 NW100G-01 [3] NW100G RSV (8) R N NW100G-01 ( ) Web 3 D 1 D 2 D 3 3 t DF DF(t; D 1),DF(t; D 2),DF(t; D 3) DF(t) D 1 D 2 D 3 t DF C DF(t; D k) D k C DF(t; D1) DF(t; D2) DF(t; D3) DF(t; D2) C C Prec(C) DF(t; D(C)) Prec(C) = DF(t) D(C) C C macroaveraged precision microaveraged precision [1] macroaveraged precision

( t j t i ) P ( t j ) } MI W (t i)= j (MI(t i,t j ) DF (t j )) {t j w(t i,t j )>0} (7) 4. 1. 2 RSV RSV (Robertson s Selection Value) [5] RSV RSV =( rdf R df N ) { α log N rdf+0.

5 microaveraged precision microaveraged precision macroaveraged precision microaveraged precision Tangibility macroaveraged precision microaveraged precision C Prec(C) DF(t; D(C)) C C Prec(C) = DF(t) C C DF DF 1 3 C 1,C 2,C 3 C 1 ( 1) DF(t; D(C 1)) = = DF(t) = C 2 ( 2) 2 DF(t; D(C 2)) = = DF(t) = 2022 C 3 ( 3) 3 DF(t; D(C 3)) = = DF(t) = Prec(C) = = 7475 = C 1 D 1 DF D 2 DF D 3 DF C 2 D 1 DF D 2 DF D 3 DF C 2 D 1 DF D 2 DF D 3 DF microaveraged precison Prec(C) τ 1 ( ) 1 1 TNG3 DFS MI RSV DFL. TNG

526 = 1771 3 DF(t) = 2401 3837 + 1867 + 1771 Prec(C) = 3911 + 2022 + 2401 = 7475 =0.8969.

6 TNG3 TNG3 1 DFS DFS DFS DFS TNG3 TNG3 Tangibility DF TNG ,3,4 Prec(C) TNG3 MI RSV TNG3 MI RSV TNG MI RSV 1 40 TNG3 0.8 MI RSV MI RSV TNG3 TNG3 MI RSV TNG3 1 MI RSV 1 DF 1 MI RSV DF TNG3 TNG3 DF MI RSV (400 1)/ MI RSV TNG DF 1 TNG3 DFS TNG3 TNG3 DF 4 DF TNG3 DF 1000 DF 100 MI RSV 0 TNG3 164 DF TNG3 DFS TNG3 400 DF 20 DFS TNG3 DF 5. Tangibility Tangibility TNG3 TNG3 Tangibility MI RSV DFS DFL TNG3 DF TNG3

001 TNG3 TNG3 MI RSV 1 150 TNG3 1 MI RSV 1 DF 1 MI RSV DF TNG3 TNG3 DF MI RSV 400 400 400(400 1)/2 8000 MI RSV TNG3 4. 3.

7 1 Microaveraged Precision. 4 RSV Microaveraged Precision TNG3 MI RSV DFS 400 DF DF TNG3 MI RSV DFS TNG3 Microaveraged Precision MI Microaveraged Precision Tangibility [10] [11] 2 [1] Soumen Chakrabarti. Mining the Web: Discovering Knowledge from Hypertext Data. Morgan-Kauffman, [2] C. Ding and X. He. Cluster merging and splitting in hierarchical clustering algorithms. In IEEE International Conference on Data Mining (ICDM 02), pp , [3] Oyama K. Ishida E. Kando N. Eguchi, K. and K. Kuriyama. Overview of the web retrieval task at the third ntcir workshop, [4] A. K. Jain, M. N. Murty, and P. J. Flynn. Data clustering: a review. ACM Computing Surveys, Vol. 31, No. 3, pp , [5] Toyoda Masashi, Kitsuregawa Masaru, Mano Hiroko, Itoh Hideo, and Ogawa Yasushi. University of tokyo/ricoh at ntcir-3 web retrieval task. In Proc. of the 3rd NTCIR Workshop Meeting, pp , [6] Y. Ohsawa, E.B. Nels,, and M. Yachida. Keygraph: Automatic indexing by co-occurrence graph based on building construction metaphor. In Proc. of IEEE ADL 98, [7] A. Takano, Y. Niwa, S. Nishioka, M. Iwayama, T. Hisamitsu, O. Imaichi, and H. Sakurai. Associative information access using dualnavi. In Proc. of ICDL 00, pp. 2 [10] [11] Articulateness Tangibility AR1 AR2 TNG3

Microaveraged Precision 0.05 0.005 0.001 3 3 MI Microaveraged Precision 0.05 0.005 0.001 3 Tangibility [10] [11] 2 [1] Soumen Chakrabarti. Mining the Web: Discovering Knowledge from Hypertext Data.

8 5 TNG D(C) TNG D(C) [8] Hisamitsu Toru, Niwa Yoshiki, Nishioka Shingo, Sakurai Hirofumi, Imaichi Osamu, Iwayama Makoto, and Takano Akihiko. Extracting terms by a combination of term frequency and a measure of term representativeness. International journal of theoretical and applied aissues in specialized communication, Vol. 6, No. 2, pp , [9] Yang Yiming and O. Pedersen Jan. A comparative study on feature selection in text categorization. In Proc. of ICML- 97, pp , [10],,,.. DBSJ Letters, Vol. 4, No. 2, pp , [11],,,.., 137, pp , [12],.., Vol. 17, pp , [13].., Vol. 41, pp , [14], E.,. Keygraph:., Vol. J82-D-I, pp ,

Extracting terms by a combination of term frequency and a measure of term representativeness. International journal of theoretical and applied aissues in specialized communication, Vol. 6, No. 2, pp.

Σχετικά έγγραφα

IPSJ SIG Technical Report Vol.2014-CE-127 No /12/6 CS Activity 1,a) CS Computer Science Activity Activity Actvity Activity Dining Eight-He

IPSJ SIG Technical Report Vol.2014-CE-127 No /12/6 CS Activity 1,a) CS Computer Science Activity Activity Actvity Activity Dining Eight-He CS Activity 1,a) 2 2 3 CS Computer Science Activity Activity Actvity Activity Dining Eight-Headed Dragon CS Unplugged Activity for Learning Scheduling Methods Hisao Fukuoka 1,a) Toru Watanabe 2 Makoto