DEIM Forum 2018 D5-1 Inho Kang 169 0072 3-4-1 Naver Corporation E-mail: marerhg@ruri.waseda.jp, once.ihkang@navercorp.com, tetsuyasakai@acm.org ( ) URL PC NAVER 3 992 2,472 5.0,,,, URL ( ) 1. Agrawal URL 2015 Google more Google searches were completed on mobile devices than desktop computers. 1 NAVER 2 NAVER URL ( ) NAVER [2], [12], [16] [3], [15] 3 Agrawal [1] URL ( ) URL URL URL 1 https://techcrunch.com/2015/10/08/mobile-searches-surpassdesktop-searches-at-google-for-the-first-time/ 2 https://www.naver.com URL NAVER ( [5], [14]) NAVER ( ) PC [11] NAVER URL ( ) ( ) 3 3 ACM CIKM 2017 short paper [9]
1 URL 1 URL A, URL B, URL C, URL D URL B Agrawal URL B URL 1 Agrawal 2. 2. 1 URL 2. 2 2. 3 ( ) 2. 1 URL Joachims [7] URL Agrawal [1] URL Agrawal 1 URL URL ( ) 2 (URL) 3 URL URL Agrawal URL 3 1 2 1 URL URL [7] URL ( URL ) URL URL ( ) URL B URL A, URL C 1 URL D 1 Joachims 2 URL 1 URL P (2, 1) URL URL URL B > URL A, URL B > URL C, URL B > URL D. URL B > URL A P (2, 1) URL B > URL D P (2, 4)( ) 1 URL B > URL A P (2, 1) URL B > URL D P (2, 4) Agrawal (4. 1 ) Kadotami [8] ( ) Agrawal Agrawal Kadotami 2 Agrawal -order [1] 1 URL B > URL A URL B URL A
2 -order 4. 2 2. 2 [4], [13], [15] Williams [15] ( ) Lagun [10] Guo Song [6] ( ) 2. 3 Lagun [10] 3 3 3. 2 8 URL 4. 2. 1 1 ( ) ( ) 2 ( ) 4. 1 1: 3 3 200 B A 200 C c( C) cardscore(c) 4 S cardscore(c) cardscore(c) = s S score(c, s), (1) score(c, s) = time(c, s) dominance(c, s) completeness(c, s), time(c, s) = c s, dominance(c, s) = c s s completeness(c, s) = c s c,.
cardscore(c) 4. 2 2: 4 6 2. 1 -order 6 3 A 100 + 100 200 100 0 B C 5. 5 ( ) ( ) c C Agrawal Kadotami 5 ( ) cardscore(c) NAVER 5. 1 2016 12 12 18 NAVER NAVER USER ID, SERP ID, QUERY, UNIXTIME, INTERACTION TYPE, VISIBLE ITEMS, CLICKED CARD VISIBLE ITEMS NAVER > 0.8 0.8 > = > = 0.2 < 0.2 10 30 7 30 70%
7,616 3 2,472 8 2 7 30 1 2 1 1 50 992 720,764 315,759 5. 2 992 4 NAVER Lancers 5 2 1 3 2 4 1000 7,476 8 (34-1 Jan-34 ) 20 60. 5 www.lancers.jp 8 ( ) 2. 1 3 1 N1 N2 2 NN 1 L-L-R N1 N2 NN 3 2 NN 2 NN 2 Fleiss κ Cohen s κ [17] Fleiss κ 3 Cohen s κ 2 95% 2 2 2 2,023
1 N1-N2-NN L-L-L 993 (40.17%) L 1,742 (70.47%) L-L-R 506 (20.47%) L-R-L 105 (4.25%) R-L-L 138 (5.58%) R-R-R 347 (14.04%) R 730 (29.53%) R-R-L 177 (7.16%) R-L-R 92 (3.72%) L-R-R 114 (4.61%) 2,472 (100%) total 2,472 (100%) 2 3 Fleiss κ 2 Cohen s κ κ 95% CI 3 0.562 [0.539, 0.584] N1 N2 0.570 [0.551, 0.588] N1 NN 0.199 [0.178, 0.219] N2 NN 0.245 [0.224, 0.265] 5. 3 = n N = n N N: N (< = N): -order 2 N N n(< = N ): 5. 4 C (Click) A (Abandonment) score cardscore random C+A(score) [] cardscore C+A(random) C A(score) cardscore A(random) C(score)+A(score) cardscore 5. 5 3 4 3 3 4 2 5 3 Tukey HSD 3 ( 3 ) n N N C+A(score) 0.6932 0.5138 1270 1832 2472 C+A(random) 0.6045 0.4482 1108 1833 2472 C 0.6431 0.3354 829 1289 2472 A(score) 0.6772 0.4490 1110 1639 2472 A(random) 0.4865 0.3135 775 1593 2472 C(score)+A(score) 0.6678 0.5489 1357 2032 2472 4 ( 2 ) n N N C+A(score) 0.7175 0.5348 1082 1508 2023 C+A(random) 0.6287 0.4696 950 1511 2023 C 0.6820 0.3584 725 1063 2023 A(score) 0.6884 0.4652 941 1367 2023 A(random) 0.4789 0.3134 634 1324 2023 C(score)+A(score) 0.6842 0.5645 1142 1669 2023 5. 6 3 C 0.6431 A(score) 0.6772 5 cardscore(c) 3 4 C+A(score) C N 3 5 (p = 0.0454)
5 Tukey HSD ( 3 ) α = 0.05 p p C+A(score) A(random) 0.2067 [ 0.1601, 0.2534 ] 0.0000 A(score) A(random) 0.1907 [ 0.1428, 0.2386 ] 0.0000 C(score)+A(score) A(random) 0.1813 [ 0.1357, 0.2269 ] 0.0000 C A(random) 0.1566 [ 0.1056, 0.2076 ] 0.0000 C+A(random) A(random) 0.1180 [ 0.0713, 0.1646 ] 0.0000 C+A(score) C+A(random) 0.0888 [ 0.0438, 0.1337 ] 0.0000 A(score) C+A(random) 0.0728 [ 0.0265, 0.1191 ] 0.0001 C(score)+A(score) C+A(random) 0.0633 [ 0.0195, 0.1072 ] 0.0006 C+A(score) C 0.0501 [ 0.0006, 0.0996 ] 0.0454 C C+A(random) 0.0387 [ 0.0108, 0.0882 ] 0.2256 A(score) C 0.0341 [ 0.0166, 0.0848 ] 0.3911 C+A(score) C(score)+A(score) 0.0254 [ 0.0185, 0.0693 ] 0.5643 C(score)+A(score) C 0.0247 [ 0.0238, 0.0732 ] 0.6956 C+A(score) A(score) 0.0160 [ 0.0303, 0.0623 ] 0.9231 A(score) C(score)+A(score) 0.0094 [ 0.0358, 0.0546 ] 0.9914 5 C+A(score) A(score) C+A(score) 3 4 C+A(score) C(score)+A(score) cardscore(c) 5 (p = 0.5643) 3 4 C+A(score) C+A(random) 5 p 0.0000 A(score) A(random) 5 p 0.0000 cardscore(c) 3 4 A(random) 4 2 5. 7 cardscore cardscore time, dominance, completeness (4. 1 ) 6 cardscore 1 2 A(score) cardscore A(score) 3 t d c time dominance completeness td time dominance Tukey HSD 7 c t completeness time 6 completeness time 7 ( c t ) (p = 0.0006) time completeness (tc) (tdc) time (t) ( p = 0.0195, p = 0.0211) completeness 6 cardscore A(score) ( 3 ) time dominance completeness t - - 0.6236 d - - 0.6486 c - - 0.6928 td - 0.6571 dc - 0.6687 tc - 0.6777 tdc 0.6772 6.
7 A(score) cardscore Tukey HSD ( 3 ) α = 0.05 p p c t 0.0692 [ 0.0202, 0.1183 ] 0.0006 tc t 0.0541 [ 0.0051, 0.1031 ] 0.0195 tdc t 0.0537 [ 0.0047, 0.1026 ] 0.0211 dc t 0.0451 [ 0.0038, 0.0941 ] 0.0940 c d 0.0442 [ 0.0044, 0.0929 ] 0.1034 c td 0.0358 [ 0.0130, 0.0845 ] 0.3157 td t 0.0335 [ 0.0155, 0.0825 ] 0.4054 tc d 0.0291 [ 0.0196, 0.0777 ] 0.5733 tdc d 0.0286 [ 0.0200, 0.0772 ] 0.5905 d t 0.0250 [ 0.0239, 0.0740 ] 0.7409 c dc 0.0241 [ 0.0245, 0.0728 ] 0.7674 tc td 0.0206 [ 0.0281, 0.0693 ] 0.8754 tdc td 0.0202 [ 0.0285, 0.0688 ] 0.8859 dc d 0.0201 [ 0.0285, 0.0687 ] 0.8867 c tdc 0.0156 [ 0.0331, 0.0643 ] 0.9654 c tc 0.0152 [ 0.0336, 0.0639 ] 0.9700 dc td 0.0116 [ 0.0370, 0.0603 ] 0.9924 tc dc 0.0090 [ 0.0397, 0.0576 ] 0.9982 tdc dc 0.0085 [ 0.0401, 0.0572 ] 0.9986 td d 0.0085 [ 0.0402, 0.0571 ] 0.9987 tc tdc 0.0004 [ 0.0482, 0.0491 ] 1.0000 time, dominance, completeness cardscore (time) (completeness) [1] Rakesh Agrawal, Alan Halverson, Krishnaram Kenthapadi, Nina Mishra, and Panayiotis Tsaparas. Generating labels from clicks. In Proceedings of ACM WSDM 2009, pp. 172 181, 2009. [2] Jaime Arguello, Fernando Diaz, and Jamie Callan. Learning to aggregate vertical results into web search results. In Proceedings of ACM CIKM 2011, pp. 201 210, 2011. [3] Lydia B. Chilton and Jaime Teevan. Addressing people s information needs directly in a web search result page. In Proceedings of WWW 2011, pp. 27 36, 2011. [4] Aleksandr Chuklin and Pavel Serdyukov. Potential good abandonment prediction. In WWW 2012 Companion, pp. 485 486, 2012. [5] Nick Craswell and Martin Szummer. Random walks on the click graph. In Proceedings of ACM SIGIR 2007, pp. 239 246, 2007. [6] Qi Guo and Yang Song. Large-scale analysis of viewing behavior: Towards measuring satisfaction with mobile proactive systems. In CIKM 2016, 2016. [7] Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, Filip Radlinski, and Geri Gay. Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM TOIS, Vol. 25, No. 2, 2007. [8] Yuta Kadotami, Yasuaki Yoshida, Sumio Fujita, and Tetsuya Sakai. Mobile vertical ranking based on preference graphs. In Proceedings of ACM ICTIR 2017, 2017. [9] Mami Kawasaki, Inho Kang, and Tetsuya Sakai. Ranking rich mobile verticals based on clicks and abandonment. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp. 2127 2130, 2017. [10] Dmitry Lagun, Chih-Hung Hsieh, and Dale Webster Vidhya Navalpakkam. Towards better measurement of attention and satisfaction in mobile search. In Proceedings of SIGIR 2014, pp. 113 122, 2014. [11] Jane Li, Scott Huffman, and Akihito Tokuda. Good abandonment in mobile and pc internet search. In Proceedings of ACM SIGIR 2009, pp. 43 50, 2009. [12] Ashok Kumar Ponnuswami, Kumaresh Pattabiraman, Qiang Wu, Ran Gilad-Bachrach, and Tapas Kanungo. On composition of federated web search result page: Using online users to provide pairwise preference for heterogeneous verticals. In Proceedings of ACM WSDM 2011, pp. 715 724, 2011. [13] Yang Song, Xiaolin Shi, Ryen White, and Ahmed Hassan Awadallah. Context-aware web search abandonment prediction. In Proceedings of ACM SIGIR 2014, pp. 93 102, 2014. [14] Kuansan Wang, Toby Walker, and Zijian Zheng. PSkip: Estimating relevance ranking quality from web search clickthrough data. In Proceedings of ACM KDD 2009, pp. 1355 1364, 2009. [15] Kyle Williams, Julia Kiseleva, Aidan C. Crook, Imed Zitouni, Ahmed Hassan Awadallah, and Madian Khabsa. Detecting good abandonment in mobile search. In Proceedings of WWW 2016, pp. 495 505, 2016. [16] Ke Zhou, Thomas Demeester, Dong Nguyen, Djoerd Hiemstra, and Dolf Trieschnigg. Aligning vertical collection relevance with user intent. In Proceedings of ACM CIKM 2014, pp. 1915 1918, 2014. [17]. :., 2015.