IT 17081 Examensarbete 15 hp November 2017 Medium Data on Big Data Predicting Disk Failures in CERNs NetApp-based Data Storage System Albin Stjerna Institutionen för informationsteknologi Department of Information Technology
Abstract Medium Data on Big Data. Predicting Disk Failures in CERNs NetApp-based Data Storage System Albin Stjerna Teknisk- naturvetenskaplig fakultet UTH-enheten Besöksadress: Ångströmlaboratoriet Lägerhyddsvägen 1 Hus 4, Plan 0 Postadress: Box 536 751 21 Uppsala Telefon: 018 471 30 03 Telefax: 018 471 30 00 I describe in this report an experimental system for using classification and regression trees to generate predictions of disk failures in a NetApp-based storage system at the European Organisation for Nuclear Research (CERN) based on a mixture of SMART data, system logs, and low-level system performance data particular to NetApp's storage solutions. Additionally, I make an attempt at profiling the system's built-in failure prediction method, and compiling statistics on historical complete-disk failures as well as bad blocks developed. Finally, I experiment with various parameters for producing classification trees and end up with two candidate models which have a true-positive rate of 86% with a false-alarm rate of 4% or a true-positive rate of 71% and a false-alarm rate of 0.9% respectively, illustrating that classification trees might be a viable method for predicting real-life disk failures in CERNs storage systems. Hemsida: http://www.teknat.uu.se/student Handledare: Miroslav Potocky Ämnesgranskare: Justin Pearson Examinator: Olle Gällmo IT 17081 Tryckt av: Reprocentralen ITC
F P TP = F P FN = F \P FP = P \ F TN = F \ P
4.3 10 5
[ t d,t[ t lead [t + t lead,t+ t p [ t p t lead t p t d
P TP F F P TP = F P FN = F \P FP = P \F TN = F \P TP FP TN FN TP TPR = TP + FN FP F AR = FP + TN
P ( ) = 57.69 P ( ) = 15.31
j i x i,j x i,j = x i,j x i 1,j x =2 x x x x 1
x i R n,i=1,...,l y R l l m Q θ =(j, t m ) j t m Q (θ) Q (θ) Q (θ) = {(x, y) Q x j t m } Q (θ) = Q \ Q (θ) H m k 1 H(Q) =1 p t t Q Q 0 0+0+ +1 2 =1 Q k 0, 1,...,k 1 n l = Q (θ) n r = Q (θ) t=0 G(Q, θ) = n lh(q (θ)) + n r H(Q (θ)) Q θ =(j, t m ) Q left (θ) Q (θ) p 2 t
Q m Q =1 χ 2 n
Power On Hours 0.0 gini = 0.5 samples = 100.0% value = [0.5, 0.5] class = Failed True False gini = 0.0 samples = 30.1% value = [1.0, 0.0] class = Failed Temperature Celsius -0.6286 gini = 0.4074 samples = 69.9% value = [0.28, 0.72] class = OK Spin Up Time 0.963 gini = 0.3967 samples = 12.7% value = [0.73, 0.27] class = Failed Reallocated Sectors Count RAW -0.9842 gini = 0.3031 samples = 57.2% value = [0.19, 0.81] class = OK gini = 0.1884 samples = 8.8% value = [0.89, 0.11] class = Failed gini = 0.4567 samples = 3.9% value = [0.35, 0.65] class = OK gini = 0.2229 samples = 52.5% value = [0.13, 0.87] class = OK gini = 0.255 samples = 4.6% value = [0.85, 0.15] class = Failed 4.3 10 5
Power On Hours 0.0 gini = 0.5 samples = 100.0% value = [0.5, 0.5] class = Failed True False gini = 0.0 samples = 30.1% value = [1.0, 0.0] class = Failed gini = 0.4074 samples = 69.9% value = [0.28, 0.72] class = OK 4.3 10 5
χ 2
True sense_3 3.0 gini = 0.1298 samples = 100.0% value = [0.07, 0.93] class = OK False blocks_written 415448236032.0 gini = 0.0386 samples = 94.4% value = [0.02, 0.98] class = OK max_q 233.5 gini = 0.1528 samples = 5.6% value = [0.92, 0.08] class = Failed io_completed_count_4_ms_delta 552209472.0 gini = 0.0198 samples = 93.0% value = [0.01, 0.99] class = OK blocks_written 447934660608.0 gini = 0.4444 samples = 1.4% value = [0.67, 0.33] class = Failed gini = 0.0 samples = 4.7% value = [1.0, 0.0] class = Failed max_q_delta 3.5 gini = 0.5 samples = 0.9% value = [0.5, 0.5] class = Failed blocks_written_delta 3444588032.0 gini = 0.01 samples = 92.6% value = [0.01, 0.99] class = OK gini = 0.0 samples = 0.5% value = [1.0, 0.0] class = Failed gini = 0.0 samples = 0.9% value = [1.0, 0.0] class = Failed gini = 0.0 samples = 0.5% value = [0.0, 1.0] class = OK gini = 0.0 samples = 0.5% value = [0.0, 1.0] class = OK gini = 0.0 samples = 0.5% value = [1.0, 0.0] class = Failed gini = 0.0 samples = 91.6% value = [0.0, 1.0] class = OK gini = 0.5 samples = 0.9% value = [0.5, 0.5] class = Failed
1 2 3 4 5 1.00/.000.00/.000.00/.000.00/.000.00/.000.00/.000 2.00/.000.00/.007.00/.003.00/.003.14/.026.29/.035 3.00/.000.00/.007.00/.003.00/.009.00/.009.29/.040 4.00/.000.00/.004.00/.001.00/.007.00/.012.29/.020 5.00/.000.00/.004.00/.001.00/.010.00/.009.29/.022 6.00/.000.00/.004.00/.002.00/.011.00/.009.14/.030 7.00/.000.00/.004.00/.001.00/.009.00/.009.14/.040 8.00/.000.00/.004.00/.001.00/.010.00/.008.14/.034 9.00/.000.00/.004.00/.001.00/.010.00/.008.14/.031 10.00/.000.00/.004.00/.001.00/.007.14/.057.57/.035 11.00/.000.00/.004.00/.001.00/.007.14/.057.57/.035 12.00/.000.00/.004.00/.002.00/.007.14/.058.43/.037 13.00/.000.00/.004.00/.001.00/.004.14/.055.57/.034 14.00/.000.00/.032.00/.009.00/.012.14/.062.29/.037 15.00/.000.00/.032.00/.009.00/.012.14/.062.29/.038 16.00/.000.00/.032.00/.009.00/.012.14/.069.29/.034 17.00/.000.00/.032.00/.009.00/.041.14/.026.29/.037 18.00/.000.00/.032.14/.012.14/.012.14/.041.43/.040 19.00/.000.00/.032.14/.009.14/.012.29/.012.57/.038 20.00/.000.00/.032.14/.009.14/.012.29/.012.57/.035 21.00/.000.00/.032.14/.009.14/.012.29/.014.43/.040 22.00/.000.00/.032.14/.009.14/.012.29/.014.43/.042 23.00/.000.00/.032.14/.009.14/.012.29/.012.43/.037 24.57/.006.57/.010.71/.010.57/.040.71/.024.71/.024 25.57/.006.57/.010.71/.010.71/.041.86/.027.86/.027 26.57/.006.57/.010.43/.010.57/.041.86/.025.86/.025 27.57/.006.57/.010.57/.009.57/.040.57/.022.57/.022 28.57/.006.57/.010.57/.010.57/.041.86/.025.86/.025 29.57/.006.57/.010.43/.009.57/.040.57/.026.57/.026 30.57/.006.57/.010.71/.009.71/.040.57/.027.57/.027 109.57/.006.57/.010.71/.009.86/.041.71/.013.71/.013
1 2 3 4 5 1.00/.000.00/.000.00/.000.00/.000.00/.000.00/.000 2.00/.000.00/.007.00/.007.00/.007.14/.030.43/.044 3.00/.000.00/.007.00/.007.00/.012.00/.017.29/.053 4.00/.000.00/.004.00/.004.00/.010.00/.016.14/.038 5.00/.000.00/.004.00/.004.00/.013.00/.013.14/.030 6.00/.000.00/.004.00/.004.00/.014.00/.014.14/.039 7.00/.000.00/.004.00/.004.00/.013.00/.013.14/.045 8.00/.000.00/.004.00/.004.00/.013.00/.013.14/.043 9.00/.000.00/.004.00/.004.00/.013.00/.013.14/.039 10.00/.000.00/.004.00/.004.00/.010.14/.060.57/.056 11.00/.000.00/.004.00/.004.00/.010.14/.060.57/.057 12.00/.000.00/.004.00/.004.00/.010.14/.060.57/.057 13.00/.000.00/.004.00/.004.00/.007.14/.058.43/.053 14.00/.000.00/.032.00/.009.00/.012.14/.062.29/.050 15.00/.000.00/.032.00/.009.00/.012.14/.062.29/.059 16.00/.000.00/.032.00/.009.00/.012.14/.069.29/.053 17.00/.000.00/.032.00/.009.00/.041.00/.054.29/.061 18.00/.000.00/.032.00/.009.00/.041.00/.054.29/.061 19.00/.000.00/.032.00/.009.00/.041.00/.054.29/.062 20.00/.000.00/.032.00/.009.00/.041.00/.054.29/.060 21.00/.000.00/.032.00/.009.00/.041.00/.054.43/.067 22.00/.000.00/.032.00/.009.00/.041.00/.054.29/.063 23.00/.000.00/.032.00/.009.00/.041.00/.054.29/.063 24.57/.006.57/.010.57/.013.71/.045.71/.045.71/.045 25.57/.006.57/.010.57/.013.71/.046.71/.046.71/.046 26.57/.006.57/.010.57/.015.71/.047.71/.047.71/.047 27.57/.006.57/.010.57/.013.71/.045.71/.045.71/.045 28.57/.006.57/.010.57/.015.71/.047.71/.047.71/.047 29.57/.006.57/.010.57/.013.71/.045.71/.045.71/.045 30.57/.006.57/.010.57/.013.71/.045.71/.045.71/.045 109.57/.006.57/.010.57/.013.71/.046.71/.046.71/.046
1 2 3 4 5 1.00/.000.00/.000.00/.000.00/.000.00/.000.00/.000 2.00/.000.00/.007.00/.007.00/.007.14/.030.00/.036 3.00/.000.00/.007.00/.007.00/.007.00/.007.00/.024 4.00/.000.00/.004.00/.004.00/.004.00/.010.00/.017 5.00/.000.00/.004.00/.004.00/.004.00/.004.00/.010 6.00/.000.00/.004.00/.004.00/.004.00/.004.00/.031 7.00/.000.00/.004.00/.004.00/.004.14/.028.14/.036 8.00/.000.00/.004.00/.004.00/.004.14/.028.14/.036 9.00/.000.00/.004.00/.004.00/.004.14/.028.14/.036 10.00/.000.00/.004.00/.004.00/.004.14/.055.14/.028 11.00/.000.00/.004.00/.004.00/.004.14/.055.14/.028 12.00/.000.00/.004.00/.004.00/.004.14/.055.14/.028 13.00/.000.00/.004.00/.004.00/.004.14/.055.14/.028 14.00/.000.00/.032.00/.009.00/.009.14/.059.14/.033 15.00/.000.00/.032.00/.009.00/.009.14/.059.14/.033 16.00/.000.00/.032.00/.009.00/.009.00/.052.00/.028 17.00/.000.00/.032.00/.009.00/.038.00/.051.00/.051 18.00/.000.00/.032.00/.009.00/.038.00/.051.00/.051 19.00/.000.00/.032.00/.009.00/.038.00/.051.00/.051 20.00/.000.00/.032.00/.009.00/.038.00/.051.00/.051 21.00/.000.00/.032.00/.009.00/.038.00/.051.00/.051 22.00/.000.00/.032.00/.009.00/.038.00/.051.00/.051 23.00/.000.00/.032.00/.009.00/.038.00/.051.00/.051 24.57/.006.57/.010.57/.010.71/.025.71/.025.71/.025 25.57/.006.57/.010.57/.010.71/.025.71/.025.71/.025 26.57/.006.57/.010.57/.010.71/.027.71/.027.71/.027 27.57/.006.57/.010.57/.010.71/.025.71/.025.71/.025 28.57/.006.57/.010.57/.010.71/.025.71/.025.71/.025 29.57/.006.57/.010.57/.010.71/.025.71/.025.71/.025 30.57/.006.57/.010.57/.010.71/.025.71/.025.71/.025 109.57/.006.57/.010.57/.010.71/.025.71/.025.71/.025
1 2 3 4 5 1.00/.000.00/.000.00/.000.00/.000.00/.000.00/.000 2.00/.010.00/.010.00/.010.14/.033.43/.039.43/.064 3.00/.010.00/.010.00/.010.00/.010.00/.010.14/.050 4.00/.010.00/.010.00/.010.00/.010.00/.010.00/.017 5.00/.010.00/.010.00/.010.00/.010.00/.010.14/.043 6.00/.010.00/.010.00/.010.00/.010.00/.022.14/.045 7.00/.010.00/.010.00/.010.14/.019.29/.042.57/.056 8.00/.010.00/.010.00/.010.14/.016.29/.039.57/.055 9.00/.010.00/.010.00/.010.14/.017.29/.040.43/.076 10.00/.010.00/.010.00/.010.14/.060.29/.042.57/.048 11.00/.010.00/.010.00/.010.14/.060.29/.042.43/.066 12.00/.010.00/.010.00/.010.14/.060.29/.040.43/.065 13.00/.000.14/.043.14/.020.29/.030.29/.030.29/.030 14.00/.032.00/.032.14/.066.29/.052.29/.052.29/.052 15.00/.032.00/.032.29/.097.29/.068.43/.078.43/.078 16.00/.032.00/.032.29/.097.29/.068.43/.078.43/.078 17.00/.032.00/.032.29/.097.29/.068.43/.078.43/.078 18.00/.032.00/.032.29/.097.29/.074.29/.074.29/.074 19.00/.032.00/.032.29/.097.29/.074.29/.074.29/.074 20.00/.032.00/.032.29/.097.29/.074.29/.074.29/.074 21.00/.032.00/.032.29/.097.29/.074.29/.074.29/.074 22.00/.032.00/.032.29/.097.29/.074.29/.074.29/.074 23.00/.032.00/.032.29/.097.29/.074.29/.074.29/.074 24.57/.006.57/.015.57/.015.71/.032.71/.032.71/.032 25.57/.006.57/.015.57/.015.71/.032.71/.032.71/.032 26.57/.006.57/.015.57/.015.71/.033.71/.033.71/.033 27.57/.006.57/.015.57/.015.71/.032.71/.032.71/.032 28.57/.006.57/.015.57/.015.71/.033.71/.033.71/.033 29.57/.006.57/.015.57/.015.71/.032.71/.032.71/.032 30.57/.006.57/.015.57/.015.71/.032.71/.032.71/.032 109.57/.006.57/.015.57/.015.71/.032.71/.032.71/.032
1 2 3 4 5 1.00/.000.00/.000.00/.000.00/.000.00/.000.00/.000 2.00/.000.00/.000.00/.000.14/.023.14/.023.14/.023 3.00/.000.00/.000.00/.000.00/.000.00/.000.00/.000 4.00/.000.00/.000.00/.000.00/.000.00/.000.00/.000 5.00/.000.00/.000.00/.000.00/.000.00/.000.00/.000 6.00/.000.00/.000.00/.000.00/.000.00/.000.00/.000 7.00/.000.14/.016.14/.016.29/.039.29/.039.29/.039 8.00/.000.14/.014.14/.014.29/.038.29/.038.29/.038 9.00/.000.14/.014.14/.014.29/.038.29/.038.29/.038 10.00/.000.14/.014.14/.014.29/.065.29/.036.29/.036 11.00/.000.14/.015.14/.014.29/.065.29/.039.29/.038 12.00/.000.14/.014.14/.014.29/.065.29/.038.29/.036 13.00/.000.14/.034.14/.034.29/.046.29/.046.29/.046 14.00/.032.00/.032.14/.066.29/.078.29/.078.29/.078 15.00/.032.00/.032.29/.097.29/.077.29/.077.29/.077 16.00/.032.00/.032.29/.097.29/.077.43/.081.43/.081 17.00/.032.00/.032.29/.097.29/.077.29/.077.29/.077 18.00/.032.00/.032.29/.097.29/.077.29/.077.29/.077 19.00/.032.00/.032.29/.097.29/.076.29/.076.29/.076 20.00/.032.00/.032.29/.097.29/.077.29/.077.29/.077 21.00/.032.00/.032.29/.097.29/.077.29/.077.29/.077 22.00/.032.00/.032.29/.097.29/.076.29/.076.29/.076 23.00/.032.00/.032.29/.097.29/.079.29/.079.29/.079 24.57/.006.57/.006.57/.006.57/.006.57/.006.57/.006 25.57/.006.57/.006.57/.006.57/.006.57/.006.57/.006 26.57/.006.57/.006.57/.006.57/.006.57/.006.57/.006 27.57/.006.57/.006.57/.006.57/.006.57/.006.57/.006 28.57/.006.57/.006.57/.006.57/.006.57/.006.57/.006 29.57/.006.57/.006.57/.006.57/.006.57/.006.57/.006 30.57/.006.57/.006.57/.006.57/.006.57/.006.57/.006 109.57/.006.57/.006.57/.006.57/.006.57/.006.57/.006
True sense_3 3.0 gini = 0.1298 samples = 100.0% value = [0.07, 0.93] class = OK False blocks_written 415448236032.0 gini = 0.0386 samples = 94.4% value = [0.02, 0.98] class = OK sense_3 6.0 gini = 0.1528 samples = 5.6% value = [0.92, 0.08] class = Failed io_completed_count_4_ms_delta 552209472.0 gini = 0.0198 samples = 93.0% value = [0.01, 0.99] class = OK blocks_written_delta 3870585344.0 gini = 0.4444 samples = 1.4% value = [0.67, 0.33] class = Failed io_completed_count_8_ms_delta 233073.5 gini = 0.5 samples = 0.9% value = [0.5, 0.5] class = Failed gini = 0.0 samples = 4.7% value = [1.0, 0.0] class = Failed gini = 0.01 samples = 92.6% value = [0.01, 0.99] class = OK gini = 0.0 samples = 0.5% value = [1.0, 0.0] class = Failed gini = 0.0 samples = 0.9% value = [1.0, 0.0] class = Failed gini = 0.0 samples = 0.5% value = [0.0, 1.0] class = OK gini = 0.0 samples = 0.5% value = [0.0, 1.0] class = OK gini = 0.0 samples = 0.5% value = [1.0, 0.0] class = Failed
True sense_3 3.0 gini = 0.1298 samples = 100.0% value = [0.07, 0.93] class = OK False blocks_written 341963636736.0 gini = 0.0386 samples = 94.4% value = [0.02, 0.98] class = OK io_completed_count_30_ms 40759144.0 gini = 0.1528 samples = 5.6% value = [0.92, 0.08] class = Failed blocks_read_delta 7073871872.0 gini = 0.0199 samples = 92.6% value = [0.01, 0.99] class = OK gini = 0.5 samples = 1.9% value = [0.5, 0.5] class = Failed gini = 0.0 samples = 3.7% value = [1.0, 0.0] class = Failed gini = 0.375 samples = 1.9% value = [0.75, 0.25] class = Failed gini = 0.0 samples = 87.9% value = [0.0, 1.0] class = OK io_completed_count_4_ms_delta 1424875.5 gini = 0.32 samples = 4.7% value = [0.2, 0.8] class = OK gini = 0.0 samples = 2.8% value = [0.0, 1.0] class = OK gini = 0.5 samples = 1.9% value = [0.5, 0.5] class = Failed
1 2 3 4 5 1.11/.029.32/.207.45/.260.46/.241.46/.241.71/.424 2.11/.029.63/.476.61/.438.83/.617.72/.539.72/.323 3.11/.029.13/.029.63/.454.63/.439.80/.587.74/.340 4.11/.029.11/.023.13/.029.13/.026.64/.360.80/.307 5.11/.029.94/.815.97/.731.95/.733.66/.343.77/.280 6.11/.029.94/.815.97/.731.97/.733.64/.344.80/.287 7.11/.029.94/.815.97/.731.97/.733.63/.344.83/.295 8.11/.029.94/.815.97/.731.47/.258.86/.651.78/.290 9.11/.029.94/.815.97/.731.48/.258.86/.648.77/.293 10.11/.029.94/.815.94/.721.48/.255.87/.637.82/.267 11.11/.029.94/.815.94/.721.48/.255.89/.635.86/.275 12.11/.029.94/.815.97/.724.48/.254.86/.637.78/.253 13.11/.029.11/.023.94/.767.69/.438.70/.434.85/.259 14.26/.021.46/.129.46/.129.55/.200.69/.255.89/.269 15.26/.021.46/.129.46/.129.55/.200.69/.256.87/.259 16.26/.021.46/.129.46/.129.55/.200.68/.254.89/.267 17.26/.021.46/.129.46/.129.55/.200.69/.256.87/.267 18.26/.021.46/.129.46/.129.55/.200.69/.256.86/.297 19.26/.021.46/.129.46/.129.51/.115.69/.191.84/.223 20.26/.021.46/.129.46/.129.51/.115.68/.189.85/.215 21.26/.021.46/.129.46/.129.51/.115.68/.192.85/.207 22.26/.021.46/.129.46/.129.51/.115.68/.186.89/.217 23.26/.021.46/.129.46/.129.48/.115.69/.190.83/.212 24.26/.021.46/.129.46/.129.51/.115.68/.187.87/.216 25.26/.021.46/.129.46/.129.51/.115.68/.190.84/.212 26.26/.021.46/.129.46/.129.51/.115.69/.189.84/.221 27.26/.021.46/.129.46/.129.48/.115.51/.079.82/.210 28.46/.007.46/.007.55/.134.53/.071.54/.087.78/.207 29.46/.007.46/.007.55/.134.53/.071.54/.087.79/.193 30.46/.007.59/.015.60/.019.60/.019.61/.042.82/.188 109.46/.007.59/.015.68/.029.68/.031.68/.031.82/.182
1 2 3 4 5 1.11/.029.32/.207.45/.260.46/.241.46/.241.77/.501 2.11/.029.63/.476.61/.438.83/.617.71/.539.76/.381 3.11/.029.13/.029.63/.454.63/.439.80/.587.76/.393 4.11/.029.11/.023.13/.029.13/.026.64/.366.78/.365 5.11/.029.94/.815.97/.731.95/.733.66/.349.84/.377 6.11/.029.94/.815.97/.731.97/.733.66/.349.80/.325 7.11/.029.94/.815.97/.731.97/.733.63/.349.82/.335 8.11/.029.94/.815.97/.731.47/.258.87/.651.85/.353 9.11/.029.94/.815.97/.731.48/.258.85/.649.85/.359 10.11/.029.94/.815.94/.721.46/.252.86/.638.85/.344 11.11/.029.94/.815.94/.721.46/.252.86/.638.84/.351 12.11/.029.94/.815.94/.724.46/.252.86/.641.78/.321 13.11/.029.11/.023.94/.767.68/.438.70/.443.85/.278 14.26/.021.46/.129.46/.129.55/.200.70/.272.87/.312 15.26/.021.46/.129.46/.129.55/.200.70/.271.87/.311 16.26/.021.46/.129.46/.129.55/.200.70/.270.89/.315 17.26/.021.46/.129.46/.129.55/.200.70/.267.87/.309 18.26/.021.46/.129.46/.129.55/.200.70/.267.89/.319 19.26/.021.46/.129.46/.129.51/.116.70/.216.84/.249 20.26/.021.46/.129.46/.129.51/.116.70/.216.85/.239 21.26/.021.46/.129.46/.129.51/.116.70/.213.86/.235 22.26/.021.46/.129.46/.129.51/.116.70/.213.85/.235 23.26/.021.46/.129.46/.129.51/.116.70/.216.86/.238 24.26/.021.46/.129.46/.129.51/.116.70/.216.86/.235 25.26/.021.46/.129.46/.129.51/.116.70/.213.86/.243 26.26/.021.46/.129.46/.129.51/.116.70/.216.86/.237 27.26/.021.46/.129.46/.129.51/.116.53/.107.89/.236 28.46/.007.46/.007.55/.134.53/.071.54/.087.80/.209 29.46/.007.46/.007.55/.134.53/.071.54/.087.76/.210 30.46/.007.59/.015.60/.019.60/.019.61/.042.80/.197 109.46/.007.59/.015.68/.029.68/.031.68/.031.79/.200
1 2 3 4 5 1.11/.029.32/.207.45/.260.46/.241.46/.241.70/.408 2.11/.029.63/.476.61/.438.82/.616.70/.545.66/.394 3.11/.029.13/.029.63/.454.62/.438.79/.586.66/.388 4.11/.029.11/.029.13/.031.13/.031.64/.372.66/.320 5.11/.029.94/.821.97/.729.95/.737.66/.353.80/.350 6.11/.029.94/.821.97/.729.97/.737.66/.353.72/.293 7.11/.029.94/.821.97/.729.97/.737.66/.353.83/.305 8.11/.029.94/.821.97/.729.48/.264.87/.655.76/.334 9.11/.029.94/.821.95/.729.48/.264.87/.655.80/.333 10.11/.029.94/.821.97/.725.48/.261.89/.647.85/.322 11.11/.029.94/.821.97/.726.48/.261.89/.646.83/.329 12.11/.029.94/.821.97/.725.48/.261.89/.647.75/.306 13.11/.029.11/.029.97/.773.70/.449.70/.449.78/.272 14.26/.021.46/.129.46/.129.55/.200.70/.268.83/.296 15.26/.021.46/.129.46/.129.55/.200.70/.268.84/.283 16.26/.021.46/.129.46/.129.55/.200.70/.266.85/.300 17.26/.021.46/.129.46/.129.55/.200.70/.264.85/.303 18.26/.021.46/.129.46/.129.55/.200.70/.264.80/.309 19.26/.021.46/.129.46/.129.51/.116.70/.212.84/.271 20.26/.021.46/.129.46/.129.51/.116.70/.212.84/.243 21.26/.021.46/.129.46/.129.51/.116.70/.210.86/.258 22.26/.021.46/.129.46/.129.51/.116.70/.210.86/.231 23.26/.021.46/.129.46/.129.51/.116.70/.212.83/.219 24.26/.021.46/.129.46/.129.51/.116.70/.212.84/.220 25.26/.021.46/.129.46/.129.51/.116.70/.210.87/.231 26.26/.021.46/.129.46/.129.51/.116.70/.212.85/.246 27.26/.021.46/.129.46/.129.51/.116.53/.103.86/.252 28.46/.007.46/.007.55/.134.53/.070.59/.100.80/.197 29.46/.007.46/.007.55/.134.53/.070.59/.100.80/.197 30.46/.007.59/.015.60/.019.60/.019.61/.042.78/.176 109.46/.007.59/.015.68/.029.68/.036.68/.036.77/.210
1 2 3 4 5 1.11/.029.32/.207.45/.260.40/.212.44/.232.66/.385 2.11/.029.63/.476.61/.438.82/.616.70/.540.70/.408 3.11/.029.13/.029.63/.454.62/.438.60/.408.74/.433 4.11/.029.11/.029.13/.032.13/.032.64/.377.59/.322 5.11/.029.94/.821.97/.731.95/.740.66/.355.75/.350 6.11/.029.94/.821.97/.731.97/.740.66/.355.74/.303 7.11/.029.94/.821.97/.731.97/.740.66/.355.74/.324 8.11/.029.94/.821.97/.731.48/.267.87/.658.82/.374 9.11/.029.94/.821.95/.731.48/.267.87/.658.84/.362 10.11/.029.94/.821.94/.719.49/.265.89/.649.80/.332 11.11/.029.94/.821.94/.719.49/.265.89/.649.76/.312 12.11/.029.94/.821.94/.719.49/.265.89/.649.77/.314 13.11/.029.11/.029.97/.774.70/.452.70/.452.80/.280 14.26/.021.46/.129.46/.129.55/.200.70/.298.80/.351 15.26/.021.46/.129.46/.129.55/.200.70/.296.84/.341 16.26/.021.46/.129.46/.129.55/.200.70/.296.84/.347 17.26/.021.46/.129.46/.129.55/.200.70/.293.84/.344 18.26/.021.46/.129.46/.129.55/.200.70/.294.85/.374 19.26/.021.46/.129.46/.129.49/.115.70/.268.90/.350 20.26/.021.46/.129.46/.129.49/.115.70/.269.87/.332 21.26/.021.46/.129.46/.129.49/.115.70/.268.86/.317 22.26/.021.46/.129.46/.129.49/.115.70/.268.87/.327 23.26/.021.46/.129.46/.129.49/.115.70/.268.87/.330 24.26/.021.46/.129.46/.129.49/.115.70/.269.86/.315 25.26/.021.46/.129.46/.129.49/.115.70/.269.85/.328 26.26/.021.46/.129.46/.129.49/.115.70/.268.86/.313 27.26/.021.46/.129.46/.129.49/.115.53/.157.84/.312 28.46/.007.46/.007.55/.134.53/.070.59/.100.79/.214 29.46/.007.46/.007.55/.134.53/.070.59/.100.79/.220 30.46/.007.59/.015.60/.019.60/.019.61/.042.77/.187 109.46/.007.59/.015.68/.029.68/.042.68/.042.78/.228
1 2 3 4 5 1.11/.029.32/.207.45/.260.40/.212.44/.232.68/.414 2.11/.029.63/.476.61/.438.82/.616.70/.531.69/.426 3.11/.029.13/.029.63/.454.62/.438.60/.408.69/.430 4.11/.029.11/.029.13/.029.13/.029.63/.368.56/.280 5.11/.029.94/.821.97/.741.97/.741.66/.357.74/.301 6.11/.029.94/.821.97/.741.97/.741.66/.357.75/.313 7.11/.029.94/.821.97/.741.97/.741.66/.357.76/.312 8.11/.029.94/.821.97/.741.48/.268.87/.660.77/.323 9.11/.029.94/.821.97/.741.48/.268.87/.660.77/.330 10.11/.029.94/.821.94/.727.49/.264.89/.648.76/.304 11.11/.029.94/.821.94/.727.49/.264.89/.648.76/.300 12.11/.029.94/.821.94/.727.49/.264.89/.648.77/.313 13.11/.029.11/.029.95/.777.69/.447.69/.447.77/.268 14.26/.021.46/.129.46/.129.55/.200.68/.281.72/.299 15.26/.021.46/.129.46/.129.55/.200.68/.281.77/.304 16.26/.021.46/.129.46/.129.55/.200.68/.281.77/.303 17.26/.021.46/.129.46/.129.55/.200.68/.281.77/.303 18.26/.021.46/.129.46/.129.55/.200.68/.281.80/.327 19.26/.021.46/.129.46/.129.49/.115.68/.256.80/.289 20.26/.021.46/.129.46/.129.49/.115.68/.256.79/.262 21.26/.021.46/.129.46/.129.49/.115.68/.256.79/.262 22.26/.021.46/.129.46/.129.49/.115.68/.256.79/.264 23.26/.021.46/.129.46/.129.49/.115.68/.256.79/.262 24.26/.021.46/.129.46/.129.49/.115.68/.256.79/.263 25.26/.021.46/.129.46/.129.49/.115.68/.256.79/.263 26.26/.021.46/.129.46/.129.49/.115.68/.256.80/.264 27.26/.021.46/.129.46/.129.49/.115.51/.144.76/.259 28.46/.007.46/.007.55/.134.51/.060.60/.103.80/.211 29.46/.007.46/.007.55/.134.51/.060.59/.104.80/.210 30.46/.007.59/.015.60/.019.60/.019.61/.042.78/.184 109.46/.007.59/.015.68/.029.70/.110.69/.049.79/.196
n [1,n]
n n n