- PDF Free Download

Μέγεθος: px

Εμφάνιση ξεκινά από τη σελίδα:

Download ""

Ξανθίππη Στεφανόπουλος
5 χρόνια πριν
Προβολές:

9 tf idf t

13 MATLAB \index{}

15 \index{}

18 tf.idf MATLAB

19 N grams

35 /

38 N N N ń ż k k + 1 k + 1 k + 1

39 tf idf

j n d j = (t 1,j, t 2,j,..., t n,j ) n d j t i https://www.

40 j n d j = (t 1,j, t 2,j,..., t n,j ) n d j t i

41 1 2 D 1 D 2 D 3 D 4 d t tf(d, t) t d

42 df(t) t w w(d, t) = tf(d, t) d tf = (tf 1, tf 2, tf 3,..., tf n ) tf(d, t) = n(d, t) ntf(d, t) ntf(d, t) = n(d, t) max t n(d, t) max t n(d, t) d

43 (idf) idf(t) = log D df(t) idf(t) = log 1 + D df(t) idf nidf nidf = idf logd t d tf(t) idf(t) w(d, t) = tf(d, t) idf(t) w(d, t) = ntf(d, t) nidf(t) = n(d, t) max t n(d, t) idf logd tf idf D 1 D 2 D 3 Q D IDF = log(d/df i )

44 tf idf m n w i,j t i d j m»n w 1,1 w 1,2 w 1,3... w 1,n w 2,1 w 2,2 w 2,3... w 2,n A = w m,1 w m,2 w m,3... w m,n m

45 f(x, y) = f(y, x) f(x, y) f(x, z) + f(z, y) f(x, y) 0 x y f(x, x) = 0 m

46 d 1 d 2 q sim(d, q) = cos( d, q) = m d i q i i=1 m m d 2 i i=1 qi 2 i=1 θ 1 θ 2 d 1 d 2 d 1 d 2 L 1 L 2 ( m )1/p dist p (d, q) = d i q i p i=1

47 N N N T w 1 w 2 w 3 w 4 w 5 w i w 1 w 2 w 3 w 4 w 5 w 1 w 2 w 2 w 3 w 3 w 4 w 4 w 5

48 w 1 w 2 w 3 w 2 w 3 w 4 w 3 w 4 w 5 N P (w1 n ) w1 n = w 1, w 2,..., w n (w n 1 ) = P (w 1 )P (w 2 w 1 )P (w 3 w 2 1)P (w 4 w 3 1)...P (w n w n 1 1 ) = n k=1 P (w k w k 1 1 ) P (w n w1 n 1 ) n (w n w n 1 1 ) P (w n w n 1, w n 1 ) (w n w n 1 1 ) P (w n w n 1 n N 1)

49 n (w1 n ) P (w k w k 1 ) k=1 (w n w n 1 ) = count(w n 1w n ) count(wn 1 w) (w n w n 1 ) = count(w n 1w n ) count(w n 1 ) (w 3 w 1, w 2 ) = count(w 1, w 2, w 3 ) count(w 1, w 2 ) n N+1w n ) n N+1) (w n w n +1 ) = count(wn 1 count(w n 1 N N

50 N S #N Grams S = W (N 1) W S

51 C c i,j j C

52 C = c 1,1 c 1,2 c 1,3... c 1,ndocs c 2,1 c 2,2 c 2,3... c 2,ndocs c N Grams,1 c N Grams,2 c N Grams,3... c N Grams,ndocs function [ngrams, C, prob_global] = test_bigrams(words,... sentence_docs, N, n_docs, stop, min_length,max_length, theme) for i = 1:length(sentences) if length(sentence(i)) > N+1) words = textscan(sentence(i), '%s') for j = 1:length(words) if ismember(words(j), stop) ismember(words(j+1),... stop) continue; else create all_ngram, docs; end end end end [un_ngrams, docs] = unique_ngrams(all_ngrams, docs); unique_words = unique(all_words); for i = 1:size(un_ngrams), t = un_ngrams{i, 1}; % ckeck token's length if length(t)<min_length length(t)>max_length, removed_words(i)=1; continue; end end doc = doc(removed_words == 0); un_ngrams = un_ngrams(removed_words == 0); %remove alphanumeric

53 for i = 1:size(un_ngrams), exp = '[!?!#@$%^&*_+"()[]{}:.;< -\]'; n_parts=regexp(un_ngrams,'\d+'); % identify any numeric part m_parts= regexp(un_ngrams{i,1},exp); if isempty(n_parts) == 0, % if there is a numeric part rem_terms_alphanumeric(i) = 1; end if isempty(m_parts) == 0 rem_terms_alphanumeric(i) = 1; end continue; end

60 P recision tp tp + fp Recall tp tp + fn F measure 2 P recision Recall P recision + Recall

64 = 10

69 MATLAB

70 MATLAB t

75 P (w 2 w 1 ) = count(w 1w 2 ) count(w 1 )

76 bigrams doc P (w 3 w 1 w 2 ) = count(w 1w 2 w 3 ) count(w 1 w 2 )

81 MATLAB TaggedWords

87 MATLAB IndexedBigrams IndexedTrigrams IndexedUnigrams \index{} \index{} addition deletion completely unrelated document

88 p 0 p 1

89 tagged_bigrams tagged_trigrams

91 recall = = 0.75 recall = 8 24 =

93 # # # #

94 # # # #

95 MATLAB

99 dlsi.ua.es/~elloret/publications/textsummarization.pdf manning.com/books/natural-language-processing-in-action

100 https: //web.stanford.edu/~jurafsky/slp3/ed3book.pdf

101 term-vector-3.pdf Text-Documents-Clustering-using-K-Means-Algorithm

102

103

104

105

106

107

108

109

110 p unigram = p bi gram = 0.5 p tr gram = 0.5

111

112

113

114

115

116

117

118

119

120

121

122

123 p unigram = p bi gram = 0.5 p tr gram = 0.5

124

125

126

127

128

129

130

131

132

133

134

135

136 p unigram = p bi gram = 0.5 p tr gram = 0.5

137

138

139

140

141

142

143

144

145

146

147

148 p unigram = p bi gram = 0.5 p tr gram = 0.5

149

150

151

152

153

154

155

156

Σχετικά έγγραφα

Διαχείριση εγγράφων. Αποθήκες και Εξόρυξη Δεδομένων Διδάσκων: Μ. Χαλκίδη

Διαχείριση εγγράφων. Αποθήκες και Εξόρυξη Δεδομένων Διδάσκων: Μ. Χαλκίδη Διαχείριση εγγράφων Αποθήκες και Εξόρυξη Δεδομένων Διδάσκων: Μ. Χαλκίδη Απεικόνιση κειμένων για Information Retrieval Δεδομένου ενός κειμένου αναζητούμε μια μεθοδολογία απεικόνισης του γραμματικού χώρου