Diae Hu LDA for Audio Music April, 00 Terms Model Terms (per sog: Variatioal Terms: p( α Γ( i α i i Γ(α i p( p(, β p(c, A j Σ i α i i i ( V / ep β (i j ij (3 q( γ Γ( i γ i i Γ(γ i q( φ q( ω { } (c A T Σ (c A i γ i ( (4 (5 φ i i (6 V ω j j ( ωj j (7 {,..., K} is a scalar represeted by a K vector, where i for a uique i. j is a biary V vector idicatig which of the pitches are o for the th segmet. c is a cotiuous V vector represetig the chromogram for that time period {,, c} are the sequece of {topics, words, chromograms} withi a sog. There are N of each. A is a V V matri Joit Distributio p(,,, c α, β, A p( α p( α p( p(, β p(c, A (8 Σ N/ { ep } (c K A T Σ (c A j V ( i β ij i j (9
Diae Hu LDA for Audio Music April, 00 3 Margial Distributio (Likelihood p(c α, β, A p( α p( p(, β p(c A (0 p( α p( p(, β p(c, A ( 4 Variatioal Distributio 5 Decompose Log-likelihood q(,, γ, φ, ω q( γ q( φ q( ω ( For readability, we drop depedece parameters below. 6 Lower-Boud l p(c α, β, A q(,, l p(,,, c q(,, [ p(,,, c q(,, l l q(,, q(,, l p(,,, c q(,, p(,, q(,, q(,, l p(,, q(,, (3 (4 (5 L(γ, φ, ω; α, β, A KL( q(,, p(,, (6 L(γ, φ, ω; α, β, A q(,, l p(,,, c q(,, d (7 q(,, l p(,,, c d q(,, l q(,, d (8 E q [l p(,,, c α, β, A E q [l p(,, γ, φ, ω (9 E q[l p( α + E q [l p( + E q [l p(c, A + E q [l p(, β E [l q( γ E q [l q( φ E [l q( ω (0
Diae Hu LDA for Audio Music April, 00 Where, E q [l p( α l Γ( α i l Γ(α i + (α i (Ψ(γ i Ψ( γ i ( E q [l p( E q [l p(c, A E q [l p(, β φ i (Ψ(γ i Ψ( γ i ( l Σ N/ j i j k c T kσ kj c j + j i j k l j k c T kσ kl A ljω j (ω j ωjm jj + ω j ω k M kj (3 φ i l β ij ω j (4 E [l q( γ l Γ( γ i l Γ(γ i + (γ i (Ψ(γ i Ψ( γ i (5 E q [l q( φ E [l q( ω j φ i l φ i (6 ωj T l ω j + ( ωj T l( ω j (7 The mai differeces from the origial LDA model occur i the terms show i eq. (3, 4, 7: i E q [l p(c, A E q [l p(c, A (8 [ N l Σ + E N/ q (c A T Σ (c A l Σ N/ l Σ N/ tr [ N l Σ N/ c T Σ c + c T Σ AE q [ c T Σ c + c T Σ Aω ( diag(ω ω + ω ω T A T Σ A j k c T kσ kj c j + j (9 E q [ T A T Σ A (30 (3 j k l j k c T kσ kl A ljω j (ω j ωjm jj + ω j ω k M kj (3 3
Diae Hu LDA for Audio Music April, 00 E q [l p(, β E q [l p(, β E q l β (i j ij (33 j j E q [l q( ω E q [l E q [ i j l β ij j j E q [E i q [ j l β ij (34 φ i l β ij ω j (35 q( ω j (36 (37 V E q l ω j j ( ω j j (38 j j E q [ j l ω j + ( E q [ j l( ω j (39 ωj T l ω j + ( ωj T l( ω j (40 Gettig from the last term i eq. (30 to eq. (3 is show i sectio A.. I eq. (34, ad are idepedet of each other, so we ca break up the epectatio. 7 Iferrig Variatioal Parameters 7. Iferrig φ i We maimie the lower boud L(γ, φ, ω; α, β, A with respect to the elemets of φ: ( K L [φi φ i (Ψ(γ i Ψ i γ ( V i + φ i j ω V j l β ij φ i l φ i + λ j φ i (4 Take the derivative of L [φi: L ( ( [φi K Ψ(γ i Ψ i φ γ i + V j ω j l β ij l φ i + λ (4 i Settig eq. (4 equal to 0 ad solvig for φ i : { ( K } { φ i ep Ψ(γ i Ψ i γ V } i ep j ω j l β ij { ( K } ep Ψ(γ i Ψ i γ V i j (43 β ij ω j (44 7. Iferrig ω j Let M A T Σ A. 4
Diae Hu LDA for Audio Music April, 00 We maimie the lower boud L(γ, φ, ω; α, β, A with respect to the elemets of ω: L [ωj Take the derivative of L [ωj: k l + c T kσ kl A ljω j (ω j ω jm jj ω j ω k M kj k φ i l β ij ω j ωj T l ω j ( ωj T l( ω j (45 L [ωj ω j k l c T kσ kl A lj M jj k j ω k M kj + ( ωj φ i l β ij l ω j (46 Set eq. (46 equal to 0 ad solve for ω : ( ωj l ω j k l ω j σ k l σ k l c T kσ kl A lj M jj k j ω k M kj + φ i l β ij (47 c T kσ kl A lj M jj k j ω k M kj + φ i l β ij (48 c T kσ kl A lj (AT Σ A jj k j ω k (A T Σ A kj + A more detailed derivatio of the derivative i eq. (46 ca be foud i sectio A.. Notes o choosig Σ ad A: Let A ai ad Σ δi. The, eq. (49 ca be writte as φ i l β ij (49 ω j c j δa a δ + (50 8 Parameter Estimatio The parameters for α ad β eist at the corpus level, ad L ow represets the overall variatioal lower boud. This is the sum of the idividual variatioal bouds for each sog i the corpus. 8. Estimatig β We maimie the lower boud L(γ, φ, ω; α, β, A with respect to the matri β ij : L [βij Take the derivative of L [βij: MN d φ i ω j l β ij + d j λ i β ij (5 j L [βij β ij MN d d φ i ω j + β ij λ i (5 5
Diae Hu LDA for Audio Music April, 00 Set eq. (5 to 0 ad solve for β ij : β ij ( M N d d φ i ω j β ij λ i (53 MN d d φ i ω j ( M ω φ d ij (54 The last term i eq. (5 is a Lagrage multiplier that restricts all rows of β to sum to. 8. Estimatig A 8.. Method : Ifer A from model We maimie the lower boud L(γ, φ, ω; α, β, A with respect to the matri A: L [A MN d d c T Σ Aω [( tr diag(ω ω + ω ω T A T Σ A (55 Take the derivative of L [A : L [A A MN d d ( c T Σ ω diag(ω ω + ω ω T A T Σ (56 Set eq. (56 equal to 0 ad solve for A: ( M N T ( d A Σ c T M Σ ω d N d d diag(ω 8.. Method : Estimate A ad Σ from groud-truth ω + ω Give that both c ad are observed, we use MLE to estimate A ad Σ from eq. (??: L l ω T (57 p(c, A, Σ (58 Maimie L with respects to A: N l Σ (c A T Σ (c A (59 N l Σ tr[ct Σ c + tr[c T Σ A tr[t A T Σ A (60 Isolate terms: L [A tr[ac T Σ tr[at Σ A T (6 L [A Take derivative: A Σ c T Σ A T (6 Set equal to 0 ad solve for A: A c T ( T (63 6
Diae Hu LDA for Audio Music April, 00 Maimie L with respect to Σ. Let Z Σ : 9 Variatioal EM Isolate terms: L [Z N l Z tr[zcct + tr[zac T tr[(at ZA (64 Take derivative: L [Z Z Z cct + c T A T AT A T (65 Set equal to 0 ad solve for Z: Z Σ cct + c T A T AT A T (66. (E-step Fi the curret model parameters α, β, ad A. Compute variatioal parameters {γ m, φ m, ω m } for each sog s m by miimiig the KL divergece: γ i α i + φ i (67 φ i ep[ψ(γ i k l V ω β j ij (68 j ω j σ c T kσ kl A lj (AT Σ A jj ω k (A T Σ A kj + k j φ i l β ij (69. (M-step Fi the curret variatioal parameters γ, φ, ad ω across all sogs from the E-step. Maimie the lower boud L(α, β, A, γ, φ, ω with respect to the model parameters: β M ω φ (70 d ( M N T ( d A Σ c T M Σ ω d N d d diag(ω ω + ω ω T (7 The variatioal parameters all deped o each other, so at each step of the variatioal EM, full variatioal iferece requires alteratig betwee eqs. (67, (68, ad (69 util the boud coverges. The update rule for α is the same as i the LDA paper; it caot be computed directly A Derivatio Details A. Computig E q [c, A We show how to derive the last term of eq. (3 from the last term of eq. (30. Give the distributio for q( ω, we first show how to calculate the term E q [ T : E q [ i j δ ij ω i + ( δ ij ω i ω j (7 δ ij (ω i ω i + ω i ω j (73 7
Diae Hu LDA for Audio Music April, 00 I matri form: E q [ T ω ω ω... ω ω V ω ω ω... ω ω V.... ω V ω ω V ω... ω V diag(ω ω + ω ω T δ ij if i j, ad 0 otherwise The first term i eq. (7 accouts for the case that i j, so E q [( i. The square does ot affect the epected value of the multiomial, so E q [( i E q [ i ω i. The secod term accouts for the case whe i j. Sice i ad j are idepedet, we ca break up the epectatio ito E q [ i j E q [ i E q [ j ω i ω j. Now, we show how to compute the last term of eq. (3: E q [ T A T Σ A E q tr[ T A T Σ A (74 E q tr[ T A T Σ A (75 [ N tr E q [ T A T Σ A (76 tr [ N ( diag(ω ω + ω ω T A T Σ A It will be useful later o to epress eq. (77 i scalar form, so we cotiue epadig. Let M A T Σ A. [ E q [ T A T Σ A N tr ( diag(ω ω + ω ω T M (77 (78 tr [ diag(ω ωm + tr [ ω ω T M (79 (ω j ωjm jj + ω j ω k M kj (80 j j k Sice, the term iside the epected value is a scalar i eq. (74, it is trivially equivalet to take the trace of it. 8
Diae Hu LDA for Audio Music April, 00 A. Computig ω j E q [ T A T Σ A f (ω j ωjm jj + ω j ω k M kj (8 j f [ωj (ω j ω jm jj + j k ω j ω k M kj (8 k V ω j M jj ωjm jj + ωjm jj + ω j ω k M kj (83 f ω j M jj + k j ω k M kj (84 k j Refereces [ Hu, D. J., Saul, L. K. A probabilistic topic model for music aalysis, Neural Iformatio Processig Systems (NIPS Applicatios for Topic Models Workshop, 009. 9