CS 675 Itroducto to Mache Learg Lecture 7 esty estmato Mlos Hausrecht mlos@cs.tt.edu 539 Seott Square
ata: esty estmato {.. } a vector of attrbute values Objectve: estmate the model of the uderlyg robablty dstrbuto over varables X X usg eamles true dstrbuto X samles {.. } estmate ˆ X
ML arameter estmato Model ˆ X X ata {.. } Mamum lelhood ML Fd that mamzes ma.. log Log lelhood has the same mamum as lelhood log Ideedet eamles
ML arameter estmato Model ˆ X X ata {.. } Mamum lelhood ML Fd that mamzes ma Log lelhood has the same mamum as lelhood log log The otmum satsfes: d log d 0 It ca be ofte solved aalytcally
MA arameter estmato Model ˆ X X ata {.. } Mamum a osteror robablty Fd that mamzes ma Lelhood of data ror dstrbuto va Bayes theorem ormalzg factor Cojugate choces: ror dstrbuto o the arameters matches the data dstrbuto osteror s the same tye of the dstrbuto as the ror
MA arameter estmato Model ˆ X X ata {.. } Mamum a osteror robablty Fd that mamzes ma The otmum satsfes: d d log 0 or d d 0 It ca be ofte solved aalytcally
Beroull dstrbuto Model for radom varable wth two outcomes Radom varable: Two outcomes: 0 or strbuto: where s the robablty of Eamle: Co toss Outcomes: Head à Tal à 0 à robablty of a Head
Beroull dstrbuto ata : d samle of outcomes co fls Lelhood of data: Loglelhood of data: l - umber of s - umber of 0s log log ML estmate: ML
Beroull dstrbuto ata : d samle of outcomes co fls osteror of data: Lelhood Cojugate ror: osteror: Γ Γ Γ Beta Beta
Bomal dstrbuto models couts of occurreces of bary outcomes orderdeedet sequece of trals Model: robablty of a outcome head robablty of a outcome 0 tal robablty dstrbuto fucto: Eamle roblem: co fls where each co fl ca have two results: head or tal * 3* - umber of outcomes - umber of outcomes
Bomal dstrbuto: Bomal dstrbuto
Mamum lelhood ML estmate. Lelhood of data: Log-lelhood!!! log log!!! log log l Costat from the ot of otmzato!!! ML ML Soluto: The same as for Beroull ad wth d sequece of eamles
osteror desty osteror desty ror choce Lelhood osteror MA estmate ma arg MA va Bayes rule Γ Γ Γ Beta Γ Γ Γ Beta MA
Multomal dstrbuto Eamle: multle rolls of a dce wth 6 results Outcome: couts of occurreces of ossble outcomes of trals: - a umber of tmes a outcome has bee see Model arameters: s.t. - robablty of a outcome robablty dstrbuto: ML estmate:!!! ML! Multomal dstrbuto
osteror ad MA estmate Choce of the ror: rchlet dstrbuto.... r r MA.. MA estmate: osteror desty.. Γ Γ r rchlet s the cojugate choce for the multomal samlg!!!!
rchlet dstrbuto rchlet dstrbuto: Assume: 3.. Γ Γ r 3
Other dstrbutos The same deas ca be aled to other dstrbutos Tycally we choose dstrbutos that behave well so that comutatos lead to ce solutos Eoetal famly of dstrbutos Cojugate choces for some of the dstrbutos from the eoetal famly: Bomal Beta Multomal - rchlet Eoetal Gamma osso Iverse Gamma Gaussa - Gaussa mea ad Wshart covarace
Gaussa ormal dstrbuto Model of a real-valued outcome Gaussa: ~ σ arameters: - mea σ - stadard devato esty fucto: σ e[ Eamle: σ π σ ] 0.4 0.35 0 0.3 0.5 0. 0.5 0. 0.05 0-4 -3 - - 0 3 4
arameter estmates Loglelhood ML estmates of the mea ad varace: ML varace estmate s based Ubased estmate: ˆ ˆ ˆ σ log σ σ l ˆ ˆ σ ˆ σ σ σ E E
Multvarate ormal dstrbuto Multvarate ormal: ~ arameters: - mea - covarace matr esty fucto: e d / / π T Eamle:
arttoed Gaussa strbutos Multvarate Gaussa: Eamle: recso matr What are the dstrbutos for margals ad codtoals? a a b
arttoed Codtoals ad Margals Codtoal desty: Margal esty:
arttoed Codtoals ad Margals
arameter estmates Loglelhood ML estmates of the mea ad covaraces: Covarace estmate s based Ubased estmate: ˆ T ˆ ˆ ˆ log l T ˆ ˆ ˆ E E T ˆ ˆ ˆ
osteror of a multvarate ormal Assume a ror o the mea that s ormally dstrbuted: The the osteror of s ormally dstrbuted T d / / e π e * / / T d π e / / T d π
osteror of a multvarate ormal The the osteror of s ormally dstrbuted e / / T d π
Other dstrbutos Gamma dstrbuto models desty over o-egatve umbers: λ a b λ a b λ e a for λ [ 0 ] Γ a b
Other dstrbutos Eoetal dstrbuto: A secal case of Gamma for a λ b e b Uform dstrbuto: λ b a b for [ a b] b a osso dstrbuto: models a umber of evets occurrg a secfc tme terval λ λ e λ for { 0 }!
Sequetal Bayesa arameter estmato Sequetal Bayesa aroach Uder the d the estmates of the osteror ca be comuted cremetally for a sequece of data ots If we use a cojugate ror we get bac the same osteror Assume we slt the data the last elemet ad the rest The: d d A ew ror
Eoetal famly Eoetal famly: all robablty mass / desty fuctos that ca be wrtte the eoetal ormal form e[ ] T f η h η t Z η η a vector of atural or caocal arameters t a fucto referred to as a suffcet statstc h a fucto of t s less mortat Zη a ormalzato costat a artto fucto { T Z η h e η t } d Other commo form: [ η t A ] T f η h e η log Z η A η
Eoetal famly: eamles Beroull dstrbuto π π π π e log log π π π e{ log π } e log π Eoetal famly [ ] T f η h e η t Z η arameters π η log ote π η π e t Z η η h π e
Eoetal famly: eamles Uvarate Gaussa dstrbuto Eoetal famly arameters e log e σ σ σ σ π / / σ σ η t π / h log 4 e log e η η η σ σ η Z [ ] e t h Z f T η η η ] e[ σ π σ σ