CS 1675 Introduction to Machine Learning Lecture 7. Density estimation. Milos Hauskrecht 5329 Sennott Square

Μέγεθος: px

Εμφάνιση ξεκινά από τη σελίδα:

Download "CS 1675 Introduction to Machine Learning Lecture 7. Density estimation. Milos Hauskrecht 5329 Sennott Square"

Θεοδώρα Γιαννόπουλος
5 χρόνια πριν
Προβολές:

1 CS 675 Itroducto to Mache Learg Lecture 7 esty estmato Mlos Hausrecht mlos@cs.tt.edu 539 Seott Square

2 ata: esty estmato {.. } a vector of attrbute values Objectve: estmate the model of the uderlyg robablty dstrbuto over varables X X usg eamles true dstrbuto X samles {.. } estmate ˆ X

3 ML arameter estmato Model ˆ X X ata {.. } Mamum lelhood ML Fd that mamzes ma.. log Log lelhood has the same mamum as lelhood log Ideedet eamles

4 ML arameter estmato Model ˆ X X ata {.. } Mamum lelhood ML Fd that mamzes ma Log lelhood has the same mamum as lelhood log log The otmum satsfes: d log d 0 It ca be ofte solved aalytcally

5 MA arameter estmato Model ˆ X X ata {.. } Mamum a osteror robablty Fd that mamzes ma Lelhood of data ror dstrbuto va Bayes theorem ormalzg factor Cojugate choces: ror dstrbuto o the arameters matches the data dstrbuto osteror s the same tye of the dstrbuto as the ror

6 MA arameter estmato Model ˆ X X ata {.. } Mamum a osteror robablty Fd that mamzes ma The otmum satsfes: d d log 0 or d d 0 It ca be ofte solved aalytcally

7 Beroull dstrbuto Model for radom varable wth two outcomes Radom varable: Two outcomes: 0 or strbuto: where s the robablty of Eamle: Co toss Outcomes: Head à Tal à 0 à robablty of a Head

8 Beroull dstrbuto ata : d samle of outcomes co fls Lelhood of data: Loglelhood of data: l - umber of s - umber of 0s log log ML estmate: ML

9 Beroull dstrbuto ata : d samle of outcomes co fls osteror of data: Lelhood Cojugate ror: osteror: Γ Γ Γ Beta Beta

10 Bomal dstrbuto models couts of occurreces of bary outcomes orderdeedet sequece of trals Model: robablty of a outcome head robablty of a outcome 0 tal robablty dstrbuto fucto: Eamle roblem: co fls where each co fl ca have two results: head or tal * 3* - umber of outcomes - umber of outcomes

11 Bomal dstrbuto: Bomal dstrbuto

12 Mamum lelhood ML estmate. Lelhood of data: Log-lelhood!!! log log!!! log log l Costat from the ot of otmzato!!! ML ML Soluto: The same as for Beroull ad wth d sequece of eamles

13 osteror desty osteror desty ror choce Lelhood osteror MA estmate ma arg MA va Bayes rule Γ Γ Γ Beta Γ Γ Γ Beta MA

14 Multomal dstrbuto Eamle: multle rolls of a dce wth 6 results Outcome: couts of occurreces of ossble outcomes of trals: - a umber of tmes a outcome has bee see Model arameters: s.t. - robablty of a outcome robablty dstrbuto: ML estmate:!!! ML! Multomal dstrbuto

15 osteror ad MA estmate Choce of the ror: rchlet dstrbuto.... r r MA.. MA estmate: osteror desty.. Γ Γ r rchlet s the cojugate choce for the multomal samlg!!!!

16 rchlet dstrbuto rchlet dstrbuto: Assume: 3.. Γ Γ r 3

17 Other dstrbutos The same deas ca be aled to other dstrbutos Tycally we choose dstrbutos that behave well so that comutatos lead to ce solutos Eoetal famly of dstrbutos Cojugate choces for some of the dstrbutos from the eoetal famly: Bomal Beta Multomal - rchlet Eoetal Gamma osso Iverse Gamma Gaussa - Gaussa mea ad Wshart covarace

18 Gaussa ormal dstrbuto Model of a real-valued outcome Gaussa: ~ σ arameters: - mea σ - stadard devato esty fucto: σ e[ Eamle: σ π σ ]

19 arameter estmates Loglelhood ML estmates of the mea ad varace: ML varace estmate s based Ubased estmate: ˆ ˆ ˆ σ log σ σ l ˆ ˆ σ ˆ σ σ σ E E

20 Multvarate ormal dstrbuto Multvarate ormal: ~ arameters: - mea - covarace matr esty fucto: e d / / π T Eamle:

21 arttoed Gaussa strbutos Multvarate Gaussa: Eamle: recso matr What are the dstrbutos for margals ad codtoals? a a b

22 arttoed Codtoals ad Margals Codtoal desty: Margal esty:

23 arttoed Codtoals ad Margals

24 arameter estmates Loglelhood ML estmates of the mea ad covaraces: Covarace estmate s based Ubased estmate: ˆ T ˆ ˆ ˆ log l T ˆ ˆ ˆ E E T ˆ ˆ ˆ

25 osteror of a multvarate ormal Assume a ror o the mea that s ormally dstrbuted: The the osteror of s ormally dstrbuted T d / / e π e * / / T d π e / / T d π

26 osteror of a multvarate ormal The the osteror of s ormally dstrbuted e / / T d π

27 Other dstrbutos Gamma dstrbuto models desty over o-egatve umbers: λ a b λ a b λ e a for λ [ 0 ] Γ a b

28 Other dstrbutos Eoetal dstrbuto: A secal case of Gamma for a λ b e b Uform dstrbuto: λ b a b for [ a b] b a osso dstrbuto: models a umber of evets occurrg a secfc tme terval λ λ e λ for { 0 }!

29 Sequetal Bayesa arameter estmato Sequetal Bayesa aroach Uder the d the estmates of the osteror ca be comuted cremetally for a sequece of data ots If we use a cojugate ror we get bac the same osteror Assume we slt the data the last elemet ad the rest The: d d A ew ror

30 Eoetal famly Eoetal famly: all robablty mass / desty fuctos that ca be wrtte the eoetal ormal form e[ ] T f η h η t Z η η a vector of atural or caocal arameters t a fucto referred to as a suffcet statstc h a fucto of t s less mortat Zη a ormalzato costat a artto fucto { T Z η h e η t } d Other commo form: [ η t A ] T f η h e η log Z η A η

31 Eoetal famly: eamles Beroull dstrbuto π π π π e log log π π π e{ log π } e log π Eoetal famly [ ] T f η h e η t Z η arameters π η log ote π η π e t Z η η h π e

32 Eoetal famly: eamles Uvarate Gaussa dstrbuto Eoetal famly arameters e log e σ σ σ σ π / / σ σ η t π / h log 4 e log e η η η σ σ η Z [ ] e t h Z f T η η η ] e[ σ π σ σ

Σχετικά έγγραφα

Latent variable models Variational approximations.

Latent variable models Variational approximations. CS 3750 Mache Learg Lectre 9 Latet varable moel Varatoal appromato. Mlo arecht mlo@c.ptt.e 539 Seott Sqare CS 750 Mache Learg Cooperatve vector qatzer Latet varable : meoalty bary var Oberve varable :