S6 - Introuction To Statistical Theory - Homework 4 (3.) Let θ be a ranom variable in (, ) with ensity where β, γ (, ) (i) Calculate the mean an moe of θ. Enrique Areyan February 3, 4 π(θ) θ γ e βθ, Solution: Since θ is proportional to θ γ e βθ, we know that θ Gamma(γ, β), where β >, γ >. Therefore, E[θ] γ γ, an Moe β β (ii) Suppose that X,..., X n are ranom variables, which, conitional on θ are inepenent an each have the Poisson istribution with parameter θ. Fin the form of the posterior ensity of θ given observe values X x,... X n x n. What is the posterior mean? Solution: We know that x θ P oisson(θ). Let us compute the posterior istribution: θ x π(θ)f(x; θ) Θ π(θ )f(x; θ )θ by efinition of posterior ensity θ γ e βθ e θ θ x constant x! constant replacing prior an Poisson istribution constant θ γ +x e βθ θ grouping like terms constant θ (γ+x) e (β+)θ rearranging exponents Gamma(γ + x, β + ), by efinition of Gamma istribution Therefore, the posterior mean is the mean of the gamma istribution with parameters γ + x an β +, i.e., E[θ x] γ + x β + Note that γ > an x,,,, so γ + x > an β > thus β + >. (ii) Suppose now that T,..., T n are ranom variables, which, conitional on θ are inepenent, each exponentially istribute with parameter θ. What is the moe of the posterior istribution of θ, given T t,..., T n t n? Solution: In this case we know that x θ Exp(θ). The posterior istribution is: θ x π(θ)f(x; θ) Θ π(θ )f(x; θ )θ by efinition of posterior ensity θ γ e βθ constant θe θt constant replacing prior an Exp. istribution constant θ γ + e βθ θt grouping like terms constant θ γ e (β+t)θ rearranging exponents Gamma(γ +, β + t), by efinition of Gamma istribution
Since γ > then γ + >, an we have a close form for the moe: Moe (γ + ) β + t γ β + t (3.3) Fin the form of the Bayes rule in an estimation problem with loss function { a(θ ) if θ, L(θ, ) b( θ) if > θ, where a an b are given positive constants. Solution: We wish to minimize the following expression: Θ L(θ, )π(θ x)θ As a function of, i.e., let f() be efine as follow: Fin f () f() a(θ )π(θ x)θ + a(θ )π(θ x)θ + b( θ)π(θ x)θ [ ] [ a θπ(θ x)θ π(θ x)θ + b [ b b( θ)π(θ x)θ π(θ x)θ ] π(θ x)θ a π(θ x)θ + a θπ(θ x)θ b θπ(θ x)θ θπ(θ x)θ (erivative of f with respect to ) using prouct rule an funamental theorem of calculus: f () ( b ) π(θ x)θ a π(θ x)θ + [bπ( x) + aπ( x)] aπ( x) bπ( x) ] b π(θ x)θ a b π(θ x)θ a π(θ x)θ + (a + b)π( x) (a + b)π( x) π(θ x)θ Set f () b π(θ x)θ a π(θ x)θ b π(θ x)θ a π(θ x)θ b a This expression represents a quantile of posterior π(θ x) in terms of a an b. Concretely: π(θ x)θ π(θ x)θ π(θ x)θ + π(θ x)θ since prob. as to π(θ x)θ + b a π(θ x)θ replacing for our minimun π(θ x)θ a a + b Solving for the posterior quantile
Hence, the Bayes rule is the quantile posterior a a + b of π(θ x). Note that we shoul check whether the minimality of the value obtain. Using the secon erivative test: f () b π(θ x)θ a π(θ x)θ bπ( x) + aπ( x) (a + b)π( x) (a + b) () The last equality is hols since the probability of taking action (x) given x is. By hypothesis a an b are given positive constants, an so is a + b, so we have a local minimun. Moreover, this expression is always positive an hence, the minimum foun is the global minimum. (3.4) Suppose that X is istribute as a binomial ranom variable with inex n an parameter θ. Calculate the Bayes rule (base on the single observation X) for estimating θ when the prior istribution is the uniform istribution on [, ] an the loss function is Is the rule you obtain minimax? L(θ, ) (θ ) /{θ( θ)} Solution: We wish to minimize the following expression: Θ L(θ, )π(θ x)θ Let us first fin the posterior istribution π(θ x). (θ ) θ( θ) π(θ x)θ θ x π(θ)f(x; θ) Θ π(θ )f(x; θ )θ by efinition of posterior ensity (n) x θ x ( θ) n x replacing prior an Binomial istribution Θ π(θ )f(x; θ )θ constant θ x ( θ) n x grouping like terms Beta(x +, n x + ), by efinition of Beta istribution Now we can minimize the function: (θ ) f() π(θ x)θ efine f() this way θ( θ) (θ ) θ x ( θ) n x θ( θ) B(x +, n x + ) θ replace posterior B(x +, n x + ) (θ ) θ x ( θ) n x θ simplifying Note that in this context is a constant, an so we can ignore this term for optimization B(x +, n x + ) purposes. Hence, optimize f() is equivalent to optimize g() efine as: g() (θ ) θ x ( θ) n x θ (θ θ + )θ x ( θ) n x θ θ x+ ( θ) n x θ θ x ( θ) n x θ + θ x ( θ) n x θ
Fin g () g () (erivative of g with respect to ): [ () θ x+ ( θ) n x θ θ x ( θ) n x θ + θ x ( θ) n x θ + θ x ( θ) n x θ ] θ x ( θ) n x θ Set g () θ x ( θ) n x θ + θ x ( θ) n x θ We can write this conition in terms of the Beta function, i.e., B(a, b) B(x +, n x) B(x, n x) θ x ( θ) n x θ θ x ( θ) n x θ t a ( t) b t, as follows: Finally, using ientities relating the Beta an Gamma function an the fact that n an x are positive integers: B(x +, n x) B(x, n x) Γ(x + )Γ(n x) Γ(n + ) Γ(x)Γ(n x) Γ(n) Γ(x + )Γ(n) x!(n )! Γ(n + )Γ(x) n!(x )! x n An hence, the Bayes rule is just the mean π X. A similar argument to that in (3.3), using secon n erivative test, shows that this is in fact the minimum. The obtaine rule π is minimax since it has constant risk, as the following computation shows: R(θ, π ) E θ L(θ, π ) efinition of risk ( E θ L θ, X ) n replacing for the rule π ( θ X n E θ θ( θ) ) by loss function θ θ X ( X n + n E θ θ( θ) ) squaring θ θ( θn n ) + nθ( θ)+n θ n θ( θ) expectation of a Binomial an Binomial square θ + nθ( θ) + n θ n θ( θ) nθ nθ + n θ n θ n θ( θ)
nθ nθ n θ( θ) n( θ) n ( θ) n Since n is fixe, this is a constant. (3.5) At a critical stage in the evelopment of a new airplane, a ecision must be taken to continue or to abanon the project. The financial viability of the project can be measure by a parameter θ, < θ <, the project being profitable if θ >. Data x provie information about θ. (i) If θ <, the cost to the taxpayer of continuing the project is ( θ) (in units of $bilion), whereas if θ > it is zero (since the project will be privatize if profitable). If θ > the cost of abanoning the project is (θ ) (ue to contractual arrangements for purchasing the airplane from the French), whereas if θ < it is zero. Derive the Bayes ecision rule in terms of the posterior mean of θ given x. Solution: Let A {, }, where enotes abanoning the project an continuing the project. Let us write the loss function as follow: L(θ, ) We want to minimize Θ L(θ, )π(θ x)θ, i.e., { { if θ ( (θ ) if < θ L(θ, ) θ) if θ if < θ Θ L(θ, )π(θ x)θ ( θ)π(θ x)θ if (θ )π(θ x)θ if The Bayes rule is that which minimizes the above expression. That is, the Bayes rule will choose to go on with the protect if ( θ)π(θ x)θ < (θ )π(θ x)θ or, equivalently, the Bayes rule will choose to abanon the protect if (θ )π(θ x)θ < ( θ)π(θ x)θ. (ii) The minister of aviation has prior ensity 6θ( θ) for θ. The Prime Minister has prior ensity 4θ 3. The prototype aeroplane is subjecte to trials, each inepenently having probability θ of success, an the ata x consist of the total number of trials require for the first successful result to be obtaine. For what values of x will there be serious ministerial isagreement? Solution: Clearly, X Geometric(θ), where X,,.... Hence π(x) ( θ) x θ. We can compute the posterior probability ensity π(θ x) for the minister of aviation (MA) an the Prime Minister (PM ): MA: π(θ x) constant θ( θ)( θ) x θ θ ( θ) x Beta(3, x + ) PM π(θ x) constant θ 3 ( θ) x θ θ 4 ( θ) x Beta(5, x) We want to fin the values of x for which the Bayes rule for MA an MP will isagree. Let π MA (θ x) an π MP (θ x) be the prior of the minister of aviation an Prime Minister respectively. Then, the values of x we are intereste in are:
π MA (θ x)θ π MP (θ x)θ x for which: ( θ)π MA(θ x)θ < ( θ)π MP (θ x)θ > (θ )π MA(θ x)θ (θ )π MP (θ x)θ Solving for x using the fact that π(θ x) is a probability istribution an computing its mean: Note that θπ MA (θ x)θ < θπ MA (θ x)θ π MA (θ x)θ θπ MP (θ x)θ > θπ MP (θ x)θ π MP (θ x)θ π MA (θ x)θ θπ MA (θ x)θ < π MP (θ x)θ θπ MP (θ x)θ > π MA (θ x)θ an θπ MA (θ x)θ E[π MA (θ x)]. Same for MP. We know each of these istributions are Beta an so we know its mean: E[π MA (θ x)] E[Beta(3, x + )] 3 x + 4 an E[π MP (θ x)] E[Beta(5, x)] 5 5 + x Replacing into our conitions: E[π MA(θ x)] < 3 x + 4 < E[π MP (θ x)] > 5 5 + x > < 3 x + 4 > 5 5 + x x + 4 < 6 x + 5 > x < x > 5 Hence, the minister of aviation will go on with the project if x < while the Prime Minister will choose to abanon the project if x > 5. This means that MA will choose to abanon if x >. So, for values x 3, 4; there will be a isagreement: MA woul want to abanon but PM will not. A ecision will have to be reache through some other argument. Note that this result makes intuitive sense: the prime minister prior is a cubic function that places lower emphasis on smaller values of θ an thus, is more tolerant to risk while the opposite is try for the minister of aviation whose prior is a quaratic function.