MLE for Beta(1, theta)

This shows how to find the maximum likelihood estimator for $\theta$ for a $Beta(1, \theta)$ random variable.

The data distribution is:

$$X \sim Beta(1, \theta) \\ $$

The likelihood function is:



$$\begin{align} L(\theta | X_1, ..., X_n) &= \prod_{i = 1}^{n} \frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)}X_i^{\alpha - 1}(1 - X_i)^{\beta - 1} \\ &= \prod_{i = 1}^{n} \frac{\Gamma(1 + \theta)}{\Gamma(1)\Gamma(\theta)}X_i^{1 - 1}(1 - X_i)^{\theta - 1} \\ &= \prod_{i = 1}^{n} \frac{\Gamma(1 + \theta)}{\Gamma(1)\Gamma(\theta)}X_i^{1 - 1}(1 - X_i)^{\theta - 1} \\ &= \prod_{i = 1}^{n} \frac{\Gamma(1 + \theta)}{\Gamma(1)\Gamma(\theta)}X_i^{0}(1 - X_i)^{\theta - 1} \\ &= \prod_{i = 1}^{n} \frac{\Gamma(1 + \theta)}{\Gamma(1)\Gamma(\theta)} (1 - X_i)^{\theta - 1} \\ \end{align}$$

Note that $\Gamma(1) = (1 - 1)! = 0! = 1$,
and $\frac{\Gamma(1 + \theta)}{\Gamma(\theta)} = \frac{(1 + \theta - 1)!}{(\theta - 1)!} = \frac{(\theta)!}{(\theta - 1)!}= \theta$,
so $\frac{\Gamma(1 + \theta)}{\Gamma(1)\Gamma(\theta)} = \theta$.

$$\begin{align} L(\theta | X_1, ..., X_n) &= \prod_{i = 1}^{n} \frac{\Gamma(1 + \theta)}{\Gamma(1)\Gamma(\theta)}(1 - X_i)^{\theta - 1} \\ &= \prod_{i = 1}^{n} \theta (1 - X_i)^{\theta - 1} \\ &= \theta^n \prod_{i = 1}^{n} (1 - X_i)^{\theta - 1} \\ \end{align}$$

The log-likelihood function is:

$$\begin{align} l(\theta | X_1, ..., X_n) &= nlog(\theta) + (\theta - 1)\sum_{i = 1}^{n}log(1 - X_i) \\ &= nlog(\theta) + \theta \sum_{i = 1}^{n}log(1 - X_i) - \sum_{i = 1}^{n}log(1 - X_i) \\ \end{align}$$

Note that the $\theta$ does not appear in the last term, so the partial derivative with respect to $\theta$ will have only two terms.

The first partial derivative of the log-likelihood function is:

$$\frac{\partial l}{\partial \theta} = \frac{n}{\theta}  + \sum_{i = 1}^{n}log(1 - X_i) \\ $$

The maximum likelihood estimator for $\theta$ is obtained by setting this equal to zero and then solving for $\theta$:
 
$$\begin{align} 0 &= \frac{n}{\theta}  + \sum_{i = 1}^{n}log(1 - X_i) \\ -\frac{n}{\theta} &= \sum_{i = 1}^{n}log(1 - X_i) \\ -n &= \theta \sum_{i = 1}^{n}log(1 - X_i) \\ \hat{\theta} &= \frac{-n}{\sum_{i = 1}^{n}log(1 - X_i)} \\ \end{align}$$

The second partial derivative with respect to $\theta$ of the log-likelihood function is:

$$\frac{\partial^{2} l}{\partial \theta^2} = -\frac{n}{\theta^2} \\ $$

The Fisher Information is:

$$I(\theta) = -E\Bigg[\frac{\partial^{2} l}{\partial \theta^2}\Bigg] = -E\Bigg[-\frac{n}{\theta^2}\Bigg] = \frac{n}{\theta^2}$$

The variance of the maximum likelihood estimator is:

$$V[\hat{\theta}] = \frac{1}{I(\theta)} = \frac{\theta^2}{n}$$