Bayesian Learning in Machine Learning

Probability for Learning

Probability for classification and modeling concepts.
Bayesian probability
- Notion of probability interpreted as partial belief
Bayesian Estimation
- It calculate the validity of a proposition
           - Based on prior estimate of its probability
           - and New relevant evidence

Bayes Theorem

Goal: To determine the most probable hypothesis, given the data D plus any initial knowledge about the prior probabilities of the various hypotheses in H.

Bayes Rule :
                      P(h|D) = P(D|h)P(h)/P(D)

P(h) = prior probability of hypothesis h
P(D) = prior probability of training data D
P(h|D) = probability of h given D (posterior density)
P(D|h) = probability of D given h (likelihood of D given h)

An Example

Does patient have cancer or not ?
A patient takes a lab test and the result comes back positive. The test returns a correct positive result in only 98% of the cases in which the disease is actually present, and a correct negative result in only 97% of the cases in which disease is not present. Furthermore, .008 of the entire population have this cancer.

Maximum A Posteriori (MAP) Hypothesis

P(h|D) = P(D|h)P(h)/P(D)

The Goal of Bayesian Learning: the most probable hypothesis given the training data (Maximum A Posteriori hypothesis)

Compute ML Hypo

Bayes Optimal Classifier

Question: Given new instance x, what is its most probable classification?
hMAP (x) is not the most probable classification!
Example: Let P(h1|D) = .4,
P(h2|D) = .3,
P(h3|D) = .3
Given new data x, we have h1(x)=+, h2(x) = -, h3 = -
What is the most probable classification of x?

Bayes optimal classification: