Bayesian Learning in Machine Learning

Probability for Learning

Probability for classification and modeling concepts.
Bayesian probability
 - Notion of probability interpreted as partial belief
Bayesian Estimation
 - It calculate the validity of a proposition
           - Based on prior estimate of its probability
           - and New relevant evidence

Bayes Theorem 

Goal: To determine the most probable hypothesis, given the data D plus any initial knowledge about the prior probabilities of the various hypotheses in H.

Bayes Rule :
                      P(h|D) = P(D|h)P(h)/P(D)

P(h) = prior probability of hypothesis h
P(D) = prior probability of training data D
P(h|D) = probability of h given D (posterior density)
P(D|h) = probability of D given h (likelihood of D given h)


An Example

Does patient have cancer or not ?
  A patient takes a lab test and the result comes back positive. The test returns a correct positive result in only 98% of the cases in which the disease is actually present, and a correct negative result in only 97%  of the cases in which disease is not present. Furthermore, .008 of the entire population have this cancer.

Maximum A Posteriori (MAP) Hypothesis

P(h|D) = P(D|h)P(h)/P(D)

The Goal of Bayesian Learning: the most probable hypothesis given the training data (Maximum A Posteriori hypothesis)

Compute ML Hypo

Bayes Optimal Classifier

Question: Given new instance x, what is its most probable classification?
 hMAP (x) is not the most probable classification!
Example: Let P(h1|D) = .4,
                       P(h2|D) = .3,
                       P(h3|D) = .3
 Given new data x, we have h1(x)=+, h2(x) = -, h3 = -
 What is the most probable classification of x?

Bayes optimal classification:

Where V is the set of all the values a classification can take and vj is one possible such classification.
Example: