Evidence Lower Bound (ELBO) Visualization
This visualization demonstrates the Evidence Lower Bound (ELBO) in variational inference. It extends the Bayesian visualization by adding a variational distribution Q and showing how ELBO relates to the marginal likelihood.
Marginal/Evidence, P(E)
What is my expectation to see a jumping thing?
Posterior, P(H|E)
Given that I see a jumping thing, what is the probability it's a frog?
Variational Distribution Q(H)
Variational Free Energy F[Q,y] (negative ELBO)
-Energy
-EQ(x)[ln P(y,x)]
Entropy
H[Q(x)]
F[Q,y]
Complexity
DKL[Q(x) || P(x)]
Accuracy
EQ(x)[ln P(y|x)]
F[Q,y]
Divergence
DKL[Q(x) || P(x|y)]
Evidence
ln P(y)
F[Q,y]
Green boxes are tractable and can be directly computed.Red boxes are intractable in the general case.
-F[Q,y] = ELBO = Evidence Lower Bound.
The third formulation shows that ELBO ≤ Evidence (hence the name). Both are negative, so |ELBO| ≥ |Evidence|.
Drag the dividing lines to adjust probabilities and the slider to adjust Q. See how they affect the ELBO and its relationship to the marginal likelihood.