MIStatlE
中文

← Knowledge map∑ Mathematical Foundations

Information Theory

4 concepts, 1 written; shares concepts or relations with 4 other subjects. Earlier ideas sit to the left, a region holds one chapter, and relations that leave the subject hang on the cards as tags.

123 earlierlater ∑Information Theory extends to applied in Shannon Entropy KL Divergence &Cross Entropy TypicalSet & AEP VAE & ELBO Law of Large Numbers → Typical Set & AEP · derives — Applying the law of large numbers to −log p(X) gives the AEP.Probability · Law of Large Num… → VAE & ELBO → Score Matching · extends to — From a likelihood bound for latent-variable models to objectives that learn the score directly.→ Generative Models · Score Matc… VAE & ELBO ↔ LeWorldModel / SIGReg · contrast — Both learn latent representations: the VAE reconstructs observations, LeWorldModel predicts only in latent space and relies on SIGReg against collapse.→ Deep Learning · LeWorldModel…
Relationsposition is order: read along the rail, left to righta tag on a card leads to another subject
NodesConceptTheoremModelwrittenplanned1 / 4 written

Select a concept to see its summary and relations1 / 4 written

Information Theory1/4

Select a node to see its summary and how it relates to the others.

Relations in this subject, in words6

  • Law of Large NumbersderivesTypical Set & AEP

    Applying the law of large numbers to −log p(X) gives the AEP.

  • Shannon EntropyrequiresTypical Set & AEP

    Entropy sets the size of the typical set, about 2^{nH}.

  • Shannon Entropyextends toKL Divergence & Cross Entropy

    KL divergence is relative entropy: entropy extended from one distribution to a pair.

  • KL Divergence & Cross Entropyapplied inVAE & ELBO

    The regularizer in the ELBO is a KL divergence.

  • VAE & ELBOextends toScore Matching

    From a likelihood bound for latent-variable models to objectives that learn the score directly.

  • VAE & ELBOcontrastLeWorldModel / SIGReg

    Both learn latent representations: the VAE reconstructs observations, LeWorldModel predicts only in latent space and relies on SIGReg against collapse.

Notes

Notes in this subject

Filter the notes page by this subject →
Information TheoryDerivation Typical Sets: Where Probability Mass Lives in High Dimensions Why does probability mass sit on a thin shell rather than at the point of highest density? Read →