Information Theory

• Information Theory


  Precisely state the limits for compression and transmission of information
• Mathematical Model for Information Source

  • Any information source produces an output that is random
  • The source output is characterized in statistical terms
• Memoryless


  • Output sequence of source is statistically independent of all past and future outputs
  • If source is discrete, it is called discrete memoryless source (DMS)
• Stationary


  • Output of discrete source is statistically dependent
  • Example is the English text
• Self-information, Uncertainty


  Shannon pursued the idea of defining the information content h(E) of an event E as a function which depends solely on the probability P{E}
• Axioms for self-information, uncertainty

  • h(E) must be a decreasing factor of P{E}: more an event is likely, the less information its occurrence brings to us
  • h(E) = 0 if P{E} = 1, since if we are certain that E will occur, we get no information from its outcome
  • h(EՈF) = h(E) + h(F) if E and F are independent
• Logarithmic Function for the 3 Axioms

  h(E) = -logn P{E}
• Logarithmic base


  • 2 (bit or Sh (Shannon))
  • e (natural unit or nepers)
  • 3 (trit)
  • 10 (decimal digit)
• Conditional self-information, uncertainty


  h(E/F) = -logn P{E/F} where P{E/F} = P{EՈF}/P{F}
• Relationship between h(EՈF), h(E/F) and h(F)

  h(EՈF) = h(E/F) + h(F)
• Mutual information i(E;F)


  i(E;F) = h(E) - h(E/F) = h(E) + h(F) - h(EՈF)
• Entropy H(x)


  Average uncertainty in the outcomes x = x1, x = x2, ..., x = xn: H(x) = Σ Pi log Pi
• Coding


  Process of converting quantized intervals assigned to individual PAM samples into binary signals
• Coding Efficiency (Ceff)


  Ratio of the minimum number of bits required to achieve a certain dynamic range to the actual PCM bits used
• Idle Channel Noise


  Random, thermal noise present only at the input of the PAM sampler when there is no analog input signal
• Midtread Quantization


  First quantization interval is made larger in amplitude than the rest of the steps to reduce idle channel noise
• Midrise Quantization


  Lowest magnitude positive and negative codes have the same voltage range as all the other codes
• Coding Methods


  • Level-at-a-Time Coding
  • Digit-at-a-Time Coding
  • Word-at-a-Time Coding
• Codec (Coder-Decoder)


  Single integrated-circuit chip that performs every function of a PCM encoder and decoder
• Vocoders (Voice Encoder-Decoder)


  Used to digitize speech signals by encoding the minimum amount of speech information necessary to reproduce a perceptible message with fewer bits
• Vocoding Techniques


  • Channel Vocoder
  • Formant Vocoder
  • Linear Predictive Coder
• PCM Line Speed


  Rate at which serial PCM bits are clocked out of the transmitter onto the transmission line, dependent on the sampling rate and the number of bits in the compressed PCM code