ECE 563 - Information Theory (Fall 2018)

Lecturer: Lav Varshney (office hours, Friday 9:30-11:00am, 314 CSL and by appointment)

Teaching Assistants: Ravi Kiran Raman (office hours, Tuesday 4:00-5:00pm, 3036 ECE) and Sam Spencer (office hours, Thursday 2:30-3:30pm, 114 CSL)

Lectures: Tuesday and Thursday, 12:30pm, 2015 Electrical and Computer Engineering Building

Problem Solving Sessions: Friday, 2:00pm, 141 Coordinated Science Laboratory [optional]

Course Goals

Catalog Description

Mathematical models for channels and sources; entropy, information, data compression, channel capacity, Shannon's theorems, and rate-distortion theory.

Prerequisites: Solid background in probability (ECE 534, MATH 464, or MATH 564).

Textbook: T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006.

Grading: Homework [including programming assignments] (25%), Midterm exam [in class] (25%), Final exam [as designated by university] (25%), Group juxtaposition paper [in groups of three, in roughly Allerton format] (25%)

Syllabus, Syllabus Attachment


Homework

Problem Solving Sessions

Exams

Juxtaposition Paper

Course Schedule

Date Topic Reading Assignment Learning Objectives Multimedia Supplements
8/28

1. The problem of communication, information theory beyond communication

[slides]

  • Chapter 1 (Introduction and Preview) of Cover & Thomas
8/30

2. The idea of error-control coding and linear codes

[slides] [handwritten]

  • Chapter 7.11 (Hamming Codes) of Cover & Thomas
9/4 3. Information measures and their axiomatic derivation
  4. Basic inequalities with information measures
  • Chapter 2.4-2.10 (Entropy, Relative Entropy and Mutual Information) of Cover & Thomas
 
9/11 5. Asymptotic Equipartition Property
  • Chapter 3.1 of Cover & Thomas
9/13 6. Source Coding Theorem
  • Chapter 3.2 of Cover & Thomas
  • Chapter 5.2 of Yeung, if you'd like
9/18 7. Variable-length Codes
  • Chapter 5 of Cover & Thomas
9/20 8. Entropy Rate of Stochastic Processes
  • Chapter 4 of Cover & Thomas
 
9/25 9. Distributed Source Coding
  • Chapter 15.4 of Cover & Thomas
 
9/27 10. Universal Source Coding
  • Chapter 13 of Cover & Thomas
10/2 11. Method of Types
  • Chapter 11.1-11.3 of Cover & Thomas
 
10/4 12. Allerton Conference [no lecture]    
10/9 13. Hypothesis Testing
  • Chapter 11.7-11.10 of Cover & Thomas
 
10/11 14. Channel Coding Theorem: Converse and Joint AEP
  • Chapter 7.9 and 7.6 of Cover & Thomas
10/16 15. Channel Coding Theorem: Achievability and Examples
  • Chapter 7.7 and 7.1 of Cover & Thomas
 
10/18 16. Midterm [no lecture]      
10/23 17. Source-Channel Separation
  • Chapter 7.13 of Cover & Thomas (and e.g. Gastpar et al., 2003)
 
10/25 18. Differential Entropy, Maximum Entropy, and Capacity of Real-Valued Channels
  • Chapter 8, 9, and 12 of Cover & Thomas
 
10/30 19. Rate-Distortion Theorem: Converse and Examples
  • Chapter 10 of Cover & Thomas
11/1 20. Rate-Distortion Theorem: Achievability and More Examples
  • Chapter 10 of Cover & Thomas (and Chapter 9 of Yeung)
 
11/6 21. Quantization Theory  
11/8 22. Blahut-Arimoto 
  • Chapter 10.8 of Cover & Thomas (and Chapter 10 of Yeung)
 
11/13 23. Strong Data Processing Inequalities  
11/15 24. Large Deviations
  • Chapter 11.4-11.5 of Cover & Thomas
 
11/27 25. Error Exponents for Channel Coding    

Topics: