If you need to add another course to your spring semester schedule, consider enrolling in CSCE 496/896 Section 009: Information Theory.
CSCE 496/896 Section 009: Information Theory
Time: MWF 11:30 a.m.-12:20 p.m.
Location: LPH 103
Instructor:
Dr. Massimiliano Pierobon
Assistant Professor
104 Schorr Center
Department of Computer Science and Engineering University of Nebraska-Lincoln
Lincoln, NE 68508
Tel: (402) 472-5021
Fax: (402) 472-7767
E-mail: pierobon@cse.unl.edu
Office Hours: TBD or by appointment
Description: This course deals with the foundations of information theory, as well as the more practical aspects of information coding. Information measures are first introduced, and then applied to the analysis of the theoretical performance achievable in data compression and propagation over noisy channels. The goal of the course is to teach students the mathematical basis of information manipulation and how concepts related to the source and channel coding are used to model, analyze and design modern computing and communication systems in order to enable efficient information processing. Another goal is to establish concrete links of these concepts with advanced technologies used to process the information in different systems (audio, video, biometrics, wireless, optical, molecular, and quantum communication/computing). Some of the topics will be presented using a more practical approach by means of examples built using commercial software tools.
Prerequisites: A grade of "P" or "C" or better in CSCE 310, CSCE 310H, CSCE 311, SOFT 260, SOFT 260H or RAIK 283H; STAT 380, ECEN 305 or RAIK 270H. Completing CSCE 462/862, CSCE 465/865, and MATH 817 prior to taking this course is recommended. Exceptions can be granted on a per-student basis by the instructor.
Required Textbooks:
· Stefan M. Moser, Information Theory, Lecture Notes, 6th Edition, Signal and Information Processing Lab ETH Zürich Zurich, Switzerland, 2018
· David J. C. MacKay, Information Theory, Inference, and Learning Course Topics
Algorithms, Cambridge University Press, 2003
Selected lectures of this course will be based on the following additional
textbooks (not required):
· M. Cover, J. A. Thomas, Elements of information theory (1st or 2nd edition), Editore: John Wiley & Sons, 1991 (1st ed.), 2006 (2nd ed.).
· R. G. Gallager, Information Theory and Reliable Communication, Editore: John Wiley & Sons, 1968
Lecture notes (PDF) will be available through the course’s homepage in Canvas.
A list of reference books and research papers will be given throughout the semester. HOMEWORKS and EXAMS will be based on what explained during the lectures and supplemental reading materials.
Course Topics:
1. Entropy and source coding
a. Introduction to information theory. Entropy of a memoryless source. Coding of memoryless sources. Prefix codes. Kraft inequality. Huffman codes and Shannon codes. Source coding theorems (for memoryless sources). Joint entropy and conditional entropy. Chain rules. Entropy of sources with memory. Source coding theorem. Practical methods for source coding. Universal codes. Arithmetic coding. Lempel-Ziv coding.
2. Channel capacity
a. Channel models. Discrete channels. Mutual
information. Data processing inequality. Channel capacity. Coding of information for transmission on unreliable channels. Entropy, mutual information, and capacity for continuous channels. Gaussian AWGN channel. Channel coding theorem. Error exponent. Fano’s inequality. Converse of the channel coding theorem. Hints for practical channel codes.
3. Rate distortion theory
a. Rate-distortion function. Coding of discrete and continuous sources with a fidelity criterion. Vector quantization. Channel coding with a fidelity
criterion.
4. Network information theory
a. Another look at source coding. Slepian-Wolf source coding. Multiple-access channels. Gaussian multiple-access channel. Gaussian broadcast channel. Capacity regions.
Course Organization: The course is lecture-based. Participation to lectures is mandatory. There will be TWO exams, FOUR homeworks. Optional projects, proposed during the course, can be selected to account for up to 10 points bonus on the final grade.
Grade Distribution
Homeworks: 35%
Exam 1 (OPEN NOTES): 30%
Exam 2 (OPEN NOTES): 30%
In-class Participation: 5%
Project: <= 10%
Final letter grades will be assigned tentatively based on the following scale:
A+: ≥ 100
A: 97% to 100%
A−: 94% to 96%
B+: 90% to 93%
B: 87% to 89%
B−: 84% to 86%
C+: 80% to 83%
C: 77% to 79%
C−: 74% to 76%
D+: 70% to 73%
D: 67% to 69%
D−: 64% to 66%
F: ≤ 63%
Homeworks: Homework submissions will be through web hand-in. Late homework is penalized 10% per day, and no homework will be accepted after the solution is posted online.
Exams: There will be TWO in-class exams.
Project: There will be half-semester-long projects, focused on the in- depth research of articles and other materials on a cutting-edge topic related to the course. The project should be executed through a review-style paper and an oral presentation at the end of the course. The presentation will be performed in front of the class within the dead week, and it will be followed by technical questions from the instructors (oral exam).
4xx Vs. 8xx: All homework assignments, quizzes, exams, etc. must be your own work. No direct collaboration with fellow students, past or current, is allowed unless otherwise stated. The Computer Science & Engineering department has an Academic Integrity Policy: http://cse.unl.edu/ugrad/resources/academic_integrity.php
All students enrolled in any computer science course are bound by this policy. You are expected to read, understand, and follow this policy. Violations will be dealt with on a case by case basis and may result in a failing assignment or a failing grade for the course itself.
Students with Disabilities: Students with disabilities are encouraged to contact the instructor for a confidential discussion of their individual needs for academic accommodation. It is the policy of the University of Nebraska-Lincoln to provide flexible and individualized accommodations to students with documented disabilities that may affect their ability to fully participate in course activities or to meet course requirements. To receive accommodation services, students must be registered with the Services for Students with Disabilities (SSD) office, 232 Canfield Administration, 472-3787 voice or TTY.
Suggestion Box: The CSE Department has an anonymous suggestion box (http://cse.unl.edu/department/suggestion.php) that you may use to voice your concerns about any problems in the course or department if you do not wish to be identified.
Stay Up-to-Date: It is CSE Department policy that all students in CSE courses are expected to regularly check their email so they do not miss important announcements.
CSE Resource Student Center: The CSE Student Resource Center (Avery Hall Rm 12) is intended to provide UNL Computer Science and Computer Engineering majors who are new to the program with a set of resources that will help them assimilate to college life and encourage them to continue their study of Computer Science and Computer Engineering (http://cse.unl.edu/src).
This syllabus will be updated and expanded as the semester progresses.