CSCE 492/892 - Section 009: Information Theory
Spring 2022
Lecture Time: MWF 11:30AM – 12:20PM
Location: Avery Hall 119
Instructor:
Dr. Massimiliano Pierobon
Associate Professor
School of Computing
College of Engineering
University of Nebraska-Lincoln
Lincoln, NE 68588
Contact Info
Office: 104 Schorr Center
Tel: (402) 472-5021
Fax: (402) 472-7767
Web: http://cse.unl.edu/~pierobon/
E-mail: pierobon@cse.unl.edu
Office Hours:
Monday, Wednesday 1:30–2:30 PM or by appointment.
Face Coverings:
Given the current transmission level of COVID-19 in our community, UNL will temporarily require face coverings indoors, including at UNL events, until further notice.
Disposable masks are available for students at no cost at the following locations:
• City Campus – a self-service wall dispenser in the hall by the west doors of the Nebraska Union or the Nebraska Union Welcome Desk located on the first floor near the north entrance
• East Campus – the Nebraska East Union Welcome Desk located on the first floor near the north entrance
• Innovation Campus – at Food Innovation Center, 1901 N 21st Street, Room 232
• Peter Kiewit Institute – at the reception desk in Room 107
Description: This course deals with the foundations of information theory, as well as the more practical aspects of information coding. Information measures are first introduced, and then applied to the analysis of the theoretical performance achievable in data compression and propagation over noisy channels. The goal of the course is to teach students the mathematical basis of information manipulation and how concepts related to the source and channel coding are used to model, analyze and design modern computing and communication systems in order to enable efficient information processing. Another goal is to establish concrete links of these concepts with advanced technologies used to process the information in different systems (audio, video, biometrics, wireless, optical, molecular, and quantum communication/computing). Some of the topics will be presented using a more practical approach by means of examples built using commercial software tools.
Prerequisites: A grade of "P" or "C" or better in CSCE 310, CSCE 310H, CSCE 311, SOFT 260, SOFT 260H or RAIK 283H; STAT 380, ECEN 305 or RAIK 270H. Completing CSCE 462/862, CSCE 465/865, and MATH 817 prior to taking this course is recommended but not required. Exceptions can be granted on a per-student basis by the instructor.
Required Textbooks:
Stefan M. Moser, Information Theory, Lecture Notes,
6th Edition, Signal and Information Processing Lab ETH Zürich
Zurich, Switzerland, 2018
David J. C. MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge University Press, 2003
Selected lectures of this course will be based on the following additional textbooks (not required):
M. Cover, J. A. Thomas, Elements of information theory (1st or 2nd edition), Editore: John Wiley & Sons, 1991 (1st ed.), 2006 (2nd ed.).
R. G. Gallager, Information Theory and Reliable Communication, Editore: John Wiley & Sons, 1968
A list of reference books and research papers will be given throughout the semester.
HOMEWORKS and EXAMS will be based on what explained during the lectures and supplemental reading materials.
Additional Materials:
All the following additional materials will be available in Canvas:
• Lecture slides
• Additional reading resources
• Homework assignments and quizzes
• Programming assignments and software
Course Topics:
1. Entropy and source coding
a. Introduction to information theory. Entropy of a memoryless source. Coding of memoryless sources. Prefix codes. Kraft inequality. Huffman codes and Shannon codes. Source coding theorems (for memoryless sources). Joint entropy and conditional entropy. Chain rules. Entropy of sources with memory. Source coding theorem. Practical methods for source coding. Universal codes. Arithmetic coding. Lempel-Ziv coding.
2. Channel capacity
a. Channel models. Discrete channels. Mutual information. Data processing inequality. Channel capacity. Coding of information for transmission on unreliable channels. Entropy, mutual information, and capacity for continuous channels. Gaussian AWGN channel. Channel coding theorem. Error exponent. Fano’s inequality. Converse of the channel coding theorem. Hints for practical channel codes.
3. Rate distortion theory
a. Rate-distortion function. Coding of discrete and continuous sources with a fidelity criterion. Vector quantization. Channel coding with a fidelity criterion.
4. Network information theory
a. Another look at source coding. Slepian-Wolf source coding. Multiple-access channels. Gaussian multiple-access channel. Gaussian broadcast channel. Capacity regions.
Course Organization:
The course is lecture-based. Lectures will be in person/online (to be decided…) following the course schedule. There will be TWO exams and SIX homework assignments. Optional projects, proposed during the course, can be selected to account for up to 10 bonus points on the final grade.
Communication Plan:
Course official announcements and messages will be delivered through Canvas. Lectures are white-board-based. Images of the white board will be saved and distributed. Homework assignments will be posted and submitted online through Canvas.
Exams will be delivered online through Canvas timed quiz + Zoom-enabled view of participants and their computer screen monitored real-time by the instructor/proctors. Students are expected to have a broadband connection available at the time of the exam to enable Zoom proctoring.
Office hours are held on Zoom or in person by appointment (set via email), as well as through the Discussions functionality in Canvas.
Back-up Plan:
To accommodate problems that could arise for the students to properly follow live lectures, lecture recording may be provided on a per-request basis.
Alternative accommodation for the exams will be provided to students experiencing connection problems during exams.
Inclement Weather:
If in-person classes are canceled and the campus follows instructional continuity plans, alternative lecture plans will be communicated through Canvas announcements.
Assessment Plan:
Homework Assignments. Homework submissions will be through Canvas. Late homework is penalized 10% per day, and no homework will be accepted after the solution is posted online
Exams:
There will be TWO exams that will be delivered delivered online through Canvas timed quiz + Zoom-enabled view of participants and their computer screen monitored real-them by the instructor/proctors.
The exams will be OPEN NOTES: students are allowed to used class notes on your computer but are not allowed to use Internet (Google, etc.).
Project:
There will be half-semester-long optional projects, focused on the in-depth research of articles and other materials on a cutting-edge topic related to the course. The project should be executed through a review-style paper and an oral presentation (via Zoom or alternative means) at the end of the course. The presentation will be performed within the dead week, and it will be followed by technical questions from the instructors.
Grade Distribution:
Homeworks: 35%
Exam 1 (OPEN NOTES): 30%
Exam 2 (OPEN NOTES): 30%
In-class Participation: 5%
Project: <= 10%
Final letter grades will be assigned tentatively based on the following scale:
A+: ≥ 100 A: 97% to 100% A−: 94% to 96%
B+: 90% to 93% B: 87% to 89% B−: 84% to 86%
C+: 80% to 83% C: 77% to 79% C−: 74% to 76%
D+: 70% to 73% D: 67% to 69% D−: 64% to 66%
F: ≤ 63%
4xx Vs. 8xx:
This course will not have major differences between the 4xx and 8xx versions in the delivery of the content. Instead, some selected questions in the exams will be mandatory for 8xx students, and optional for 4xx students.
UNL Course Policies and Resources:
Students are responsible for knowing the university policies and resources found on this page (https://go.unl.edu/coursepolicies):
• University-wide Attendance Policy
• Academic Honesty Policy
• Services for Students with Disabilities
• Mental Health and Well-Being Resources
• Final Exam Schedule
• Fifteenth Week Policy
• Emergency Procedures
• Diversity & Inclusiveness
• Title IX Policy
• Other Relevant University-Wide Policies
Suggestion Box:
The School of Computing has an anonymous suggestion box (https://computing.unl.edu/anonymous-department-feedback-form) that you may use to voice your concerns about any problems in the course or department if you do not wish to be identified.
Stay up-to-date:
It is School of Computing policy that all students in SoC courses are expected to regularly check their email, so they do not miss important announcements.
CSE Resource Student Center:
The SoC Student Resource Center (Avery Hall Rm 12 and online via Canvas) is intended to provide UNL Computer Science and Computer Engineering majors who are new to the program with a set of resources that will help them assimilate to college life and encourage them to continue their study of Computer Science and Computer Engineering (https://computing.unl.edu/current-undergraduate#SRC)
This syllabus will be updated and expanded as the semester progresses.