CSCE 496/896 Section 009: Information Theory
Online Course with Remote Access through Canvas
Synchronous Lectures from 01/25
MWF11:30PM – 12:20PM (Recorded and Posted on Canvas)
Zoom Link: TBD
Zoom Password: TBD
Instructor: Dr. Massimiliano Pierobon, Associate Professor
Department of Computer Science and Engineering
University of Nebraska-Lincoln
Lincoln, NE 68588
Office: 104 Schorr Center
Office Hours: @ same Zoom link as lectures on M W 2:30PM – 3:30PM or by appointment.
Tel: (402) 472-5021
Fax: (402) 472-7767
Web: http://cse.unl.edu/~pierobon/
E-mail: pierobon@cse.unl.edu
Office Hours: Monday, Wednesday 3:30PM – 4:30PM or by appointment.
Description: This course deals with the foundations of information theory, as well as the more practical aspects of information coding. Information measures are first introduced, and then applied to the analysis of the theoretical performance achievable in data compression and propagation over noisy channels. The goal of the course is to teach students the mathematical basis of information manipulation and how concepts related to the source and channel coding are used to model, analyze and design modern computing and communication systems in order to enable efficient information processing. Another goal is to establish concrete links of these concepts with advanced technologies used to process the information in different systems (audio, video, biometrics, wireless, optical, molecular, and quantum communication/computing). Some of the topics will be presented using a more practical approach by means of examples built using commercial software tools.
Prerequisites: A grade of "P" or "C" or better in CSCE 310, CSCE 310H, CSCE 311, SOFT 260, SOFT 260H or RAIK 283H; STAT 380, ECEN 305 or RAIK 270H. Completing CSCE 462/862, CSCE 465/865, and MATH 817 prior to taking this course is recommended but not required. Exceptions can be granted on a per-student basis by the instructor.
Required Textbooks:
• Stefan M. Moser, Information Theory, Lecture Notes, 6th Edition, Signal and Information Processing Lab ETH Zürich, Zurich, Switzerland, 2018
• David J. C. MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge University Press, 2003
Selected lectures of this course will be based on the following additional textbooks (not required):
• M. Cover, J. A. Thomas, Elements of information theory (1st or 2nd edition), Editore: John Wiley & Sons, 1991 (1st ed.), 2006 (2nd ed.).
• R. G. Gallager, Information Theory and Reliable Communication, Editore: John Wiley & Sons, 1968
A list of reference books and research papers will be given throughout the semester.
HOMEWORKS and EXAMS will be based on what explained during the lectures and supplemental reading materials.
Additional Materials: All the following additional materials will be available in Canvas:
• Lecture slides
• Additional reading resources
• Homework assignments and quizzes
• Programming assignments and software
Course Topics:
1. Entropy and source coding
a. Introduction to information theory. Entropy of a memoryless source. Coding of memoryless sources. Prefix codes. Kraft inequality. Huffman codes and Shannon codes. Source coding theorems (for memoryless sources). Joint entropy and conditional entropy. Chain rules. Entropy of sources with memory. Source coding theorem. Practical methods for source coding. Universal codes. Arithmetic coding. Lempel-Ziv coding.
2. Channel capacity
a. Channel models. Discrete channels. Mutual information. Data processing inequality. Channel capacity. Coding of information for transmission on unreliable channels. Entropy, mutual information, and capacity for continuous channels. Gaussian AWGN channel. Channel coding theorem. Error exponent. Fano’s inequality. Converse of the channel coding theorem. Hints for practical channel codes.
3. Rate distortion theory
a. Rate-distortion function. Coding of discrete and continuous sources with a fidelity criterion. Vector quantization. Channel coding with a fidelity criterion.
4. Network information theory
a. Another look at source coding. Slepian-Wolf source coding. Multiple-access channels. Gaussian multiple-access channel. Gaussian broadcast channel. Capacity regions.
Course Organization: The course is lecture-based. Lectures will be delivered through Zoom videoconferencing and recording of all the lectures is available in Canvas. There will be TWO exams, SIX homeworks. Optional projects, proposed during the course, can be selected to account for up to 10 points bonus on the final grade.
Communication and Remote Teaching Plan:
• Course official announcements and messages will be delivered through Canvas.
• Lectures are white-board based and will be delivered through Zoom videoconference with recording enabled and white-board functionality through iPad and Apple stylus pen. Audio and audio captions will be also generated and images of the white board will be saved and distributed along with the audio and caption files to avoid potential problem streaming videos.
• Homework assignments will be posted and submitted online through Canvas.
• Exams will be delivered online through Canvas timed quiz + Zoom-enabled view of participants and their computer screen monitored real-them by the instructor/proctors. Students are expected to have a broadband connection available at the time of the exam to enable Zoom proctoring.
• Office hours are held on Zoom by appointment (set via email), as well as through the Discussions functionality in Canvas.
Back-up Plan: To accommodate problems that could arise for the students to properly follow live lectures, the presence on Zoom during lecture time is not mandatory, while students are otherwise expected to access the online videos of the lecture, as well as download the white board pictures, both available after lecture time on Canvas.
In case of problems for the instructor to establish a proper Zoom videconference at the time of the lecture, the instructor will record lecture using VidGrid, and distribute the video through Canvas.
Alternative accommodation for the exams will be provided to students experiencing connection problems.
Assessment Plan:
• Homework Assignments:
Homework submissions will be through Canvas. Late homework is penalized 10% per day, and no homework will be accepted after the solution is posted online
• Exams:
There will be TWO exams that will be delivered delivered online through Canvas timed quiz + Zoom-enabled view of participants and their computer screen monitored real-them by the instructor/proctors. The exams will be OPEN NOTES: students are allowed to used class notes on your computer but are not allowed to use Internet (Google, etc.). The exam will include both open-ended, and multiple-choice questions.
• Project:
There will be half-semester-long projects, focused on the in-depth research of articles and other materials on a cutting-edge topic related to the course. The project should be executed through a review-style paper and an oral presentation (via Zoom or alternative means) at the end of the course. The presentation will be performed within the dead week, and it will be followed by technical questions from the instructors (oral exam).
Grade Distribution:
Homeworks: 35%
Exam 1 (OPEN NOTES): 30%
Exam 2 (OPEN NOTES): 30%
In-class Participation: 5%
Project: <= 10%
Final letter grades will be assigned tentatively based on the following scale:
A+: ≥ 100 A: 97% to 100% A−: 94% to 96%
B+: 90% to 93% B: 87% to 89% B−: 84% to 86%
C+: 80% to 83% C: 77% to 79% C−: 74% to 76%
D+: 70% to 73% D: 67% to 69% D−: 64% to 66%
F: ≤ 63%
4xx Vs. 8xx: This course will not have major differences between the 4xx and 8xx versions in the delivery of the content. Instead, some selected questions in the exams and lab assignments will be mandatory for 8xx students, and optional for 4xx students.
Attendance: Attendance at all officially scheduled class meetings in Zoom is expected but not mandatory. Students are responsible for knowing all materials discussed in class meetings. Changes to class and assignments will be announced on Canvas.
Academic Integrity: All homework assignments, quizzes, exams, etc. must be your own work. No direct collaboration with fellow students, past or current, is allowed unless otherwise stated. The Computer Science & Engineering department has an Academic Integrity Policy:
http://cse.unl.edu/ugrad/resources/academic_integrity.php
All students enrolled in any computer science course are bound by this policy. You are expected to read, understand, and follow this policy. Violations will be dealt with on a case by case basis and may result in a failing assignment or a failing grade for the course itself.
Dealing with Stress and Adversity:
UNL offers a variety of options to students to aid them in dealing with stress and adversity. Counseling and Psychological Services (CAPS) is a multidisciplinary team of psychologists and counselors that works collaboratively with Nebraska students to help them explore their feelings and thoughts and learn helpful ways to improve their mental, psychological and emotional well-being when issues arise. CAPS can be reached by calling 402-472-7450. Big Red Resilience & Well-Being provides fun events, innovative education, and dynamic services to help students understand emotions, manage stress, build strength, connect with others, develop grit and navigate transitions.
Students with Disabilities: Students with disabilities are encouraged to contact the instructor for a confidential discussion of their individual needs for academic accommodation. It is the policy of the University of Nebraska-Lincoln to provide flexible and individualized accommodations to students with documented disabilities that may affect their ability to fully participate in course activities or to meet course requirements. To receive accommodation services, students must be registered with the Services for Students with Disabilities (SSD) office, 232 Canfield Administration, 472-3787 voice or TTY.
Suggestion Box: The CSE Department has an anonymous suggestion box (http://cse.unl.edu/department/suggestion.php) that you may use to voice your concerns about any problems in the course or department if you do not wish to be identified.
Stay Up-to-Date: It is CSE Department policy that all students in CSE courses are expected to regularly check their email so they do not miss important announcements.
CSE Resource Student Center: The CSE Student Resource Center (Avery Hall Rm 12) is intended to provide UNL Computer Science and Computer Engineering majors who are new to the program with a set of resources that will help them assimilate to college life and encourage them to continue their study of Computer Science and Computer Engineering (http://cse.unl.edu/src).
This syllabus will be updated and expanded as the semester progresses.