Ph.D. Dissertation Defense: Balaji Balasubramaniam
Friday November 15
10:30 AM
211 Schorr Center
"Controller Software: Evolution, Identification, and Implementation"
In the annals of automation history and advancement, one can find control technology is at the core. Modern-day controllers rely heavily on software capability to provide stability and improve the system’s performance. In particular, drone flight controllers use autopilot control software to accomplish autonomous navigation from take-off to landing. However, we know very little about how the controller code modifications and its impact, particularly at the software level. No general framework has been developed to identify the control code changes and observe the real values of software control loops at the kernel layer. In this thesis, we lay the foundation of a general framework for studying, thinking about, implementing, and assessing control software as a discipline distinct from software engineering and control communities but informed by them. We argue that control software is sufficiently critical to warrant its own field of study focused on the safety- critical aspects of its realization. Here, we present 3 distinct pieces of that foundation by performing a first-of-its-kind study on the evolution of two popular open-source safety-critical autopilot control software – ArduPilot and Paparazzi, to better understand how controllers evolve and the space of potential mismatches between control design and their software implementation. We report on three major findings. First, control software evolves quickly and controllers are rewritten in their entirety, many times over through the controller’s lifetime, which implies that the design, synthesis, and implementation of controllers must support not just the initial baseline system but also their incremental evolution. Second, many software changes stem from an inherent mismatch between the continuous time/space physical model and its corresponding discrete software implementation, but also from the mishandling of exceptional conditions, and limitations and distinct data representation of the underlying computing architecture. Third, using our mutation tool that we developed, we show that small code changes can have a dramatic effect in a controller’s behavior, which implies that further support is needed to bridge these mismatches as carefully verified model properties may not necessarily translate to its software implementation. For the second piece of our foundation, we present an automated technique based on a novel application of machine learning to detect commits related to control software, its changes, and how the control software evolves. We identify the safety-critical control software, its boundaries with other embedded software in the system, and the way in which control software evolves could help developers isolate, test, and verify control implementation, and improve control software development. We leverage messages from developers (e.g., commit comments), and code changes themselves to understand how control software is refined, extended, and adapted over time. We examine three distinct, popular, real-world, safety-critical autopilots – ArduPilot, Paparazzi UAV, and LibrePilot to test our method demonstrating an effective detection rate of 0.95 for control-related code changes. Lastly, we present a fully-implemented runtime code manipulation system to improve the control software based on our studies, we monitor real values of software control loops at the kernel layer and implement an additional controller to enhance the system reliability that is not easy-to-reach by typical coders. Using software engineering techniques, we leverage dynamic analysis that works with real values during run-time execution of autopilot software to precisely monitor and modify control signals in control loops. Specifically, we propose an implementation of Dynamic instrumentation-based Real-time code manipulation by adding a System-near (DRS) controller in the autopilot system. We verify the DRS controller under normal and abnormal flight behavior using the Software In The Loop (SITL) simulation environment. Tests using six traditional metrics show an improved performance under abnormal behavior.
Committee:
Dr. Justin Bradley, Chair
Dr. Hamid Bagheri, Co-Chair
Dr. Benjamin Riggan
Dr. Witty Srisa-an
Ph.D. Dissertation Defense: Ji Young Lee
Friday, November 15
12 PM
211 Schorr Center and Zoom
Zoom link: https://unl.zoom.us/j/98957381254?pwd=ysfwO8muXwoNhpqvdYGZJjahCaSgBr.1
"Deep Learning for Vision-Based Bridge Inspection by Resource-Constrained Unmanned Aircraft Systems"
Bridge inspection is critical for ensuring structural integrity, extending the service life of infrastructure, and minimizing maintenance costs. As bridges age and endure increasing loads, regular inspections help detect early signs of wear, such as cracks or corrosion, that could impact safety and performance. However, traditional inspection methods are labor-intensive, requiring significant time, specialized equipment, and manual access to challenging areas, which can lead to costly disruptions. Additionally, reliance on human inspectors introduces subjectivity, with assessments varying by individual expertise. These factors highlight the inefficiencies and safety risks in current inspection practices, underscoring the need for more objective, efficient solutions.
In this work, we propose advanced computer vision techniques, leveraging deep learning, to streamline and enhance the bridge inspection process. The vision techniques are specifically designed to work onboard small, Group 1 Unmanned Aircraft Systems (UASs) and hence are SWaP (Size, Weight, and Power) constrained friendly. Specifically, we address challenges in detecting and assessing defects in both concrete and steel bridge components under real-world outdoor conditions. For concrete bridges, we construct robust datasets capturing transverse surface cracks and design a deep learning segmentation model for crack width measurement. For steel bridge inspection, we examined object detection algorithms focused on critical connection components like rivets and bolts, emphasizing computational efficiency and precision to deploy the model for SWaP-constrained devices. Additionally, a novel classification approach is presented to evaluate corrosion severity in steel structures, offering an automated solution for tracking progressive structural deterioration.
Through these contributions, this thesis advances the field of autonomous bridge inspection by integrating scalable deep learning methodologies into Group 1 UAS, providing a practical and transformative approach to structural health monitoring that supports safer, more cost-effective, and data-driven infrastructure maintenance.
Committee:
Dr. Justin Bradley, Co-chair
Dr. Chungwook Sim, Co-Chair
Dr. Carrick Detweiler
Dr. Armin Roohi
Dr. Daniel Linzell