Presenting my research at the University of Michigan’s Summer Research Opportunity Symposium

Author: Johnathan Ivy | Major: Data Science and Mathematics | Semester: Fall 2024

Hello, my name is Jonathan Ivey. I am a data science and mathematics major, and in the spring and fall semesters of 2024, I have been working with Dr. Susan Gauch in the Computer Science Department on differentiating diverse perspectives in training data for AI models.

Often when developing AI models for specific tasks (e.g., determining what emotions are present in a text), we use human annotators to label the data. Then, we use that data to train models, and ultimately, those annotations determine what ground truth is. In this process, annotators often give different labels, and traditional methods treat these disagreements as noise to be removed from the data. However, there are many cases where annotators have a valid disagreement (e.g., whether a tweet is offensive), and by removing that we are ignoring alternative points of view. In my project, I assess the weaknesses of these current noise removal techniques and develop a new method that preserves annotator disagreement.

I originally started this project in the fall while working in Dr. Susan Gauch’s lab. At the time it had a stronger focus on labeling the reasons for disagreements, but during the summer, I participated in the Summer Research Opportunity Program at the University of Michigan where I worked with Dr. David Jurgens. While working there, we saw how my topic could be more focused and impactful by working on methods for improving training data across many tasks.

After clarifying the research goals, I began conducting experiments with hierarchical Bayesian modeling. I generated synthetic data to measure how noise reduction affects disagreement in different scenarios, and I found that current methods actively suppress minority dissent from ground truth estimates. To address this, I needed to create new Bayesian methods, which was especially challenging because I had never done this type of modeling before. However, through self-study and discussions of statistical theory with my lab mates, I was able to grow my skill set and develop a solution.

When I returned to the University of Arkansas in the fall, I understood the theory of the problem well, but there were many technical challenges during implementation. It is especially important that my models scale to large datasets so other researchers and applications can use my work. While working on these challenges, I met with Dr. Gauch and Dr. Jurgens regularly and they were able to give me both practical and philosophical advice on how to solve these problems. I was also able to begin writing the paper for this project, which is important because the way you communicate research often determines its ultimate impact.

Now that the semester is over, I will continue to work on additional experiments, evaluation, and writing to ensure that my work is thorough and useful for other researchers. This project will develop through the beginning of the spring semester, and we hope to have it completed in February so it can be submitted for peer review and eventually publication. In future projects, I am interested in continuing to research ways to enable diverse perspectives in language models, and I hope that my work can be used to support nuance in AI.