Link
Organization: What AI engineers can learn from qualitative research methods in HCI
Context
Human-computer interaction (HCI) research studies how people use computers and how to design computer systems effectively.
When I saw Dr. Ian Arawjo’s article about applying HCI methods to AI development, I was curious about what AI Engineers could learn from this established field.
Dr. Arawjo uses an example from Hamel Husain, an AI/ML engineering consultant, to show how AI Engineers are often reinventing methods already existing in HCI research under different names.
Key Insight
Two important realizations from Dr. Arawjo’s article:
- HCI and UX research methods have been refined over decades of studying human-computer interactions
- Since AI systems are fundamentally computer systems, many HCI insights about human interaction directly apply to AI development
Dr. Arawjo then shares specific HCI research methods that could benefit AI development, particularly in LLMOps:
HCI & UX Qualitative Research Methods Useful for AI Engineers
- Qualitative Coding: the process of systematically categorizing excerpts in your qualitative data to find themes and patterns
- Inductive Coding: creating codes and themes from raw data without using predetermined categories or theories
- Theoretical Saturation: refers to the point when no new data can be found to develop a phenomenon further
- Grounded Theory: collecting and analyzing data to create theories
- Focused Coding: refining codes and categories in qualitative data analysis to create a final set that aligns with the research objectives
- Heuristic Evaluation: a method for identifying design problems in a user interface
- Interpretivism: understanding the subjective meanings and experiences of people in a social context
- Positivist: a school of thought, that believes that knowledge is objective and can be developed through reason and logic (which is often seen as the default mindset in engineering)
Though the above seems a bit light compared to the heavy math of building a working deep learning model and the infrastructure around building an AI system, HCI is a heavily researched area that has “seen things” and “been around.”
Given the apparent similarity, it’s worthwhile for AI Engineers to learn about HCI and UX methods to help understand what works (and doesn’t) when prototyping, building, studying, and developing systems that interact with humans.
LLMOps Connection
Dr. Arawjo makes a compelling case that LLM development should be treated as an interpretivist rather than positivist endeavor. This means:
- Accepting that “perfect” solutions may not exist
- Focusing on understanding patterns rather than eliminating all errors
- Using qualitative methods to improve system behavior iteratively
Strategic Implications
- Develop a basic understanding of HCI
- Map out how AI Systems and AI Engineering concepts link to HCI and UX concepts
- Learn to apply the best practices from HCI and UX research
Key Takeaways for AI Engineers
- Some of what you are struggling with may have already been studying in HCI and UX, just under different names
- If you learn the different names, you’ll be able to apply years of research to your project
- Learning from fields that deal with humans is always helpful, given AI Agents still interact with humans
Personal Notes
Having been deeply involved in the Data Visualization community, I’ve seen this pattern before.
When web-based data visualization took off, developers initially recreated many concepts that HCI research had already solved, simply because they weren’t aware of the existing work.
The data viz community eventually embraced HCI principles, leading to better tools and practices.
I see the same opportunity for AI Engineers today: a chance to build on decades of HCI research rather than reinventing these methods from scratch.
For those interested in learning more, I highly recommend Georgia Tech’s Online Master of Science in Computer Science (OMSCS) class OMS CS6750: Human-Computer Interaction.