In January 2024 interviews and usability testing were conducted with 11 Harvard students to understand their usage of generative AI for academic work, as well as to solicit feedback on the usability of Talk with HOLLIS. Talk with HOLLIS is an experimental generative AI chatbot that retrieves results from HOLLIS via natural language prompting.

Methodology & Participants

Through in-depth interviews, the team sought to learn:

• How do students use ChatGPT for academic work?
• Why do students use Google versus HOLLIS versus ChatGPT?
• How is the overall usability of Talk with HOLLIS (TWH)?
• What feature ideas do students have for future library chatbots?

Participants included five undergraduate students and six graduate students. All participants were familiar with HOLLIS. Some participants were expert users of ChatGPT, while others had never used it. They were asked questions about ChatGPT use and then tested Talk with HOLLIS.

Interview Insights on Student Use of HOLLIS & ChatGPT

• HOLLIS is a secondary step in the research process for students. They go to HOLLIS with known item searches or intentional search goals. Preliminary topic exploration, when they're at the idea stage, is done elsewhere, like Google or ChatGPT.
• Participants expect different functionality from a chatbot than they would from a search engine. They want to engage in conversation and think of ChatGPT as a partner rather than a source of information.
• ChatGPT is used for quick answers when the source or context isn’t as important to students.
• Students use ChatGPT as a learning partner, writing editor, translation tool, and to find “lost citations” but they don’t trust it for everything. They say that they have to “fact-check” the output.
• Most students know that ChatGPT won’t connect them to Harvard resources.

Harvard students on how they use ChatGPT for academic work

“I think ChatGPT is good at definitions. If I cared more about the contextual lens in which it's being defined I'd go to Google. But if I just want a simple definition, I would ask ChatGPT.” - Junior, History of Science & Global Health

“I'll ask ChatGPT to recommend search terms or ask it, what should I read? Like, what are the papers that I can read to understand this and then it won't tell me because it doesn't have access in some cases, but then it will give me a list of search terms or broader topics to understand.” - Junior, Bioengineering & Classics

“When I want something really specific, I'll use ChatGPT. I think it allows you to hone in more whereas Google will give you more breadth…I definitely have used it if something's more mainstream. Say there's a famous book and I want to know all the stuff that the author's written and similar titles or people that they've collaborated with. I've used it for that.” - Graduate School of Education

“I don't think I, I would use ChatGPT for finding sources directly. I'd probably go to HOLLIS for that. But maybe if it was something like tell me about the five biggest findings in reinforcement learning in the past decade…From there, it'll probably say this author wrote about this in this particular paper, and then I'd look for that particular paper.” - Junior, Neuroscience

“I do literature reviews with ChatGPT maybe, 3-4 times a week. I would say to it: Do you know if any major critics wrote on this specific edition of this specific work by Dante in the 16th century? They've fallen off their life cycle in terms of criticism. And then, I would get a list of editions from the 16th century or a summary. And then I can go find those secondary resources that have been lost to time in this cycle of popularity.”- PhD Student, Romance Languages