E-Valuating E-Reference: Transforming Digital Reference Through Research and Evaluation
Presenters: Marie Radford, Lynn Connaway, Jeffrey Pomerantz, Lorri Mon and Joseph Janes
E-Valuating Virtual Viewpoints: User, Non-User and Librarians’ Perspectives on Live Chat-Based Reference
Marie Radford, Rutgers University
Lynn Connaway, OCLC
-Project funded by IMLS, Rutgers, State Lib of NJ and OCLC. Focus group interviews of libs, users & non-users of VR. Talking today about online surveys (critical incidents – focus on most memorable event). Also currently analyzing phone interviews. Used QuestionPoint. Create themes from the data.
-Libs & Users – think about an encounter that achieved positive results. Non – positive results in any type of reference interaction.
-Relational facilitators: interpersonal aspects that have a positive impact (enhanced communication). Relational barriers: interpersonal aspects that have a negative impact (impeded communication).
-Librarians: positive results. Demographics: female, 31-60, caucasian, majority in academic libs, next public libs, mainly urban & suburban. 60% described positive encounters had positive relational aspect and got desired content; 38% positive content.
-Lib Positive results: relational themes: 49& attitude of librarian & user; 23% relationship quality.
-Lib positive result for content: 85% said providing content was a positive result. 35% said instruction; 10% demonstrating subject knowledge.
-Lib negative results – primarily relational 43%l. Poor rapport, something didn’t work, poor communication skills. 54% attitude problem. Saw transcripts where lib mirrored poor attitude of patron. Better results when lib tried to keep a good attitude. Also lack of info, knowledge and time made for a neg result.
-Users mostly female, caucasian, suburbab, 12-65 fairly evenly distributed w/ exception of outliers.
-Users pos result: 61% primarily content; 26% relational & content; 13% relational.
-Users content themes: 71% providing info desired. Anecdotal: quotes also mentioned speedy.
-Users relational: 28% attitude of librarian.
-Users negative: lack of info 71%, lack of knowledge 12%. Users content negative: 68% did not get what they needed.
-Users relationship quality more important (25%) than attitude (19%).
-Nonusers again mostly caucasian females but mainly 28 and under (college age). Why aren’t they using it? Not non-users of library, just of chat.
-NonUsers positive result would be 49% info, 23% instruction.
-They found people are willing to wait longer for a subject specialist.
-NonUsers positive relational: [crap missed the slide]
-NonUser negative: information 60% [Now she is running out of time going real fast] attitude was paramount for relational aspect.
-Implications: librarians value delivering accurate answers, polite & interested users. Find rude/impatient users disruptive. Number of disruptive/rude users is decreasing. Users/NonUsers value accuracy of answers/info; younger users don’t want instruction they want the content. Value pos attitude & good comm skills from librarian.
-Recommendations: provide info & ask “wanna know how I found that?” Provide variety of formats for reference. NonUsers primary reason for non-use — no idea it existed, a few unsure if it was safe, security risk.
-Slides on Seeking Synchronicity website. (OCLC)
Jeff Pomerantz: I Have No Slides
University of North Carolina at Chapel Hill
-Guerrilla reference – Slam the Boards. Get libs active in online answer boards (Yahoo! Answers, etc.) 10th of every month is slam the boards day. Happening since 9/07. Most participation is on Yahoo! Answers (as of last study he saw had 96% market share for public access answer boards).
-Yahoo! Answers (Y!A) – simple text-box interface for your question, you self-categorize your question and it goes into a subject-specific queue. Anyone at all can answer. No vetting process (just need an account). Another simple text-box and a second optional box for sources. (People refer to selves & spouses a lot.)
-Piggybacked on data collection of some of his grad students. They have over a years’ worth of data from Yahoo! Answers (provides an API to download ?/answer records.) [Interesting – we’re obsessed with privacy of questions but Yahoo! is letting anyone download the data.]
-What was extent of librarian involvement on boards for Slam the Boards days? Were librarian answers voted as best answer more often than rest of population? (?er can vote an answer as best one.)
-Questions answered by librarians averaged 11 answers/question. (Pretty normal.) Questioner can rank 1-5 quality of answer (5 highest). StB librarians answers were voted best from 9% – 70% of the time. (70% of a particular librarians’ answers voted best.) Better than norm across Y!A.
-Extent of librarian involvement in StB is “microscopically low.” Number of ?s asked to Y!A each day is about 100,000. Is StB the way to go? We can’t even come close to touching the body of ?s being asked – huge information need. Do we want to insert ourselves into these forums or is it too much to bite off?
-Methodological problems – next to impossible to identify librarians on Y!A. User accounts have a profile associated but is all optional. Mostly all you have for user info is username. Probably missed a lot of librarians because they did not self-identify; so my numbers are probably lowball estimates.
-Flies in the face of the kind of reference eval we usually do (can definitely tell librarian answer vs outsourced services). Jessamyn has a script that can ID self-ID’d librarians in Ask Metafilter.
-Y!A has so many Q&As submitted that it’s impossible to collect them all. Not a problem we are used to facing in typical library evals. Y!A API limits 5,000 records per IP per day.
-Big infrastructure & technical issues to resolve before we can really evaluate this kind of thing. [Do we need to evaluate it per se, though.]
-How do we guide people to the right place to get their ? answered int he first place? Lots of Y!A quesions are ridiculous. Easily answerable from opinion questions to things that are more jokes between friends. Serious ?s get lost in the shuffle. What can we do to direct users to appropriate services before they get to where they want to ask? Mechanisms to embed within people’s searches to direct the to the right place to ask.
-Looking at StB has raised methodological issues. Need to address this if we are going to claim to serve info needs in the wild.
Lorri Mon: The Internet Public Library: Implications for Virtual Reference Evaluation from a Digital Library Learning Community
Florida State University
-Research within IPL and some related outside the IPL. Users, trends, practices, in LIS education. Talking about stuff from alot of these studies today. Do some training of students “learning community.”
-Community blogging as e-ref? (06 article in ref services review Pomerantz 34(2)) Drexel & Florida State students required to post weekly; not required to comment but wound up doing a lot in the space.
-Students are doing diff types of 1-to-many and many-to-many patterns. Q&A patterns: blog on experience; comments saying it was helpful (answered a ? on what to expect, etc.). Straight answering: post has a ?; students comment with answers. Community of users can answer questions for themselves pretty well and lift burden for facilitators to focus on tough questions.
-Another type of comment – reassurance. Filled a deep-seated info need. “Felt great relief.”
-Are boards like Y!A serving these types of needs better than library e-ref services?
-Another study at Florida State – geography of virtual ?s. Info from statewide service in 8/06. GIS mapping to see how people approach service from portals at libraries; compare to where users were actually logging in from based on IP. How did it work when people were asking library-specific ?s like borrowing, services, etc?
-Mostly service was entered through library in same county as their IP. Nearly half of ?s were on local services, etc.
-Not 100% accuracy rate for converting IP to lat/lon for GIS. (Large ISPs cause the problem – buy blocks of IP that users are routed through.) Also get a data point from users if you are doing this.
-Statewide services are doing more to authenticate via geography. Issue for assessment?
-Surveys and interviews with libraries nationwide to find out how they integrated lib 2.0 / web 2.0 stuff. (Respondents — skewed towards those that have technologies. A few would reply to say we don’t have it, here’s why.)
-Social networking 44% of PLs; 90% were MySpace.
-Blogging was most common tech that PLs reported. 153/242 (60+%) had blogs. 91 had wikis (lots behind the scenes – staff).
-Blurring of tech lines between e-ref techs. Meebo widgets; ref via SMS; Meebo widget embedded in FB page. Chat ref in Second Life.
-Libraries where social networking page was all they had for a website. How does this impact e-ref assessment?
-What’s happening in LIS education re: all this stuff? Looked at publicly available syllabi for ref services or digital libraries. 35 had e-ref listed as focus (chat & email), mention of IM in 8, 2 mentioned virtual worlds. Most common way to teach was indirect learning (read article, do a paper, guest speaker, etc). 11 listed direct experiences for students; 6 had role-playing. 31/40 had e-ref readings.
Response from Joe Janes
University of Washington
-Come & listen & react in the moment.
-People aren’t looking to make friends on e-ref – the affective is not that important. Generational issue? Yet first research showed attitude mattered.
-Customer service – users want services to be nice, helpful. People want a professional. Matters that you’re nice, but it’s not what they’re looking for.
-Good news, implies that they see us as professionals.
-If we really felt like it, we could make a dent in Y!A ?s. We could maybe do a good job with all the actual ?s in Y!A.
-Library reference services are a niche enterprise. Most useful niche for the effort is a local community. “Best librarianship is local.”
-Incredibly interesting & telling that Google Answers evaporated. Yahoo! has paid more attention to community – more Yahoo! people than Google people in terms of community-based interpersonal things.
-JP: Putting a monetary value on your questions didn’t work. Community effort is the key.
-Out of the queries that go to search engines, very small number wind up in Y!A, and many of those are junk questions. StB effort is somewhat misplaced but nowhere else to place it.
-Question answering software is coming. When? So far not impressive. Pops up and goes away. Difficult.
-Increasing virtuality of users. If the local is important, what happens when the local is going away? Virtual worlds, distance ed, etc. Where do we fit now? Libraries will be physical for a long time, question of where support comes from vs where services are going.
-Give people a tool and they will interact with it. Don’t care what the tool is – Meebo widget, IM, VR, Second Life, etc.
-Overwhelmingly large mass of people with tons of info needs. Increasingly “mushy.” Small cadre of people who want to do something about that. How to fit together – geographic or subject or affinity thing to grab onto.
Q: Looking for Yahoo! directory on Kindle. Browser is primarily text – page said ‘please get a more modern browser’ but still gave her directory.
JJ: Lots of work to get to it last time he wanted it. Nostalgia.
Q: Talking about librarians as people, but really you’re researching communication.
MR: One of the reasons we talked to non-users. Easier to find than users (privacy issues). Always talk to ourselves to evaluate our services. Problem.
LM: I think people are looking for their personal librarian. The one-to-one. Like your hairstylist, mechanic, etc. In Second Life – embedded librarians in communities. People are looking for this even online – want someone who they know and can go back to.
LC: Looked at how academics got info. They want their personal librarian. Undergrads referred to their lifeline – someone they knew & could call. Undergrads wanted to communicate with someone on the phone because they’re always texting – no idea we had telephone reference available. If faculty find one librarian who knows what they need in their subject area, they want to keep that.
LM: We deliberately try to stop that by random rotation of librarians at the desk.
MR: Willingness to return – wanting the same person. Familiarity trumped everything but eye contact – willing to wait if you are busy because you were nice before, you’ll be nice again. In chant, willing to return if they got good service.
JJ: Classical customer services. Don’t go back to a store if they’re mean to you.
MR: We found that young people want you there for a variety of modes – chat, phone, walk-in. Different types of service at different points in time. Variety of approaches.
LC: Sometimes younger people have personal questions & are more comfy asking virtually, so also context.
Q: Back to story told earlier about a teacher who had 17 students log into VR at once to teach them about netiquette & was annoyed by bad service. We’re not helping teachers know what we can do for their kids.
JJ: Not different now. Students coming in, everyone told to use same book, library not made aware.
LM: Touching into a rant of mine! We have Blackboard – more & more done through it, but going out to library is a different thing. Google Scholar will be eventually be embedded in Blackboard. Struck by how we are outside of that realm. At least an easy access point.
MR: Study found 30-50% of chatters at undergrad level were IN the building. Grad students were on-campus elsewhere. Don’t want to leave laptop unattended, leave public computer. Idea of merging and morphing, something that starts as a chat turns into a live interaction.
JJ: Library at his institute adopted single graphic identity for asking a question – on the website and flyers in the building. One big service, one big library. Successful to help orient people and reassure them that it’s the same people no matter whether you go in person, call, chat, etc.
LC: We talked to people who said that they didn’t know what the “ask a librarian” button meant. I guess there’s a librarian somewhere.MR: Doctoral student – saw that box but never clicked it.