The Research Footprint: Libraries Tracking and Enhancing Scholarly and Scientific Impact (http://ala13.ala.org/node/10128)
Jason Priem, Impact Story, UNC Royster Scholar, University of North Carolina at Chapel Hill, School of Information and Library Science @jasonpriem
- Altmetrics guy
- Managing the flow of research information is one of the most important parts of the research process
- Communication – we’ve progressed from letters between researchers to journals to the “DcJ” (huh?)
- “Online journals are paper journals delivered by faster horses.”
- Industrial Revolution was about homogeneity. Produce many of the same thing. Age of the web – cheap to publish anything.
- Scholarship starts with some kind of data – no longer staying in the file drawer, becoming a product that you publish.
- The revolution is when scholarship becomes a web-native product, not just posting reproductions of journal articles
- scholarship – can tell the story in a monograph or article. changing – now can use video, blogs, post notebooks, infographics, slides, etc etc
- previously, conversations disappeared. increasingly they happen on the web so they don’t disappear. website – mathoverflow where mathematicians post problems and discuss. twitter backchannel – conference conversations. scholarly communities are that are on twitter and just having those conference conversations all the time
- “Web-native science means we can start making public, not merely ‘publishing.'”
- posted a google doc of a prepublication article. doesn’t have patience to wait a year for his article to be published, got the publisher to agree to let him do this right away. Allows him to narrowcast his work to the people who care what he does. helped him improve it through their questions and criticism. runs into people who have read it already when he mentions it. wonders why bother publishing.
- How to measure usefulness/impact? Past – citations. That’s only part of the story though. Acknowledged that this is not a good metric of the article’s impact.
- Altmetrics measures a variety of impacts. how we’re going to mine impact on the next scholarly web.
- ImpactStory (http://impactstory.org/) – sharing altmetrics. open source nonprofit. shows scholarly and general public, tracks saves (mendeley, etc), cites,
- Altmetrics – faster evaluation, rewards broader impacts, rewards web-native products, builds web-native filters.
- readership networks – tracks what are people reading in the same reading sessions
- “at web scale, the value isn’t in manual curation” – Yahoo! learned this. needs to be automated – citation mining. (Google) “use the network to filter the network”
- second revolution has started – altmetrics are “too useful to ignore,” like citation tracking.
Cathy Sarli, Scholarly Communications Specialist, Becker Medical Library, Washington University School of Medicine
Kristi Holmes, Bioinformaticist, Becker Medical Library, Washington University School of Medicine
The Becker Model – https://becker.wustl.edu/impact-assessment
- Dr. Mae Gordon – researcher. 2007 looking for citation info on articles that came out of a big project (Ocular Hypertension Treatment Study) she PI’d from ’94-2010. found 6 ore publications that were highly cited. interesting, impressive. but – “so what?” what’s the reason? the story? indicative of anything? predictive? etc
- “It is no longer enough to measure what we can – we need to measure what matters.” – Wells & Whitworth, 2007. Aus & NZ Health Policy 4:14
- talked to a bunch of people. collected anecdotal info and used it to continue investigating. talked to investigators, colleagues, policy makers, support staff, etc etc – like genealogy
- used any database they could get their hands on. Google, Google Scholar to pick up grey lit (curriculum, continuing ed, trade pubs, etc). consulted with orgs and assns.
- found clinical guidelines, diagnostic criteria, measurement instruments, curriculum guidelines, iphone app, all kinds of things!
- Gordon flabbergasted – focused on the research and the articles, not all this other stuff.
- website includes a section with strategies for researchers to enhance impact of their work.
- general issues – time lag between research discovery and application. when to start this type of study? difficult to establish direct correlations. not a linear process. supporting documentation may not be available to be released to the public.
- project has helped them better understand users and what they’re producing. also how they can use the outputs to understand how the work is being disseminated and built upon – see how scientific process moves
- research impact matters. individually for P&T. group-level – are we fulfilling our goals? institutional level – what’s our place in the scholarly ecosystem?
- Kristi worked with a WashU bench researcher (anonymous for now). retroactive assessment with publication data (still important info!) did an interview with Becker Model checklist to dig into what the investigator was doing
- Research Center that’s applying to renew a grant. needed evidence of collaboration. looked at publication data and did coauthorship analysis. found a 30% increase in collaboration in the 4 years investigated. also saw doubling in geographic diversity of collaboration.
- moving forward – have ideas for how they can be proactive – checklists for investigators, will use some strategies Cathy talked about to enhance impact of research
- WashU Inst of Clinical and Translational Sciences – huge research awards. absolutely need better measures to track impact.
- ID papers and projects with cite data, info from funding reports, survey data and interviews with investigators. also encouraging proactivity – workshops and consultations on reporting ROI, enhancing research impact, NIH Public Access Policy
- value of the individual taking responsibility for curating their work.
- strategies for libraries – get to know your resources & users. why do people want this info? various service models you can try. identify various indicators that can be used – different disciplines are developing different ones. operationalize it – ontologies.
Rush G. Miller, The Hillman University Librarian, Director of the University Library System & Professor, University of Pittsburgh
- PlumX – ways to measure scholarly output
- why? library strategic goal to innovate in scholarly publishing. publish 60-70 OA journals, have major repositories for 5-6 disciplines.
- PlumX (Plum Analytics – http://www.plumanalytics.com/pr/plum-analytics-new-product-plumx-gives-researchers-an-edge.html)
- create profiles of researchers, mine various sources. looking at traditional and alt metrics.
- library did data entry, developed guidelines to standardize data, ID’d and coordinated with faculty for the pilot, found objects in their IRs and found all sorts of things to pull into profiles – articles, books, etc.
- sunburst graph with researcher in the middle, working out to individual artifacts they’ve produced and from there how each has been used/mentioned
- researchers mostly found that it was useful to see this stuff, but split on whether they could easily see the traditional and altmetrics. also split about usefulness of altmetrics in general – about 50/50. and again 50/50 that they felt they learned something new about their research. very mixed results.
- future – widgets faculty can embed in websites or dept directory, IR profile pages, coming. also one for individual artifacts. in theory could put one on everything in their IR. going to roll out to rest of campus later this year, fully implement & expand it.
- How do you scale this out to all your library staff? how do you get them on board to do this and drill as deeply as needed?
- Rush – Pitt – reorg’d into a liaison model rather than particular functions. more oriented to researchers, embedding – so just another thing we ask them to do as part of that. huge training programs within library system – continuous. won’t get thousands of faculty taking advantage of this. have a mandate for IRs but they still don’t get more than 50% participation unless they go knocking on doors. “I’m not sure how to scale, frankly. Or how many faculty will want to do this when we open it up.”
- Kristi – was pulled in by Cathy. already providing heavy duty research support – PhD biochemist. hiring someone to work on this. hot area for them – integrated at highest levels at their institution.
- Becker Model for health and clinical tracking – other models for other sciences?
- Similar frameworks – mainly Europe, Aus, UK, etc. US now starting to catch up. Many are also theoretical but don’t give specifics.
- where are the tools? don’t want to gather this data manually.
- Kristi – semantic web, semantically structured data. machine readable data. number of groups looking at scholarly data in humanities. Northwestern project – you tell it what you want and it is automatically all set up.
- Jason – Plum is doing it, altmetrics.com, people are working on it. just the beginning of the infrastructure. separation between who collects the data and who builds things on top of it. he thinks that’s a useful separation.
- useful for P&T?
- Jason – neat stuff coming out of this, quite possibly. someone who is building a network of people who’ve been on dissertation committees.