Beyond Journals: Altmetrics for Mapping and Measuring Scholarly Publishing

Following on from our look at archiving and analysing Twitter, this week in DITA we examined a use case for APIs and social media analytics: alternative metrics for scholarly publishing. This article covers how alternative metrics are extending traditional bibliometrics, thinks about how altmetrics work and some of the limitations and talks about one of the leading altmetric platforms

What are Alternative Metrics?

Academic research costs effort and usually requires money. It is normally provided to researchers via research grants that can be publicly or privately funded. Research outcomes are usually published in scholarly literature, journal articles or monographs, and disseminated through ‘grey literature’ such as speaking at conferences. Research publication enables outcomes to be shared with others and provides mechanisms for measuring the impact of the research via the analysis of citations: which other research references, and therefore builds upon, this work and impact factors. Research impact is of interest to academics and their institutions who derive prestige from it and also research funders to evaluate whether funding is being directed at research that is useful. Other researchers are interested in research that receives a lot of interest so they can track developments in their field.

Citation analysis and indexing measures the impact of a research publication within a particular network: scholarly literature. Alternative metrics, known as altmetrics takes the same idea, that research mentioned and propagated through networks is a measure of impact, and extends it to broader networks such as social media and news publishing. Altmetrics began to emerge around 2010 arguing that existing metrics didn’t take enough account of the growing dissemination ecosystems afford by the web. The 2010 altmetrics manifesto argues that more diverse metrics were required to fully reflect a more diverse scholarly publication ecosystem. This diversity can include not just the variety of online dissemination channels but also the evolving research output: for example the growing publication of datasets alongside articles.

How do Alternative Metrics work?

For this type of tracking to happen two core things need to be in place:

  1. The research publication or output needs to have a persistent identifier that can be referenced in the target network. Digital Object Identifiers (DOIs) and permanent URLs are two examples of this.
  2. The target network data needs to be accessible programatically for example via an API.

Target networks can then be scanned for mentions of research identifiers and the prevalence of this research output within the network can be measured. Services that measure altmetrics often track research outputs across several networks. Our practical DITA experiments this week involve working with perhaps the biggest current altmetrics provider

It is probably worth mentioning that this type of analysis does not measure the quality of the research. Quality is something incredibly slippery to define and difficult to mention. Alternative metrics can only tell you how much a research output has been amplified across public networks. It is also difficult to say as yet that altmetrics translate into influence or impact. It anything it is a measure of its virality. The potential of altmetrics, and comparing it with traditional metrics is however itself a growing field of research (and debate).

About provide a number of tools that provide access to their measurement the attention a scholarly output receives across a range of sources. These sources can include articles in policy documents (about 20 and counting), mainstream media, discussion in blog posts or mentions on social media. also tracks inclusion in an online reference manager such as Mendeley or CiteULike, but not yet Zotero but doesn’t count this as sharing activity.

They compute an attention score that considers the type of source (ranked so that a mention in a newspaper article is weighted more heavily than a blog post which is weighted more heavily than tweeting), the type of author (for example is this an academic researcher, a journalist or a promotional tweet from a publisher) and the number mentions of for article (unique per person per source. The weighted ranking per source and type of author probably reflects that ‘attention’ here is rather subjective. Discussion in a policy paper or mainstream media feature probably is attention: mention on social media or inclusion in a reference manager doesn’t even necessarily indicate the article has been read. ‘Amplification’ might be a better term. Still, it is good to see attempting to take account of context. This algorithm is important because understanding and trusting it is key to altmetrics having authority and being able to gain traction as a useful measure of evaluation.

Example Altmetric Badge

This analysis results in an Altmetric score for the article that is usually depicted within a graphic known as an altimetric donut with the colours of the graphic representing different sources. A full score and demographic breakdown for an article is also available. See this article, one of the highest scoring ever, Pathology in the Hundred Acre Wood: a neurodevelopmental perspective on A.A. Milne for an example or track the increasing score of  the topical Landing on a Comet: a guide to Rosetta’s perilous mission:

Screen shot of an article score breakdown accessed via the bookmarklet for a trending article today #CometLanding

Badges can be licensed for embedding on other web pages. Organisations can also subscribe to the Altmetric Explorer that provides access to the data along with aggregated summary dashboards targeted towards data relevant to that institution. For example institutions can track research amplification across sources or drill down to particular departments or academics. Individuals can use a bookmarklet to see the altmetrics for an article they are reading. The final product is an API that allows licensed access to the underlying data. The API currently serves about 11 million queries per week (source: webinar).

Exploring have been kind enough to allow us access to the platform during this module so we are able to try the full suite of tools and think about how their tools might fit into the scholarly landscape for researchers and institutions. For those without access Altmetric run a very informative webinar series.

The main working space in Altmetric is Explore the Data. This is a faceted search interface to the outputs within the database. Facets include:

  • time period
  • keywords
  • identifiers
  • subjects
  • journals
  • publishers
  • funders
  • source

One you have composed and executed a search the Articles tab lists matching articles. Clicking on the score leads to the full score breakdown and clicking on the output title links to the online web location for the output. There is also a quick button press to access the listing or article as JSON data via the API or generate an emailed CSV of the data for analysis offline in a suitable tool.

A second view is the  Activity tab that lists the each mention ordered by the most recent. Mentions are audible. Clicking on the activity will link through to the original source.

A third view of the data is via the Journals tab. This aggregates the matching articles by their publishing container. The journals can be listed by overall score or by article timelines to track patterns of mentions over time.

A search can also be saved. It will then appear in your personal workspace so it can be executed again. Each execution represents a snapshot of that output’s altmetric in time with some information on change over time. This is dynamic data and reacts to each mention that is tracked.

The user interface is very easy and intuitive to pick up. It is easy to put together quite complicated queries that can be precisely targeted or extremely broad and the results are beautifully visualised. The donut provides a very clear way of instantly seeing not just the score but, assuming no issues with colour blindness, the summary of sources featured. The colour emphasis indicates whether the output has been mentioned across a wide variety of sources or whether a particular type of source predominates. This is particular useful in the tiled view when you can quickly scan article scores with just the title and their containing journal.

Tile View available via Altmetric Explorer provides a quick scan of scores and sources

Within the breakdown the score benchmarking and demographic breakdown provide a level of instant insight that is difficult to do with the simple Twitter archiving tools we have tried. The one click access to JSON data – no programming required to access – is quick and easy too (assuming you have the means to parse and wrangle JSON data subsequently).

Experimenting with

I tried three types of searches that I saved to my workspace:

  • keyword searches for topics of interest
  • a journal based search made up of journals whose RSS table of contents service I currently subscribe to plus some others I only discovered from using
  • a subject search for LIS

Searches can be saved as workspaces with email reports and data export available

The first is highly focused and is typical of how a researcher might attempt to keep up trends in their research areas, how publishers might measure target article of campaign and how an institution might target specific publications, research centres or themes or even individuals.

The second has a broader scope but is still focused more closely on selected sources. Using RSS table of content services allows me to receive updates from journals when new articles are posted. Combining this with would allow an understanding of how those are propagated. I could do an initial selection of reading based on what I think I might be interested in from the table of content that might be modified as I see particular articles getting more or less attention (there is only so much time and so much to read!). The use of critical selection and altmetrics may help researchers shape and prioritise their reading lists without becoming too subjected to preferential attachment. I think this would be a danger of only deciding what to read based on altmetrics, or indeed any citation measure. Being able to extend time periods may help not only avoid but also identify social information effects (e.g. filter bubbles or infostorms) in the data.

The third is my broadest query and checks for mentions across an entire subject area. I could see publishers doing this to track how their journals or articles are doing in a field and institutions their departments and academics. As a researcher there is maybe some interesting analysis to be done on how a subject is defined by seeing which articles are also classified in other subject areas and whether higher ranked articles are say more interdisciplinary. It would also be really interesting if you could join up altmetrics with pricing fluctuations say.

Some of this data I could get from other sources and existing systems but I haven’t. By compiling lots of data in one place and adding a really clean and simple user interface not only introduced me to altmetrics but also exposed data I should have been able to see easily before but haven’t. There are lessons to be learnt here about how discovery systems work and what are suitable entry points to discovery. Most library resource discovery systems expect you to have some sense of your information need to help you look for it whereas searching based on altmetrics provided that serendipitous discovery of what’s buzzing that I might be interested in.

Reflecting on Altmetrics

Altmetrics are to be welcomed as a useful addition to established metrics given scholarly publishing is transforming around web practices and technologies. Almetrics, citation analysis and impact factor are measuring different things so draw upon different sources and algorithms for their calculations. They may be complementary although they do not necessarily correlate or agree. Altmetrics themselves may change the way scholarly publication is practiced as researchers, institutions and publishers think about how to use them and that dynamic remains to be seen and researched.

Altmetrics likely to be an ongoing feature of bibliometrics and research impact studies

I was conscious that some of the scores seen within have been influence by #citylis folk tweeting articles during our practical exercise exploring altmetrics. Should altmetrics work out that this ‘attention’ is given to papers because of a classroom activity and weight it accordingly? Will altmetric services be able to get more sophisticated at understanding context and weighting for even more factors as they have more data to mine and develop more sophisticated algorithms? Will those algorithms be proprietary or will they remain transparent and trustworthy?

Measuring research impact is notoriously difficult and debates on how best to do this will be ongoing as long as research exists I suspect. Like any analysis, including citation based measurement, there are limitations and potential pitfalls with altmetrics if not interpreted appropriately but within the boundaries of those limitations they provide useful insight derived from open and accessible data that we should make use of whilst research on that use continues.

I also think services like will make us think harder about user interface design for resource discovery and data sharing and how data mining and visualisation can be used to facilitate resource discovery and research.

If anything one of the most important lessons I’ve taken from looking at altmetrics is a reminder that no one approach should be the be all and end all and that diversity should be embraced and can be tamed via APIs, open data, trustworthy algorithms and accessible user interfaces.


One thought on “Beyond Journals: Altmetrics for Mapping and Measuring Scholarly Publishing

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s