7.1.2Scientometrics 2.0 – Promises, challenges and limitations

Recently, the conception of Scientometrics 2.0 was proposed to embrace a big step towards the measurement of societal impact and “broader impacts” of research and to cover “open science” – ‘social media metrics’ or ‘alternative metrics’ as groundwork and components for a “Scientometrics 2.0” (Priem and Hemminger, 2010). As possible sources Priem and Hemminger recommended to include bookmarking, reference managers, recommendation systems, comments on articles, microblogging, Wikipedia, blogging, and other sources such as social networks, video, and open data repositories. In the recently launched Handbook of Science and Technology Indicators (Glänzel et al., 2019) we see social media metrics, book reviews, scholarly twitter metrics, readership,  web citation indicators and online indicators were all considered and introduced as new indicators for reseach assessment in the context of Scientometrics 2.0.

Promises

One of the most important promises is, of course, to overcome a number of limitations of the scientometrics 1.x model, above all, the restriction to the measurement of scholarly communication and impact. Within this broader scope of new version of Scientometrics 2.0, in general, and altmetrics, in particular, a number of important features and promises have been addressed. Thus Sugimoto (2016) pointed to the increasing demand for showing impact of research beyond academia, and democratising the impact by giving greater voice and vote, e.g., to underrepresented groups (gender, ethnicity, disability, geographic etc.) in determining impact. The other main promise of Scientometrics 2.0 is from social networks. Network-based approaches based on social media data may also contribute to a more diversified system of scientific impact assessment by adding a relational and social capital-based perspective (Hoffman et al., 2014). In particular, Wouters et al. (2019) provided three basic examples of this kind of network-based application analyzing the relationships and interactions among different actors: communities of attention, hashtag coupling analysis, and reader pattern analysis.
 

Challenges and limitations

The promises are contrasted by a number of challenges and limitations have been summarised by Wouters and Costas (2012), Sugimoto (2016), Gumpenberger et al. (2016) and Kousha (2019), including:

  • Analyses are usually conducted at the individual (micro) level and most benefits of Scientometrics 2.0 are at the micro level. However, the aggregation at higher levels is questionable, so that the validity, reliability and feasibility of the large scale studies are one of the main challenges.
  • A number of assumptions are not yet validated and tested but the high dynamics and rapid development of online and electronic communications (Web 2.0 – and beyond?) would increase the difficulties for altmetrics to keep pace with this development once validated and implemented.
  • More transparency and clarity in the data covered is needed. There is not yet any clear definition of actors on both sides. Thus, if we talk about impact – impact upon whom is meant? And what are the potential biases in terms of actor and user profiles? Without clarification the standardization and normalization of measures is hardly conceivable.
  • Data quality: Automated processes produce errors and influence social media metrics. Web search also needs to tackle false matches and duplicate results for a better data quality.
  • In contrast to the previous scientometrics model, altmetrics still lacks mathematical background and proper models, which impedes the clear interpretation of indicators. Issues caused by composite indicators and the arbitrariness of their construction make their interpretation and comparability even more difficult. One of the goals of the altmetrics movement was to overcome the flaws of the traditional citation-based indicators but instead new ‘all-in-one’ indicators are created (“old habits die hard”).