Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userThe case for caution in predicting scientists future impact
139
0
0.0
(
0
)
Ask ChatGPT about the research
No Arabic abstract
We stress-test the career predictability model proposed by Acuna et al. [Nature 489, 201-202 2012] by applying their model to a longitudinal career data set of 100 Assistant professors in physics, two from each of the top 50 physics departments in the US. The Acuna model claims to predict h(t+Delta t), a scientists h-index Delta t years into the future, using a linear combination of 5 cumulative career measures taken at career age t. Here we investigate how the predictability depends on the aggregation of career data across multiple age cohorts. We confirm that the Acuna model does a respectable job of predicting h(t+Delta t) up to roughly 6 years into the future when aggregating all age cohorts together. However, when calculated using subsets of specific age cohorts (e.g. using data for only t=3), we find that the models predictive power significantly decreases, especially when applied to early career years. For young careers, the model does a much worse job of predicting future impact, and hence, exposes a serious limitation. The limitation is particularly concerning as early career decisions make up a significant portion, if not the majority, of cases where quantitative approaches are likely to be applied.
rate research
Read More
Reputation is an important social construct in science, which enables informed quality assessments of both publications and careers of scientists in the absence of complete systemic information. However, the relation between reputation and career growth of an individual remains poorly understood, despite recent proliferation of quantitative research evaluation methods. Here we develop an original framework for measuring how a publications citation rate $Delta c$ depends on the reputation of its central author $i$, in addition to its net citation count $c$. To estimate the strength of the reputation effect, we perform a longitudinal analysis on the careers of 450 highly-cited scientists, using the total citations $C_{i}$ of each scientist as his/her reputation measure. We find a citation crossover $c_{times}$ which distinguishes the strength of the reputation effect. For publications with $c < c_{times}$, the authors reputation is found to dominate the annual citation rate. Hence, a new publication may gain a significant early advantage corresponding to roughly a 66% increase in the citation rate for each tenfold increase in $C_{i}$. However, the reputation effect becomes negligible for highly cited publications meaning that for $cgeq c_{times}$ the citation rate measures scientific impact more transparently. In addition we have developed a stochastic reputation model, which is found to reproduce numerous statistical observations for real careers, thus providing insight into the microscopic mechanisms underlying cumulative advantage in science.
We present a simple generalization of Hirschs h-index, Z = sqrt{h^{2}+C}/sqrt{5}, where C is the total number of citations. Z is aimed at correcting the potentially excessive penalty made by h on a scientists highly cited papers, because for the majority of scientists analyzed, we find the excess citation fraction (C-h^{2})/C to be distributed closely around the value 0.75, meaning that 75 percent of the authors impact is neglected. Additionally, Z is less sensitive to local changes in a scientists citation profile, namely perturbations which increase h while only marginally affecting C. Using real career data for 476 physicists careers and 488 biologist careers, we analyze both the distribution of $Z$ and the rank stability of Z with respect to the Hirsch index h and the Egghe index g. We analyze careers distributed across a wide range of total impact, including top-cited physicists and biologists for benchmark comparison. In practice, the Z-index requires the same information needed to calculate h and could be effortlessly incorporated within career profile databases, such as Google Scholar and ResearcherID. Because Z incorporates information from the entire publication profile while being more robust than h and g to local perturbations, we argue that Z is better suited for ranking comparisons in academic decision-making scenarios comprising a large number of scientists.
Correctly assessing a scientists past research impact and potential for future impact is key in recruitment decisions and other evaluation processes. While a candidates future impact is the main concern for these decisions, most measures only quantify the impact of previous work. Recently, it has been argued that linear regression models are capable of predicting a scientists future impact. By applying that future impact model to 762 careers drawn from three disciplines: physics, biology, and mathematics, we identify a number of subtle, but critical, flaws in current models. Specifically, cumulative non-decreasing measures like the h-index contain intrinsic autocorrelation, resulting in significant overestimation of their predictive power. Moreover, the predictive power of these models depend heavily upon scientists career age, producing least accurate estimates for young researchers. Our results place in doubt the suitability of such models, and indicate further investigation is required before they can be used in recruiting decisions.
In the book The Essential Tension Thomas Kuhn described the conflict between tradition and innovation in scientific research --i.e., the desire to explore new promising areas, counterposed to the need to capitalize on the work done in the past. While it is true that along their careers many scientists probably felt this tension, only few works have tried to quantify it. Here, we address this question by analyzing a large-scale dataset, containing all the papers published by the American Physical Society (APS) in more than $25$ years, which allows for a better understanding of scientists careers evolution in Physics. We employ the Physics and Astronomy Classification Scheme (PACS) present in each paper to map the scientific interests of $181,397$ authors and their evolution along the years. Our results indeed confirm the existence of the `essential tension with scientists balancing between exploring the boundaries of their area and exploiting previous work. In particular, we found that although the majority of physicists change the topics of their research, they stay within the same broader area thus exploring with caution new scientific endeavors. Furthermore, we quantify the flows of authors moving between different subfields and pinpoint which areas are more likely to attract or donate researchers to the other ones. Overall, our results depict a very distinctive portrait of the evolution of research interests in Physics and can help in designing specific policies for the future.
Do scientists follow hot topics in their scientific investigations? In this paper, by performing analysis to papers published in the American Physical Society (APS) Physical Review journals, it is found that papers are more likely to be attracted by hot fields, where the hotness of a field is measured by the number of papers belonging to the field. This indicates that scientists generally do follow hot topics. However, there are qualitative differences among scientists from various countries, among research works regarding different number of authors, different number of affiliations and different number of references. These observations could be valuable for policy makers when deciding research funding and also for individual researchers when searching for scientific projects.
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
