Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userSolving reviewer assignment problem in software peer review: An approach based on preference matrix and asymmetric TSP model
295
0
0.0
(
0
)
Added by
Yanqing Wang
Publication date
2014
fields
Informatics Engineering
and research's language is
English
Ask ChatGPT about the research
No Arabic abstract
Optimized reviewer assignment can effectively utilize limited intellectual resources and significantly assure review quality in various scenarios such as paper selection in conference or journal, proposal selection in funding agencies and so on. However, little research on reviewer assignment of software peer review has been found. In this study, an optimization approach is proposed based on students preference matrix and the model of asymmetric traveling salesman problem (ATSP). Due to the most critical role of rule matrix in this approach, we conduct a questionnaire to obtain students preference matrices and convert them to rule matrices. With the help of software ILOG CPLEX, the approach is accomplished by controlling the exit criterion of ATSP model. The comparative study shows that the assignment strategies with both reviewers preference matrix and authors preference matrix get better performance than the random assignment. Especially, it is found that the performance is just a little better than that of random assignment when the reviewers and authors preference matrices are merged. In other words, the majority of students have a strong wish of harmonious development even though high-level students are not willing to do that.
rate research
Read More
We consider three important challenges in conference peer review: (i) reviewers maliciously attempting to get assigned to certain papers to provide positive reviews, possibly as part of quid-pro-quo arrangements with the authors; (ii) torpedo reviewing, where reviewers deliberately attempt to get assigned to certain papers that they dislike in order to reject them; (iii) reviewer de-anonymization on release of the similarities and the reviewer-assignment code. On the conceptual front, we identify connections between these three problems and present a framework that brings all these challenges under a common umbrella. We then present a (randomized) algorithm for reviewer assignment that can optimally solve the reviewer-assignment problem under any given constraints on the probability of assignment for any reviewer-paper pair. We further consider the problem of restricting the joint probability that certain suspect pairs of reviewers are assigned to certain papers, and show that this problem is NP-hard for arbitrary constraints on these joint probabilities but efficiently solvable for a practical special case. Finally, we experimentally evaluate our algorithms on datasets from past conferences, where we observe that they can limit the chance that any malicious reviewer gets assigned to their desired paper to 50% while producing assignments with over 90% of the total optimal similarity. Our algorithms still achieve this similarity while also preventing reviewers with close associations from being assigned to the same paper.
A `peer-review system in the context of judging research contributions, is one of the prime steps undertaken to ensure the quality of the submissions received, a significant portion of the publishing budget is spent towards successful completion of the peer-review by the publication houses. Nevertheless, the scientific community is largely reaching a consensus that peer-review system, although indispensable, is nonetheless flawed. A very pertinent question therefore is could this system be improved?. In this paper, we attempt to present an answer to this question by considering a massive dataset of around $29k$ papers with roughly $70k$ distinct review reports together consisting of $12m$ lines of review text from the Journal of High Energy Physics (JHEP) between 1997 and 2015. In specific, we introduce a novel textit{reviewer-reviewer interaction network} (an edge exists between two reviewers if they were assigned by the same editor) and show that surprisingly the simple structural properties of this network such as degree, clustering coefficient, centrality (closeness, betweenness etc.) serve as strong predictors of the long-term citations (i.e., the overall scientific impact) of a submitted paper. These features, when plugged in a regression model, alone achieves a high $R^2$ of 0.79 and a low $RMSE$ of 0.496 in predicting the long-term citations. In addition, we also design a set of supporting features built from the basic characteristics of the submitted papers, the authors and the referees (e.g., the popularity of the submitting author, the acceptance rate history of a referee, the linguistic properties laden in the text of the review reports etc.), which further results in overall improvement with $R^2$ of 0.81 and $RMSE$ of 0.46.
A semi-supervised model of peer review is introduced that is intended to overcome the bias and incompleteness of traditional peer review. Traditional approaches are reliant on human biases, while consensus decision-making is constrained by sparse information. Here, the architecture for one potential improvement (a semi-supervised, human-assisted classifier) to the traditional approach will be introduced and evaluated. To evaluate the potential advantages of such a system, hypothetical receiver operating characteristic (ROC) curves for both approaches will be assessed. This will provide more specific indications of how automation would be beneficial in the manuscript evaluation process. In conclusion, the implications for such a system on measurements of scientific impact and improving the quality of open submission repositories will be discussed.
This paper designs a market platform for Peer-to-Peer (P2P) energy trading in Transactive Energy (TE) systems, where prosumers and consumers actively participate in the market as seller or buyer to trade energy. An auction-based approach is used for market clearing in the proposed platform and a review of different types of auction is performed. The appropriate auction approach for market clearing in the proposed platform is designed. The proposed auction mechanism is implemented in three steps namely determination, allocation and payment. This paper identifies important P2P market clearing performance indices, which are used to compare and contrast the designed auction with different types of auction mechanisms. Comparative studies demonstrate the efficacy of the proposed auction mechanism for market clearing in the P2P platform.
One of the outstanding challenges of cross-identification is multiplicity: detections in crowded regions of the sky are often linked to more than one candidate associations of similar likelihoods. We map the resulting maximum likelihood partitioning to the fundamental assignment problem of discrete mathematics and efficiently solve the two-way catalog-level matching in the realm of combinatorial optimization using the so-called Hungarian algorithm. We introduce the method, demonstrate its performance in a mock universe where the true associations are known, and discuss the applicability of the new procedure to large surveys.
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
