CAS Evaluation Research Center Established

A new establishment of the Chinese Academy of Sciences (CAS), the Evaluation Research Center, was officially inaugurated on September 20 in Beijing. Affiliated to the CAS

Patent Citations Analysis and Its Value in Research Evaluation： A Review and a New Approach to Map Technology-relevant Research

Purpose： First, to review the state-of-the-art in patent citation analysis, particularly characteristics of patent citations to scientific literature （scientific non-patent references, SNPRs）. Second, to present a novel mapping approach to identify technology-relevant research based on the papers cited by and referring to the SNPRs. Design/methodology/approach： In the review part we discuss the context of SNPRs such as the time lags between scientific achievements and inventions. Also patent-to-patent citation is addressed particularly because this type of patent citation analysis is a major element in the assessment of the economic value of patents. We also review the research on the role of universities and researchers in technological development, with important issues such as universities as sources of technological knowledge and inventor-author relations. We conclude the review part of this paper with an overview of recent research on mapping and network analysis of the science and technology interface and of technological progress in interaction with science. In the second part we apply new techniques for the direct visualization of the cited and citing relations of SNPRs, the mapping of the landscape around SNPRs by bibliographic coupling and co-citation analysis, and the mapping of the conceptual environment of SNPRs by keyword co-occurrence analysis. Findings： We discuss several properties of SNPRs. Only a small minority of publications covered by the Web of Science or Scopus are cited by patents, about 3%-4%. However, for publications based on university-industry collaboration the number of SNPRs is considerably higher, around 15%. The proposed mapping methodology based on a ＂second order SNPR approach＂ enables a better assessment of the technological relevance of research. Research limitations： The main limitation is that a more advanced merging of patent and publication data, in particular unification of author and inventor names, in still a necessity. Practical implications： The proposed mapping methodology enables the creation of a database of technology-relevant papers （TRPs）. In a bibliometric assessment the publications of research groups, research programs or institutes can be matched with the TRPs and thus the extent to which the work of groups, programs or institutes are relevant for technological development can be measured. Originality/value： The review part examines a wide range of findings in the research of patent citation analysis. The mapping approach to identify a broad range of technologyrelevant papers is novel and offers new opportunities in research evaluation practices.

Evaluations on social network analysis (SNA) research in China's Mainland:A critical assessment of journal literatures

Social network analysis(SNA) has been introduced to China's Mainland since the end of last century. It is often stated that SNA research has experienced rapid growth in China over these years, but few studies have been conducted to prove the statement. This paper aims at exploring the research status and development of SNA in China by a critical assessment of journal articles. Our findings show that SNA is an evolving and diversified research area which has rich themes and topics, and can be applied to those studies on different levels, context and disciplines, and attract researchers and scholars from various fields and domains. In addition, the information community(Library & Information Science and Information Systems) plays a leading role in the SNA related researches. The paper also points out the research on SNA in China has some limitations, so it proposes several implications for the future development of SNA research from perspectives of information science.

Research and perspectives on criteria for evaluation of nursing research achievements

This paper introduces concepts related to scientific research achievements, analyzes current evaluation methods with regard to nursing research achievements and their application at home and abroad, and summarizes findings from the investigation of obstacles to the application of nursing research results in China, aiming to provide reference points for the evaluation and application of nursing research results in China.

RESEARCH ON THE THEORY AND METHODS OF COMPREHENSIVE EVALUATION OF REGIONAL COAL RESOURCES

consists of forecasting for the total reserves of regional mineral resources and evaluation,compreIn general,based on evaluated objects and factors involved,evaluation of mineral resources hensive evaluation of regional mineral resources, and economic and technical evaluation of deposits. Here a region can be a country, a province, or a mining area, while a deposit is mainly related to a single type of mineral, although it can contain more than one types of minerals. In the past, to assess mineral resources was primarily to evaluate and forecast the total reserves of regional mineral resources. With the advancement of science, technology, and economy, the evaluation of mineral resources has developed from forecasting the reserves of them to comprehensively evaluating them on the basis of numerous factors.

A Framework for the Assessment of Research and Its Impacts

This paper proposes a holistic framework for the development of models for the assessment of research activities and their impacts. It distinguishes three dimensions, including in an original way, data as a main dimension, together with theory and methodology. Each dimension of the framework is further characterized by three main building blocks： education, research, and innovation （theory）; efficiency, effectiveness, and impact （methodology）; and availability, interoperability, and ＂unit-free＂ property （data）. The different dimensions and their nine constituent building blocks are attributes of an overarching concept, denoted as ＂quality.＂ Three additional quality attributes are identified as implementation factors （tailorability, transparency, and openness） and three ＂enabling＂ conditions （convergence, mixed methods, and knowledge infrastructures） complete the fi-amework. A framework is required to develop models of metrics. Models of metrics are necessary to assess the meaning, validity, and robustness of metrics. The proposed framework can be a useful reference for the development of the ethics of research evaluation. It can act as a common denominator for different analytical levels and relevant aspects and is able to embrace many different and heterogeneous streams of literature. Directions for future research are provided.

Measuring and Visualizing Research Collaboration and Productivity

Purpose： This paper presents findings of a quasi-experimental assessment to gauge the research productivity and degree of interdisciplinarity of research center outputs. Of special interest, we share an enriched visualization of research co-authoring patterns. Design/methodology/approach： We compile publications by 45 researchers in each of 1） the iUTAH project, which we consider here to be analogous to a ＂research center,＂ 2） CG1-a comparison group of participants in two other Utah environmental research centers, and 3） CG2--a comparison group of Utah university environmental researchers not associated with a research center. We draw bibliometric data from Web of Science and from Google Scholar. We gather publications for a period before iUTAH had been established （2010-2012） and a period after （2014-2016）. We compare these research outputs in terms of publications and citations thereto. We also measure interdisciplinarity using Integration scoring and generate science overlay maps to locate the research publications across disciplines. Findings： We find that participation in the iUTAH project appears to increase research outputs （publications in the After period） and increase research citation rates relative to the comparison group researchers （although CG 1 research remains most cited, as it was in the Before period）. Most notably, participation in iUTAH markedly increases co-authoring among researchers--in general; and for junior, as well as senior, faculty; for men and women： across organizations; and across disciplines. Research limitations： The quasi-experimental design necessarily generates suggestive, not definitively causal, findings because of the imperfect controls. Practical implications： This study demonstrates a viable approach for research assessment of a center or program for which random assignment of control groups is not possible. It illustrates use of bibliometric indicators to inform R＆D program management. Originality/value： New visualizations of researcher collaboration provide compelling comparisons of the extent and nature of social networking among target cohorts.

Minimum Representative Size in Comparing Research Performance of Universities:the Case of Medicine Faculties in Romania

Purpose： The main goal of this study is to provide reliable comparison of performance in higher education. In this respect, we use scientometric measures associated with faculties of medicine in the six health studies universities in Romania.Design/methodology/approach： The method to estimate the minimum necessary size, proposed in in Shen et al.（2017）, is applied in this article. We collected data from the Scopus data-base for the academics of the departments of medicine within the six health studies universities in Romania during the 2009 to 2014. And two kind of statistic treatments based on that method are implemented, pair-wise comparison and one-to-the-rest comparison. All the results of these comparisons are shown.Findings： According to the results： We deem that Cluj and Tg. Mure？ have the superior and inferior performance respectively, since their reasonably small value of the minimum representative size, in either of the kinds of comparison, whichever indexes of citations, h-index, or g-index is used. we can not reliably distinguish differences among the rest of the faculties, since the quite large value of their minimum representative size.Research limitations： There is only six faculties of medicine in health studies universities in Romania are analyzed.Practical implications： Our methods of comparison play an important role in ranking data sets associated with different collective units, such as faculties, universities, institutions, based on some aggregate scores like mean and totality. Originality/value： We applied the minimum representative size to a new emprical context- that of the departments of medicine in the health studies universities in Romania.

Evaluating grant proposals:lessons from using metrics as screening device

Purpose:This study examines the effects of using publication-based metrics for the initial screening in the application process for a project leader.The key questions are whether formal policy affects the allocation of funds to researchers with a better publication record and how the previous academic performance of principal investigators is related to future project results.Design/methodology/approach:We compared two competitions,before and after the policy raised the publication threshold for the principal investigators.We analyzed 9,167 papers published by 332 winners in physics and the social sciences and humanities(SSH),and 11,253 publications resulting from each funded project.Findings:We found that among physicists,even in the first period,grants tended to be allocated to prolific authors publishing in high-quality journals.In contrast,the SSH project grantees had been less prolific in publishing internationally in both periods;however,in the second period,the selection of grant recipients yielded better results regarding awarding grants to more productive authors in terms of the quantity and quality of publications.There was no evidence that this better selection of grant recipients resulted in better publication records during grant realization.Originality:This study contributes to the discussion of formal policies that rely on metrics for the evaluation of grant proposals.The Russian case shows that such policy may have a profound effect on changing the supply side of applicants,especially in disciplines that are less suitable for metric-based evaluations.In spite of the criticism given to metrics,they might be a useful additional instrument in academic systems where professional expertise is corrupted and prevents allocation of funds to prolific researchers.

Functions of Uni- and Multi-citations： Implications for Weighted Citation Analysis

Purpose： （1） To test basic assumptions underlying frequency-weighted citation analysis： （a） Uni-citations correspond to citations that are nonessential to the citing papers; （b） The influence of a cited paper on the citing paper increases with the frequency with which it is cited in the citing paper. （2） To explore the degree to which citation location may be used to help identify nonessential citations. Design/methodology/approach： Each of the in-text citations in all research articles published in Issue 1 of the Journal of the Association for Information Science and Technology （JASIST） 2016 was manually classified into one of these five categories： Applied, Contrastive, Supportive, Reviewed, and Perfunctory. The distributions of citations at different in-text frequencies and in different locations in the text by these functions were analyzed. Findings： Filtering out nonessential citations before assigning weight is important for frequency-weighted citation analysis. For this purpose, removing citations by location is more effective than re-citation analysis that simply removes uni-citations. Removing all citation occurrences in the Background and Literature Review sections and uni-citations in the Introduction section appears to provide a good balance between filtration and error rates. Research limitations： This case study suffers from the limitation of scalability and generalizability. We took careful measures to reduce the impact of other limitations of the data collection approach used. Relying on the researcher＇s judgment to attribute citation functions, this approach is unobtrusive but speculative, and can suffer from a low degree of confidence, thus creating reliability concerns. Practical implications： Weighted citation analysis promises to improve citation analysis for research evaluation, knowledge network analysis, knowledge representation, and information retrieval. The present study showed the importance of filtering out nonessential citations before assigning weight in a weighted citation analysis, which may be a significant step forward to realizing these promises. Originality/value： Weighted citation analysis has long been proposed as a theoretical solution to the problem of citation analysis that treats all citations equally, and has attracted increasing research interest in recent years. The present study showed, for the first time, the importance of filtering out nonessential citations in weighted citation analysis, pointing research in this area in a new direction.

Appraisal of Clinical Practice Guidelines for Ischemic Stroke Management in Chinese Medicine with Appraisal of Guidelines for Research and Evaluation Instrument:A Systematic Review

Objective： To systematically review the clinical practice guidelines （CPGs） for ischemic stroke in Chinese medicine （CM） with the Appraisal of Guidelines for Research and Evaluation （AGREE II） instrument. Methods： CM CPGs for ischemic stroke were searched in 5 online databases and hand-searches in CPG- related handbooks published from January 1990 to December 2012. The CPGs were categorized into evidence based （EB） guideline, consensus based with no explicit consideration of evidence based （CB-EB） guideline and consensus based （CB） guideline according to the development method. Three reviewers independently appraised the CPGs based on AGREE II instrument, and compared the CPGs＇ recommendations on CM pattern classification and treatment. Results： Five CM CPGs for ischemic stroke were identified and included. Among them, one CPG was EB guideline, two were CB guidelines and two were CB-EB guidelines. The quality score of the EB guideline was higher than those of the CB-EB and CB guidelines. Five CM patterns in the CPGs were recommended in the EB CPG. The comprehensive protocol of integrative Chinese and Western medicine recommended in the EB CPG was mostly recommended for ischemic stroke in the CPGs. The recommendations varied based on the CM patterns. Conclusion： The quality of EB CPG was higher than those of CB and CB-EB CPGs in CM for ischemic stroke and integrative approaches were included in CPGs as major interventions.

Co-author Weighting in Bibliometric Methodology and Subfields of a Scientific Discipline

Purpose: To give a theoretical framework to measure the relative impact of bibliometric methodology on the subfields of a scientific discipline, and how that impact depends on the method of evaluation used to credit individual scientists with citations and publications. The authors include a study of the discipline of physics to illustrate the method. Indicators are introduced to measure the proportion of a credit space awarded to a subfield or a set of authors.Design/methodology/approach: The theoretical methodology introduces the notion of credit spaces for a discipline. These quantify the total citation or publication credit accumulated by the scientists in the discipline. One can then examine how the credit is divided among the subfields. The design of the physics study uses the American Physical Society print journals to assign subdiscipline classifications to articles and gather citation, publication, and author information. Credit spaces for the collection of Physical Review Journal articles are computed as a proxy for physics.Findings: There is a substantial difference in the value or impact of a specific subfield depending on the credit system employed to credit individual authors.Research limitations: Subfield classification information is difficult to obtain. In the illustrative physics study, subfields are treated in groups designated by the Physical Review journals. While this collection of articles represents a broad part of the physics literature, it is not all the literature nor a random sample.Practical implications: The method of crediting individual scientists has consequences beyond the individual and affects the perceived impact of whole subfields and institutions. Originality/value: The article reveals the consequences of bibliometric methodology on subfields of a disciple by introducing a systematic theoretical framework for measuring the consequences.

Mining Related Articles for Automatic Journal Cataloging

Purpose： This paper is an investigation of the effectiveness of the method of clustering biomedical journals through mining the content similarity of journal articles. Design/methodology/approach： 3,265 journals in Pub Med are analyzed based on article content similarity and Web usage, respectively. Comparisons of the two analysis approaches and a citation-based approach are given.Findings： Our results suggest that article content similarity is useful for clustering biomedical journals, and the content-similarity-based journal clustering method is more robust and less subject to human factors compared with the usage-based approach and the citation-based approach. Research limitations： Our paper currently focuses on clustering journals in the biomedical domain because there are a large volume of freely available resources such as Pub Med and Me SH in this field. Further investigation is needed to improve this approach to fit journals in other domains.Practical implications： Our results show that it is feasible to catalog biomedical journals by mining the article content similarity. This work is also significant in serving practical needs in research portfolio analysis.Originality/value： To the best of our knowledge, we are among the first to report on clustering journals in the biomedical field through mining the article content similarity. This method can be integrated with existing approaches to create a new paradigm for future studies of journal clustering.

Evaluation of Scientific and Technological Research in China’s Colleges: A Review of Policy Reforms, 2000–2020

Purpose:This study conducts a systematic review of policy reforms for the evaluation of scientific and technological research(E-STR)in China’s colleges.In doing so,it identifies changes to past policies,as well as the objectives and realizability of the most recent policy reform.Design/Approach/Methods:This study systematically reviews the contexts of E-STR policy reforms in China’s colleges post 2000 using three important related policy documents from 2003,2013,and 2020 as samples.Research findings were obtained via content analysis.Findings:Unlike previous attempts,the most recent policy reform has incorporated the lessons learned while introducing highly targeted measures and a monitoring mechanism.Such reform should accelerate the promotion of major original scientific research in China’s colleges,enhance the contribution of scientific research to socioeconomic development,and strengthen support for the cultivation of undergraduate talents.Originality/Value:Initiated in the early part of 2000,China’s latest E-STR policy reform constitutes an important turning point in the initiative to burst the research bubble and return to the essence of innovation.It will drive China’s colleges to embark on a research path that places value on original innovation and contribution to socioeconomic development.

Measuring National-level Research Production under Fractional Counting:China versus the U.S.

This paper aims to examine the heterogeneity of research production between China and the United States by disaggregating the national research production into its constituent components.Different from previous studies,we introduce the share counting method to determine the number of publications attributed to each country.We analyse bibliographic metadata from over 36 million SCI/SSCI-indexed journal publications published in the period from 2000 to 2021.The research production of China and the United States is decomposed according to the document types,disciplines,and high-impact journals.In the quantitative analysis,the first finding is that China emerged as the world’s largest contributor to SCIindexed publications in 2019 under fractional counting,two years earlier than under whole counting.Surpassing the U.S.in publication count does not indicate a completely surpassing position for China in its scientific production strength,however.When it is divided by document types,China has published a smaller proportion of review-type journal publications than the U.S.;when filtered by disciplines,in the period from 2016 to 2021,China’s research production leads in only 100 of 178 natural science fields and 2of 58 social science fields.The second finding is,when only the number of papers on high-impact journals is considered,China also surpassed the U.S.in 2019;meanwhile,the proportion of high-impact journal papers of China is still lower than that of the U.S.These results reveal that there are different knowledge production patterns in China and the United States.This study contributes to a better understanding of the disparities in research productivity between the top two nations,and suggests several policy implications for China.

Towards a Framework for Evaluating the Research Performance of Chinese Double First-Class Universities

The study focuses on the research performance of Double First-Class(DFC)universities in China.A theoretical framework based on economic theory and the Analytic Hierarchy Process(AHP)method for a novel evaluation model is proposed.The model is performed on a sample of 41 participant DFC universities in China.Using data collected from these universities,it was found that there is no consistency between performance ranking,input ranking,and output ranking,with the best ranked universities far from the most efficient.These findings provide empirical evidence of DFC universities’research performance situation and suggest strategies that the government can use to propel their sustainable development.

Research methods in complementary and alternative medicine： an integrative review

The scientific literature presents a modest amount ot evidence m the use or complementary ana al[erna- tire medicine （CAM）. On the other hand, in practice, relevant results are common. The debates among CAM practitioners about the quality and execution of scientific research are important, Therefore, the aim of this review is to gather, synthesize and describe the differentiated methodological models that encompass the complexity of therapeutic interventions. The process of bringing evidence-based medicine into clinical practice in CAM is essential for the growth and strengthening of complementary medicines worldwide.

Development and preliminary verification of the evaluation system for clinical practice guidelines in China

Objective:Clinical practice guidelines can improve healthcare processes and patient outcomes;however,the quality of these guidelines varies greatly in China.The aim of this study was to construct a comprehensive instrument for the appraisal of clinical practice guidelines in China(AGREE-CHINA),and to validate its reliability as a tool for helping potential guideline users in assessing guideline quality.Methods:First,an interdisciplinary working group was established for developing the methods.They also created a checklist as a tool according to the Appraisal of Guidelines,Research and EvaluationⅡ(AGREEⅡ)standards,considering the particularity of Chinese clinical practice.Next,the first draft of AGREE-China was developed by vote,modification,preliminary trial,and crossverification.To ensure the objectivity,credibility,and reproducibility of the draft assessment,all of the checklists and standards were cross-reviewed fairly widely.Fin ally,AGREE-CHINA and AGREEⅡwere used to assess the Chinese guideli nes published in the past five years,and the results were compared.Results:The presented AGREE-CHINA covered five main checkpoints(science and rigor,effectiveness and safety,economy,usability and feasibility,and conflicts of interest)with each point divided into several more specific checkpoints.Definitions and rationales for each main checkpoint appear in the Appendix.The quality ratings based on the total scores of AGREE-China and AGREEⅡwere consistent(r=0.508,P=0.020).Compared with AGREEⅡ,the study showed a higher level of interraterreliability for AGREE-CHINA overall(ICC=0.957,P<0.001).The mean time required for AGREE-CHINA was less than that for AGREEⅡ:this was approximately 30 minutes for every assessment.User satisfaction was generally high.Conclusions:This paper has presented the first edition of the AGREE-CHINA appraisal tool for clinical guidelines.It is quick and easy to use;it assesses and performs well in comparison to AGREEⅡ.This first version of AGREE-CHINA will require further development and validation.

Reflections on the Use of SSCI Papers in Evaluating Social Sciences Research in Chinese Universities

Purpose:This review demonstrates how to position Social Sciences Citation Index(SSCI)papers reasonably in order to promote the reform and development of the system for evaluating social sciences research(E-SSR)in China.Design/Approach/Methods:This review examines the contributions made by SSCI papers after such papers became a tool in the E-SSR system in Chinese universities,and the resultant issues.This review analyzes documents pertaining to the E-SSR systems of more than 50 world-class universities with consideration to the inherent characteristics and historical mission of social sciences research in China.The findings serve as the basis from which to examine the future trends in the reform of the E-SSR system in Chinese universities.Findings:The application of SSCI papers as an E-SSR tool is not common in world-class universities.To date,the reform of the E-SSR system in Chinese universities has involved:(i)establishing a pluralistic evaluation mechanism,with equal importance placed on SSCI papers and other research achievements;(ii)emphasizing the need for caution in using SSCI papers as an E-SSR tool and instituting distinct treatments for various disciplines;(iii)reducing the importance attached to journal language and ranking,while emphasizing innovation quality and practical contributions;and(iv)establishing China’s E-SSR standards in order to achieve an equilibrium between internationalization and localization.Originality/Value:This review argues that although SSCI papers constitute an indicator of E-SSR system,their importance must not be overstated.The main purpose of the E-SSR system is to facilitate the development of social sciences with a style and characteristics unique to China.