Can ChatGPT evaluate research quality?

Purpose:Assess whether ChatGPT 4.0 is accurate enough to perform research evaluations on journal articles to automate this time-consuming task.Design/methodology/approach:Test the extent to which ChatGPT-4 can assess the quality of journal articles using a case study of the published scoring guidelines of the UK Research Excellence Framework(REF)2021 to create a research evaluation ChatGPT.This was applied to 51 of my own articles and compared against my own quality judgements.Findings:ChatGPT-4 can produce plausible document summaries and quality evaluation rationales that match the REF criteria.Its overall scores have weak correlations with my self-evaluation scores of the same documents(averaging r=0.281 over 15 iterations,with 8 being statistically significantly different from 0).In contrast,the average scores from the 15 iterations produced a statistically significant positive correlation of 0.509.Thus,averaging scores from multiple ChatGPT-4 rounds seems more effective than individual scores.The positive correlation may be due to ChatGPT being able to extract the author’s significance,rigour,and originality claims from inside each paper.If my weakest articles are removed,then the correlation with average scores(r=0.200)falls below statistical significance,suggesting that ChatGPT struggles to make fine-grained evaluations.Research limitations:The data is self-evaluations of a convenience sample of articles from one academic in one field.Practical implications:Overall,ChatGPT does not yet seem to be accurate enough to be trusted for any formal or informal research quality evaluation tasks.Research evaluators,including journal editors,should therefore take steps to control its use.Originality/value:This is the first published attempt at post-publication expert review accuracy testing for ChatGPT.

Can first or last name uniqueness help to identify diaspora researchers from any country?

Purpose:Diaspora researchers work in one country but have ancestral origins in another,either through moves during a research career(mobile diaspora researchers)or by starting research in the target country(embedded diaspora researchers).Whilst mobile researchers might be tracked through affiliation changes in bibliometric databases,embedded researchers cannot.This article reports an evidence-based discussion of which countries’diaspora researchers can be partially tracked using first or last names,addressing this limitation.Design/methodology/approach:A frequency analysis of first and last names of authors of all Scopus journal articles 2001-2021 for 200 countries or regions.Findings:There are great variations in the extent to which first or last names are uniquely national,from Monserrat(no unique first names)to Thailand(81%unique last names).Nevertheless,most countries have a subset of first or last names that are relatively unique.For the 50 countries with the most researchers,authors with relatively national names are always more likely to research their name-associated country,suggesting a continued national association.Lists of researchers’first and last name frequencies and proportions are provided for 200 countries/regions.Research limitations:Only one period is tracked(2001-2021)and no attempt was made to validate the ancestral origins of any researcher.Practical implications:Simple name heuristics can be used to identify the international spread of a sample of most countries’diaspora researchers,but some manual checks of individual names are needed to weed out false matches.This can supplement mobile researcher data from bibliometric databases.Originality/value:This is the first attempt to list name associations for the authors of all countries and large regions,and to identify the countries for which diaspora researchers could be tracked by name.

Is big team research fair in national research assessments? The case of the UK Research Excellence Framework 2021

Collaborative research causes problems for research assessments because of the difficulty in fairly crediting its authors.Whilst splitting the rewards for an article amongst its authors has the greatest surface-level fairness,many important evaluations assign full credit to each author,irrespective of team size.The underlying rationales for this are labour reduction and the need to incentivise collaborative work because it is necessary to solve many important societal problems.This article assesses whether full counting changes results compared to fractional counting in the case of the UK’s Research Excellence Framework(REF)2021.For this assessment,fractional counting reduces the number of journal articles to as little as 10%of the full counting value,depending on the Unit of Assessment(UoA).Despite this large difference,allocating an overall grade point average(GPA)based on full counting or fractional counting gives results with a median Pearson correlation within UoAs of 0.98.The largest changes are for Archaeology(r=0.84)and Physics(r=0.88).There is a weak tendency for higher scoring institutions to lose from fractional counting,with the loss being statistically significant in 5 of the 34 UoAs.Thus,whilst the apparent over-weighting of contributions to collaboratively authored outputs does not seem too problematic from a fairness perspective overall,it may be worth examining in the few UoAs in which it makes the most difference.

“My ADHD Hellbrain”: A Twitter Data Science Perspective on a Behavioural Disorder

Purpose: Attention deficit hyperactivity disorder(ADHD) is a common behavioural condition. This article introduces a new data science method, word association thematic analysis, to investigate whether ADHD tweets can give insights into patient concerns and online communication needs. Design/methodology/approach: Tweets matching "my ADHD"(n=58,893) and 99 other conditions(n=1,341,442) were gathered and two thematic analyses conducted. Analysis 1: A standard thematic analysis of ADHD-related tweets. Analysis 2: A word association thematic analysis of themes unique to ADHD.Findings: The themes that emerged from the two analyses included people ascribing their brains agency to explain and justify their symptoms and using the concept of neurodivergence for a positive self-image. Research limitations: This is a single case study and the results may differ for other topics.Practical implications: Health professionals should be sensitive to patients' needs to understand their behaviour, find ways to justify and explain it to others and to be positive about their condition.Originality/value: Word association thematic analysis can give new insights into the(self-reported) patient perspective.

An Automatic Method to Identify Citations to Journals in News Stories: A Case Study of UK Newspapers Citing Web of Science Journals

Purpose: Communicating scientific results to the public is essential to inspire future researchers and ensure that discoveries are exploited. News stories about research are a key communication pathway for this and have been manually monitored to assess the extent of press coverage of scholarship.Design/methodology/Approach: To make larger scale studies practical, this paper introduces an automatic method to extract citations from newspaper stories to large sets of academic journals. Curated ProQuest queries were used to search for citations to 9,639 Science and3,412 Social Science Web of Science(WoS) journals from eight UK daily newspapers during2006–2015. False matches were automatically filtered out by a new program, with 94% of the remaining stories meaningfully citing research.Findings: Most Science(95%) and Social Science(94%) journals were never cited by these newspapers. Half of the cited Science journals covered medical or health-related topics,whereas 43% of the Social Sciences journals were related to psychiatry or psychology. From the citing news stories, 60% described research extensively and 53% used multiple sources,but few commented on research quality.Research Limitations: The method has only been tested in English and from the ProQuest Newspapers database.Practical implications: Others can use the new method to systematically harvest press coverage of research.Originality/value: An automatic method was introduced and tested to extract citations from newspaper stories to large sets of academic journals.

I’m Nervous about Sharing This Secret with You:Youtube Influencers Generate Strong Parasocial Interactions by Discussing Personal Issues

Purpose:Performers may generate loyalty partly through eliciting illusory personal connections with their audience,parasocial relationships(PSRs),and individual illusory exchanges,parasocial interactions(PSIs).On social media,semi-PSIs are real but imbalanced exchanges with audiences,including through comments on influencers’videos,and strong semi-PSIs are those that occur within PSRs.This article introduces and assesses an automatic method to detect videos with strong PSI potential.Design/methodology/approach:Strong semi-PSIs were hypothesized to occur when commenters used a variant of the pronoun“you”,typically addressing the influencer.Comments on the videos of UK female influencer channels were used to test whether the proportion of you pronoun comments could be an automated indicator of strong PSI potential,and to find factors associating with the strong PSI potential of influencer videos.The highest and lowest strong PSI potential videos for 117 influencers were classified with content analysis for strong PSI potential and evidence of factors that might elicit PSIs.Findings:The you pronoun proportion was effective at indicating video strong PSI potential,the first automated method to detect any type of PSI.Gazing at the camera,head and shoulders framing,discussing personal issues,and focusing on the influencer associated with higher strong PSI potential for influencer videos.New social media factors found include requesting feedback and discussing the channel itself.Research limitations:Only one country,genre and social media platform was analysed.Practical implications:The method can be used to automatically detect YouTube videos with strong PSI potential,helping influencers to monitor their performance.Originality/value:This is the first automatic method to detect any aspect of PSI or PSR.

How Has Covid-19 Affected Published Academic Research?A Content Analysis of Journal Articles Mentioning the Virus

Purpose:Methods to tackle Covid-19 have been developed by a wave of biomedical research but the pandemic has also influenced many aspects of society,generating a need for research into its consequences,and potentially changing the way existing topics are investigated.This article investigates the nature of this influence on the wider academic research mission.Design/methodology/approach:This article reports an inductive content analysis of 500 randomly selected journal articles mentioning Covid-19,as recorded by the Dimensions scholarly database on 19 March 2021.Covid-19 mentions were coded for the influence of the disease on the research.Findings:Whilst two thirds of these articles were about biomedicine(e.g.treatments,vaccines,virology),or health services in response to Covid-19,others covered the pandemic economy,society,safety,or education.In addition,some articles were not about the pandemic but stated that Covid-19 had increased or decreased the value of the reported research or changed the context in which it was conducted.Research limitations:The findings relate only to Covid-19 influences declared in published journal articles.Practical implications:Research managers and funders should consider whether their current procedures are effective in supporting researchers to address the evolving demands of pandemic societies,particularly in terms of timeliness.Originality/value:The results show that although health research dominates the academic response to Covid-19,it is more widely disrupting academic research with new demands and challenges.

Male, Female, and Nonbinary Differences in UK Twitter Self-descriptions: A Fine-grained Systematic Exploration

Purpose: Although gender identities influence how people present themselves on social media, previous studies have tested pre-specified dimensions of difference, potentially overlooking other differences and ignoring nonbinary users.Design/methodology/approach: Word association thematic analysis was used to systematically check for fine-grained statistically significant gender differences in Twitter profile descriptions between 409,487 UK-based female, male, and nonbinary users in 2020. A series of statistical tests systematically identified 1,474 differences at the individual word level, and a follow up thematic analysis grouped these words into themes.Findings: The results reflect offline variations in interests and in jobs. They also show differences in personal disclosures, as reflected by words, with females mentioning qualifications, relationships, pets, and illnesses much more, nonbinaries discussing sexuality more, and males declaring political and sports affiliations more. Other themes were internally imbalanced, including personal appearance(e.g. male: beardy;female: redhead), selfevaluations(e.g. male: legend;nonbinary: witch;female: feisty), and gender identity(e.g. male: dude;nonbinary: enby;female: queen).Research limitations: The methods are affected by linguistic styles and probably underreport nonbinary differences.Practical implications: The gender differences found may inform gender theory, and aid social web communicators and marketers.Originality/value: The results show a much wider range of gender expression differences than previously acknowledged for any social media site.

Data Science Altmetrics

Introduction Within the field of scientometrics,which involves quantitative studies of science,the citation analysis specialism counts citations between academic papers in order to help evaluate the impact of the cited work（Moed,2006）.

Autism Spectrum Disorder on Twitter during COVID-19:Account types, self-descriptions, and tweeting themes

Social media can be used to share experiences of health-related conditions, such as Autism Spectrum Disorder(ASD), and to advertise support. Knowledge about how this occurs can inform those seeking to guide those with the condition. This article reports three related studies to investigate ASD on Twitter in the USA, derived from Covid-19 tweets between March 10 and June 30, 2020. Study 1: Twitter accounts mentioning ASD in author biographies were classified with content analysis by type, finding parents and people declaring ASD to be both common,with support, advocates and specialists also represented. Study 2: The biographies of these accounts were analysed using word association thematic analysis(WATA), finding a strong family relationships theme amongst the parent tweeters, for example. The results also suggested common identity aspects of people declaring ASD, including gaming and artistic interests. Study3: Covid-19-related tweets from the same accounts were analysed using WATA, finding no ASD-specific themes for parents or people declaring ASD. The results suggest that ASD in the USA is represented for Covid-19 through parents, individuals declaring it, and supporters, but without raising concerns particular to the disorder.