83:I[191445,["$4","$5","$6","$7","$8","$9","$a","$b","$c","$d","$e","$f","$10","$11","$12","$13","$14","$15","$16","$17","$18","$19","$1a","$1b","$1c","$1f","$20","$21","$22","$23","$24","$25","$26","$27","$28","$29","$2a","$2b"],"WorkTabProvider"] 84:I[649892,["$4","$5","$6","$7","$8","$9","$a","$b","$c","$d","$e","$f","$10","$11","$12","$13","$14","$15","$16","$17","$18","$19","$1a","$1b","$1c","$1f","$20","$21","$22","$23","$24","$25","$26","$27","$28","$29","$2a","$2b"],"WorkHeader"] 85:T6625,Authors and AffiliationsJeffrey Koury, ResearchHub FoundationDominikus Brian, ResearchHub FoundationAbstractThe peer review process, essential for maintaining scientific rigor, faces significant challenges including reviewer fatigue, bias, and insufficient rigor. The rapid increase in scientific publications has placed an unsustainable burden on a limited pool of reviewers, resulting in the inability of reviewers to detect falsified data, screen errors with high fidelity, and even establish consensus between reviewers [1]. ResearchHub Foundation aims to address these issues by 1) incentivizing reviewers and 2) integrating Agentic AI systems powered by recent breakthroughs in large language and multi-modal foundational models. Our unique approach with validated expert human reviews, and trustworthy automated tools to complement human reviewers, will enhance the quality and credibility of peer reviews to benefit both authors and the larger scientific community.We have observed from our own experiments (200+ AI peer review samples) that AI reviewers have capabilities that excel in domains such as detecting technical errors and statistical flaws, while human reviewers provide greater insight into the broader relevance and context of research. We hypothesize that incorporating human feedback into AI models of peer review can improve their performance beyond existing state-of-the-art AI systems without human feedback. The project has two aims:Establish a Reliable Protocol to Assess Reviewer Performance: Using a standardized rubric, human editors with specific sets of expertise will evaluate AI and human reviews across methods, results, discussion, rigor, novelty, and error detection. AI reviewers are expected to outperform in technical rigor, while humans excel in contextual analysis.Develop a Versatile Agentic Solutions for Scientific Auditing : By leveraging upon the growing number of toolkits for deploying AI Agents driven by LLM, Large multi-modal models (LMMs), long-term memories, and orchestration infrastructure, we ought to develop specialized agentic solutions honed for various aspects of scientific document auditing.This initiative leverages ResearchHub’s extensive reviewer database, standardized prompts, and expertise-weighted evaluations to enhance the peer review process. By addressing systemic issues and integrating AI and human expertise, this project seeks to improve the accuracy, efficiency, and scalability of scientific peer review, supporting the advancement of open and validated science.Introduction and SignificanceScientific Auditing - Peer review is a cornerstone of scientific integrity, serving as the primary mechanism for validating research and granting manuscripts the status of “verified” work. However, the process is increasingly strained due to systemic challenges. Peer reviewers face difficulties in thoroughly assessing statistical analyses, verifying references, and evaluating claims against the broader scientific literature. Despite the critical nature of their role, reviewers are not compensated for their efforts, which often results in diminished rigor, personal biases, superficial feedback, and gaps in understanding key aspects of the studies they assess. These challenges are further exacerbated by the rapid growth in scientific publications, which has significantly increased the workload on an already limited pool of reviewers. The number of scientific publications has increased an estimated 47% from 2016 to 2022 [2]. This combination of rising demand and insufficient support has created a critical bottleneck in the peer review process, highlighting the urgent need for innovative solutions to enhance the accuracy, efficiency, and sustainability of scientific auditing. With the rise of techniques resulting in large terabyte sized datasets (i.e. single-cell RNAseq, BARseq) the complexity for detecting errors and the time demand to do so for human reviewers is amplified. Additionally, this unfortunate combination has resulted in excessive burden on a small number of reviewers resulting in reviewer fatigue. The use of ResearchCoin incentives on ResearchHub has provided the impetus to overcome this apprehension of reviewers to peer review, and provides a new found incentive and motivation. Nonetheless, even with appropriate incentives, human reviewers continue to have blindspots during the review process. Agentic AI solution - With the rapid mass-adoption of LLMs and AI Agents, the ability to holistically assess a manuscript has become feasible at rapid speeds and low costs, providing immensely valuable insights into the paper that often reside within human reviewer’s blind spots. These tools can fill the gaps of human reviewers limitations and provide a more expedited turnaround time on feedback and requested edits, ultimately adding a more holistic set of feedback and context to the academic manuscript. Foundational models (o1, Claude, Gemini, LLaMA, DeepSeek, etc.) have made strides and provide a basis for this approach. With smaller amounts of data, such as peer review data on ResearchHub, it is already possible to substantially leverage these foundation models with minimal additional training data to construct a much more accurate peer review system greater than the sum of its parts. AI reviewers are not a panacea, and it’s becoming clear that human reviewers are still needed, but will have AI reviewers as a powerful supplement [3]. Hypothesis and AimsHypothesis: Published and preprinted papers have errors or issues within the manuscript that AI reviewers can catch that are often overlooked by human reviewers, and integrating human reviewer/editor reinforcement learning can enhance capabilities of the AI review system when domain specific context is essential. Aim 1: Assess the strengths and weaknesses of human reviewers and AI reviewers using a standardized rubric. Human editors will evaluate the outputs of both human reviewers and AI reviewers for methods, results, discussion, rigor, novelty and error detection rate. Anticipated result: Human peer reviewers will be superior in interpreting the implications of the methods and results that are NOT explicitly mentioned in the manuscript, while AI reviewers will be more thorough in assessing statistics, detecting errors etc based on information available explicitly in the manuscript. Aim 2: Develop versatile systems of agentic solutions driven by LLM, LMMs, and AI Agents, empowered by domain specific knowledge, in combination with human evaluation and feedback in producing thorough evaluation of scientific documents. Anticipated result: After tuning and including post-training of foundational models with the human evaluation and feedback, the resulting agentic solutions will perform better than the direct use of benchmark foundational models alone when used to a particular scientific auditing task (manuscript review, editor’s feedback, journal decision, etc.)Methods and DesignCurrently, the use of already existing foundation models equipped with simple prompt engineering has provided a viable starting point for pinpointing errors and giving additional context in scientific manuscripts. We’ve developed a minimum viable system in the form of a web app shown in Fig. 1 below. We also made several observations throughout our pilot experiment in which, holistically, using the same optimized prompt, Gemini 2.0 has been the most robust foundation model in producing consistent results human editors and readers generally consider more useful. This is in part due to the 1M+ context window for token processing, roughly 5-7 times greater than the typical 128k and 280k context windows found in other foundation models. Figure 1. Web-application developed by ResearchHub Foundation using AI agents (AI Editor and AI Peer Reviewers). Figure 2. Proposed ARTEMIS system architecture and workflow design.In Fig. 2 , we present the proposed ARTEMIS system architecture and workflow design, here we follow on with describing the key actions happening within each step of the process: Scientific Document Assigned for Decision-Making.Human users who make decisions based on scientific documents submit audit requests to gain further insights regarding the document and the scientific works it represents.Document Pre-Processing for Metadata Extraction.Depending on the metadata content, a specific full extraction protocol is performed to obtain the maximum amount of information from the document.Full extraction results are sent to the AI Quality Control agent to ensure they meet the required standards and can be used further.(6b, 7b, 8b.) Extracted document content that fails to satisfy the minimum requirements is sent to humans for manual decision-making, annotation of reasons for exclusion, and archiving within the Data Lakehouse. (6a & 6a’.) Extracted documents that pass the QC requirements, either automatically or manually after human consideration, are passed on to the AI Editor.(7a.) The AI Editor generates or orchestrates the assignment of the document to the appropriate AI reviewer(s).(8a.) AI Peer Reviewer(s) generate their assessment of the document and send it to the AI QC agent to ensure correct structure and compliance with the required review standards.All reviews that pass the AI QC check are curated into a raw report ready for human evaluation.The raw report is prepared (and reformatted if necessary) to be presented to humans for Human Evaluation.Human evaluation results are submitted to the AI QC for further checks, including verifying whether the contribution is authentically human and backed by proper login credentials.Human feedback from verified editors or validators is requested to justify the quality of the given human evaluation results.All data collected thus far is curated into a coherent dataset corresponding to the scientific document and the auditing task at hand.The dataset is organized into its respective zones and/or warehouses within the Data Lakehouse.Actionable insights are produced based on the knowledge available within the Data Lakehouse to answer the questions posed in the original audit request. This process can also be further facilitated by interactive Data Librarian/Analyst Agents that help users explore or generate additional insights beyond the original request.Aim 1 - Strengths & Weaknesses of Human vs AI ReviewersThe first stage of the proposal is focused on producing the “Human Wisdom” components described across steps 10 to 13 of the ARTEMIS architecture and work flow (Fig. 2). In the process of producing our human evaluation components of our desired dataset, we will prompt the AI Reviewers and Human Reviewers to perform assessment based on two requirements:Classic ResearchHub Foundation prompt:ResearchHub Foundation bounty prompt (italicized):Your review must be submitted during the 14-day submission window from the day this bounty was initiated.Mention your credentials and include your areas of relevant expertise and limitations in assessing this preprint.Include the version of the preprint you are reviewing (e.g. 1st, 2nd, etc.). Make sure you review the most updated version available.Use the rating system in the "Peer Reviews" tab for all 5 criteria (overall assessment, introduction, methods, results, discussion) but the content within each is flexible (in-line comments can be used instead of a block of text in each section).For each figure, including supplementary material, you must provide a detailed assessment (pros or cons).Plagiarism will not be tolerated. You must comprehensively disclose any use of artificial intelligence (AI) platforms used in the review process. Please refer to our AI Policy for additional details.Currently, ResearchHub processes approximately 170 peer reviews a week (Fig 3a), with 68.7% of all preprints who receive a peer review incentive receiving at least 1 peer review that meets the bar of quality (Fig 3b). Only reviews meeting this minimum bar, as deemed by ResearchHub’s editorial team, will be leveraged for this study.The fields of expertise that ResearchHub peer reviewers specialize in are namely in the biomedical and hard sciences including molecular biology, immunology, oncology, neuroscience, etc (Fig 3c, 3d).Error Detection prompt:Please carefully review the following academic paper and identify any errors, issues, inconsistencies, or questionable reasoning, with a special focus on the mathematical content. In particular, examine the correctness and clarity of all formulas, derivations, and calculations. Check for doubtful hypotheses, shaky assumptions, incorrect mathematical statements, undefined terms, unjustified steps, missing references for key theorems, or discrepancies between the described methods and the results presented. If any part of the argument seems incomplete, unclear, or unconvincing, highlight these sections and explain why they might be problematic. Additionally, look for subtle errors such as misapplied formulas, overlooked boundary conditions, or unsupported assumptions in modeling steps. Make sure to check the figures, captions, and meta data of the paper whenever relevant. Finally, suggest potential ways to strengthen or clarify the mathematical aspects of the paper. { "overall_assessment": "", "mathematical_rigor_and_correctness": { "equation_summary": { "inline_equations": , "display_equations": }, "equation_details" :{ "inline_equations": , "display_equations": , }, "abbreviation_and_symbol_definitions": "", "sign_and_operation_checks": "" }, "issues_list": [ { "issue": "", "severity_rating": , "severity_explanation": "", "suggestion": "" }, { "issue": "", "severity_rating": , "severity_explanation": "", "suggestion": "" } // Repeat for issues 3 to 10 ] } Standardized Scoring Approach for Severity Rating: 1.0 - 2.9: Minor typographical errors or negligible issues that do not affect understanding. 3.0 - 5.9: Moderate issues that may cause some confusion or misunderstandings but are correctable. 6.0 - 8.9: Significant errors that impact the validity of results or the clarity of the paper. 9.0 - 10.0: Critical flaws that fundamentally undermine the paper's conclusions or correctness. In the issues_list, provide a list of the 10 most important or most glaring mistakes or issues, following the standardized scoring approach to maintain objectivity across papers. Be specific to the current paper. **Ensure that your response is formatted as valid JSON only, without any additional text or commentary. At ALL COST, NEVER INCLUDES ```json, in your valid JSON answer. a valid json should start with { and ends with }Figure 3a. Total Number of Completed Peer Reviews per Week on ResearchHub from May 2022 to present. ResearchHub has housed almost 4000 peer reviews at a rate of roughly 170 peer reviews a week. Blue indicates a concerted effort to promote the peer review process on ResearchHub. Figure 3b. Total number of manuscripts separated by if ResearchHub Foundation received peer reviews where we distributed the bounty (green, 68.7%) or didn’t distribute the bounty (red, 31.3%). Green signals a manuscript which received at least 1 peer review that met a minimum threshold of quality to receive the bounty. Red signals a manuscript which either didn’t receive any peer review or received peer review but didn’t meet a minimum threshold of quality or abiding by the stipulations set by ResearchHub Foundation.Figure 3c. Number of Peer Reviews previously completed on ResearchHub stratified by OpenAlex Subfields Figure 3d. Number of Peer Reviews previously completed on ResearchHub stratified by OpenAlex Topics The Human Feedback component will be primarily driven by human editors/validators who will assess the precision and accuracy of the human and AI reviewers based on the claims made. The collection of human feedback will then be used to benchmark and improve the AI-generated reviews. An example of how this benchmarking can be performed is shown in the recent work ReviewEval by Kumar et al.[4] and the references therein. Part of which is already available within the ARTEMIS system under development. With the growth in community interest for open peer reviews, of which ResearchHub has adopted, human validators can quickly view and provide input to the reviews being produced in an open transparent marketplace.[5] The editors/validators will use a flash cards style approach for simple validation. The flash cards approach will appear in several forms, depending on the relevant context. Typical cases would include: Fact-Check, Logic-Check, Math-Check, or Expert Confidence. This alternative approach to consolidating expert feedback is designed as a "bite-sized" task that an expert in the field can answer or reason through without needing to read the entire paper. All context necessary for understanding or providing meaningful feedback on the specific excerpt will be provided on the flashcard. Peer reviewers are assumed to be knowledgeable about the particular topic and their claims will be subject to various weights depending on their background/reputation.ResearchHub’s verification and reputation system: Using OpenAlex, an open repository for scientific data, ResearchHub has ascribed a reputation score for verified users within their various fields of expertise (Fig. 4). This will be used to value or weigh the statements made by various peer reviewers and validators.Reviewer’s track record will be checked based on their past performances and previous editorial feedback, if any available. Additionally, each peer reviewer may need to complete a mini-quiz before receiving a tag for specific expertise, ensuring that they actually possess the required expertise. If the reputation system isn’t sufficient for properly confirming a user’s expertise, the mini-quiz will be employed and include single line domain specific questions, with each question restricted to a 5-10 second countdown to receive an answer.Figure 4. Example of a verified user account on ResearchHub with a quantitative percentile ranking as a “reputation” based on publication history. Alternative Approach: Assuming reviewers and/or editors will be difficult to source for the work, alternatively we can validate the AI reviews against other pre-existing datasets [6]. Additionally, in the context where the flash cards approach doesn’t prove to be a good source of validation, we can have validators assess the reviewers based on an adapted version of the ResearchHub Foundation’s standardized unified rubric (not public, but unified recommendations to reviewers can be found here) to give weighted scores on methods, results, discussion, rigor, novelty and error detection rate. Editors evaluations will be weighted contingent on their reputation score in that field according to ResearchHub’s reputation system.Aim 2 - Develop Versatile Agentic Solutions for Scientific AuditingFunctions described in steps 5 to 9 (Fig. 2) for developing AI agents for scientific auditing, we propose to develop 3 agentic roles and 5 auxiliary tooling functions. The 3 agentic roles will be composed of: Quality Control Agent, AI Reviewer Agent, AI Editor. Each of these roles may be deployed under different context and configuration to achieve the desired performance. For example, this may include an AI Editor in Neuroscience, AI Reviewer in Neuroscience, AI quality control in Neuroscience etc parallelized for a plethora of fields of expertise. The 5 Auxiliary toolings are : Long-Term Memory, Knowledge Base, Visual Understanding and Reasoning, Scientific References, Consistency of Structured Output. The design of these toolings will be made as modular as possible to allow replacement of the tech-stack when necessary to boost performance and efficiency.RigorHuman reviewers can openly review any preprints, allowing for expedited reviews. The presence of a standardized prompt for human and AI reviewers, and standardized rubrics for editors allows for apples to apples direct comparisons on specific components of the reviewers. The verification and reputation v2 system on ResearchHub will be employed to weight evaluations according to the expertise of the individual making the claims. Additionally, prior to this pre-registration, we have performed preliminary studies that attempt to understand the dynamics, economics, and quality of incentivized peer review process of which is being employed on ResearchHub [7]. All three aspects of open peer review submissions addressed above (dynamics, economics, and quality) are highly dependent with at least 9 other coupled factors beyond economic and non-economic incentives (Table 1). Therefore, any correlation found would be contextual and apply only to a certain combination of peer review configurations that involved at least the 9 factors. As it pertains to aim 1 of this pre-registration, we will ensure that the peer review context will be fixed ensuring maximal reproducibility.Table 1. The Nine Identified factors influencing correlation between incentives on the quality of open peer reviews of scientific manuscriptsFactor Influencing Incentive-Quality CorrelationVariables Order of IncentivizationAward First (Commissioned Task)Award Later (Rewarded)Bidding Mechanism (Stake for Chance)Reviewer Pool Size Small [90%)Time RestrictionIntense (15 Days) BudgetTable 2. Budget description and breakdown for an estimated 12 month timeline. SectionNeedDetailsCostDevelopmentComputeAPIOutsourced developer work for data cleaning and UI/UX designDigital Ocean, o1/claude/gemini APIs, GPUs, observability system, auxiliary tools for AI Agents, and vector storage, etc.$10,000Human reviewer costs Incentive for Peer Reviewers for ResearchHub Foundation selected papersIncentive for Peer Reviewers for YesNoError (YNE) selected papers - 50 papersIncentive for Peer Reviewers for Gareth and Maria’s selected papers - 50 papers Preprints in the fields of biology, chemistry, neuroscience, immunology, genetics, molecular biology, biochemistry, oncology, mathematics, physics, computer science, etc $150,000 for 1000 human reviews (assuming $150 per peer review and 1 review per preprint)Human editor costsIncentive for Validators (Editors or Authors) to assess AI and Human Reviewer OutputsValidators will be asked to provide either full or partial feedback on both the original preprint and/or the AI review submissions.$50,000 for 1000 editors (assuming $50 to assess all peer reviews per manuscript) Administrative CostMarketing The cost associated with marketing efforts to ensure campaign visibility and sustain interest for contributions from the community.$5,000Reserve FundFor unexpected needs and for insurance against volatility. $15,000, If unused, will be put toward getting a larger dataset. Disclosure and Licensing StatementThis document outlines the terms under which the content, outputs, and tools resulting from our study on AI reviewers for scientific manuscripts are shared and utilized. Transparency and collaboration are core to our research philosophy, and we aim to strike a balance between open science principles and the potential for practical, real-world applications.Open Source CommitmentContent and Outputs:All data, algorithms, models, and other outputs derived from this study will be made available under an open-source license, allowing unrestricted access for academic, research, and non-commercial purposes.The specific license will be the MIT License to ensure clarity regarding permissible uses and attribution requirements.Repository Access:Public repositories (e.g., on GitHub, Zenodo, HuggingFace or similar platforms) will host the content and outputs, accompanied by detailed documentation to facilitate their reuse and extension by the broader research community.Commercialization ClauseWhile the content and outputs of this study will be open-sourced, we reserve the right to commercialize tools or technologies derived from these resources. By participating in this study or utilizing its outputs, users acknowledge and agree to the terms outlined in this disclosure and licensing statement.Declaration of Generative AI in Scientific WritingDuring the preparation of this work, ChatGPT was used to enhance the readability and language of some parts of the manuscript. Additionally, it was used to consolidate the original content written in the introduction, methods, and preliminary data to construct an Abstract. Following any ChatGPT usage, content was reviewed and edited as needed.AcknowledgementsWe would like to acknowledge Matt Schlicht, founder of YesNoError for being a Strategic Data Partner and Gareth Dyke & Maria Machado as Strategic Expert Community Partners. References[1] Howard Bauchner, Frederick P Rivara, Use of artificial intelligence and the future of peer review, Health Affairs Scholar, Volume 2, Issue 5, May 2024, qxae058, https://doi.org/10.1093/haschl/qxae058[2] Hanson, M. A., Gómez Barreiro, P., Crosetto, P., & Brockington, D. (2024). The strain on scientific publishing. arXiv. https://arxiv.org/abs/2309.15884v2[3] Seghier, M.L. (2025), "AI-powered peer review needs human supervision", Journal of Information, Communication and Ethics in Society, Vol. 23 No. 1, pp. 104-116. https://doi.org/10.1108/JICES-09-2024-0132[4] Kirtani, C., Garg, M. K., Prasad, T., Singhal, T., Mandal, M., & Kumar, D. (2025). ReviewEval: An evaluation framework for AI-generated reviews. arXiv. https://arxiv.org/abs/2502.11736v2[5] Yang, J. (2025). Paper Copilot: The artificial intelligence and machine learning community should adopt a more transparent and regulated peer review process. arXiv. https://arxiv.org/abs/2502.00874[6] Kuznetsov, I., Gurevych, I. et. a. (2024). What can natural language processing do for peer review? arXiv. https://arxiv.org/abs/2405.06563 GitHub Repository: Afzal, O. M. (n.d.). nlp-for-peer-review. GitHub. https://github.com/OAfzal/nlp-for-peer-review[7] Brian, D. (2024). Incentivized vs non-incentivized open peer reviews: Dynamics, economics, and quality. ResearchHub. https://doi.org/10.55277/ResearchHub.taescjxh86:T607e,ARTEMIS - Automated Review and Trustworthy Evaluation for Manuscripts in Science Authors and Affiliations Jeffrey Koury1, Dominikus Brian1 1 ResearchHub Foundation Abstract The peer review process, essential for maintaining scientific rigor, faces significant challenges including reviewer fatigue, bias, and insufficient rigor. The rapid increase in scientific publications has placed a burden on a limited pool of editors and reviewers, resulting in the inability of reviewers to efficiently detect falsified data, screen errors with high fidelity, and even establish consensus between reviewers [1]. ResearchHub Foundation aims to address these issues by 1) incentivizing reviewers and 2) integrating Agentic AI systems powered by recent breakthroughs in large language and multi-modal foundational models. Our unique approach with validated expert human reviews, and trustworthy automated tools to complement human reviewers, will enhance the quality and credibility of peer reviews to benefit both authors and the larger scientific community. We have observed from our own experiments (200+ AI peer review samples) that AI reviewers have capabilities that excel in domains such as detecting technical errors and statistical flaws, while human reviewers provide greater insight into the broader relevance and context of research. We hypothesize that incorporating human feedback into AI models of peer review can improve their performance beyond existing state-of-the-art AI systems without human feedback. The project has two aims: Establish a Reliable Protocol to Assess Reviewer Performance: Using a standardized rubric, human editors with specific sets of expertise will evaluate AI and human reviews across introduction, methods, results, discussion, and figure analysis. AI reviewers are expected to outperform in technical rigor, while humans excel in contextual analysis. Develop a Versatile Agentic Solutions for Scientific Auditing : By leveraging upon the growing number of toolkits for deploying AI Agents driven by LLM, Large multi-modal models (LMMs), long-term memories, and orchestration infrastructure, we will seek to develop specialized agentic solutions honed for various aspects of scientific document auditing. This initiative leverages ResearchHub’s extensive reviewer database, standardized prompts, and expertise-weighted evaluations to enhance the peer review process. By addressing systemic issues and integrating AI and human expertise, this project seeks to improve the accuracy, efficiency, and scalability of scientific peer review, supporting the advancement of open science and minimizing exhaustion for human reviewers. Introduction and Significance Scientific Auditing - Peer review is a cornerstone of scientific integrity, serving as the primary mechanism for validating research and granting manuscripts the status of “verified” work. Peer reviewers face difficulties in thoroughly assessing statistical analyses, verifying quotation errors, and evaluating claims in a time efficient manner [2, 3]. These challenges are further exacerbated by the rapid growth in scientific publications, which has significantly increased the workload on an already limited pool of reviewers and editors. The number of scientific publications has increased an estimated 47% from 2016 to 2022 [4]. Despite the critical nature of their role, reviewers (excluding domain specific reviewers such as methods and statistics reviewers for example), are usually not monetarily compensated for their efforts. 87.5% (n=320) of survey participants have reported never being paid for peer review [5]. This combination of rising publications and insufficient monetary support has created a critical bottleneck in the peer review process, including burdens on editors, highlighting the urgent need for innovative solutions to enhance the accuracy, efficiency, and sustainability of scientific auditing [6]. With the rise of techniques resulting in large terabyte sized datasets (i.e. single-cell RNAseq, BARseq) the complexity for detecting errors and the time demand to do so for human reviewers is amplified. The use of ResearchCoin incentives on ResearchHub has provided the impetus to overcome this apprehension of reviewers to expeditiously peer review, and provides a new found incentive and motivation. Nonetheless, even with appropriate incentives, human reviewers continue to have difficulties assessing statistical analyses, verifying quotation errors and identifying other issues during the review process. Agentic AI solution - With the rapid mass-adoption of LLMs and AI Agents, the ability to holistically assess a manuscript has become feasible at rapid speeds and low costs, providing immensely valuable insights into the paper that often reside within human reviewer’s blind spots. These tools can fill the gaps of human reviewers limitations and provide a more expedited turnaround time on feedback and requested edits, ultimately adding a more holistic set of feedback and context to the academic manuscript. Foundational models (o1, Claude, Gemini, LLaMA, DeepSeek, etc.) have made strides and provide a basis for this approach. With smaller amounts of data, such as peer review data on ResearchHub, it is already possible to substantially leverage these foundation models with minimal additional training data to construct a much more accurate peer review system greater than the sum of its parts. AI reviewers are not a panacea, and it’s becoming clear that human reviewers are still needed, but will have AI reviewers as a powerful supplement [7]. Hypothesis and Aims Hypothesis: Published and preprinted papers have errors or issues within the manuscript that AI reviewers can catch that are often not caught by human reviewers, and integrating human reviewer/editor reinforcement learning can enhance capabilities of the AI review system when domain specific context is essential. Aim 1: Assess the strengths and weaknesses of human reviewers and AI reviewers using a standardized rubric. Human editors will evaluate the outputs of both human reviewers and AI reviewers for methods, results, discussion, rigor, novelty and error detection rate. Anticipated result: Human peer reviewers will be superior in interpreting the implications of the methods and results that are NOT explicitly mentioned in the manuscript, while AI reviewers will be more thorough in assessing statistics, detecting errors etc based on information available explicitly in the manuscript. Aim 2: Develop versatile systems of agentic solutions driven by LLM, LMMs, and AI Agents, empowered by domain specific knowledge, in combination with human evaluation and feedback in producing thorough evaluation of scientific documents. Anticipated result: After tuning and including post-training of foundational models with the human evaluation and feedback, the resulting agentic solutions will perform better than the direct use of benchmark foundational models alone when used to a particular scientific auditing task (manuscript review, editor’s feedback, journal decision, etc.) Methods and Design Currently, the use of already existing foundation models equipped with simple prompt engineering has provided a viable starting point for pinpointing errors and giving additional context in scientific manuscripts. We’ve developed a minimum viable system in the form of a web app shown in Fig. 1 below. We also made several observations throughout our pilot experiment in which, holistically, using the same optimized prompt, Gemini 2.0 has been the most robust foundation model in producing consistent results human editors and readers generally consider more useful (data not shown). This may be in part due to the 1M+ context window for token processing, roughly 5-7 times greater than the typical 128k and 280k context windows found in other foundation models. Figure 1. Web-application developed by ResearchHub Foundation using AI agents (AI Editor and AI Peer Reviewers). Figure 2. Proposed ARTEMIS system architecture and workflow design. In Fig. 2 , we present the proposed ARTEMIS system architecture and workflow design, here we follow on with describing the key actions happening within each step of the process: Scientific Document Assigned for Decision-Making. Human users who make decisions based on scientific documents submit audit requests to gain further insights regarding the document and the scientific works it represents. Document Pre-Processing for Metadata Extraction. Depending on the metadata content, a specific full extraction protocol is performed to obtain the maximum amount of information from the document. Full extraction results are sent to the AI Quality Control agent to ensure they meet the required standards and can be used further. (6b, 7b, 8b.) Extracted document content that fails to satisfy the minimum requirements is sent to humans for manual decision-making, annotation of reasons for exclusion, and archiving within the Data Lakehouse. (6a & 6a’.) Extracted documents that pass the QC requirements, either automatically or manually after human consideration, are passed on to the AI Editor. (7a.) The AI Editor generates or orchestrates the assignment of the document to the appropriate AI reviewer(s). (8a.) AI Peer Reviewer(s) generate their assessment of the document and send it to the AI QC agent to ensure correct structure and compliance with the required review standards. All reviews that pass the AI QC check are curated into a raw report ready for human evaluation. The raw report is prepared (and reformatted if necessary) to be presented to humans for Human Evaluation. Human evaluation results are submitted to the AI QC for further checks, including verifying whether the contribution is authentically human and backed by proper login credentials. Human feedback from verified editors or validators is requested to justify the quality of the given human evaluation results. All data collected thus far is curated into a coherent dataset corresponding to the scientific document and the auditing task at hand. The dataset is organized into its respective zones and/or warehouses within the Data Lakehouse. Actionable insights are produced based on the knowledge available within the Data Lakehouse to answer the questions posed in the original audit request. This process can also be further facilitated by interactive Data Librarian/Analyst Agents that help users explore or generate additional insights beyond the original request. Aim 1 - Strengths & Weaknesses of Human vs AI Reviewers The first stage of the proposal is focused on producing the “Human Wisdom” components described across steps 10 to 13 of the ARTEMIS architecture and work flow (Fig. 2). In the process of producing our human evaluation components of our desired dataset, we will prompt the AI Reviewers and Human Reviewers to perform assessment based on two requirements: Classic ResearchHub Foundation prompt: ResearchHub Foundation bounty prompt (italicized): Your review must be submitted during the 14-day submission window from the day this bounty was initiated. Mention your credentials and include your areas of relevant expertise and limitations in assessing this preprint. Include the version of the preprint you are reviewing (e.g. 1st, 2nd, etc.). Make sure you review the most updated version available. Use the rating system in the "Peer Reviews" tab for all 5 criteria (overall assessment, introduction, methods, results, discussion) but the content within each is flexible (in-line comments can be used instead of a block of text in each section). For each figure, including supplementary material, you must provide a detailed assessment (pros or cons). Plagiarism will not be tolerated. You must comprehensively disclose any use of artificial intelligence (AI) platforms used in the review process. Please refer to our AI Policy for additional details. ResearchHub houses many reviewers with expertise in biomedical sciences and hard sciences including molecular biology, immunology, oncology, neuroscience, etc (Fig 3a, 3b).. These same set of reviewers can be validators when discerning human vs AI reviews. Figure 3a. Number of Peer Reviews previously completed on ResearchHub stratified by OpenAlex Subfields. Figure 3b. Number of Peer Reviews previously completed on ResearchHub stratified by OpenAlex Topics. The Human Feedback component will be primarily driven by human validators who will assess the precision and accuracy of the human and AI reviewers based on the claims made. The collection of human feedback will then be used to benchmark and improve the AI-generated reviews. An example of how this benchmarking can be performed is shown in the recent work ReviewEval by Kumar et al. and the references therein [8]. Partially implemented within the currently available ARTEMIS system. With the growth in community interest for open peer reviews, of which ResearchHub has adopted, human validators can quickly view and provide input to the reviews being produced in an open transparent marketplace [9]. The validators will use a flash cards style approach for simple validation. The flash cards approach will appear in several forms, depending on the relevant context. Typical cases would include: Fact-Check, Logic-Check, Math-Check, or Expert Confidence. This approach to consolidating expert feedback is designed as a "bite-sized" task that an expert in the field can answer or reason through without needing to read the entire paper. All context necessary for understanding or providing meaningful feedback on the specific excerpt will be provided on the flashcard. Peer reviewer’s claims will be subject to various weights depending on their background/reputation. ResearchHub’s verification and reputation system: Using OpenAlex, an open repository for scientific data, ResearchHub has ascribed a reputation score for verified users within their various fields of expertise (Fig. 4). This will be used to value or weigh the statements made by various peer reviewers and validators. Reviewer’s track record will be checked based on their past performances and previous editorial feedback, if any available. Additionally, each peer reviewer may need to complete a mini-quiz before receiving a tag for specific expertise, ensuring that they actually possess the required expertise. If the reputation system isn’t sufficient for properly confirming a user’s expertise, the mini-quiz will be employed and include single line domain specific questions, with each question restricted to a 5-10 second countdown to receive an answer. Figure 4. Example of a verified user account on ResearchHub with a quantitative percentile ranking as a “reputation” based on publication history. Alternative Approach: If reviewers and/or validators will be difficult to source for the work, alternatively we can validate the AI reviews against other pre-existing datasets [10]. Additionally, in the context where the flash cards approach doesn’t prove to be a good source of validation, we can have validators assess the reviewers based on an adapted version of the ResearchHub Foundation’s standardized unified rubric (not public, but unified recommendations to reviewers can be found here) to give weighted scores on methods, results, discussion, rigor, novelty and error detection rate. Editors evaluations will be weighted contingent on their reputation score in that field according to ResearchHub’s reputation system. Aim 2 - Develop Versatile Agentic Solutions for Scientific Auditing Functions described in steps 5 to 9 (Fig. 2) for developing AI agents for scientific auditing, we propose to develop 3 agentic roles and 5 auxiliary tooling functions. The 3 agentic roles will be composed of: Quality Control Agent, AI Reviewer Agent, AI Editor. Each of these roles may be deployed under different context and configuration to achieve the desired performance. For example, this may include an AI Editor in Neuroscience, AI Reviewer in Neuroscience, AI quality control in Neuroscience etc parallelized for a plethora of fields of expertise. The 5 Auxiliary toolings are : Long-Term Memory, Knowledge Base, Visual Understanding and Reasoning, Scientific References, Consistency of Structured Output. The design of these toolings will be made as modular as possible to allow replacement of the tech-stack when necessary to boost performance and efficiency. Rigor Human reviewers can openly review any preprints, allowing for expedited reviews. The presence of a standardized prompt for human and AI reviewers, and standardized rubrics for editors allows for apples to apples direct comparisons on specific components of the reviewers. The verification and reputation v2 system on ResearchHub will be employed to weight evaluations according to the expertise of the individual making the claims. Additionally, prior to this pre-registration, we have performed preliminary studies that attempt to understand the dynamics, economics, and quality of incentivized peer review process of which is being employed on ResearchHub [11]. All three aspects of open peer review submissions addressed above (dynamics, economics, and quality) are highly dependent with at least 9 other coupled factors beyond economic and non-economic incentives (Table 1). Therefore, any correlation found would be contextual and apply only to a certain combination of peer review configurations that involved at least the 9 factors. As it pertains to aim 1 of this pre-registration, we will ensure that the peer review context will be fixed ensuring maximal reproducibility. Table 1. The Nine Identified factors influencing correlation between incentives on the quality of open peer reviews of scientific manuscripts. Factor Influencing Incentive-Quality Correlation Variables Order of Incentivization Award First (Commissioned Task) Award Later (Rewarded) Bidding Mechanism (Stake for Chance) Reviewer Pool Size Small [<100] Medium [100-1000] Large [1000+] Expertise Diversity Ranging from 0 to 1 0: All reviewers are from the exact same research subfield 1: For each subfield there are exactly equal number of reviewer * Each reviewer can have more than 1 expertise, but maximum 3 notable expertise. Complete list of subfields, will be at most 3 x number of reviewers in the pool. Reviewer Selection Criteria Single Chief-Editor (or Campaign Host) make the final call First Come First Served [Based on Objective Quality Scores or Time Order] Democratic Decision [Community Votes or Expert Scoring] Type of Incentive and Size/Intensity Monetary Reputation Monetary + Reputation Requested Review Structure Simple (Yes/No answer and inline comments) Moderate (Structured Comments on Different Aspects) Rigor (Complete Assessment on All Sections, Figures, References, and more) Difficulties of Review Task Easy (General Knowledgeable Public) Medium (Graduate Student Level on the Topic) Hard (Subject Expert on the Topic) Likelihood & Proportion of AI Usage Low (<10%) Moderate (10-90%) High (>90%) Time Restriction Intense (<3 Days) Moderate (3-14 Days) Leisurely ( >15 Days) Budget Table 2. Budget description and breakdown for an estimated 12 month timeline. Section Need Details Cost Development Compute API Outsourced developer work for data cleaning and UI/UX design Digital Ocean, o1/claude/gemini APIs, GPUs, observability system, auxiliary tools for AI Agents, and vector storage, etc. $10,000 Human editor costs Incentive for Validators (Editors or Authors) to assess AI and Human Reviewer Outputs. Including incentives for validating papers, N = 50, from YesNoError (YNE) Validators will be asked to provide either full or partial feedback on both the original preprint and/or the AI review submissions. $50,000 for 1000 editors (assuming $50 to assess all peer reviews per manuscript) Administrative Cost Marketing The cost associated with marketing efforts to ensure campaign visibility and sustain interest for contributions from the community. $5,000 Reserve Fund For unexpected needs and for insurance against volatility. $15,000, If unused, will be put toward getting a larger dataset. Disclosure and Licensing Statement This document outlines the terms under which the content, outputs, and tools resulting from our study on AI reviewers for scientific manuscripts are shared and utilized. Transparency and collaboration are core to our research philosophy, and we aim to strike a balance between open science principles and the potential for practical, real-world applications. Open Source Commitment Content and Outputs: All data, algorithms, models, and other outputs derived from this study will be made available under an open-source license, allowing unrestricted access for academic, research, and non-commercial purposes. The specific license will be the MIT License to ensure clarity regarding permissible uses and attribution requirements. Repository Access: Public repositories (e.g., on GitHub, Zenodo, HuggingFace or similar platforms) will host the content and outputs, accompanied by detailed documentation to facilitate their reuse and extension by the broader research community. Commercialization Clause While the content and outputs of this study will be open-sourced, we reserve the right to commercialize tools or technologies derived from these resources. By participating in this study or utilizing its outputs, users acknowledge and agree to the terms outlined in this disclosure and licensing statement. Declaration of Generative AI in Scientific Writing During the preparation of this work, ChatGPT was used to enhance the readability and language of some parts of the manuscript. Additionally, it was used to consolidate the original content written in the introduction, methods, and preliminary data to construct an Abstract. Following any ChatGPT usage, content was reviewed and edited as needed. Acknowledgements We would like to acknowledge Matt Schlicht, founder of YesNoError for being a Strategic Data Partner and Gareth Dyke & Maria Machado as Strategic Expert Community Partners. References [1] Howard Bauchner, Frederick P Rivara, Use of artificial intelligence and the future of peer review, Health Affairs Scholar, Volume 2, Issue 5, May 2024, qxae058, https://doi.org/10.1093/haschl/qxae058 [2] Aczel B, Barwich AS, Diekman AB, et al. The present and future of peer review: Ideas, interventions, and evidence. Proc Natl Acad Sci U S A. 2025;122(5):e2401232121. doi:10.1073/pnas.2401232121 [3] Smith Nail and Cumberledge Aaron. 2020 Quotation errors in general science journals Proc. R. Soc. A. 47620200538. http://doi.org/10.1098/rspa.2020.0538 [4] Hanson, M. A., Gómez Barreiro, P., Crosetto, P., & Brockington, D. (2024). The strain on scientific publishing. arXiv. https://arxiv.org/abs/2309.15884v2 [5] LeBlanc, A.G., Barnes, J.D., Saunders, T.J. et al. Scientific sinkhole: estimating the cost of peer review based on survey data with snowball sampling. Res Integr Peer Rev 8, 3 (2023). https://doi.org/10.1186/s41073-023-00128-2 [6] https://www.biorxiv.org/content/10.1101/2025.03.18.644032v1.full.pdf [7] Seghier, M.L. (2025), "AI-powered peer review needs human supervision", Journal of Information, Communication and Ethics in Society, Vol. 23 No. 1, pp. 104-116. https://doi.org/10.1108/JICES-09-2024-0132 [8] Kirtani, C., Garg, M. K., Prasad, T., Singhal, T., Mandal, M., & Kumar, D. (2025). ReviewEval: An evaluation framework for AI-generated reviews. arXiv. https://arxiv.org/abs/2502.11736v2 [9] Yang, J. (2025). Paper Copilot: The artificial intelligence and machine learning community should adopt a more transparent and regulated peer review process. arXiv. https://arxiv.org/abs/2502.00874 [10] Kuznetsov, I., Gurevych, I. et. a. (2024). What can natural language processing do for peer review? arXiv. https://arxiv.org/abs/2405.06563 GitHub Repository: Afzal, O. M. (n.d.). nlp-for-peer-review. GitHub. https://github.com/OAfzal/nlp-for-peer-review [11] Brian, D. (2024). Incentivized vs non-incentivized open peer reviews: Dynamics, economics, and quality. ResearchHub. https://doi.org/10.55277/ResearchHub.taescjxh 87:T100a7,{"type":"doc","content":[{"type":"heading","attrs":{"id":"8c038a7e-4227-4ec0-9963-7222b0bed545","data-toc-id":"8c038a7e-4227-4ec0-9963-7222b0bed545","textAlign":null,"level":1},"content":[{"type":"text","text":"ARTEMIS - Automated Review and Trustworthy Evaluation for Manuscripts in Science"}]},{"type":"heading","attrs":{"id":"81fb7381-2743-4bfd-b8ab-754c0d398c13","data-toc-id":"81fb7381-2743-4bfd-b8ab-754c0d398c13","textAlign":null,"level":2},"content":[{"type":"text","text":"Authors and Affiliations"}]},{"type":"paragraph","attrs":{"id":"68586fb8-71c7-4ad2-8f22-3385791b5b9d","class":null,"textAlign":null},"content":[{"type":"text","text":"Jeffrey Koury"},{"type":"text","marks":[{"type":"superscript"}],"text":"1"},{"type":"text","text":", Dominikus Brian"},{"type":"text","marks":[{"type":"superscript"}],"text":"1"},{"type":"text","text":" "},{"type":"hardBreak"},{"type":"hardBreak"},{"type":"text","marks":[{"type":"superscript"}],"text":"1"},{"type":"text","text":" ResearchHub Foundation"}]},{"type":"heading","attrs":{"id":"9fd38a75-c58f-433a-8bc7-0b232a40e192","data-toc-id":"9fd38a75-c58f-433a-8bc7-0b232a40e192","textAlign":null,"level":2},"content":[{"type":"text","text":"Abstract"}]},{"type":"paragraph","attrs":{"id":"abca777a-b88b-4bec-b25f-cc32d669003f","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"The peer review process, essential for maintaining scientific rigor, faces significant challenges including reviewer fatigue, bias, and insufficient rigor. The rapid increase in scientific publications has placed a burden on a limited pool of editors and reviewers, resulting in the inability of reviewers to "},{"type":"text","marks":[{"type":"italic"}],"text":"efficiently"},{"type":"text","text":" detect falsified data, screen errors with high fidelity, and even establish consensus between reviewers [1]. ResearchHub Foundation aims to address these issues by 1) incentivizing reviewers and 2) integrating Agentic AI systems powered by recent breakthroughs in large language and multi-modal foundational models. Our unique approach with validated expert human reviews, and trustworthy automated tools to complement human reviewers, will enhance the quality and credibility of peer reviews to benefit both authors and the larger scientific community."}]},{"type":"paragraph","attrs":{"id":"19293bde-95e9-4b41-be4e-7d25fa380808","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"We have observed from our own experiments (200+ AI peer review samples) that AI reviewers have capabilities that excel in domains such as detecting technical errors and statistical flaws, while human reviewers provide greater insight into the broader relevance and context of research. We hypothesize that incorporating human feedback into AI models of peer review can improve their performance beyond existing state-of-the-art AI systems without human feedback. The project has two aims:"}]},{"type":"orderedList","attrs":{"start":1},"content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"b2217a67-1853-429b-ae21-14f231611507","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Establish a Reliable Protocol to Assess Reviewer Performance"},{"type":"text","text":": Using a standardized rubric, human editors with specific sets of expertise will evaluate AI and human reviews across introduction, methods, results, discussion, and figure analysis. AI reviewers are expected to outperform in technical rigor, while humans excel in contextual analysis."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"e5481b4c-d192-43be-89c2-86e48d79c067","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Develop a Versatile Agentic Solutions for Scientific Auditing "},{"type":"text","text":": By leveraging upon the growing number of toolkits for deploying AI Agents driven by LLM, Large multi-modal models (LMMs), long-term memories, and orchestration infrastructure, we will seek to develop specialized agentic solutions honed for various aspects of scientific document auditing."}]}]}]},{"type":"paragraph","attrs":{"id":"ae1d4a34-bb4e-4b9d-8d01-a810e3627438","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"This initiative leverages ResearchHub’s extensive reviewer database, standardized prompts, and expertise-weighted evaluations to enhance the peer review process. By addressing systemic issues and integrating AI and human expertise, this project seeks to improve the accuracy, efficiency, and scalability of scientific peer review, supporting the advancement of open science and minimizing exhaustion for human reviewers."}]},{"type":"heading","attrs":{"id":"cf4cc6b9-33c0-4ddf-a261-65dafde77a1e","data-toc-id":"cf4cc6b9-33c0-4ddf-a261-65dafde77a1e","textAlign":"justify","level":2},"content":[{"type":"text","text":"Introduction and Significance"}]},{"type":"paragraph","attrs":{"id":"b5809a87-5dce-47bc-8b47-3e98f385d567","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Scientific Auditing -"},{"type":"text","text":" Peer review is a cornerstone of scientific integrity, serving as the primary mechanism for validating research and granting manuscripts the status of “verified” work. Peer reviewers face difficulties in thoroughly assessing statistical analyses, verifying quotation errors, and evaluating claims in a time efficient manner [2, 3]. These challenges are further exacerbated by the rapid growth in scientific publications, which has significantly increased the workload on an already limited pool of reviewers and editors. The number of scientific publications has increased an estimated 47% from 2016 to 2022 [4]. Despite the critical nature of their role, reviewers (excluding domain specific reviewers such as methods and statistics reviewers for example), are usually not monetarily compensated for their efforts. 87.5% (n=320) of survey participants have reported never being paid for peer review [5]. This combination of rising publications and insufficient monetary support has created a critical bottleneck in the peer review process, including burdens on editors, highlighting the urgent need for innovative solutions to enhance the accuracy, efficiency, and sustainability of scientific auditing [6]. With the rise of techniques resulting in large terabyte sized datasets (i.e. single-cell RNAseq, BARseq) the complexity for detecting errors and the time demand to do so for human reviewers is amplified. The use of ResearchCoin incentives on ResearchHub has provided the impetus to overcome this apprehension of reviewers to expeditiously peer review, and provides a new found incentive and motivation. Nonetheless, even with appropriate incentives, human reviewers continue to have difficulties assessing statistical analyses, verifying quotation errors and identifying other issues during the review process. "}]},{"type":"paragraph","attrs":{"id":"818fa2d4-c55c-4189-a133-cf9913aea48f","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Agentic AI solution - "},{"type":"text","text":"With the rapid mass-adoption of LLMs and AI Agents, the ability to holistically assess a manuscript has become feasible at rapid speeds and low costs, providing immensely valuable insights into the paper that often reside within human reviewer’s blind spots. These tools can fill the gaps of human reviewers limitations and provide a more expedited turnaround time on feedback and requested edits, ultimately adding a more holistic set of feedback and context to the academic manuscript. Foundational models (o1, Claude, Gemini, LLaMA, DeepSeek, etc.) have made strides and provide a basis for this approach. With smaller amounts of data, such as peer review data on ResearchHub, it is already possible to substantially leverage these foundation models with minimal additional training data to construct a much more accurate peer review system greater than the sum of its parts. AI reviewers are not a panacea, and it’s becoming clear that human reviewers are still needed, but will have AI reviewers as a powerful supplement [7]. "}]},{"type":"heading","attrs":{"id":"c840d50a-64fd-47c2-8cc2-198a24f2fe56","data-toc-id":"c840d50a-64fd-47c2-8cc2-198a24f2fe56","textAlign":"justify","level":2},"content":[{"type":"text","text":"Hypothesis and Aims"}]},{"type":"paragraph","attrs":{"id":"0c2801de-fc45-4310-adb1-414a721eff5a","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Hypothesis: "},{"type":"text","text":"Published and preprinted papers have errors or issues within the manuscript that AI reviewers can catch that are often not caught by human reviewers, and integrating human reviewer/editor reinforcement learning can enhance capabilities of the AI review system when domain specific context is essential. "}]},{"type":"paragraph","attrs":{"id":"8e728bb1-77a5-4166-bd11-12271de71e67","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Aim 1:"},{"type":"text","text":" Assess the strengths and weaknesses of human reviewers and AI reviewers using a standardized rubric. Human editors will evaluate the outputs of both human reviewers and AI reviewers for methods, results, discussion, rigor, novelty and error detection rate. "},{"type":"text","marks":[{"type":"underline"}],"text":"Anticipated result"},{"type":"text","text":": Human peer reviewers will be superior in interpreting the implications of the methods and results that are NOT explicitly mentioned in the manuscript, while AI reviewers will be more thorough in assessing statistics, detecting errors etc based on information available explicitly in the manuscript. "}]},{"type":"paragraph","attrs":{"id":"68bff7e6-7dce-4542-96e1-0dc0100b8e33","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Aim 2: "},{"type":"text","text":"Develop versatile systems of agentic solutions driven by LLM, LMMs, and AI Agents, empowered by domain specific knowledge, in combination with human evaluation and feedback in producing thorough evaluation of scientific documents. "},{"type":"text","marks":[{"type":"underline"}],"text":"Anticipated result"},{"type":"text","text":": After tuning and including post-training of foundational models with the human evaluation and feedback, the resulting agentic solutions will perform better than the direct use of benchmark foundational models alone when used to a particular scientific auditing task (manuscript review, editor’s feedback, journal decision, etc.)"}]},{"type":"heading","attrs":{"id":"0ff95663-7735-4089-97dc-858bed34ef57","data-toc-id":"0ff95663-7735-4089-97dc-858bed34ef57","textAlign":"justify","level":2},"content":[{"type":"text","text":"Methods and Design"}]},{"type":"paragraph","attrs":{"id":"ae78d807-0af6-4b18-b84a-c2b06bc42f45","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"Currently, the use of already existing foundation models equipped with simple prompt engineering has provided a viable starting point for pinpointing errors and giving additional context in scientific manuscripts. We’ve developed a minimum viable system in the form of a web app shown in "},{"type":"text","marks":[{"type":"bold"}],"text":"Fig. 1"},{"type":"text","text":" below. We also made several observations throughout our pilot experiment in which, holistically, using the same optimized prompt, Gemini 2.0 has been the most robust foundation model in producing consistent results human editors and readers generally consider more useful (data not shown). This "},{"type":"text","marks":[{"type":"italic"}],"text":"may"},{"type":"text","text":" be in part due to the 1M+ context window for token processing, roughly 5-7 times greater than the typical 128k and 280k context windows found in other foundation models. "}]},{"type":"imageBlock","attrs":{"src":"https://storage.prod.researchhub.com/uploads/notes/users/34416/780924e3-bd10-44d8-9c95-fb7a146bd58c/screenshot-2025-04-02-at-3.png","width":"100%","align":"center"}},{"type":"paragraph","attrs":{"id":"4b70b38e-7d2d-4b2f-a5b5-4281c9d54e37","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Figure 1. "},{"type":"text","text":"Web-application developed by ResearchHub Foundation using AI agents (AI Editor and AI Peer Reviewers). "}]},{"type":"imageBlock","attrs":{"src":"https://storage.prod.researchhub.com/uploads/notes/users/34416/cd271a3f-286d-4688-be4c-4ecd9e0d6a5c/image.png","width":"100%","align":"center"}},{"type":"paragraph","attrs":{"id":"f736fb42-1b05-496e-92a4-77790f8cf555","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Figure 2."},{"type":"text","text":" Proposed ARTEMIS system architecture and workflow design."}]},{"type":"paragraph","attrs":{"id":"01349d6f-f3e3-4ea1-a201-332a408a3600","class":null,"textAlign":null},"content":[{"type":"text","text":"In "},{"type":"text","marks":[{"type":"bold"}],"text":"Fig. 2 "},{"type":"text","text":", we present the proposed ARTEMIS system architecture and workflow design, here we follow on with describing the key actions happening within each step of the process: "}]},{"type":"orderedList","attrs":{"start":1},"content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"7ac7e37f-6f3c-4555-931c-84b5bb47e179","class":null,"textAlign":null},"content":[{"type":"text","text":"Scientific Document Assigned for Decision-Making."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"e385dca3-7e4f-4316-b33d-d69f4e87caf5","class":null,"textAlign":null},"content":[{"type":"text","text":"Human users who make decisions based on scientific documents submit audit requests to gain further insights regarding the document and the scientific works it represents."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"31ad7192-0ddf-4dfc-a4b5-d6667aaaeb27","class":null,"textAlign":null},"content":[{"type":"text","text":"Document Pre-Processing for Metadata Extraction."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"4abfb5d5-0ac6-473a-9d70-483c82f04e7a","class":null,"textAlign":null},"content":[{"type":"text","text":"Depending on the metadata content, a specific full extraction protocol is performed to obtain the maximum amount of information from the document."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"0ad8d9ab-5ac5-4cee-8708-b9b9fd796ac0","class":null,"textAlign":null},"content":[{"type":"text","text":"Full extraction results are sent to the AI Quality Control agent to ensure they meet the required standards and can be used further."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"1ed7895e-f5f9-449c-a375-197b2e1d7e46","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"(6b, 7b, 8b.) "},{"type":"text","text":"Extracted document content that fails to satisfy the minimum requirements is sent to humans for manual decision-making, annotation of reasons for exclusion, and archiving within the Data Lakehouse. "},{"type":"hardBreak"},{"type":"text","marks":[{"type":"italic"}],"text":"(6a & 6a’.) "},{"type":"text","text":"Extracted documents that pass the QC requirements, either automatically or manually after human consideration, are passed on to the AI Editor."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"7d099bff-8bc2-4468-85bf-f8de1362c0cd","class":null,"textAlign":null},"content":[{"type":"text","text":"(7a.) The AI Editor generates or orchestrates the assignment of the document to the appropriate AI reviewer(s)."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"bcc62bd2-6e7d-4cc7-95dd-66e0aad4bd84","class":null,"textAlign":null},"content":[{"type":"text","text":"(8a.) AI Peer Reviewer(s) generate their assessment of the document and send it to the AI QC agent to ensure correct structure and compliance with the required review standards."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"6769daa3-cf65-42a2-b557-1df7f486fa3d","class":null,"textAlign":null},"content":[{"type":"text","text":"All reviews that pass the AI QC check are curated into a raw report ready for human evaluation."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"193e5537-e372-4247-a062-79f47817b0c3","class":null,"textAlign":null},"content":[{"type":"text","text":"The raw report is prepared (and reformatted if necessary) to be presented to humans for Human Evaluation."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"b00910ac-9458-4cc1-97a6-37ddc3f0290c","class":null,"textAlign":null},"content":[{"type":"text","text":"Human evaluation results are submitted to the AI QC for further checks, including verifying whether the contribution is authentically human and backed by proper login credentials."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"ef5cd76d-e9a0-4c8f-86f2-d2377ae0b08a","class":null,"textAlign":null},"content":[{"type":"text","text":"Human feedback from verified editors or validators is requested to justify the quality of the given human evaluation results."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"e75ed64b-6d1e-4810-8dad-1601020bc72a","class":null,"textAlign":null},"content":[{"type":"text","text":"All data collected thus far is curated into a coherent dataset corresponding to the scientific document and the auditing task at hand."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"c83c8c3e-ad5e-436a-b6d1-806a7eefa24f","class":null,"textAlign":null},"content":[{"type":"text","text":"The dataset is organized into its respective zones and/or warehouses within the Data Lakehouse."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"828ab71b-af7c-4587-a316-b257e476f7c8","class":null,"textAlign":null},"content":[{"type":"text","text":"Actionable insights are produced based on the knowledge available within the Data Lakehouse to answer the questions posed in the original audit request. This process can also be further facilitated by interactive Data Librarian/Analyst Agents that help users explore or generate additional insights beyond the original request."}]}]}]},{"type":"heading","attrs":{"id":"ecd66d59-3df0-4b47-a62e-21969fa6bab9","data-toc-id":"ecd66d59-3df0-4b47-a62e-21969fa6bab9","textAlign":"justify","level":2},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Aim 1 - Strengths & Weaknesses of Human vs AI Reviewers"}]},{"type":"paragraph","attrs":{"id":"053a38a7-f954-4172-bcb3-6dc0f60ab21f","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"The first stage of the proposal is focused on producing the “Human Wisdom” components described across steps 10 to 13 of the ARTEMIS architecture and work flow "},{"type":"text","marks":[{"type":"bold"}],"text":"(Fig. 2)"},{"type":"text","text":". In the process of producing our human evaluation components of our desired dataset, we will prompt the AI Reviewers and Human Reviewers to perform assessment based on"},{"type":"text","marks":[{"type":"textStyle","attrs":{"fontSize":null,"fontFamily":null,"color":"#NaNNaNNaN"}},{"type":"highlight","attrs":{"color":"#NaNNaNNaN"}}],"text":" two requirements:"}]},{"type":"orderedList","attrs":{"start":1},"content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"a55dae78-3f69-4d4e-a5a2-208058c4af35","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"underline"}],"text":"Classic ResearchHub Foundation prompt:"}]},{"type":"orderedList","attrs":{"start":1},"content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"20930a05-7073-40a9-8aa6-6c3d00a5cdcc","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"ResearchHub Foundation bounty prompt (italicized):"}]},{"type":"orderedList","attrs":{"start":1},"content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"5e7e1884-494e-4967-b0d0-1cf937691201","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"Your review must be submitted during the 14-day submission window from the day this bounty was initiated."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"c7eddb43-a6ac-4c05-aac1-a1ff997e8650","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"Mention your credentials and include your areas of relevant expertise and limitations in assessing this preprint."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"2a021933-5f86-474d-b976-fe705375328b","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"Include the version of the preprint you are reviewing (e.g. 1st, 2nd, etc.). Make sure you review the most updated version available."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"d3f2e23e-cfef-412d-852c-9c5b878165f4","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"Use the rating system in the \"Peer Reviews\" tab for all 5 criteria (overall assessment, introduction, methods, results, discussion) but the content within each is flexible (in-line comments can be used instead of a block of text in each section)."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"b73775a4-9859-419a-ab93-d78692463659","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"For each figure, including supplementary material, you must provide a detailed assessment (pros or cons)."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"e04f1bfb-98ec-4606-a304-af54c4f2bd58","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"Plagiarism will not be tolerated. You must comprehensively disclose any use of artificial intelligence (AI) platforms used in the review process. Please refer to our AI Policy for additional details."}]},{"type":"paragraph","attrs":{"id":"0a355a31-e2d6-4e82-b28c-e31fdfa24328","class":null,"textAlign":null}}]}]}]}]}]}]},{"type":"paragraph","attrs":{"id":"6695000a-7aae-48b5-aa31-dddd589b4b17","class":null,"textAlign":null},"content":[{"type":"text","text":"ResearchHub houses many reviewers with expertise in biomedical sciences and hard sciences including molecular biology, immunology, oncology, neuroscience, etc "},{"type":"text","marks":[{"type":"bold"}],"text":"(Fig 3a, 3b)"},{"type":"text","text":".. These same set of reviewers can be validators when discerning human vs AI reviews."}]},{"type":"imageBlock","attrs":{"src":"https://storage.prod.researchhub.com/uploads/notes/users/34416/c3e676a5-18c6-4249-87d7-391c76dfa009/image.png","width":"100%","align":"center"}},{"type":"paragraph","attrs":{"id":"5fc98250-cf05-4487-95df-3235d8caffa4","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Figure 3a."},{"type":"text","text":" Number of Peer Reviews previously completed on ResearchHub stratified by OpenAlex Subfields."}]},{"type":"imageBlock","attrs":{"src":"https://storage.prod.researchhub.com/uploads/notes/users/34416/6ec2f15c-09b4-4d55-bd2d-a3d79d24d9ac/image.png","width":"100%","align":"center"}},{"type":"paragraph","attrs":{"id":"640927b6-9194-41fb-9542-9fd4692bb81e","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Figure 3b"},{"type":"text","text":". Number of Peer Reviews previously completed on ResearchHub stratified by OpenAlex Topics."},{"type":"hardBreak"},{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"id":"eddef2d4-680b-43dc-b4eb-57d8c249e19b","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"The Human Feedback component will be primarily driven by human validators who will assess the precision and accuracy of the human and AI reviewers based on the claims made. The collection of human feedback will then be used to benchmark and improve the AI-generated reviews. An example of how this benchmarking can be performed is shown in the recent work ReviewEval by Kumar et al. and the references therein [8]. Partially implemented within the currently available ARTEMIS system. With the growth in community interest for open peer reviews, of which ResearchHub has adopted, human validators can quickly view and provide input to the reviews being produced in an open transparent marketplace [9]. The validators will use a flash cards style approach for simple validation. The flash cards approach will appear in several forms, depending on the relevant context. Typical cases would include: Fact-Check, Logic-Check, Math-Check, or Expert Confidence. This approach to consolidating expert feedback is designed as a \"bite-sized\" task that an expert in the field can answer or reason through without needing to read the entire paper. All context necessary for understanding or providing meaningful feedback on the specific excerpt will be provided on the flashcard. Peer reviewer’s claims will be subject to various weights depending on their background/reputation."}]},{"type":"orderedList","attrs":{"start":1},"content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"019562d3-7ccf-4cca-853b-fddab36dfcb7","class":null,"textAlign":null},"content":[{"type":"text","text":"ResearchHub’s verification and reputation system: Using OpenAlex, an open repository for scientific data, ResearchHub has ascribed a reputation score for verified users within their various fields of expertise "},{"type":"text","marks":[{"type":"bold"}],"text":"(Fig. 4)."},{"type":"text","text":" This will be used to value or weigh the statements made by various peer reviewers and validators."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"99f28769-4c82-4af5-814a-e77fc7b2e2db","class":null,"textAlign":null},"content":[{"type":"text","text":"Reviewer’s track record will be checked based on their past performances and previous editorial feedback, if any available. Additionally, each peer reviewer may need to complete a mini-quiz before receiving a tag for specific expertise, ensuring that they actually possess the required expertise. If the reputation system isn’t sufficient for properly confirming a user’s expertise, the mini-quiz will be employed and include single line domain specific questions, with each question restricted to a 5-10 second countdown to receive an answer."}]}]}]},{"type":"imageBlock","attrs":{"src":"https://storage.prod.researchhub.com/uploads/notes/users/34416/4b99c3bf-2b94-43dd-b57e-ef35fad2fc5f/image.png","width":"100%","align":"center"}},{"type":"paragraph","attrs":{"id":"44b12a0d-ed46-429e-a64f-22563ba717f0","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Figure 4."},{"type":"text","text":" Example of a verified user account on ResearchHub with a quantitative percentile ranking as a “reputation” based on publication history."},{"type":"hardBreak"},{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"id":"75fb822f-dc2c-4e60-bffa-aed017e655f1","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"underline"}],"text":"Alternative Approach:"},{"type":"text","text":" If reviewers and/or validators will be difficult to source for the work, alternatively we can validate the AI reviews against other pre-existing datasets [10]. Additionally, in the context where the flash cards approach doesn’t prove to be a good source of validation, we can have validators assess the reviewers based on an adapted version of the ResearchHub Foundation’s standardized unified rubric (not public, but unified recommendations to reviewers can be found "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://drive.google.com/file/d/1t7NpL39ghnBY9ImWjuunbc6gzmzrhqUt/view?ref=blog.researchhub.foundation","target":"_blank","rel":"noopener noreferrer nofollow","class":null}},{"type":"underline"}],"text":"here"},{"type":"text","text":") to give weighted scores on "},{"type":"text","marks":[{"type":"underline"}],"text":"methods, results, discussion, rigor, novelty and error detection rate."},{"type":"text","text":" Editors evaluations will be weighted contingent on their reputation score in that field according to ResearchHub’s reputation system."}]},{"type":"heading","attrs":{"id":"38af1211-e9e3-44b2-9e9f-7a1707749993","data-toc-id":"38af1211-e9e3-44b2-9e9f-7a1707749993","textAlign":"justify","level":2},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Aim 2 - Develop Versatile Agentic Solutions for Scientific Auditing"}]},{"type":"paragraph","attrs":{"id":"daaa21b1-ffc7-433a-becd-b728128dc738","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"Functions described in steps 5 to 9 "},{"type":"text","marks":[{"type":"bold"}],"text":"(Fig. 2) "},{"type":"text","text":"for developing AI agents for scientific auditing, we propose to develop 3 agentic roles and 5 auxiliary tooling functions. The 3 agentic roles will be composed of: Quality Control Agent, AI Reviewer Agent, AI Editor. Each of these roles may be deployed under different context and configuration to achieve the desired performance. For example, this may include an AI Editor in Neuroscience, AI Reviewer in Neuroscience, AI quality control in Neuroscience etc parallelized for a plethora of fields of expertise. "}]},{"type":"paragraph","attrs":{"id":"22c29048-9a73-4b5b-8bcb-d9de08595c23","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"The 5 Auxiliary toolings are : Long-Term Memory, Knowledge Base, Visual Understanding and Reasoning, Scientific References, Consistency of Structured Output. The design of these toolings will be made as modular as possible to allow replacement of the tech-stack when necessary to boost performance and efficiency."}]},{"type":"heading","attrs":{"id":"3328a9e7-f117-4306-ba8a-c9b58da032fc","data-toc-id":"3328a9e7-f117-4306-ba8a-c9b58da032fc","textAlign":"justify","level":2},"content":[{"type":"text","text":"Rigor"}]},{"type":"paragraph","attrs":{"id":"2b74a16c-76c5-425c-b1c8-0194a72ac5d4","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"Human reviewers can openly review any preprints, allowing for expedited reviews. The presence of a standardized prompt for human and AI reviewers, and standardized rubrics for editors allows for apples to apples direct comparisons on specific components of the reviewers. The verification and reputation v2 system on ResearchHub will be employed to weight evaluations according to the expertise of the individual making the claims. "},{"type":"hardBreak"},{"type":"hardBreak"},{"type":"text","text":"Additionally, prior to this pre-registration, we have performed preliminary studies that attempt to understand the dynamics, economics, and quality of incentivized peer review process of which is being employed on ResearchHub [11]. All three aspects of open peer review submissions addressed above (dynamics, economics, and quality) are highly dependent with at least 9 other coupled factors beyond economic and non-economic incentives "},{"type":"text","marks":[{"type":"bold"}],"text":"(Table 1)"},{"type":"text","text":". Therefore, any correlation found would be contextual and apply only to a certain combination of peer review configurations that involved at least the 9 factors. As it pertains to aim 1 of this pre-registration, we will ensure that the peer review context will be fixed ensuring maximal reproducibility."}]},{"type":"paragraph","attrs":{"id":"8a42c0d7-576f-4b01-a673-ceab9b957e70","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Table 1. "},{"type":"text","text":"The Nine Identified factors influencing correlation between incentives on the quality of open peer reviews of scientific manuscripts."}]},{"type":"table","attrs":{"id":"a31d9ea1-fe1e-4dd6-8594-c0e761ce1e5d"},"content":[{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"2cde18c4-512f-4534-8474-254914651956","class":null,"textAlign":"center"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Factor Influencing Incentive-Quality Correlation"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"b950d364-5d4a-42e7-a5dc-dbd1085e4e38","class":null,"textAlign":"center"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Variables "}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"ecda05f8-f5e5-4e8a-a6b1-5b8fe340c461","class":null,"textAlign":"center"},"content":[{"type":"text","text":"Order of Incentivization"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"57c937fa-3fb0-4edf-a868-0653665b4b5b","class":null,"textAlign":null},"content":[{"type":"text","text":"Award First (Commissioned Task)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"a56be9d0-7929-40a4-891a-ee71c67a79c5","class":null,"textAlign":null},"content":[{"type":"text","text":"Award Later (Rewarded)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"198bfdc3-aab4-418a-ac31-132db6d8574f","class":null,"textAlign":null},"content":[{"type":"text","text":"Bidding Mechanism (Stake for Chance)"}]}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"1e550c0e-2696-4891-aef8-866381c6c599","class":null,"textAlign":"center"},"content":[{"type":"text","text":"Reviewer Pool Size "}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"f274d629-3543-4b9b-9811-4d5658f7fd52","class":null,"textAlign":null},"content":[{"type":"text","text":"Small [<100]"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"c4fcf379-fdcd-4169-ae52-6f48e1e4131f","class":null,"textAlign":null},"content":[{"type":"text","text":"Medium [100-1000]"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"b6defc34-ea05-455f-8244-e410f0618d22","class":null,"textAlign":null},"content":[{"type":"text","text":"Large [1000+]"}]}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"6aa1cae5-baaa-4875-b9e0-185bd05db637","class":null,"textAlign":"center"},"content":[{"type":"text","text":"Expertise Diversity"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"08635dc4-a9d1-42f8-8656-a03aaed285cf","class":null,"textAlign":null},"content":[{"type":"text","text":"Ranging from 0 to 1 "}]},{"type":"paragraph","attrs":{"id":"c852dd85-c959-43b6-82aa-f72ade152eda","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"0: All reviewers are from the exact same research subfield"}]},{"type":"paragraph","attrs":{"id":"c5cdb7ba-657a-48f2-94ef-addf2887c6bb","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"1: For each subfield there are exactly equal number of reviewer"}]},{"type":"paragraph","attrs":{"id":"df6ad422-4ebf-4cb3-b442-8e953cf357c2","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"* Each reviewer can have more than 1 expertise, but maximum 3 notable expertise. Complete list of subfields, will be at most 3 x number of reviewers in the pool"},{"type":"text","text":"."}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"ff2746d8-58a3-41fb-8477-be5d0f1cb277","class":null,"textAlign":"center"},"content":[{"type":"text","text":"Reviewer Selection Criteria"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"c79d4bde-7209-47a6-81da-15133f933229","class":null,"textAlign":null},"content":[{"type":"text","text":"Single Chief-Editor (or Campaign Host) make the final call"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"47bd5ec6-4bab-4fc1-b6b0-eae363d38b93","class":null,"textAlign":null},"content":[{"type":"text","text":"First Come First Served [Based on Objective Quality Scores or Time Order]"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"8479ad19-942a-4165-9f59-e8aa325b2d40","class":null,"textAlign":null},"content":[{"type":"text","text":"Democratic Decision [Community Votes or Expert Scoring]"}]}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"764cf305-3cc0-4259-9188-d087bbcf5d63","class":null,"textAlign":"center"},"content":[{"type":"text","text":"Type of Incentive and Size/Intensity"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"84107737-8591-419e-8d01-3e7f360f97ca","class":null,"textAlign":null},"content":[{"type":"text","text":"Monetary"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"f596d762-1c78-4c92-99f1-bec880022e77","class":null,"textAlign":null},"content":[{"type":"text","text":"Reputation"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"efe69d58-4cef-40e5-b12f-a7de38ef1a14","class":null,"textAlign":null},"content":[{"type":"text","text":"Monetary + Reputation"}]}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"909c9445-5b86-46c4-85ae-b9a7a38d200f","class":null,"textAlign":"center"},"content":[{"type":"text","text":"Requested Review Structure"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"02aeac4e-0015-4e7d-a769-5be9253b9b52","class":null,"textAlign":null},"content":[{"type":"text","text":"Simple (Yes/No answer and inline comments)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"91dbf093-3720-48a3-adc8-aa389cbd8647","class":null,"textAlign":null},"content":[{"type":"text","text":"Moderate (Structured Comments on Different Aspects)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"edb06ee9-ee1e-4dca-93e3-7862a16d7295","class":null,"textAlign":null},"content":[{"type":"text","text":"Rigor (Complete Assessment on All Sections, Figures, References, and more)"}]}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"840138d4-92ac-4a6f-bffc-b336af664c5e","class":null,"textAlign":"center"},"content":[{"type":"text","text":"Difficulties of Review Task"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"dcb52fa7-7e80-426d-b042-0a14f6d5539f","class":null,"textAlign":null},"content":[{"type":"text","text":"Easy (General Knowledgeable Public)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"3a0643e6-8af6-41d3-b642-e3d035ab9704","class":null,"textAlign":null},"content":[{"type":"text","text":"Medium (Graduate Student Level on the Topic)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"43bac5c6-6e8c-44e6-8647-d361c6d6a798","class":null,"textAlign":null},"content":[{"type":"text","text":"Hard (Subject Expert on the Topic)"}]}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"d8c76cc4-0f5a-405c-bef9-c31a001f7624","class":null,"textAlign":"center"},"content":[{"type":"text","text":"Likelihood & Proportion of AI Usage "}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"ec301d4e-1b1b-436e-9694-630b7f319bb4","class":null,"textAlign":null},"content":[{"type":"text","text":"Low (<10%)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"d9fd2e8d-a9fc-4645-96ce-8823a02ea131","class":null,"textAlign":null},"content":[{"type":"text","text":"Moderate (10-90%)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"6f1df4b2-871f-4768-9567-346d96fb878a","class":null,"textAlign":null},"content":[{"type":"text","text":"High (>90%)"}]}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"fd0628b7-b305-445e-9c15-5855d20a38b6","class":null,"textAlign":"center"},"content":[{"type":"text","text":"Time Restriction"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"4fcadfdd-278c-4d90-a8c7-9e52371e21f8","class":null,"textAlign":null},"content":[{"type":"text","text":"Intense (<3 Days)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"76d9a0b3-b978-4010-878c-42dd17a3200e","class":null,"textAlign":null},"content":[{"type":"text","text":"Moderate (3-14 Days)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"74e8147e-5962-4a0d-9334-8fd0f82be551","class":null,"textAlign":null},"content":[{"type":"text","text":"Leisurely ( >15 Days)"}]}]}]}]}]}]},{"type":"heading","attrs":{"id":"fb707edd-1fee-4288-be60-092c8c89af90","data-toc-id":"fb707edd-1fee-4288-be60-092c8c89af90","textAlign":null,"level":2},"content":[{"type":"hardBreak"},{"type":"text","text":"Budget"}]},{"type":"paragraph","attrs":{"id":"8119d08e-40a0-4797-9040-b4c59407c4f0","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Table 2."},{"type":"text","text":" Budget description and breakdown for an estimated 12 month timeline. "}]},{"type":"table","attrs":{"id":"d8e69946-2f95-4f4a-97e5-11a271c263d0"},"content":[{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":null},"content":[{"type":"paragraph","attrs":{"id":"653966ac-5536-4b07-a9de-a5dbd5a8c981","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Section"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":null},"content":[{"type":"paragraph","attrs":{"id":"ffd1cd07-cfa6-4a54-ae47-b76c4f9be60f","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Need"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":null},"content":[{"type":"paragraph","attrs":{"id":"3e6666dd-4d63-49fc-a583-7bf6a0a21d80","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Details"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":null},"content":[{"type":"paragraph","attrs":{"id":"7dd97198-b77b-487a-b661-99b1748ba291","class":null,"textAlign":"center"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Cost"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"8eaad482-43a8-4b6a-9d4f-123eae5c64bd","class":null,"textAlign":null},"content":[{"type":"text","text":"Development"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"orderedList","attrs":{"start":1},"content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"bc269868-78cd-452e-8fff-1d098fce4cc3","class":null,"textAlign":null},"content":[{"type":"text","text":"Compute"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"e4a9bbb7-e054-4dc9-bf17-a37d6be4324f","class":null,"textAlign":null},"content":[{"type":"text","text":"API"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"ea7eedb5-af72-41fd-a825-c27fde22efba","class":null,"textAlign":null},"content":[{"type":"text","text":"Outsourced developer work for data cleaning and UI/UX design"}]}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"438316bb-9839-4cf3-94d6-07a8fc34e382","class":null,"textAlign":null},"content":[{"type":"text","text":"Digital Ocean, o1/claude/gemini APIs, GPUs, observability system, auxiliary tools for AI Agents, and vector storage, etc."}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"cb20a808-1e68-49be-8e3c-6e1bc0b1efe7","class":null,"textAlign":null},"content":[{"type":"text","text":"$10,000"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"da7ee106-0cb6-4279-83b3-ae5c6a5a605f","class":null,"textAlign":null},"content":[{"type":"text","text":"Human editor costs"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"2f70821e-fc49-4b0f-84fe-be1be8e835d8","class":null,"textAlign":null},"content":[{"type":"text","text":"Incentive for Validators (Editors or Authors) to assess AI and Human Reviewer Outputs. Including incentives for validating papers, N = 50, from YesNoError (YNE)"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"63d2745b-ded9-49fe-ae7c-9bffe9deb005","class":null,"textAlign":null},"content":[{"type":"text","text":"Validators will be asked to provide either full or partial feedback on both the original preprint and/or the AI review submissions."}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"5e9e5273-4995-4bfe-afd5-552fcae69ad0","class":null,"textAlign":null},"content":[{"type":"text","text":"$50,000 for 1000 editors (assuming $50 to assess all peer reviews per manuscript)"}]},{"type":"paragraph","attrs":{"id":"6168b153-6272-4dc6-a773-5a38e1f52fad","class":null,"textAlign":null},"content":[{"type":"hardBreak"},{"type":"text","text":" "}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"79914fc8-5f3e-4009-80f2-36d2e95d13fd","class":null,"textAlign":null},"content":[{"type":"text","text":"Administrative Cost"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"b55aaf23-b5b1-4536-940b-1535f1e22f71","class":null,"textAlign":null},"content":[{"type":"text","text":"Marketing"},{"type":"hardBreak"},{"type":"hardBreak"},{"type":"text","text":" "}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"3bb8ac27-814c-4de8-8ffb-aef38a5f31bb","class":null,"textAlign":null},"content":[{"type":"text","text":"The cost associated with marketing efforts to ensure campaign visibility and sustain interest for contributions from the community."}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"a546b8f1-4856-437b-ac70-8d0e1797dfb7","class":null,"textAlign":null},"content":[{"type":"text","text":"$5,000"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"3c44806c-d5f2-4665-8b90-e0b0c8a96809","class":null,"textAlign":null},"content":[{"type":"text","text":"Reserve Fund"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"ed887d93-f21f-4e13-a83e-cb738aa4a863","class":null,"textAlign":null},"content":[{"type":"text","text":"For unexpected needs and for insurance against volatility. "}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"de0ccca9-4192-4106-95a8-f9f57d46c76a","class":null,"textAlign":null},"content":[{"type":"text","text":" "}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"d17a9064-7a62-4bf1-aa80-2a327316d9b7","class":null,"textAlign":null},"content":[{"type":"text","text":"$15,000, If unused, will be put toward getting a larger dataset."}]}]}]}]},{"type":"heading","attrs":{"id":"7c501115-d85f-492e-9d43-0bfc85ae253b","data-toc-id":"7c501115-d85f-492e-9d43-0bfc85ae253b","textAlign":"justify","level":2},"content":[{"type":"text","text":"Disclosure and Licensing Statement"}]},{"type":"paragraph","attrs":{"id":"a27c9522-23b3-47d9-891b-b7fd844342d5","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"This document outlines the terms under which the content, outputs, and tools resulting from our study on AI reviewers for scientific manuscripts are shared and utilized. Transparency and collaboration are core to our research philosophy, and we aim to strike a balance between open science principles and the potential for practical, real-world applications."}]},{"type":"heading","attrs":{"id":"f1f7a7cb-beaf-4874-acd5-c0c54c36fb74","data-toc-id":"f1f7a7cb-beaf-4874-acd5-c0c54c36fb74","textAlign":"justify","level":3},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Open Source Commitment"}]},{"type":"orderedList","attrs":{"start":1},"content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"e47d185a-026f-4a5c-b3ae-6cae5b8f6d00","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Content and Outputs"},{"type":"text","text":":"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"529957c9-986a-454f-ab72-0677e3645da1","class":null,"textAlign":null},"content":[{"type":"text","text":"All data, algorithms, models, and other outputs derived from this study will be made available under an open-source license, allowing unrestricted access for academic, research, and non-commercial purposes."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"367f6e15-fd6e-4350-a45e-fef3500af411","class":null,"textAlign":null},"content":[{"type":"text","text":"The specific license will be the MIT License to ensure clarity regarding permissible uses and attribution requirements."}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"45878005-acbe-4323-a56a-8c41e47b31bd","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Repository Access"},{"type":"text","text":":"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"ec653f0d-cc28-49b2-863f-aefdd1749693","class":null,"textAlign":null},"content":[{"type":"text","text":"Public repositories (e.g., on GitHub, Zenodo, HuggingFace or similar platforms) will host the content and outputs, accompanied by detailed documentation to facilitate their reuse and extension by the broader research community."}]},{"type":"paragraph","attrs":{"id":"80017aa2-be42-406c-b51d-d39e474737dc","class":null,"textAlign":null},"content":[{"type":"hardBreak"},{"type":"text","marks":[{"type":"bold"}],"text":"Commercialization Clause"}]}]}]}]}]},{"type":"paragraph","attrs":{"id":"4539fd2c-dcd5-4e75-8a93-af056260278e","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"While the content and outputs of this study will be open-sourced, we reserve the right to commercialize tools or technologies derived from these resources. "}]},{"type":"paragraph","attrs":{"id":"5b993517-4350-4d63-b66a-de5b2a35d538","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"By participating in this study or utilizing its outputs, users acknowledge and agree to the terms outlined in this disclosure and licensing statement."}]},{"type":"heading","attrs":{"id":"8d89c32b-dcfc-4b18-b4f8-b2c70c0e4337","data-toc-id":"8d89c32b-dcfc-4b18-b4f8-b2c70c0e4337","textAlign":"justify","level":2},"content":[{"type":"text","text":"Declaration of Generative AI in Scientific Writing"}]},{"type":"paragraph","attrs":{"id":"238437c9-6767-4329-b2eb-cfbc4fb4bba6","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"During the preparation of this work, ChatGPT was used to enhance the readability and language of some parts of the manuscript. Additionally, it was used to consolidate the original content written in the introduction, methods, and preliminary data to construct an Abstract. Following any ChatGPT usage, content was reviewed and edited as needed."}]},{"type":"heading","attrs":{"id":"69fc03a6-1a50-4537-9a9f-a8861a858bdb","data-toc-id":"69fc03a6-1a50-4537-9a9f-a8861a858bdb","textAlign":"justify","level":2},"content":[{"type":"text","text":"Acknowledgements"}]},{"type":"paragraph","attrs":{"id":"a6fd71d4-7abb-43e2-8e68-990bda383b72","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"We would like to acknowledge Matt Schlicht, founder of YesNoError for being a Strategic Data Partner and Gareth Dyke & Maria Machado as Strategic Expert Community Partners. "}]},{"type":"heading","attrs":{"id":"e810c287-cbe9-4fd1-869d-01225474c7a5","data-toc-id":"e810c287-cbe9-4fd1-869d-01225474c7a5","textAlign":null,"level":2},"content":[{"type":"text","text":"References"}]},{"type":"paragraph","attrs":{"id":"e81ebc0d-6d7b-4724-9f82-c0d1da682177","class":null,"textAlign":null},"content":[{"type":"text","text":"[1] Howard Bauchner, Frederick P Rivara, Use of artificial intelligence and the future of peer review, "},{"type":"text","marks":[{"type":"italic"}],"text":"Health Affairs Scholar"},{"type":"text","text":", Volume 2, Issue 5, May 2024, qxae058, "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://doi.org/10.1093/haschl/qxae058","target":"_blank","rel":"noopener noreferrer nofollow","class":null}}],"text":"https://doi.org/10.1093/haschl/qxae058"}]},{"type":"paragraph","attrs":{"id":"1e8c4c3e-7700-4d5b-874e-9f4b6dedd201","class":null,"textAlign":null},"content":[{"type":"text","text":"[2] Aczel B, Barwich AS, Diekman AB, et al. The present and future of peer review: Ideas, interventions, and evidence. Proc Natl Acad Sci U S A. 2025;122(5):e2401232121. doi:10.1073/pnas.2401232121"}]},{"type":"paragraph","attrs":{"id":"9143643e-ae79-4c34-95db-8c5c999455e5","class":null,"textAlign":null},"content":[{"type":"text","text":"[3] Smith Nail and Cumberledge Aaron. 2020 Quotation errors in general science journals Proc. R. Soc. A. 47620200538. "},{"type":"text","marks":[{"type":"textStyle","attrs":{"fontSize":"14px","fontFamily":"\"Proxima Nova\", sans-serif","color":"rgb(51, 49, 50)"}}],"text":" "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://doi.org/10.1098/rspa.2020.0538","target":"_blank","rel":"noopener noreferrer nofollow","class":"doi-link link"}}],"text":"http://doi.org/10.1098/rspa.2020.0538"}]},{"type":"paragraph","attrs":{"id":"be1fc96a-5803-48f8-b7ac-36096bb36d6f","class":null,"textAlign":null},"content":[{"type":"text","text":"[4] Hanson, M. A., Gómez Barreiro, P., Crosetto, P., & Brockington, D. (2024). "},{"type":"text","marks":[{"type":"italic"}],"text":"The strain on scientific publishing"},{"type":"text","text":". arXiv. "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://arxiv.org/abs/2309.15884v2","target":"_blank","rel":"noopener noreferrer nofollow","class":null}},{"type":"underline"}],"text":"https://arxiv.org/abs/2309.15884v2"}]},{"type":"paragraph","attrs":{"id":"5d2e96b9-753b-4876-83b0-402841585b4e","class":null,"textAlign":null},"content":[{"type":"text","text":"[5] "},{"type":"text","marks":[{"type":"textStyle","attrs":{"fontSize":"16px","fontFamily":"-apple-system, \"system-ui\", \"Segoe UI\", Roboto, Oxygen-Sans, Ubuntu, Cantarell, \"Helvetica Neue\", sans-serif","color":"rgb(51, 51, 51)"}}],"text":"LeBlanc, A.G., Barnes, J.D., Saunders, T.J. "},{"type":"text","marks":[{"type":"italic"}],"text":"et al."},{"type":"text","marks":[{"type":"textStyle","attrs":{"fontSize":"16px","fontFamily":"-apple-system, \"system-ui\", \"Segoe UI\", Roboto, Oxygen-Sans, Ubuntu, Cantarell, \"Helvetica Neue\", sans-serif","color":"rgb(51, 51, 51)"}}],"text":" Scientific sinkhole: estimating the cost of peer review based on survey data with snowball sampling. "},{"type":"text","marks":[{"type":"italic"}],"text":"Res Integr Peer Rev"},{"type":"text","marks":[{"type":"textStyle","attrs":{"fontSize":"16px","fontFamily":"-apple-system, \"system-ui\", \"Segoe UI\", Roboto, Oxygen-Sans, Ubuntu, Cantarell, \"Helvetica Neue\", sans-serif","color":"rgb(51, 51, 51)"}}],"text":" "},{"type":"text","marks":[{"type":"bold"}],"text":"8"},{"type":"text","marks":[{"type":"textStyle","attrs":{"fontSize":"16px","fontFamily":"-apple-system, \"system-ui\", \"Segoe UI\", Roboto, Oxygen-Sans, Ubuntu, Cantarell, \"Helvetica Neue\", sans-serif","color":"rgb(51, 51, 51)"}}],"text":", 3 (2023). "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://doi.org/10.1186/s41073-023-00128-2","target":"_blank","rel":"noopener noreferrer nofollow","class":"link"}},{"type":"textStyle","attrs":{"fontSize":"16px","fontFamily":"-apple-system, \"system-ui\", \"Segoe UI\", Roboto, Oxygen-Sans, Ubuntu, Cantarell, \"Helvetica Neue\", sans-serif","color":"rgb(51, 51, 51)"}}],"text":"https://doi.org/10.1186/s41073-023-00128-2"}]},{"type":"paragraph","attrs":{"id":"7740f77a-dff9-4955-8d6c-405906aa016d","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"textStyle","attrs":{"fontSize":"16px","fontFamily":"-apple-system, \"system-ui\", \"Segoe UI\", Roboto, Oxygen-Sans, Ubuntu, Cantarell, \"Helvetica Neue\", sans-serif","color":"rgb(51, 51, 51)"}}],"text":"[6] "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://www.biorxiv.org/content/10.1101/2025.03.18.644032v1.full.pdf","target":"_blank","rel":"noopener noreferrer nofollow","class":"link"}},{"type":"textStyle","attrs":{"fontSize":"16px","fontFamily":"-apple-system, \"system-ui\", \"Segoe UI\", Roboto, Oxygen-Sans, Ubuntu, Cantarell, \"Helvetica Neue\", sans-serif","color":"rgb(51, 51, 51)"}}],"text":"https://www.biorxiv.org/content/10.1101/2025.03.18.644032v1.full.pdf"}]},{"type":"paragraph","attrs":{"id":"0ac6549c-8ed8-4136-aaa4-e2f5dbe71ccf","class":null,"textAlign":null},"content":[{"type":"text","text":"[7] Seghier, M.L. (2025), \"AI-powered peer review needs human supervision\", "},{"type":"text","marks":[{"type":"italic"}],"text":"Journal of Information, Communication and Ethics in Society"},{"type":"text","text":", Vol. 23 No. 1, pp. 104-116. "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://doi.org/10.1108/JICES-09-2024-0132","target":"_blank","rel":"noopener noreferrer nofollow","class":null}}],"text":"https://doi.org/10.1108/JICES-09-2024-0132"}]},{"type":"paragraph","attrs":{"id":"6643dce1-77e8-494d-8a18-a0ffbcde8440","class":null,"textAlign":null},"content":[{"type":"text","text":"[8] Kirtani, C., Garg, M. K., Prasad, T., Singhal, T., Mandal, M., & Kumar, D. (2025). "},{"type":"text","marks":[{"type":"italic"}],"text":"ReviewEval: An evaluation framework for AI-generated reviews"},{"type":"text","text":". arXiv. "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://arxiv.org/abs/2502.11736v2","target":"_blank","rel":"noopener noreferrer nofollow","class":null}},{"type":"underline"}],"text":"https://arxiv.org/abs/2502.11736v2"}]},{"type":"paragraph","attrs":{"id":"9daf6a9f-3bbf-4dc5-ad2a-14da53f280a5","class":null,"textAlign":null},"content":[{"type":"text","text":"[9] Yang, J. (2025). "},{"type":"text","marks":[{"type":"italic"}],"text":"Paper Copilot: The artificial intelligence and machine learning community should adopt a more transparent and regulated peer review process"},{"type":"text","text":". arXiv. "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://arxiv.org/abs/2502.00874","target":"_blank","rel":"noopener noreferrer nofollow","class":null}},{"type":"underline"}],"text":"https://arxiv.org/abs/2502.00874"}]},{"type":"paragraph","attrs":{"id":"72228cce-524a-438a-be5a-9d7bf25babe8","class":null,"textAlign":null},"content":[{"type":"text","text":"[10] Kuznetsov, I., Gurevych, I. et. a. (2024). "},{"type":"text","marks":[{"type":"italic"}],"text":"What can natural language processing do for peer review?"},{"type":"text","text":" arXiv. "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://arxiv.org/abs/2405.06563","target":"_blank","rel":"noopener noreferrer nofollow","class":null}},{"type":"underline"}],"text":"https://arxiv.org/abs/2405.06563"},{"type":"text","marks":[{"type":"underline"}],"text":" "},{"type":"text","text":"GitHub Repository: Afzal, O. M. (n.d.). "},{"type":"text","marks":[{"type":"italic"}],"text":"nlp-for-peer-review"},{"type":"text","text":". GitHub. "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/OAfzal/nlp-for-peer-review","target":"_blank","rel":"noopener noreferrer nofollow","class":null}},{"type":"underline"}],"text":"https://github.com/OAfzal/nlp-for-peer-review"}]},{"type":"paragraph","attrs":{"id":"aa6206f3-2b22-4e99-a73f-72cf75dfd2e0","class":null,"textAlign":null},"content":[{"type":"text","text":"[11] Brian, D. (2024). "},{"type":"text","marks":[{"type":"italic"}],"text":"Incentivized vs non-incentivized open peer reviews: Dynamics, economics, and quality"},{"type":"text","text":". ResearchHub. "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://doi.org/10.55277/ResearchHub.taescjxh","target":"_blank","rel":"noopener noreferrer nofollow","class":null}},{"type":"underline"}],"text":"https://doi.org/10.55277/ResearchHub.taescjxh"}]},{"type":"paragraph","attrs":{"id":"69cb50e0-8f88-456f-aedb-a9d807b6eb36","class":null,"textAlign":null}},{"type":"paragraph","attrs":{"id":"14e9c72c-3792-48fc-859a-608010825fef","class":null,"textAlign":null},"content":[{"type":"hardBreak"},{"type":"hardBreak"},{"type":"hardBreak"},{"type":"hardBreak"},{"type":"hardBreak"},{"type":"hardBreak"},{"type":"hardBreak"},{"type":"hardBreak"},{"type":"hardBreak"},{"type":"text","text":" "}]}]}88:T607e,ARTEMIS - Automated Review and Trustworthy Evaluation for Manuscripts in Science Authors and Affiliations Jeffrey Koury1, Dominikus Brian1 1 ResearchHub Foundation Abstract The peer review process, essential for maintaining scientific rigor, faces significant challenges including reviewer fatigue, bias, and insufficient rigor. The rapid increase in scientific publications has placed a burden on a limited pool of editors and reviewers, resulting in the inability of reviewers to efficiently detect falsified data, screen errors with high fidelity, and even establish consensus between reviewers [1]. ResearchHub Foundation aims to address these issues by 1) incentivizing reviewers and 2) integrating Agentic AI systems powered by recent breakthroughs in large language and multi-modal foundational models. Our unique approach with validated expert human reviews, and trustworthy automated tools to complement human reviewers, will enhance the quality and credibility of peer reviews to benefit both authors and the larger scientific community. We have observed from our own experiments (200+ AI peer review samples) that AI reviewers have capabilities that excel in domains such as detecting technical errors and statistical flaws, while human reviewers provide greater insight into the broader relevance and context of research. We hypothesize that incorporating human feedback into AI models of peer review can improve their performance beyond existing state-of-the-art AI systems without human feedback. The project has two aims: Establish a Reliable Protocol to Assess Reviewer Performance: Using a standardized rubric, human editors with specific sets of expertise will evaluate AI and human reviews across introduction, methods, results, discussion, and figure analysis. AI reviewers are expected to outperform in technical rigor, while humans excel in contextual analysis. Develop a Versatile Agentic Solutions for Scientific Auditing : By leveraging upon the growing number of toolkits for deploying AI Agents driven by LLM, Large multi-modal models (LMMs), long-term memories, and orchestration infrastructure, we will seek to develop specialized agentic solutions honed for various aspects of scientific document auditing. This initiative leverages ResearchHub’s extensive reviewer database, standardized prompts, and expertise-weighted evaluations to enhance the peer review process. By addressing systemic issues and integrating AI and human expertise, this project seeks to improve the accuracy, efficiency, and scalability of scientific peer review, supporting the advancement of open science and minimizing exhaustion for human reviewers. Introduction and Significance Scientific Auditing - Peer review is a cornerstone of scientific integrity, serving as the primary mechanism for validating research and granting manuscripts the status of “verified” work. Peer reviewers face difficulties in thoroughly assessing statistical analyses, verifying quotation errors, and evaluating claims in a time efficient manner [2, 3]. These challenges are further exacerbated by the rapid growth in scientific publications, which has significantly increased the workload on an already limited pool of reviewers and editors. The number of scientific publications has increased an estimated 47% from 2016 to 2022 [4]. Despite the critical nature of their role, reviewers (excluding domain specific reviewers such as methods and statistics reviewers for example), are usually not monetarily compensated for their efforts. 87.5% (n=320) of survey participants have reported never being paid for peer review [5]. This combination of rising publications and insufficient monetary support has created a critical bottleneck in the peer review process, including burdens on editors, highlighting the urgent need for innovative solutions to enhance the accuracy, efficiency, and sustainability of scientific auditing [6]. With the rise of techniques resulting in large terabyte sized datasets (i.e. single-cell RNAseq, BARseq) the complexity for detecting errors and the time demand to do so for human reviewers is amplified. The use of ResearchCoin incentives on ResearchHub has provided the impetus to overcome this apprehension of reviewers to expeditiously peer review, and provides a new found incentive and motivation. Nonetheless, even with appropriate incentives, human reviewers continue to have difficulties assessing statistical analyses, verifying quotation errors and identifying other issues during the review process. Agentic AI solution - With the rapid mass-adoption of LLMs and AI Agents, the ability to holistically assess a manuscript has become feasible at rapid speeds and low costs, providing immensely valuable insights into the paper that often reside within human reviewer’s blind spots. These tools can fill the gaps of human reviewers limitations and provide a more expedited turnaround time on feedback and requested edits, ultimately adding a more holistic set of feedback and context to the academic manuscript. Foundational models (o1, Claude, Gemini, LLaMA, DeepSeek, etc.) have made strides and provide a basis for this approach. With smaller amounts of data, such as peer review data on ResearchHub, it is already possible to substantially leverage these foundation models with minimal additional training data to construct a much more accurate peer review system greater than the sum of its parts. AI reviewers are not a panacea, and it’s becoming clear that human reviewers are still needed, but will have AI reviewers as a powerful supplement [7]. Hypothesis and Aims Hypothesis: Published and preprinted papers have errors or issues within the manuscript that AI reviewers can catch that are often not caught by human reviewers, and integrating human reviewer/editor reinforcement learning can enhance capabilities of the AI review system when domain specific context is essential. Aim 1: Assess the strengths and weaknesses of human reviewers and AI reviewers using a standardized rubric. Human editors will evaluate the outputs of both human reviewers and AI reviewers for methods, results, discussion, rigor, novelty and error detection rate. Anticipated result: Human peer reviewers will be superior in interpreting the implications of the methods and results that are NOT explicitly mentioned in the manuscript, while AI reviewers will be more thorough in assessing statistics, detecting errors etc based on information available explicitly in the manuscript. Aim 2: Develop versatile systems of agentic solutions driven by LLM, LMMs, and AI Agents, empowered by domain specific knowledge, in combination with human evaluation and feedback in producing thorough evaluation of scientific documents. Anticipated result: After tuning and including post-training of foundational models with the human evaluation and feedback, the resulting agentic solutions will perform better than the direct use of benchmark foundational models alone when used to a particular scientific auditing task (manuscript review, editor’s feedback, journal decision, etc.) Methods and Design Currently, the use of already existing foundation models equipped with simple prompt engineering has provided a viable starting point for pinpointing errors and giving additional context in scientific manuscripts. We’ve developed a minimum viable system in the form of a web app shown in Fig. 1 below. We also made several observations throughout our pilot experiment in which, holistically, using the same optimized prompt, Gemini 2.0 has been the most robust foundation model in producing consistent results human editors and readers generally consider more useful (data not shown). This may be in part due to the 1M+ context window for token processing, roughly 5-7 times greater than the typical 128k and 280k context windows found in other foundation models. Figure 1. Web-application developed by ResearchHub Foundation using AI agents (AI Editor and AI Peer Reviewers). Figure 2. Proposed ARTEMIS system architecture and workflow design. In Fig. 2 , we present the proposed ARTEMIS system architecture and workflow design, here we follow on with describing the key actions happening within each step of the process: Scientific Document Assigned for Decision-Making. Human users who make decisions based on scientific documents submit audit requests to gain further insights regarding the document and the scientific works it represents. Document Pre-Processing for Metadata Extraction. Depending on the metadata content, a specific full extraction protocol is performed to obtain the maximum amount of information from the document. Full extraction results are sent to the AI Quality Control agent to ensure they meet the required standards and can be used further. (6b, 7b, 8b.) Extracted document content that fails to satisfy the minimum requirements is sent to humans for manual decision-making, annotation of reasons for exclusion, and archiving within the Data Lakehouse. (6a & 6a’.) Extracted documents that pass the QC requirements, either automatically or manually after human consideration, are passed on to the AI Editor. (7a.) The AI Editor generates or orchestrates the assignment of the document to the appropriate AI reviewer(s). (8a.) AI Peer Reviewer(s) generate their assessment of the document and send it to the AI QC agent to ensure correct structure and compliance with the required review standards. All reviews that pass the AI QC check are curated into a raw report ready for human evaluation. The raw report is prepared (and reformatted if necessary) to be presented to humans for Human Evaluation. Human evaluation results are submitted to the AI QC for further checks, including verifying whether the contribution is authentically human and backed by proper login credentials. Human feedback from verified editors or validators is requested to justify the quality of the given human evaluation results. All data collected thus far is curated into a coherent dataset corresponding to the scientific document and the auditing task at hand. The dataset is organized into its respective zones and/or warehouses within the Data Lakehouse. Actionable insights are produced based on the knowledge available within the Data Lakehouse to answer the questions posed in the original audit request. This process can also be further facilitated by interactive Data Librarian/Analyst Agents that help users explore or generate additional insights beyond the original request. Aim 1 - Strengths & Weaknesses of Human vs AI Reviewers The first stage of the proposal is focused on producing the “Human Wisdom” components described across steps 10 to 13 of the ARTEMIS architecture and work flow (Fig. 2). In the process of producing our human evaluation components of our desired dataset, we will prompt the AI Reviewers and Human Reviewers to perform assessment based on two requirements: Classic ResearchHub Foundation prompt: ResearchHub Foundation bounty prompt (italicized): Your review must be submitted during the 14-day submission window from the day this bounty was initiated. Mention your credentials and include your areas of relevant expertise and limitations in assessing this preprint. Include the version of the preprint you are reviewing (e.g. 1st, 2nd, etc.). Make sure you review the most updated version available. Use the rating system in the "Peer Reviews" tab for all 5 criteria (overall assessment, introduction, methods, results, discussion) but the content within each is flexible (in-line comments can be used instead of a block of text in each section). For each figure, including supplementary material, you must provide a detailed assessment (pros or cons). Plagiarism will not be tolerated. You must comprehensively disclose any use of artificial intelligence (AI) platforms used in the review process. Please refer to our AI Policy for additional details. ResearchHub houses many reviewers with expertise in biomedical sciences and hard sciences including molecular biology, immunology, oncology, neuroscience, etc (Fig 3a, 3b).. These same set of reviewers can be validators when discerning human vs AI reviews. Figure 3a. Number of Peer Reviews previously completed on ResearchHub stratified by OpenAlex Subfields. Figure 3b. Number of Peer Reviews previously completed on ResearchHub stratified by OpenAlex Topics. The Human Feedback component will be primarily driven by human validators who will assess the precision and accuracy of the human and AI reviewers based on the claims made. The collection of human feedback will then be used to benchmark and improve the AI-generated reviews. An example of how this benchmarking can be performed is shown in the recent work ReviewEval by Kumar et al. and the references therein [8]. Partially implemented within the currently available ARTEMIS system. With the growth in community interest for open peer reviews, of which ResearchHub has adopted, human validators can quickly view and provide input to the reviews being produced in an open transparent marketplace [9]. The validators will use a flash cards style approach for simple validation. The flash cards approach will appear in several forms, depending on the relevant context. Typical cases would include: Fact-Check, Logic-Check, Math-Check, or Expert Confidence. This approach to consolidating expert feedback is designed as a "bite-sized" task that an expert in the field can answer or reason through without needing to read the entire paper. All context necessary for understanding or providing meaningful feedback on the specific excerpt will be provided on the flashcard. Peer reviewer’s claims will be subject to various weights depending on their background/reputation. ResearchHub’s verification and reputation system: Using OpenAlex, an open repository for scientific data, ResearchHub has ascribed a reputation score for verified users within their various fields of expertise (Fig. 4). This will be used to value or weigh the statements made by various peer reviewers and validators. Reviewer’s track record will be checked based on their past performances and previous editorial feedback, if any available. Additionally, each peer reviewer may need to complete a mini-quiz before receiving a tag for specific expertise, ensuring that they actually possess the required expertise. If the reputation system isn’t sufficient for properly confirming a user’s expertise, the mini-quiz will be employed and include single line domain specific questions, with each question restricted to a 5-10 second countdown to receive an answer. Figure 4. Example of a verified user account on ResearchHub with a quantitative percentile ranking as a “reputation” based on publication history. Alternative Approach: If reviewers and/or validators will be difficult to source for the work, alternatively we can validate the AI reviews against other pre-existing datasets [10]. Additionally, in the context where the flash cards approach doesn’t prove to be a good source of validation, we can have validators assess the reviewers based on an adapted version of the ResearchHub Foundation’s standardized unified rubric (not public, but unified recommendations to reviewers can be found here) to give weighted scores on methods, results, discussion, rigor, novelty and error detection rate. Editors evaluations will be weighted contingent on their reputation score in that field according to ResearchHub’s reputation system. Aim 2 - Develop Versatile Agentic Solutions for Scientific Auditing Functions described in steps 5 to 9 (Fig. 2) for developing AI agents for scientific auditing, we propose to develop 3 agentic roles and 5 auxiliary tooling functions. The 3 agentic roles will be composed of: Quality Control Agent, AI Reviewer Agent, AI Editor. Each of these roles may be deployed under different context and configuration to achieve the desired performance. For example, this may include an AI Editor in Neuroscience, AI Reviewer in Neuroscience, AI quality control in Neuroscience etc parallelized for a plethora of fields of expertise. The 5 Auxiliary toolings are : Long-Term Memory, Knowledge Base, Visual Understanding and Reasoning, Scientific References, Consistency of Structured Output. The design of these toolings will be made as modular as possible to allow replacement of the tech-stack when necessary to boost performance and efficiency. Rigor Human reviewers can openly review any preprints, allowing for expedited reviews. The presence of a standardized prompt for human and AI reviewers, and standardized rubrics for editors allows for apples to apples direct comparisons on specific components of the reviewers. The verification and reputation v2 system on ResearchHub will be employed to weight evaluations according to the expertise of the individual making the claims. Additionally, prior to this pre-registration, we have performed preliminary studies that attempt to understand the dynamics, economics, and quality of incentivized peer review process of which is being employed on ResearchHub [11]. All three aspects of open peer review submissions addressed above (dynamics, economics, and quality) are highly dependent with at least 9 other coupled factors beyond economic and non-economic incentives (Table 1). Therefore, any correlation found would be contextual and apply only to a certain combination of peer review configurations that involved at least the 9 factors. As it pertains to aim 1 of this pre-registration, we will ensure that the peer review context will be fixed ensuring maximal reproducibility. Table 1. The Nine Identified factors influencing correlation between incentives on the quality of open peer reviews of scientific manuscripts. Factor Influencing Incentive-Quality Correlation Variables Order of Incentivization Award First (Commissioned Task) Award Later (Rewarded) Bidding Mechanism (Stake for Chance) Reviewer Pool Size Small [<100] Medium [100-1000] Large [1000+] Expertise Diversity Ranging from 0 to 1 0: All reviewers are from the exact same research subfield 1: For each subfield there are exactly equal number of reviewer * Each reviewer can have more than 1 expertise, but maximum 3 notable expertise. Complete list of subfields, will be at most 3 x number of reviewers in the pool. Reviewer Selection Criteria Single Chief-Editor (or Campaign Host) make the final call First Come First Served [Based on Objective Quality Scores or Time Order] Democratic Decision [Community Votes or Expert Scoring] Type of Incentive and Size/Intensity Monetary Reputation Monetary + Reputation Requested Review Structure Simple (Yes/No answer and inline comments) Moderate (Structured Comments on Different Aspects) Rigor (Complete Assessment on All Sections, Figures, References, and more) Difficulties of Review Task Easy (General Knowledgeable Public) Medium (Graduate Student Level on the Topic) Hard (Subject Expert on the Topic) Likelihood & Proportion of AI Usage Low (<10%) Moderate (10-90%) High (>90%) Time Restriction Intense (<3 Days) Moderate (3-14 Days) Leisurely ( >15 Days) Budget Table 2. Budget description and breakdown for an estimated 12 month timeline. Section Need Details Cost Development Compute API Outsourced developer work for data cleaning and UI/UX design Digital Ocean, o1/claude/gemini APIs, GPUs, observability system, auxiliary tools for AI Agents, and vector storage, etc. $10,000 Human editor costs Incentive for Validators (Editors or Authors) to assess AI and Human Reviewer Outputs. Including incentives for validating papers, N = 50, from YesNoError (YNE) Validators will be asked to provide either full or partial feedback on both the original preprint and/or the AI review submissions. $50,000 for 1000 editors (assuming $50 to assess all peer reviews per manuscript) Administrative Cost Marketing The cost associated with marketing efforts to ensure campaign visibility and sustain interest for contributions from the community. $5,000 Reserve Fund For unexpected needs and for insurance against volatility. $15,000, If unused, will be put toward getting a larger dataset. Disclosure and Licensing Statement This document outlines the terms under which the content, outputs, and tools resulting from our study on AI reviewers for scientific manuscripts are shared and utilized. Transparency and collaboration are core to our research philosophy, and we aim to strike a balance between open science principles and the potential for practical, real-world applications. Open Source Commitment Content and Outputs: All data, algorithms, models, and other outputs derived from this study will be made available under an open-source license, allowing unrestricted access for academic, research, and non-commercial purposes. The specific license will be the MIT License to ensure clarity regarding permissible uses and attribution requirements. Repository Access: Public repositories (e.g., on GitHub, Zenodo, HuggingFace or similar platforms) will host the content and outputs, accompanied by detailed documentation to facilitate their reuse and extension by the broader research community. Commercialization Clause While the content and outputs of this study will be open-sourced, we reserve the right to commercialize tools or technologies derived from these resources. By participating in this study or utilizing its outputs, users acknowledge and agree to the terms outlined in this disclosure and licensing statement. Declaration of Generative AI in Scientific Writing During the preparation of this work, ChatGPT was used to enhance the readability and language of some parts of the manuscript. Additionally, it was used to consolidate the original content written in the introduction, methods, and preliminary data to construct an Abstract. Following any ChatGPT usage, content was reviewed and edited as needed. Acknowledgements We would like to acknowledge Matt Schlicht, founder of YesNoError for being a Strategic Data Partner and Gareth Dyke & Maria Machado as Strategic Expert Community Partners. References [1] Howard Bauchner, Frederick P Rivara, Use of artificial intelligence and the future of peer review, Health Affairs Scholar, Volume 2, Issue 5, May 2024, qxae058, https://doi.org/10.1093/haschl/qxae058 [2] Aczel B, Barwich AS, Diekman AB, et al. The present and future of peer review: Ideas, interventions, and evidence. Proc Natl Acad Sci U S A. 2025;122(5):e2401232121. doi:10.1073/pnas.2401232121 [3] Smith Nail and Cumberledge Aaron. 2020 Quotation errors in general science journals Proc. R. Soc. A. 47620200538. http://doi.org/10.1098/rspa.2020.0538 [4] Hanson, M. A., Gómez Barreiro, P., Crosetto, P., & Brockington, D. (2024). The strain on scientific publishing. arXiv. https://arxiv.org/abs/2309.15884v2 [5] LeBlanc, A.G., Barnes, J.D., Saunders, T.J. et al. Scientific sinkhole: estimating the cost of peer review based on survey data with snowball sampling. Res Integr Peer Rev 8, 3 (2023). https://doi.org/10.1186/s41073-023-00128-2 [6] https://www.biorxiv.org/content/10.1101/2025.03.18.644032v1.full.pdf [7] Seghier, M.L. (2025), "AI-powered peer review needs human supervision", Journal of Information, Communication and Ethics in Society, Vol. 23 No. 1, pp. 104-116. https://doi.org/10.1108/JICES-09-2024-0132 [8] Kirtani, C., Garg, M. K., Prasad, T., Singhal, T., Mandal, M., & Kumar, D. (2025). ReviewEval: An evaluation framework for AI-generated reviews. arXiv. https://arxiv.org/abs/2502.11736v2 [9] Yang, J. (2025). Paper Copilot: The artificial intelligence and machine learning community should adopt a more transparent and regulated peer review process. arXiv. https://arxiv.org/abs/2502.00874 [10] Kuznetsov, I., Gurevych, I. et. a. (2024). What can natural language processing do for peer review? arXiv. https://arxiv.org/abs/2405.06563 GitHub Repository: Afzal, O. M. (n.d.). nlp-for-peer-review. GitHub. https://github.com/OAfzal/nlp-for-peer-review [11] Brian, D. (2024). Incentivized vs non-incentivized open peer reviews: Dynamics, economics, and quality. ResearchHub. https://doi.org/10.55277/ResearchHub.taescjxh 89:T100a7,{"type":"doc","content":[{"type":"heading","attrs":{"id":"8c038a7e-4227-4ec0-9963-7222b0bed545","data-toc-id":"8c038a7e-4227-4ec0-9963-7222b0bed545","textAlign":null,"level":1},"content":[{"type":"text","text":"ARTEMIS - Automated Review and Trustworthy Evaluation for Manuscripts in Science"}]},{"type":"heading","attrs":{"id":"81fb7381-2743-4bfd-b8ab-754c0d398c13","data-toc-id":"81fb7381-2743-4bfd-b8ab-754c0d398c13","textAlign":null,"level":2},"content":[{"type":"text","text":"Authors and Affiliations"}]},{"type":"paragraph","attrs":{"id":"68586fb8-71c7-4ad2-8f22-3385791b5b9d","class":null,"textAlign":null},"content":[{"type":"text","text":"Jeffrey Koury"},{"type":"text","marks":[{"type":"superscript"}],"text":"1"},{"type":"text","text":", Dominikus Brian"},{"type":"text","marks":[{"type":"superscript"}],"text":"1"},{"type":"text","text":" "},{"type":"hardBreak"},{"type":"hardBreak"},{"type":"text","marks":[{"type":"superscript"}],"text":"1"},{"type":"text","text":" ResearchHub Foundation"}]},{"type":"heading","attrs":{"id":"9fd38a75-c58f-433a-8bc7-0b232a40e192","data-toc-id":"9fd38a75-c58f-433a-8bc7-0b232a40e192","textAlign":null,"level":2},"content":[{"type":"text","text":"Abstract"}]},{"type":"paragraph","attrs":{"id":"abca777a-b88b-4bec-b25f-cc32d669003f","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"The peer review process, essential for maintaining scientific rigor, faces significant challenges including reviewer fatigue, bias, and insufficient rigor. The rapid increase in scientific publications has placed a burden on a limited pool of editors and reviewers, resulting in the inability of reviewers to "},{"type":"text","marks":[{"type":"italic"}],"text":"efficiently"},{"type":"text","text":" detect falsified data, screen errors with high fidelity, and even establish consensus between reviewers [1]. ResearchHub Foundation aims to address these issues by 1) incentivizing reviewers and 2) integrating Agentic AI systems powered by recent breakthroughs in large language and multi-modal foundational models. Our unique approach with validated expert human reviews, and trustworthy automated tools to complement human reviewers, will enhance the quality and credibility of peer reviews to benefit both authors and the larger scientific community."}]},{"type":"paragraph","attrs":{"id":"19293bde-95e9-4b41-be4e-7d25fa380808","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"We have observed from our own experiments (200+ AI peer review samples) that AI reviewers have capabilities that excel in domains such as detecting technical errors and statistical flaws, while human reviewers provide greater insight into the broader relevance and context of research. We hypothesize that incorporating human feedback into AI models of peer review can improve their performance beyond existing state-of-the-art AI systems without human feedback. The project has two aims:"}]},{"type":"orderedList","attrs":{"start":1},"content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"b2217a67-1853-429b-ae21-14f231611507","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Establish a Reliable Protocol to Assess Reviewer Performance"},{"type":"text","text":": Using a standardized rubric, human editors with specific sets of expertise will evaluate AI and human reviews across introduction, methods, results, discussion, and figure analysis. AI reviewers are expected to outperform in technical rigor, while humans excel in contextual analysis."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"e5481b4c-d192-43be-89c2-86e48d79c067","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Develop a Versatile Agentic Solutions for Scientific Auditing "},{"type":"text","text":": By leveraging upon the growing number of toolkits for deploying AI Agents driven by LLM, Large multi-modal models (LMMs), long-term memories, and orchestration infrastructure, we will seek to develop specialized agentic solutions honed for various aspects of scientific document auditing."}]}]}]},{"type":"paragraph","attrs":{"id":"ae1d4a34-bb4e-4b9d-8d01-a810e3627438","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"This initiative leverages ResearchHub’s extensive reviewer database, standardized prompts, and expertise-weighted evaluations to enhance the peer review process. By addressing systemic issues and integrating AI and human expertise, this project seeks to improve the accuracy, efficiency, and scalability of scientific peer review, supporting the advancement of open science and minimizing exhaustion for human reviewers."}]},{"type":"heading","attrs":{"id":"cf4cc6b9-33c0-4ddf-a261-65dafde77a1e","data-toc-id":"cf4cc6b9-33c0-4ddf-a261-65dafde77a1e","textAlign":"justify","level":2},"content":[{"type":"text","text":"Introduction and Significance"}]},{"type":"paragraph","attrs":{"id":"b5809a87-5dce-47bc-8b47-3e98f385d567","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Scientific Auditing -"},{"type":"text","text":" Peer review is a cornerstone of scientific integrity, serving as the primary mechanism for validating research and granting manuscripts the status of “verified” work. Peer reviewers face difficulties in thoroughly assessing statistical analyses, verifying quotation errors, and evaluating claims in a time efficient manner [2, 3]. These challenges are further exacerbated by the rapid growth in scientific publications, which has significantly increased the workload on an already limited pool of reviewers and editors. The number of scientific publications has increased an estimated 47% from 2016 to 2022 [4]. Despite the critical nature of their role, reviewers (excluding domain specific reviewers such as methods and statistics reviewers for example), are usually not monetarily compensated for their efforts. 87.5% (n=320) of survey participants have reported never being paid for peer review [5]. This combination of rising publications and insufficient monetary support has created a critical bottleneck in the peer review process, including burdens on editors, highlighting the urgent need for innovative solutions to enhance the accuracy, efficiency, and sustainability of scientific auditing [6]. With the rise of techniques resulting in large terabyte sized datasets (i.e. single-cell RNAseq, BARseq) the complexity for detecting errors and the time demand to do so for human reviewers is amplified. The use of ResearchCoin incentives on ResearchHub has provided the impetus to overcome this apprehension of reviewers to expeditiously peer review, and provides a new found incentive and motivation. Nonetheless, even with appropriate incentives, human reviewers continue to have difficulties assessing statistical analyses, verifying quotation errors and identifying other issues during the review process. "}]},{"type":"paragraph","attrs":{"id":"818fa2d4-c55c-4189-a133-cf9913aea48f","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Agentic AI solution - "},{"type":"text","text":"With the rapid mass-adoption of LLMs and AI Agents, the ability to holistically assess a manuscript has become feasible at rapid speeds and low costs, providing immensely valuable insights into the paper that often reside within human reviewer’s blind spots. These tools can fill the gaps of human reviewers limitations and provide a more expedited turnaround time on feedback and requested edits, ultimately adding a more holistic set of feedback and context to the academic manuscript. Foundational models (o1, Claude, Gemini, LLaMA, DeepSeek, etc.) have made strides and provide a basis for this approach. With smaller amounts of data, such as peer review data on ResearchHub, it is already possible to substantially leverage these foundation models with minimal additional training data to construct a much more accurate peer review system greater than the sum of its parts. AI reviewers are not a panacea, and it’s becoming clear that human reviewers are still needed, but will have AI reviewers as a powerful supplement [7]. "}]},{"type":"heading","attrs":{"id":"c840d50a-64fd-47c2-8cc2-198a24f2fe56","data-toc-id":"c840d50a-64fd-47c2-8cc2-198a24f2fe56","textAlign":"justify","level":2},"content":[{"type":"text","text":"Hypothesis and Aims"}]},{"type":"paragraph","attrs":{"id":"0c2801de-fc45-4310-adb1-414a721eff5a","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Hypothesis: "},{"type":"text","text":"Published and preprinted papers have errors or issues within the manuscript that AI reviewers can catch that are often not caught by human reviewers, and integrating human reviewer/editor reinforcement learning can enhance capabilities of the AI review system when domain specific context is essential. "}]},{"type":"paragraph","attrs":{"id":"8e728bb1-77a5-4166-bd11-12271de71e67","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Aim 1:"},{"type":"text","text":" Assess the strengths and weaknesses of human reviewers and AI reviewers using a standardized rubric. Human editors will evaluate the outputs of both human reviewers and AI reviewers for methods, results, discussion, rigor, novelty and error detection rate. "},{"type":"text","marks":[{"type":"underline"}],"text":"Anticipated result"},{"type":"text","text":": Human peer reviewers will be superior in interpreting the implications of the methods and results that are NOT explicitly mentioned in the manuscript, while AI reviewers will be more thorough in assessing statistics, detecting errors etc based on information available explicitly in the manuscript. "}]},{"type":"paragraph","attrs":{"id":"68bff7e6-7dce-4542-96e1-0dc0100b8e33","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Aim 2: "},{"type":"text","text":"Develop versatile systems of agentic solutions driven by LLM, LMMs, and AI Agents, empowered by domain specific knowledge, in combination with human evaluation and feedback in producing thorough evaluation of scientific documents. "},{"type":"text","marks":[{"type":"underline"}],"text":"Anticipated result"},{"type":"text","text":": After tuning and including post-training of foundational models with the human evaluation and feedback, the resulting agentic solutions will perform better than the direct use of benchmark foundational models alone when used to a particular scientific auditing task (manuscript review, editor’s feedback, journal decision, etc.)"}]},{"type":"heading","attrs":{"id":"0ff95663-7735-4089-97dc-858bed34ef57","data-toc-id":"0ff95663-7735-4089-97dc-858bed34ef57","textAlign":"justify","level":2},"content":[{"type":"text","text":"Methods and Design"}]},{"type":"paragraph","attrs":{"id":"ae78d807-0af6-4b18-b84a-c2b06bc42f45","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"Currently, the use of already existing foundation models equipped with simple prompt engineering has provided a viable starting point for pinpointing errors and giving additional context in scientific manuscripts. We’ve developed a minimum viable system in the form of a web app shown in "},{"type":"text","marks":[{"type":"bold"}],"text":"Fig. 1"},{"type":"text","text":" below. We also made several observations throughout our pilot experiment in which, holistically, using the same optimized prompt, Gemini 2.0 has been the most robust foundation model in producing consistent results human editors and readers generally consider more useful (data not shown). This "},{"type":"text","marks":[{"type":"italic"}],"text":"may"},{"type":"text","text":" be in part due to the 1M+ context window for token processing, roughly 5-7 times greater than the typical 128k and 280k context windows found in other foundation models. "}]},{"type":"imageBlock","attrs":{"src":"https://storage.prod.researchhub.com/uploads/notes/users/34416/780924e3-bd10-44d8-9c95-fb7a146bd58c/screenshot-2025-04-02-at-3.png","width":"100%","align":"center"}},{"type":"paragraph","attrs":{"id":"4b70b38e-7d2d-4b2f-a5b5-4281c9d54e37","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Figure 1. "},{"type":"text","text":"Web-application developed by ResearchHub Foundation using AI agents (AI Editor and AI Peer Reviewers). "}]},{"type":"imageBlock","attrs":{"src":"https://storage.prod.researchhub.com/uploads/notes/users/34416/cd271a3f-286d-4688-be4c-4ecd9e0d6a5c/image.png","width":"100%","align":"center"}},{"type":"paragraph","attrs":{"id":"f736fb42-1b05-496e-92a4-77790f8cf555","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Figure 2."},{"type":"text","text":" Proposed ARTEMIS system architecture and workflow design."}]},{"type":"paragraph","attrs":{"id":"01349d6f-f3e3-4ea1-a201-332a408a3600","class":null,"textAlign":null},"content":[{"type":"text","text":"In "},{"type":"text","marks":[{"type":"bold"}],"text":"Fig. 2 "},{"type":"text","text":", we present the proposed ARTEMIS system architecture and workflow design, here we follow on with describing the key actions happening within each step of the process: "}]},{"type":"orderedList","attrs":{"start":1},"content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"7ac7e37f-6f3c-4555-931c-84b5bb47e179","class":null,"textAlign":null},"content":[{"type":"text","text":"Scientific Document Assigned for Decision-Making."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"e385dca3-7e4f-4316-b33d-d69f4e87caf5","class":null,"textAlign":null},"content":[{"type":"text","text":"Human users who make decisions based on scientific documents submit audit requests to gain further insights regarding the document and the scientific works it represents."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"31ad7192-0ddf-4dfc-a4b5-d6667aaaeb27","class":null,"textAlign":null},"content":[{"type":"text","text":"Document Pre-Processing for Metadata Extraction."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"4abfb5d5-0ac6-473a-9d70-483c82f04e7a","class":null,"textAlign":null},"content":[{"type":"text","text":"Depending on the metadata content, a specific full extraction protocol is performed to obtain the maximum amount of information from the document."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"0ad8d9ab-5ac5-4cee-8708-b9b9fd796ac0","class":null,"textAlign":null},"content":[{"type":"text","text":"Full extraction results are sent to the AI Quality Control agent to ensure they meet the required standards and can be used further."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"1ed7895e-f5f9-449c-a375-197b2e1d7e46","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"(6b, 7b, 8b.) "},{"type":"text","text":"Extracted document content that fails to satisfy the minimum requirements is sent to humans for manual decision-making, annotation of reasons for exclusion, and archiving within the Data Lakehouse. "},{"type":"hardBreak"},{"type":"text","marks":[{"type":"italic"}],"text":"(6a & 6a’.) "},{"type":"text","text":"Extracted documents that pass the QC requirements, either automatically or manually after human consideration, are passed on to the AI Editor."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"7d099bff-8bc2-4468-85bf-f8de1362c0cd","class":null,"textAlign":null},"content":[{"type":"text","text":"(7a.) The AI Editor generates or orchestrates the assignment of the document to the appropriate AI reviewer(s)."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"bcc62bd2-6e7d-4cc7-95dd-66e0aad4bd84","class":null,"textAlign":null},"content":[{"type":"text","text":"(8a.) AI Peer Reviewer(s) generate their assessment of the document and send it to the AI QC agent to ensure correct structure and compliance with the required review standards."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"6769daa3-cf65-42a2-b557-1df7f486fa3d","class":null,"textAlign":null},"content":[{"type":"text","text":"All reviews that pass the AI QC check are curated into a raw report ready for human evaluation."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"193e5537-e372-4247-a062-79f47817b0c3","class":null,"textAlign":null},"content":[{"type":"text","text":"The raw report is prepared (and reformatted if necessary) to be presented to humans for Human Evaluation."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"b00910ac-9458-4cc1-97a6-37ddc3f0290c","class":null,"textAlign":null},"content":[{"type":"text","text":"Human evaluation results are submitted to the AI QC for further checks, including verifying whether the contribution is authentically human and backed by proper login credentials."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"ef5cd76d-e9a0-4c8f-86f2-d2377ae0b08a","class":null,"textAlign":null},"content":[{"type":"text","text":"Human feedback from verified editors or validators is requested to justify the quality of the given human evaluation results."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"e75ed64b-6d1e-4810-8dad-1601020bc72a","class":null,"textAlign":null},"content":[{"type":"text","text":"All data collected thus far is curated into a coherent dataset corresponding to the scientific document and the auditing task at hand."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"c83c8c3e-ad5e-436a-b6d1-806a7eefa24f","class":null,"textAlign":null},"content":[{"type":"text","text":"The dataset is organized into its respective zones and/or warehouses within the Data Lakehouse."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"828ab71b-af7c-4587-a316-b257e476f7c8","class":null,"textAlign":null},"content":[{"type":"text","text":"Actionable insights are produced based on the knowledge available within the Data Lakehouse to answer the questions posed in the original audit request. This process can also be further facilitated by interactive Data Librarian/Analyst Agents that help users explore or generate additional insights beyond the original request."}]}]}]},{"type":"heading","attrs":{"id":"ecd66d59-3df0-4b47-a62e-21969fa6bab9","data-toc-id":"ecd66d59-3df0-4b47-a62e-21969fa6bab9","textAlign":"justify","level":2},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Aim 1 - Strengths & Weaknesses of Human vs AI Reviewers"}]},{"type":"paragraph","attrs":{"id":"053a38a7-f954-4172-bcb3-6dc0f60ab21f","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"The first stage of the proposal is focused on producing the “Human Wisdom” components described across steps 10 to 13 of the ARTEMIS architecture and work flow "},{"type":"text","marks":[{"type":"bold"}],"text":"(Fig. 2)"},{"type":"text","text":". In the process of producing our human evaluation components of our desired dataset, we will prompt the AI Reviewers and Human Reviewers to perform assessment based on"},{"type":"text","marks":[{"type":"textStyle","attrs":{"fontSize":null,"fontFamily":null,"color":"#NaNNaNNaN"}},{"type":"highlight","attrs":{"color":"#NaNNaNNaN"}}],"text":" two requirements:"}]},{"type":"orderedList","attrs":{"start":1},"content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"a55dae78-3f69-4d4e-a5a2-208058c4af35","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"underline"}],"text":"Classic ResearchHub Foundation prompt:"}]},{"type":"orderedList","attrs":{"start":1},"content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"20930a05-7073-40a9-8aa6-6c3d00a5cdcc","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"ResearchHub Foundation bounty prompt (italicized):"}]},{"type":"orderedList","attrs":{"start":1},"content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"5e7e1884-494e-4967-b0d0-1cf937691201","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"Your review must be submitted during the 14-day submission window from the day this bounty was initiated."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"c7eddb43-a6ac-4c05-aac1-a1ff997e8650","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"Mention your credentials and include your areas of relevant expertise and limitations in assessing this preprint."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"2a021933-5f86-474d-b976-fe705375328b","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"Include the version of the preprint you are reviewing (e.g. 1st, 2nd, etc.). Make sure you review the most updated version available."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"d3f2e23e-cfef-412d-852c-9c5b878165f4","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"Use the rating system in the \"Peer Reviews\" tab for all 5 criteria (overall assessment, introduction, methods, results, discussion) but the content within each is flexible (in-line comments can be used instead of a block of text in each section)."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"b73775a4-9859-419a-ab93-d78692463659","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"For each figure, including supplementary material, you must provide a detailed assessment (pros or cons)."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"e04f1bfb-98ec-4606-a304-af54c4f2bd58","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"Plagiarism will not be tolerated. You must comprehensively disclose any use of artificial intelligence (AI) platforms used in the review process. Please refer to our AI Policy for additional details."}]},{"type":"paragraph","attrs":{"id":"0a355a31-e2d6-4e82-b28c-e31fdfa24328","class":null,"textAlign":null}}]}]}]}]}]}]},{"type":"paragraph","attrs":{"id":"6695000a-7aae-48b5-aa31-dddd589b4b17","class":null,"textAlign":null},"content":[{"type":"text","text":"ResearchHub houses many reviewers with expertise in biomedical sciences and hard sciences including molecular biology, immunology, oncology, neuroscience, etc "},{"type":"text","marks":[{"type":"bold"}],"text":"(Fig 3a, 3b)"},{"type":"text","text":".. These same set of reviewers can be validators when discerning human vs AI reviews."}]},{"type":"imageBlock","attrs":{"src":"https://storage.prod.researchhub.com/uploads/notes/users/34416/c3e676a5-18c6-4249-87d7-391c76dfa009/image.png","width":"100%","align":"center"}},{"type":"paragraph","attrs":{"id":"5fc98250-cf05-4487-95df-3235d8caffa4","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Figure 3a."},{"type":"text","text":" Number of Peer Reviews previously completed on ResearchHub stratified by OpenAlex Subfields."}]},{"type":"imageBlock","attrs":{"src":"https://storage.prod.researchhub.com/uploads/notes/users/34416/6ec2f15c-09b4-4d55-bd2d-a3d79d24d9ac/image.png","width":"100%","align":"center"}},{"type":"paragraph","attrs":{"id":"640927b6-9194-41fb-9542-9fd4692bb81e","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Figure 3b"},{"type":"text","text":". Number of Peer Reviews previously completed on ResearchHub stratified by OpenAlex Topics."},{"type":"hardBreak"},{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"id":"eddef2d4-680b-43dc-b4eb-57d8c249e19b","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"The Human Feedback component will be primarily driven by human validators who will assess the precision and accuracy of the human and AI reviewers based on the claims made. The collection of human feedback will then be used to benchmark and improve the AI-generated reviews. An example of how this benchmarking can be performed is shown in the recent work ReviewEval by Kumar et al. and the references therein [8]. Partially implemented within the currently available ARTEMIS system. With the growth in community interest for open peer reviews, of which ResearchHub has adopted, human validators can quickly view and provide input to the reviews being produced in an open transparent marketplace [9]. The validators will use a flash cards style approach for simple validation. The flash cards approach will appear in several forms, depending on the relevant context. Typical cases would include: Fact-Check, Logic-Check, Math-Check, or Expert Confidence. This approach to consolidating expert feedback is designed as a \"bite-sized\" task that an expert in the field can answer or reason through without needing to read the entire paper. All context necessary for understanding or providing meaningful feedback on the specific excerpt will be provided on the flashcard. Peer reviewer’s claims will be subject to various weights depending on their background/reputation."}]},{"type":"orderedList","attrs":{"start":1},"content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"019562d3-7ccf-4cca-853b-fddab36dfcb7","class":null,"textAlign":null},"content":[{"type":"text","text":"ResearchHub’s verification and reputation system: Using OpenAlex, an open repository for scientific data, ResearchHub has ascribed a reputation score for verified users within their various fields of expertise "},{"type":"text","marks":[{"type":"bold"}],"text":"(Fig. 4)."},{"type":"text","text":" This will be used to value or weigh the statements made by various peer reviewers and validators."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"99f28769-4c82-4af5-814a-e77fc7b2e2db","class":null,"textAlign":null},"content":[{"type":"text","text":"Reviewer’s track record will be checked based on their past performances and previous editorial feedback, if any available. Additionally, each peer reviewer may need to complete a mini-quiz before receiving a tag for specific expertise, ensuring that they actually possess the required expertise. If the reputation system isn’t sufficient for properly confirming a user’s expertise, the mini-quiz will be employed and include single line domain specific questions, with each question restricted to a 5-10 second countdown to receive an answer."}]}]}]},{"type":"imageBlock","attrs":{"src":"https://storage.prod.researchhub.com/uploads/notes/users/34416/4b99c3bf-2b94-43dd-b57e-ef35fad2fc5f/image.png","width":"100%","align":"center"}},{"type":"paragraph","attrs":{"id":"44b12a0d-ed46-429e-a64f-22563ba717f0","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Figure 4."},{"type":"text","text":" Example of a verified user account on ResearchHub with a quantitative percentile ranking as a “reputation” based on publication history."},{"type":"hardBreak"},{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"id":"75fb822f-dc2c-4e60-bffa-aed017e655f1","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"underline"}],"text":"Alternative Approach:"},{"type":"text","text":" If reviewers and/or validators will be difficult to source for the work, alternatively we can validate the AI reviews against other pre-existing datasets [10]. Additionally, in the context where the flash cards approach doesn’t prove to be a good source of validation, we can have validators assess the reviewers based on an adapted version of the ResearchHub Foundation’s standardized unified rubric (not public, but unified recommendations to reviewers can be found "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://drive.google.com/file/d/1t7NpL39ghnBY9ImWjuunbc6gzmzrhqUt/view?ref=blog.researchhub.foundation","target":"_blank","rel":"noopener noreferrer nofollow","class":null}},{"type":"underline"}],"text":"here"},{"type":"text","text":") to give weighted scores on "},{"type":"text","marks":[{"type":"underline"}],"text":"methods, results, discussion, rigor, novelty and error detection rate."},{"type":"text","text":" Editors evaluations will be weighted contingent on their reputation score in that field according to ResearchHub’s reputation system."}]},{"type":"heading","attrs":{"id":"38af1211-e9e3-44b2-9e9f-7a1707749993","data-toc-id":"38af1211-e9e3-44b2-9e9f-7a1707749993","textAlign":"justify","level":2},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Aim 2 - Develop Versatile Agentic Solutions for Scientific Auditing"}]},{"type":"paragraph","attrs":{"id":"daaa21b1-ffc7-433a-becd-b728128dc738","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"Functions described in steps 5 to 9 "},{"type":"text","marks":[{"type":"bold"}],"text":"(Fig. 2) "},{"type":"text","text":"for developing AI agents for scientific auditing, we propose to develop 3 agentic roles and 5 auxiliary tooling functions. The 3 agentic roles will be composed of: Quality Control Agent, AI Reviewer Agent, AI Editor. Each of these roles may be deployed under different context and configuration to achieve the desired performance. For example, this may include an AI Editor in Neuroscience, AI Reviewer in Neuroscience, AI quality control in Neuroscience etc parallelized for a plethora of fields of expertise. "}]},{"type":"paragraph","attrs":{"id":"22c29048-9a73-4b5b-8bcb-d9de08595c23","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"The 5 Auxiliary toolings are : Long-Term Memory, Knowledge Base, Visual Understanding and Reasoning, Scientific References, Consistency of Structured Output. The design of these toolings will be made as modular as possible to allow replacement of the tech-stack when necessary to boost performance and efficiency."}]},{"type":"heading","attrs":{"id":"3328a9e7-f117-4306-ba8a-c9b58da032fc","data-toc-id":"3328a9e7-f117-4306-ba8a-c9b58da032fc","textAlign":"justify","level":2},"content":[{"type":"text","text":"Rigor"}]},{"type":"paragraph","attrs":{"id":"2b74a16c-76c5-425c-b1c8-0194a72ac5d4","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"Human reviewers can openly review any preprints, allowing for expedited reviews. The presence of a standardized prompt for human and AI reviewers, and standardized rubrics for editors allows for apples to apples direct comparisons on specific components of the reviewers. The verification and reputation v2 system on ResearchHub will be employed to weight evaluations according to the expertise of the individual making the claims. "},{"type":"hardBreak"},{"type":"hardBreak"},{"type":"text","text":"Additionally, prior to this pre-registration, we have performed preliminary studies that attempt to understand the dynamics, economics, and quality of incentivized peer review process of which is being employed on ResearchHub [11]. All three aspects of open peer review submissions addressed above (dynamics, economics, and quality) are highly dependent with at least 9 other coupled factors beyond economic and non-economic incentives "},{"type":"text","marks":[{"type":"bold"}],"text":"(Table 1)"},{"type":"text","text":". Therefore, any correlation found would be contextual and apply only to a certain combination of peer review configurations that involved at least the 9 factors. As it pertains to aim 1 of this pre-registration, we will ensure that the peer review context will be fixed ensuring maximal reproducibility."}]},{"type":"paragraph","attrs":{"id":"8a42c0d7-576f-4b01-a673-ceab9b957e70","class":null,"textAlign":"justify"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Table 1. "},{"type":"text","text":"The Nine Identified factors influencing correlation between incentives on the quality of open peer reviews of scientific manuscripts."}]},{"type":"table","attrs":{"id":"a31d9ea1-fe1e-4dd6-8594-c0e761ce1e5d"},"content":[{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"2cde18c4-512f-4534-8474-254914651956","class":null,"textAlign":"center"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Factor Influencing Incentive-Quality Correlation"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"b950d364-5d4a-42e7-a5dc-dbd1085e4e38","class":null,"textAlign":"center"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Variables "}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"ecda05f8-f5e5-4e8a-a6b1-5b8fe340c461","class":null,"textAlign":"center"},"content":[{"type":"text","text":"Order of Incentivization"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"57c937fa-3fb0-4edf-a868-0653665b4b5b","class":null,"textAlign":null},"content":[{"type":"text","text":"Award First (Commissioned Task)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"a56be9d0-7929-40a4-891a-ee71c67a79c5","class":null,"textAlign":null},"content":[{"type":"text","text":"Award Later (Rewarded)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"198bfdc3-aab4-418a-ac31-132db6d8574f","class":null,"textAlign":null},"content":[{"type":"text","text":"Bidding Mechanism (Stake for Chance)"}]}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"1e550c0e-2696-4891-aef8-866381c6c599","class":null,"textAlign":"center"},"content":[{"type":"text","text":"Reviewer Pool Size "}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"f274d629-3543-4b9b-9811-4d5658f7fd52","class":null,"textAlign":null},"content":[{"type":"text","text":"Small [<100]"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"c4fcf379-fdcd-4169-ae52-6f48e1e4131f","class":null,"textAlign":null},"content":[{"type":"text","text":"Medium [100-1000]"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"b6defc34-ea05-455f-8244-e410f0618d22","class":null,"textAlign":null},"content":[{"type":"text","text":"Large [1000+]"}]}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"6aa1cae5-baaa-4875-b9e0-185bd05db637","class":null,"textAlign":"center"},"content":[{"type":"text","text":"Expertise Diversity"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"08635dc4-a9d1-42f8-8656-a03aaed285cf","class":null,"textAlign":null},"content":[{"type":"text","text":"Ranging from 0 to 1 "}]},{"type":"paragraph","attrs":{"id":"c852dd85-c959-43b6-82aa-f72ade152eda","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"0: All reviewers are from the exact same research subfield"}]},{"type":"paragraph","attrs":{"id":"c5cdb7ba-657a-48f2-94ef-addf2887c6bb","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"1: For each subfield there are exactly equal number of reviewer"}]},{"type":"paragraph","attrs":{"id":"df6ad422-4ebf-4cb3-b442-8e953cf357c2","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"* Each reviewer can have more than 1 expertise, but maximum 3 notable expertise. Complete list of subfields, will be at most 3 x number of reviewers in the pool"},{"type":"text","text":"."}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"ff2746d8-58a3-41fb-8477-be5d0f1cb277","class":null,"textAlign":"center"},"content":[{"type":"text","text":"Reviewer Selection Criteria"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"c79d4bde-7209-47a6-81da-15133f933229","class":null,"textAlign":null},"content":[{"type":"text","text":"Single Chief-Editor (or Campaign Host) make the final call"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"47bd5ec6-4bab-4fc1-b6b0-eae363d38b93","class":null,"textAlign":null},"content":[{"type":"text","text":"First Come First Served [Based on Objective Quality Scores or Time Order]"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"8479ad19-942a-4165-9f59-e8aa325b2d40","class":null,"textAlign":null},"content":[{"type":"text","text":"Democratic Decision [Community Votes or Expert Scoring]"}]}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"764cf305-3cc0-4259-9188-d087bbcf5d63","class":null,"textAlign":"center"},"content":[{"type":"text","text":"Type of Incentive and Size/Intensity"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"84107737-8591-419e-8d01-3e7f360f97ca","class":null,"textAlign":null},"content":[{"type":"text","text":"Monetary"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"f596d762-1c78-4c92-99f1-bec880022e77","class":null,"textAlign":null},"content":[{"type":"text","text":"Reputation"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"efe69d58-4cef-40e5-b12f-a7de38ef1a14","class":null,"textAlign":null},"content":[{"type":"text","text":"Monetary + Reputation"}]}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"909c9445-5b86-46c4-85ae-b9a7a38d200f","class":null,"textAlign":"center"},"content":[{"type":"text","text":"Requested Review Structure"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"02aeac4e-0015-4e7d-a769-5be9253b9b52","class":null,"textAlign":null},"content":[{"type":"text","text":"Simple (Yes/No answer and inline comments)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"91dbf093-3720-48a3-adc8-aa389cbd8647","class":null,"textAlign":null},"content":[{"type":"text","text":"Moderate (Structured Comments on Different Aspects)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"edb06ee9-ee1e-4dca-93e3-7862a16d7295","class":null,"textAlign":null},"content":[{"type":"text","text":"Rigor (Complete Assessment on All Sections, Figures, References, and more)"}]}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"840138d4-92ac-4a6f-bffc-b336af664c5e","class":null,"textAlign":"center"},"content":[{"type":"text","text":"Difficulties of Review Task"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"dcb52fa7-7e80-426d-b042-0a14f6d5539f","class":null,"textAlign":null},"content":[{"type":"text","text":"Easy (General Knowledgeable Public)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"3a0643e6-8af6-41d3-b642-e3d035ab9704","class":null,"textAlign":null},"content":[{"type":"text","text":"Medium (Graduate Student Level on the Topic)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"43bac5c6-6e8c-44e6-8647-d361c6d6a798","class":null,"textAlign":null},"content":[{"type":"text","text":"Hard (Subject Expert on the Topic)"}]}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"d8c76cc4-0f5a-405c-bef9-c31a001f7624","class":null,"textAlign":"center"},"content":[{"type":"text","text":"Likelihood & Proportion of AI Usage "}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"ec301d4e-1b1b-436e-9694-630b7f319bb4","class":null,"textAlign":null},"content":[{"type":"text","text":"Low (<10%)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"d9fd2e8d-a9fc-4645-96ce-8823a02ea131","class":null,"textAlign":null},"content":[{"type":"text","text":"Moderate (10-90%)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"6f1df4b2-871f-4768-9567-346d96fb878a","class":null,"textAlign":null},"content":[{"type":"text","text":"High (>90%)"}]}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"fd0628b7-b305-445e-9c15-5855d20a38b6","class":null,"textAlign":"center"},"content":[{"type":"text","text":"Time Restriction"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:0.75pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"4fcadfdd-278c-4d90-a8c7-9e52371e21f8","class":null,"textAlign":null},"content":[{"type":"text","text":"Intense (<3 Days)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"76d9a0b3-b978-4010-878c-42dd17a3200e","class":null,"textAlign":null},"content":[{"type":"text","text":"Moderate (3-14 Days)"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"74e8147e-5962-4a0d-9334-8fd0f82be551","class":null,"textAlign":null},"content":[{"type":"text","text":"Leisurely ( >15 Days)"}]}]}]}]}]}]},{"type":"heading","attrs":{"id":"fb707edd-1fee-4288-be60-092c8c89af90","data-toc-id":"fb707edd-1fee-4288-be60-092c8c89af90","textAlign":null,"level":2},"content":[{"type":"hardBreak"},{"type":"text","text":"Budget"}]},{"type":"paragraph","attrs":{"id":"8119d08e-40a0-4797-9040-b4c59407c4f0","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Table 2."},{"type":"text","text":" Budget description and breakdown for an estimated 12 month timeline. "}]},{"type":"table","attrs":{"id":"d8e69946-2f95-4f4a-97e5-11a271c263d0"},"content":[{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":null},"content":[{"type":"paragraph","attrs":{"id":"653966ac-5536-4b07-a9de-a5dbd5a8c981","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Section"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":null},"content":[{"type":"paragraph","attrs":{"id":"ffd1cd07-cfa6-4a54-ae47-b76c4f9be60f","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Need"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":null},"content":[{"type":"paragraph","attrs":{"id":"3e6666dd-4d63-49fc-a583-7bf6a0a21d80","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Details"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":null},"content":[{"type":"paragraph","attrs":{"id":"7dd97198-b77b-487a-b661-99b1748ba291","class":null,"textAlign":"center"},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Cost"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"8eaad482-43a8-4b6a-9d4f-123eae5c64bd","class":null,"textAlign":null},"content":[{"type":"text","text":"Development"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"orderedList","attrs":{"start":1},"content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"bc269868-78cd-452e-8fff-1d098fce4cc3","class":null,"textAlign":null},"content":[{"type":"text","text":"Compute"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"e4a9bbb7-e054-4dc9-bf17-a37d6be4324f","class":null,"textAlign":null},"content":[{"type":"text","text":"API"}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"ea7eedb5-af72-41fd-a825-c27fde22efba","class":null,"textAlign":null},"content":[{"type":"text","text":"Outsourced developer work for data cleaning and UI/UX design"}]}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"438316bb-9839-4cf3-94d6-07a8fc34e382","class":null,"textAlign":null},"content":[{"type":"text","text":"Digital Ocean, o1/claude/gemini APIs, GPUs, observability system, auxiliary tools for AI Agents, and vector storage, etc."}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"cb20a808-1e68-49be-8e3c-6e1bc0b1efe7","class":null,"textAlign":null},"content":[{"type":"text","text":"$10,000"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"da7ee106-0cb6-4279-83b3-ae5c6a5a605f","class":null,"textAlign":null},"content":[{"type":"text","text":"Human editor costs"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"2f70821e-fc49-4b0f-84fe-be1be8e835d8","class":null,"textAlign":null},"content":[{"type":"text","text":"Incentive for Validators (Editors or Authors) to assess AI and Human Reviewer Outputs. Including incentives for validating papers, N = 50, from YesNoError (YNE)"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"63d2745b-ded9-49fe-ae7c-9bffe9deb005","class":null,"textAlign":null},"content":[{"type":"text","text":"Validators will be asked to provide either full or partial feedback on both the original preprint and/or the AI review submissions."}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"5e9e5273-4995-4bfe-afd5-552fcae69ad0","class":null,"textAlign":null},"content":[{"type":"text","text":"$50,000 for 1000 editors (assuming $50 to assess all peer reviews per manuscript)"}]},{"type":"paragraph","attrs":{"id":"6168b153-6272-4dc6-a773-5a38e1f52fad","class":null,"textAlign":null},"content":[{"type":"hardBreak"},{"type":"text","text":" "}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"79914fc8-5f3e-4009-80f2-36d2e95d13fd","class":null,"textAlign":null},"content":[{"type":"text","text":"Administrative Cost"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"b55aaf23-b5b1-4536-940b-1535f1e22f71","class":null,"textAlign":null},"content":[{"type":"text","text":"Marketing"},{"type":"hardBreak"},{"type":"hardBreak"},{"type":"text","text":" "}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"3bb8ac27-814c-4de8-8ffb-aef38a5f31bb","class":null,"textAlign":null},"content":[{"type":"text","text":"The cost associated with marketing efforts to ensure campaign visibility and sustain interest for contributions from the community."}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"a546b8f1-4856-437b-ac70-8d0e1797dfb7","class":null,"textAlign":null},"content":[{"type":"text","text":"$5,000"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"3c44806c-d5f2-4665-8b90-e0b0c8a96809","class":null,"textAlign":null},"content":[{"type":"text","text":"Reserve Fund"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"ed887d93-f21f-4e13-a83e-cb738aa4a863","class":null,"textAlign":null},"content":[{"type":"text","text":"For unexpected needs and for insurance against volatility. "}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"de0ccca9-4192-4106-95a8-f9f57d46c76a","class":null,"textAlign":null},"content":[{"type":"text","text":" "}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"style":"border:1pt solid #000000;padding:5pt;vertical-align:top;"},"content":[{"type":"paragraph","attrs":{"id":"d17a9064-7a62-4bf1-aa80-2a327316d9b7","class":null,"textAlign":null},"content":[{"type":"text","text":"$15,000, If unused, will be put toward getting a larger dataset."}]}]}]}]},{"type":"heading","attrs":{"id":"7c501115-d85f-492e-9d43-0bfc85ae253b","data-toc-id":"7c501115-d85f-492e-9d43-0bfc85ae253b","textAlign":"justify","level":2},"content":[{"type":"text","text":"Disclosure and Licensing Statement"}]},{"type":"paragraph","attrs":{"id":"a27c9522-23b3-47d9-891b-b7fd844342d5","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"This document outlines the terms under which the content, outputs, and tools resulting from our study on AI reviewers for scientific manuscripts are shared and utilized. Transparency and collaboration are core to our research philosophy, and we aim to strike a balance between open science principles and the potential for practical, real-world applications."}]},{"type":"heading","attrs":{"id":"f1f7a7cb-beaf-4874-acd5-c0c54c36fb74","data-toc-id":"f1f7a7cb-beaf-4874-acd5-c0c54c36fb74","textAlign":"justify","level":3},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Open Source Commitment"}]},{"type":"orderedList","attrs":{"start":1},"content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"e47d185a-026f-4a5c-b3ae-6cae5b8f6d00","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Content and Outputs"},{"type":"text","text":":"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"529957c9-986a-454f-ab72-0677e3645da1","class":null,"textAlign":null},"content":[{"type":"text","text":"All data, algorithms, models, and other outputs derived from this study will be made available under an open-source license, allowing unrestricted access for academic, research, and non-commercial purposes."}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"367f6e15-fd6e-4350-a45e-fef3500af411","class":null,"textAlign":null},"content":[{"type":"text","text":"The specific license will be the MIT License to ensure clarity regarding permissible uses and attribution requirements."}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"45878005-acbe-4323-a56a-8c41e47b31bd","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"bold"}],"text":"Repository Access"},{"type":"text","text":":"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","attrs":{"id":"ec653f0d-cc28-49b2-863f-aefdd1749693","class":null,"textAlign":null},"content":[{"type":"text","text":"Public repositories (e.g., on GitHub, Zenodo, HuggingFace or similar platforms) will host the content and outputs, accompanied by detailed documentation to facilitate their reuse and extension by the broader research community."}]},{"type":"paragraph","attrs":{"id":"80017aa2-be42-406c-b51d-d39e474737dc","class":null,"textAlign":null},"content":[{"type":"hardBreak"},{"type":"text","marks":[{"type":"bold"}],"text":"Commercialization Clause"}]}]}]}]}]},{"type":"paragraph","attrs":{"id":"4539fd2c-dcd5-4e75-8a93-af056260278e","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"While the content and outputs of this study will be open-sourced, we reserve the right to commercialize tools or technologies derived from these resources. "}]},{"type":"paragraph","attrs":{"id":"5b993517-4350-4d63-b66a-de5b2a35d538","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"By participating in this study or utilizing its outputs, users acknowledge and agree to the terms outlined in this disclosure and licensing statement."}]},{"type":"heading","attrs":{"id":"8d89c32b-dcfc-4b18-b4f8-b2c70c0e4337","data-toc-id":"8d89c32b-dcfc-4b18-b4f8-b2c70c0e4337","textAlign":"justify","level":2},"content":[{"type":"text","text":"Declaration of Generative AI in Scientific Writing"}]},{"type":"paragraph","attrs":{"id":"238437c9-6767-4329-b2eb-cfbc4fb4bba6","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"During the preparation of this work, ChatGPT was used to enhance the readability and language of some parts of the manuscript. Additionally, it was used to consolidate the original content written in the introduction, methods, and preliminary data to construct an Abstract. Following any ChatGPT usage, content was reviewed and edited as needed."}]},{"type":"heading","attrs":{"id":"69fc03a6-1a50-4537-9a9f-a8861a858bdb","data-toc-id":"69fc03a6-1a50-4537-9a9f-a8861a858bdb","textAlign":"justify","level":2},"content":[{"type":"text","text":"Acknowledgements"}]},{"type":"paragraph","attrs":{"id":"a6fd71d4-7abb-43e2-8e68-990bda383b72","class":null,"textAlign":"justify"},"content":[{"type":"text","text":"We would like to acknowledge Matt Schlicht, founder of YesNoError for being a Strategic Data Partner and Gareth Dyke & Maria Machado as Strategic Expert Community Partners. "}]},{"type":"heading","attrs":{"id":"e810c287-cbe9-4fd1-869d-01225474c7a5","data-toc-id":"e810c287-cbe9-4fd1-869d-01225474c7a5","textAlign":null,"level":2},"content":[{"type":"text","text":"References"}]},{"type":"paragraph","attrs":{"id":"e81ebc0d-6d7b-4724-9f82-c0d1da682177","class":null,"textAlign":null},"content":[{"type":"text","text":"[1] Howard Bauchner, Frederick P Rivara, Use of artificial intelligence and the future of peer review, "},{"type":"text","marks":[{"type":"italic"}],"text":"Health Affairs Scholar"},{"type":"text","text":", Volume 2, Issue 5, May 2024, qxae058, "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://doi.org/10.1093/haschl/qxae058","target":"_blank","rel":"noopener noreferrer nofollow","class":null}}],"text":"https://doi.org/10.1093/haschl/qxae058"}]},{"type":"paragraph","attrs":{"id":"1e8c4c3e-7700-4d5b-874e-9f4b6dedd201","class":null,"textAlign":null},"content":[{"type":"text","text":"[2] Aczel B, Barwich AS, Diekman AB, et al. The present and future of peer review: Ideas, interventions, and evidence. Proc Natl Acad Sci U S A. 2025;122(5):e2401232121. doi:10.1073/pnas.2401232121"}]},{"type":"paragraph","attrs":{"id":"9143643e-ae79-4c34-95db-8c5c999455e5","class":null,"textAlign":null},"content":[{"type":"text","text":"[3] Smith Nail and Cumberledge Aaron. 2020 Quotation errors in general science journals Proc. R. Soc. A. 47620200538. "},{"type":"text","marks":[{"type":"textStyle","attrs":{"fontSize":"14px","fontFamily":"\"Proxima Nova\", sans-serif","color":"rgb(51, 49, 50)"}}],"text":" "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://doi.org/10.1098/rspa.2020.0538","target":"_blank","rel":"noopener noreferrer nofollow","class":"doi-link link"}}],"text":"http://doi.org/10.1098/rspa.2020.0538"}]},{"type":"paragraph","attrs":{"id":"be1fc96a-5803-48f8-b7ac-36096bb36d6f","class":null,"textAlign":null},"content":[{"type":"text","text":"[4] Hanson, M. A., Gómez Barreiro, P., Crosetto, P., & Brockington, D. (2024). "},{"type":"text","marks":[{"type":"italic"}],"text":"The strain on scientific publishing"},{"type":"text","text":". arXiv. "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://arxiv.org/abs/2309.15884v2","target":"_blank","rel":"noopener noreferrer nofollow","class":null}},{"type":"underline"}],"text":"https://arxiv.org/abs/2309.15884v2"}]},{"type":"paragraph","attrs":{"id":"5d2e96b9-753b-4876-83b0-402841585b4e","class":null,"textAlign":null},"content":[{"type":"text","text":"[5] "},{"type":"text","marks":[{"type":"textStyle","attrs":{"fontSize":"16px","fontFamily":"-apple-system, \"system-ui\", \"Segoe UI\", Roboto, Oxygen-Sans, Ubuntu, Cantarell, \"Helvetica Neue\", sans-serif","color":"rgb(51, 51, 51)"}}],"text":"LeBlanc, A.G., Barnes, J.D., Saunders, T.J. "},{"type":"text","marks":[{"type":"italic"}],"text":"et al."},{"type":"text","marks":[{"type":"textStyle","attrs":{"fontSize":"16px","fontFamily":"-apple-system, \"system-ui\", \"Segoe UI\", Roboto, Oxygen-Sans, Ubuntu, Cantarell, \"Helvetica Neue\", sans-serif","color":"rgb(51, 51, 51)"}}],"text":" Scientific sinkhole: estimating the cost of peer review based on survey data with snowball sampling. "},{"type":"text","marks":[{"type":"italic"}],"text":"Res Integr Peer Rev"},{"type":"text","marks":[{"type":"textStyle","attrs":{"fontSize":"16px","fontFamily":"-apple-system, \"system-ui\", \"Segoe UI\", Roboto, Oxygen-Sans, Ubuntu, Cantarell, \"Helvetica Neue\", sans-serif","color":"rgb(51, 51, 51)"}}],"text":" "},{"type":"text","marks":[{"type":"bold"}],"text":"8"},{"type":"text","marks":[{"type":"textStyle","attrs":{"fontSize":"16px","fontFamily":"-apple-system, \"system-ui\", \"Segoe UI\", Roboto, Oxygen-Sans, Ubuntu, Cantarell, \"Helvetica Neue\", sans-serif","color":"rgb(51, 51, 51)"}}],"text":", 3 (2023). "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://doi.org/10.1186/s41073-023-00128-2","target":"_blank","rel":"noopener noreferrer nofollow","class":"link"}},{"type":"textStyle","attrs":{"fontSize":"16px","fontFamily":"-apple-system, \"system-ui\", \"Segoe UI\", Roboto, Oxygen-Sans, Ubuntu, Cantarell, \"Helvetica Neue\", sans-serif","color":"rgb(51, 51, 51)"}}],"text":"https://doi.org/10.1186/s41073-023-00128-2"}]},{"type":"paragraph","attrs":{"id":"7740f77a-dff9-4955-8d6c-405906aa016d","class":null,"textAlign":null},"content":[{"type":"text","marks":[{"type":"textStyle","attrs":{"fontSize":"16px","fontFamily":"-apple-system, \"system-ui\", \"Segoe UI\", Roboto, Oxygen-Sans, Ubuntu, Cantarell, \"Helvetica Neue\", sans-serif","color":"rgb(51, 51, 51)"}}],"text":"[6] "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://www.biorxiv.org/content/10.1101/2025.03.18.644032v1.full.pdf","target":"_blank","rel":"noopener noreferrer nofollow","class":"link"}},{"type":"textStyle","attrs":{"fontSize":"16px","fontFamily":"-apple-system, \"system-ui\", \"Segoe UI\", Roboto, Oxygen-Sans, Ubuntu, Cantarell, \"Helvetica Neue\", sans-serif","color":"rgb(51, 51, 51)"}}],"text":"https://www.biorxiv.org/content/10.1101/2025.03.18.644032v1.full.pdf"}]},{"type":"paragraph","attrs":{"id":"0ac6549c-8ed8-4136-aaa4-e2f5dbe71ccf","class":null,"textAlign":null},"content":[{"type":"text","text":"[7] Seghier, M.L. (2025), \"AI-powered peer review needs human supervision\", "},{"type":"text","marks":[{"type":"italic"}],"text":"Journal of Information, Communication and Ethics in Society"},{"type":"text","text":", Vol. 23 No. 1, pp. 104-116. "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://doi.org/10.1108/JICES-09-2024-0132","target":"_blank","rel":"noopener noreferrer nofollow","class":null}}],"text":"https://doi.org/10.1108/JICES-09-2024-0132"}]},{"type":"paragraph","attrs":{"id":"6643dce1-77e8-494d-8a18-a0ffbcde8440","class":null,"textAlign":null},"content":[{"type":"text","text":"[8] Kirtani, C., Garg, M. K., Prasad, T., Singhal, T., Mandal, M., & Kumar, D. (2025). "},{"type":"text","marks":[{"type":"italic"}],"text":"ReviewEval: An evaluation framework for AI-generated reviews"},{"type":"text","text":". arXiv. "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://arxiv.org/abs/2502.11736v2","target":"_blank","rel":"noopener noreferrer nofollow","class":null}},{"type":"underline"}],"text":"https://arxiv.org/abs/2502.11736v2"}]},{"type":"paragraph","attrs":{"id":"9daf6a9f-3bbf-4dc5-ad2a-14da53f280a5","class":null,"textAlign":null},"content":[{"type":"text","text":"[9] Yang, J. (2025). "},{"type":"text","marks":[{"type":"italic"}],"text":"Paper Copilot: The artificial intelligence and machine learning community should adopt a more transparent and regulated peer review process"},{"type":"text","text":". arXiv. "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://arxiv.org/abs/2502.00874","target":"_blank","rel":"noopener noreferrer nofollow","class":null}},{"type":"underline"}],"text":"https://arxiv.org/abs/2502.00874"}]},{"type":"paragraph","attrs":{"id":"72228cce-524a-438a-be5a-9d7bf25babe8","class":null,"textAlign":null},"content":[{"type":"text","text":"[10] Kuznetsov, I., Gurevych, I. et. a. (2024). "},{"type":"text","marks":[{"type":"italic"}],"text":"What can natural language processing do for peer review?"},{"type":"text","text":" arXiv. "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://arxiv.org/abs/2405.06563","target":"_blank","rel":"noopener noreferrer nofollow","class":null}},{"type":"underline"}],"text":"https://arxiv.org/abs/2405.06563"},{"type":"text","marks":[{"type":"underline"}],"text":" "},{"type":"text","text":"GitHub Repository: Afzal, O. M. (n.d.). "},{"type":"text","marks":[{"type":"italic"}],"text":"nlp-for-peer-review"},{"type":"text","text":". GitHub. "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/OAfzal/nlp-for-peer-review","target":"_blank","rel":"noopener noreferrer nofollow","class":null}},{"type":"underline"}],"text":"https://github.com/OAfzal/nlp-for-peer-review"}]},{"type":"paragraph","attrs":{"id":"aa6206f3-2b22-4e99-a73f-72cf75dfd2e0","class":null,"textAlign":null},"content":[{"type":"text","text":"[11] Brian, D. (2024). "},{"type":"text","marks":[{"type":"italic"}],"text":"Incentivized vs non-incentivized open peer reviews: Dynamics, economics, and quality"},{"type":"text","text":". ResearchHub. "},{"type":"text","marks":[{"type":"link","attrs":{"href":"https://doi.org/10.55277/ResearchHub.taescjxh","target":"_blank","rel":"noopener noreferrer nofollow","class":null}},{"type":"underline"}],"text":"https://doi.org/10.55277/ResearchHub.taescjxh"}]},{"type":"paragraph","attrs":{"id":"69cb50e0-8f88-456f-aedb-a9d807b6eb36","class":null,"textAlign":null}},{"type":"paragraph","attrs":{"id":"14e9c72c-3792-48fc-859a-608010825fef","class":null,"textAlign":null},"content":[{"type":"hardBreak"},{"type":"hardBreak"},{"type":"hardBreak"},{"type":"hardBreak"},{"type":"hardBreak"},{"type":"hardBreak"},{"type":"hardBreak"},{"type":"hardBreak"},{"type":"hardBreak"},{"type":"text","text":" "}]}]}8a:T9d60,

ARTEMIS - Automated Review and Trustworthy Evaluation for Manuscripts in Science

Authors and Affiliations

Jeffrey Koury, ResearchHub Foundation

Dominikus Brian, ResearchHub Foundation

Abstract

The peer review process, essential for maintaining scientific rigor, faces significant challenges including reviewer fatigue, bias, and insufficient rigor. The rapid increase in scientific publications has placed an unsustainable burden on a limited pool of reviewers, resulting in the inability of reviewers to detect falsified data, screen errors with high fidelity, and even establish consensus between reviewers [1]. ResearchHub Foundation aims to address these issues by 1) incentivizing reviewers and 2) integrating Agentic AI systems powered by recent breakthroughs in large language and multi-modal foundational models. Our unique approach with validated expert human reviews, and trustworthy automated tools to complement human reviewers, will enhance the quality and credibility of peer reviews to benefit both authors and the larger scientific community.

We have observed from our own experiments (200+ AI peer review samples) that AI reviewers have capabilities that excel in domains such as detecting technical errors and statistical flaws, while human reviewers provide greater insight into the broader relevance and context of research. We hypothesize that incorporating human feedback into AI models of peer review can improve their performance beyond existing state-of-the-art AI systems without human feedback. The project has two aims:

Establish a Reliable Protocol to Assess Reviewer Performance: Using a standardized rubric, human editors with specific sets of expertise will evaluate AI and human reviews across methods, results, discussion, rigor, novelty, and error detection. AI reviewers are expected to outperform in technical rigor, while humans excel in contextual analysis.
Develop a Versatile Agentic Solutions for Scientific Auditing : By leveraging upon the growing number of toolkits for deploying AI Agents driven by LLM, Large multi-modal models (LMMs), long-term memories, and orchestration infrastructure, we ought to develop specialized agentic solutions honed for various aspects of scientific document auditing.

This initiative leverages ResearchHub’s extensive reviewer database, standardized prompts, and expertise-weighted evaluations to enhance the peer review process. By addressing systemic issues and integrating AI and human expertise, this project seeks to improve the accuracy, efficiency, and scalability of scientific peer review, supporting the advancement of open and validated science.

Introduction and Significance

Scientific Auditing - Peer review is a cornerstone of scientific integrity, serving as the primary mechanism for validating research and granting manuscripts the status of “verified” work. However, the process is increasingly strained due to systemic challenges. Peer reviewers face difficulties in thoroughly assessing statistical analyses, verifying references, and evaluating claims against the broader scientific literature. Despite the critical nature of their role, reviewers are not compensated for their efforts, which often results in diminished rigor, personal biases, superficial feedback, and gaps in understanding key aspects of the studies they assess. These challenges are further exacerbated by the rapid growth in scientific publications, which has significantly increased the workload on an already limited pool of reviewers. The number of scientific publications has increased an estimated 47% from 2016 to 2022 [2]. This combination of rising demand and insufficient support has created a critical bottleneck in the peer review process, highlighting the urgent need for innovative solutions to enhance the accuracy, efficiency, and sustainability of scientific auditing. With the rise of techniques resulting in large terabyte sized datasets (i.e. single-cell RNAseq, BARseq) the complexity for detecting errors and the time demand to do so for human reviewers is amplified. Additionally, this unfortunate combination has resulted in excessive burden on a small number of reviewers resulting in reviewer fatigue. The use of ResearchCoin incentives on ResearchHub has provided the impetus to overcome this apprehension of reviewers to peer review, and provides a new found incentive and motivation. Nonetheless, even with appropriate incentives, human reviewers continue to have blindspots during the review process.

Agentic AI solution - With the rapid mass-adoption of LLMs and AI Agents, the ability to holistically assess a manuscript has become feasible at rapid speeds and low costs, providing immensely valuable insights into the paper that often reside within human reviewer’s blind spots. These tools can fill the gaps of human reviewers limitations and provide a more expedited turnaround time on feedback and requested edits, ultimately adding a more holistic set of feedback and context to the academic manuscript. Foundational models (o1, Claude, Gemini, LLaMA, DeepSeek, etc.) have made strides and provide a basis for this approach. With smaller amounts of data, such as peer review data on ResearchHub, it is already possible to substantially leverage these foundation models with minimal additional training data to construct a much more accurate peer review system greater than the sum of its parts. AI reviewers are not a panacea, and it’s becoming clear that human reviewers are still needed, but will have AI reviewers as a powerful supplement [3].

Hypothesis and Aims

Hypothesis: Published and preprinted papers have errors or issues within the manuscript that AI reviewers can catch that are often overlooked by human reviewers, and integrating human reviewer/editor reinforcement learning can enhance capabilities of the AI review system when domain specific context is essential.

Aim 1: Assess the strengths and weaknesses of human reviewers and AI reviewers using a standardized rubric. Human editors will evaluate the outputs of both human reviewers and AI reviewers for methods, results, discussion, rigor, novelty and error detection rate. Anticipated result: Human peer reviewers will be superior in interpreting the implications of the methods and results that are NOT explicitly mentioned in the manuscript, while AI reviewers will be more thorough in assessing statistics, detecting errors etc based on information available explicitly in the manuscript.

Aim 2: Develop versatile systems of agentic solutions driven by LLM, LMMs, and AI Agents, empowered by domain specific knowledge, in combination with human evaluation and feedback in producing thorough evaluation of scientific documents. Anticipated result: After tuning and including post-training of foundational models with the human evaluation and feedback, the resulting agentic solutions will perform better than the direct use of benchmark foundational models alone when used to a particular scientific auditing task (manuscript review, editor’s feedback, journal decision, etc.)

Methods and Design

Currently, the use of already existing foundation models equipped with simple prompt engineering has provided a viable starting point for pinpointing errors and giving additional context in scientific manuscripts. We’ve developed a minimum viable system in the form of a web app shown in Fig. 1 below. We also made several observations throughout our pilot experiment in which, holistically, using the same optimized prompt, Gemini 2.0 has been the most robust foundation model in producing consistent results human editors and readers generally consider more useful. This is in part due to the 1M+ context window for token processing, roughly 5-7 times greater than the typical 128k and 280k context windows found in other foundation models.

Figure 1. Web-application developed by ResearchHub Foundation using AI agents (AI Editor and AI Peer Reviewers).

Figure 2. Proposed ARTEMIS system architecture and workflow design.

In Fig. 2 , we present the proposed ARTEMIS system architecture and workflow design, here we follow on with describing the key actions happening within each step of the process:

Scientific Document Assigned for Decision-Making.
Human users who make decisions based on scientific documents submit audit requests to gain further insights regarding the document and the scientific works it represents.
Document Pre-Processing for Metadata Extraction.
Depending on the metadata content, a specific full extraction protocol is performed to obtain the maximum amount of information from the document.
Full extraction results are sent to the AI Quality Control agent to ensure they meet the required standards and can be used further.
(6b, 7b, 8b.) Extracted document content that fails to satisfy the minimum requirements is sent to humans for manual decision-making, annotation of reasons for exclusion, and archiving within the Data Lakehouse.
(6a & 6a’.) Extracted documents that pass the QC requirements, either automatically or manually after human consideration, are passed on to the AI Editor.
(7a.) The AI Editor generates or orchestrates the assignment of the document to the appropriate AI reviewer(s).
(8a.) AI Peer Reviewer(s) generate their assessment of the document and send it to the AI QC agent to ensure correct structure and compliance with the required review standards.
All reviews that pass the AI QC check are curated into a raw report ready for human evaluation.
The raw report is prepared (and reformatted if necessary) to be presented to humans for Human Evaluation.
Human evaluation results are submitted to the AI QC for further checks, including verifying whether the contribution is authentically human and backed by proper login credentials.
Human feedback from verified editors or validators is requested to justify the quality of the given human evaluation results.
All data collected thus far is curated into a coherent dataset corresponding to the scientific document and the auditing task at hand.
The dataset is organized into its respective zones and/or warehouses within the Data Lakehouse.
Actionable insights are produced based on the knowledge available within the Data Lakehouse to answer the questions posed in the original audit request. This process can also be further facilitated by interactive Data Librarian/Analyst Agents that help users explore or generate additional insights beyond the original request.

Aim 1 - Strengths & Weaknesses of Human vs AI Reviewers

The first stage of the proposal is focused on producing the “Human Wisdom” components described across steps 10 to 13 of the ARTEMIS architecture and work flow (Fig. 2). In the process of producing our human evaluation components of our desired dataset, we will prompt the AI Reviewers and Human Reviewers to perform assessment based on two requirements:

Classic ResearchHub Foundation prompt:
1. ResearchHub Foundation bounty prompt (italicized):
  1. Your review must be submitted during the 14-day submission window from the day this bounty was initiated.
  2. Mention your credentials and include your areas of relevant expertise and limitations in assessing this preprint.
  3. Include the version of the preprint you are reviewing (e.g. 1st, 2nd, etc.). Make sure you review the most updated version available.
  4. Use the rating system in the "Peer Reviews" tab for all 5 criteria (overall assessment, introduction, methods, results, discussion) but the content within each is flexible (in-line comments can be used instead of a block of text in each section).
  5. For each figure, including supplementary material, you must provide a detailed assessment (pros or cons).
  6. Plagiarism will not be tolerated. You must comprehensively disclose any use of artificial intelligence (AI) platforms used in the review process. Please refer to our AI Policy for additional details.
2. Currently, ResearchHub processes approximately 170 peer reviews a week (Fig 3a), with 68.7% of all preprints who receive a peer review incentive receiving at least 1 peer review that meets the bar of quality (Fig 3b). Only reviews meeting this minimum bar, as deemed by ResearchHub’s editorial team, will be leveraged for this study.
3. The fields of expertise that ResearchHub peer reviewers specialize in are namely in the biomedical and hard sciences including molecular biology, immunology, oncology, neuroscience, etc (Fig 3c, 3d).
Error Detection prompt:

Please carefully review the following academic paper and identify any errors, issues, inconsistencies, or 
questionable reasoning, with a special focus on the mathematical content. In particular, examine the 
correctness and clarity of all formulas, derivations, and calculations. Check for doubtful hypotheses, shaky
assumptions, incorrect mathematical statements, undefined terms, unjustified steps, missing references for 
key theorems, or discrepancies between the described methods and the results presented. If any part of the 
argument seems incomplete, unclear, or unconvincing, highlight these sections and explain why they might be 
problematic. Additionally, look for subtle errors such as misapplied formulas, overlooked boundary 
conditions, or unsupported assumptions in modeling steps. Make sure to check the figures, captions, and meta
data of the paper whenever relevant. Finally, suggest potential ways to strengthen or clarify the 
mathematical aspects of the paper.

{
  "overall_assessment": "<Overall Assessment and Key Takeaways>",
  "mathematical_rigor_and_correctness": {
    "equation_summary": {
      "inline_equations": <number_of_inline_equations>,
      "display_equations": <number_of_display_equations>
    },
   "equation_details" :{
      "inline_equations": <list of identified inline_equations>,
      "display_equations": <list of identified display_equations>,
    },
    "abbreviation_and_symbol_definitions": "<Comment on whether all abbreviations and mathematical symbols are well-defined. 
    Explicitly explain ill defined items.>",
    "sign_and_operation_checks": "<Comment on the correctness of signs and mathematical operations. 
    Explicitly explain what is the issue and where are the errors.>"
  },
  "issues_list": [
    {
      "issue": "<Description of Issue 1>",
      "severity_rating": <numerical_rating_from_1.0_to_10.0>,
      "severity_explanation": "<Explanation of the rating with rationale>",
      "suggestion": "<Suggestion with concrete actionable steps to address this issue>"
    },
    {
      "issue": "<Description of Issue 2>",
      "severity_rating": <numerical_rating_from_1.0_to_10.0>,
      "severity_explanation": "<Explanation of the rating with rationale>",
      "suggestion": "<Suggestion with concrete actionable steps to address this issue>"
    }
    // Repeat for issues 3 to 10
  ]
}

Standardized Scoring Approach for Severity Rating:

1.0 - 2.9: Minor typographical errors or negligible issues that do not affect understanding.
3.0 - 5.9: Moderate issues that may cause some confusion or misunderstandings but are correctable.
6.0 - 8.9: Significant errors that impact the validity of results or the clarity of the paper.
9.0 - 10.0: Critical flaws that fundamentally undermine the paper's conclusions or correctness.
In the issues_list, provide a list of the 10 most important or most glaring mistakes or issues, following the standardized 
scoring approach to maintain objectivity across papers. Be specific to the current paper.

**Ensure that your response is formatted as valid JSON only, without any additional text or commentary.
At ALL COST, NEVER INCLUDES ```json, in your valid JSON answer. a valid json should start with { and ends with }

Figure 3a. Total Number of Completed Peer Reviews per Week on ResearchHub from May 2022 to present. ResearchHub has housed almost 4000 peer reviews at a rate of roughly 170 peer reviews a week. Blue indicates a concerted effort to promote the peer review process on ResearchHub.

Figure 3b. Total number of manuscripts separated by if ResearchHub Foundation received peer reviews where we distributed the bounty (green, 68.7%) or didn’t distribute the bounty (red, 31.3%). Green signals a manuscript which received at least 1 peer review that met a minimum threshold of quality to receive the bounty. Red signals a manuscript which either didn’t receive any peer review or received peer review but didn’t meet a minimum threshold of quality or abiding by the stipulations set by ResearchHub Foundation.

Figure 3c. Number of Peer Reviews previously completed on ResearchHub stratified by OpenAlex Subfields

Figure 3d. Number of Peer Reviews previously completed on ResearchHub stratified by OpenAlex Topics

The Human Feedback component will be primarily driven by human editors/validators who will assess the precision and accuracy of the human and AI reviewers based on the claims made. The collection of human feedback will then be used to benchmark and improve the AI-generated reviews. An example of how this benchmarking can be performed is shown in the recent work ReviewEval by Kumar et al.[4] and the references therein. Part of which is already available within the ARTEMIS system under development. With the growth in community interest for open peer reviews, of which ResearchHub has adopted, human validators can quickly view and provide input to the reviews being produced in an open transparent marketplace.[5] The editors/validators will use a flash cards style approach for simple validation. The flash cards approach will appear in several forms, depending on the relevant context. Typical cases would include: Fact-Check, Logic-Check, Math-Check, or Expert Confidence. This alternative approach to consolidating expert feedback is designed as a "bite-sized" task that an expert in the field can answer or reason through without needing to read the entire paper. All context necessary for understanding or providing meaningful feedback on the specific excerpt will be provided on the flashcard. Peer reviewers are assumed to be knowledgeable about the particular topic and their claims will be subject to various weights depending on their background/reputation.

ResearchHub’s verification and reputation system: Using OpenAlex, an open repository for scientific data, ResearchHub has ascribed a reputation score for verified users within their various fields of expertise (Fig. 4). This will be used to value or weigh the statements made by various peer reviewers and validators.
Reviewer’s track record will be checked based on their past performances and previous editorial feedback, if any available. Additionally, each peer reviewer may need to complete a mini-quiz before receiving a tag for specific expertise, ensuring that they actually possess the required expertise. If the reputation system isn’t sufficient for properly confirming a user’s expertise, the mini-quiz will be employed and include single line domain specific questions, with each question restricted to a 5-10 second countdown to receive an answer.

Figure 4. Example of a verified user account on ResearchHub with a quantitative percentile ranking as a “reputation” based on publication history.

Alternative Approach: Assuming reviewers and/or editors will be difficult to source for the work, alternatively we can validate the AI reviews against other pre-existing datasets [6]. Additionally, in the context where the flash cards approach doesn’t prove to be a good source of validation, we can have validators assess the reviewers based on an adapted version of the ResearchHub Foundation’s standardized unified rubric (not public, but unified recommendations to reviewers can be found here) to give weighted scores on methods, results, discussion, rigor, novelty and error detection rate. Editors evaluations will be weighted contingent on their reputation score in that field according to ResearchHub’s reputation system.

Aim 2 - Develop Versatile Agentic Solutions for Scientific Auditing

Functions described in steps 5 to 9 (Fig. 2) for developing AI agents for scientific auditing, we propose to develop 3 agentic roles and 5 auxiliary tooling functions. The 3 agentic roles will be composed of: Quality Control Agent, AI Reviewer Agent, AI Editor. Each of these roles may be deployed under different context and configuration to achieve the desired performance. For example, this may include an AI Editor in Neuroscience, AI Reviewer in Neuroscience, AI quality control in Neuroscience etc parallelized for a plethora of fields of expertise.

The 5 Auxiliary toolings are : Long-Term Memory, Knowledge Base, Visual Understanding and Reasoning, Scientific References, Consistency of Structured Output. The design of these toolings will be made as modular as possible to allow replacement of the tech-stack when necessary to boost performance and efficiency.

Rigor

Human reviewers can openly review any preprints, allowing for expedited reviews. The presence of a standardized prompt for human and AI reviewers, and standardized rubrics for editors allows for apples to apples direct comparisons on specific components of the reviewers. The verification and reputation v2 system on ResearchHub will be employed to weight evaluations according to the expertise of the individual making the claims.

Additionally, prior to this pre-registration, we have performed preliminary studies that attempt to understand the dynamics, economics, and quality of incentivized peer review process of which is being employed on ResearchHub [7]. All three aspects of open peer review submissions addressed above (dynamics, economics, and quality) are highly dependent with at least 9 other coupled factors beyond economic and non-economic incentives (Table 1). Therefore, any correlation found would be contextual and apply only to a certain combination of peer review configurations that involved at least the 9 factors. As it pertains to aim 1 of this pre-registration, we will ensure that the peer review context will be fixed ensuring maximal reproducibility.

Table 1. The Nine Identified factors influencing correlation between incentives on the quality of open peer reviews of scientific manuscripts

Factor Influencing Incentive-Quality Correlation	Variables
Order of Incentivization	Award First (Commissioned Task) Award Later (Rewarded) Bidding Mechanism (Stake for Chance)
Reviewer Pool Size	Small [<100] Medium [100-1000] Large [1000+]
Expertise Diversity	Ranging from 0 to 1 0: All reviewers are from the exact same research subfield 1: For each subfield there are exactly equal number of reviewer * Each reviewer can have more than 1 expertise, but maximum 3 notable expertise. Complete list of subfields, will be at most 3 x number of reviewers in the pool.
Reviewer Selection Criteria	Single Chief-Editor (or Campaign Host) make the final call First Come First Served [Based on Objective Quality Scores or Time Order] Democratic Decision [Community Votes or Expert Scoring]
Type of Incentive and Size/Intensity	Monetary Reputation Monetary + Reputation
Requested Review Structure	Simple (Yes/No answer and inline comments) Moderate (Structured Comments on Different Aspects) Rigor (Complete Assessment on All Sections, Figures, References, and more)
Difficulties of Review Task	Easy (General Knowledgeable Public) Medium (Graduate Student Level on the Topic) Hard (Subject Expert on the Topic)
Likelihood & Proportion of AI Usage	Low (<10%) Moderate (10-90%) High (>90%)
Time Restriction	Intense (<3 Days) Moderate (3-14 Days) Leisurely ( >15 Days)

Budget

Table 2. Budget description and breakdown for an estimated 12 month timeline.

Section	Need	Details	Cost
Development	Compute API Outsourced developer work for data cleaning and UI/UX design	Digital Ocean, o1/claude/gemini APIs, GPUs, observability system, auxiliary tools for AI Agents, and vector storage, etc.	$10,000
Human reviewer costs	Incentive for Peer Reviewers for ResearchHub Foundation selected papers Incentive for Peer Reviewers for YesNoError (YNE) selected papers - 50 papers Incentive for Peer Reviewers for Gareth and Maria’s selected papers - 50 papers	Preprints in the fields of biology, chemistry, neuroscience, immunology, genetics, molecular biology, biochemistry, oncology, mathematics, physics, computer science, etc	$150,000 for 1000 human reviews (assuming $150 per peer review and 1 review per preprint)
Human editor costs	Incentive for Validators (Editors or Authors) to assess AI and Human Reviewer Outputs	Validators will be asked to provide either full or partial feedback on both the original preprint and/or the AI review submissions.	$50,000 for 1000 editors (assuming $50 to assess all peer reviews per manuscript)
Administrative Cost	Marketing	The cost associated with marketing efforts to ensure campaign visibility and sustain interest for contributions from the community.	$5,000
Reserve Fund	For unexpected needs and for insurance against volatility.		$15,000, If unused, will be put toward getting a larger dataset.

Disclosure and Licensing Statement

This document outlines the terms under which the content, outputs, and tools resulting from our study on AI reviewers for scientific manuscripts are shared and utilized. Transparency and collaboration are core to our research philosophy, and we aim to strike a balance between open science principles and the potential for practical, real-world applications.

Open Source Commitment

Content and Outputs:
- All data, algorithms, models, and other outputs derived from this study will be made available under an open-source license, allowing unrestricted access for academic, research, and non-commercial purposes.
- The specific license will be the MIT License to ensure clarity regarding permissible uses and attribution requirements.
Repository Access:
- Public repositories (e.g., on GitHub, Zenodo, HuggingFace or similar platforms) will host the content and outputs, accompanied by detailed documentation to facilitate their reuse and extension by the broader research community.

Commercialization Clause

While the content and outputs of this study will be open-sourced, we reserve the right to commercialize tools or technologies derived from these resources.

By participating in this study or utilizing its outputs, users acknowledge and agree to the terms outlined in this disclosure and licensing statement.

Declaration of Generative AI in Scientific Writing

During the preparation of this work, ChatGPT was used to enhance the readability and language of some parts of the manuscript. Additionally, it was used to consolidate the original content written in the introduction, methods, and preliminary data to construct an Abstract. Following any ChatGPT usage, content was reviewed and edited as needed.

Acknowledgements

We would like to acknowledge Matt Schlicht, founder of YesNoError for being a Strategic Data Partner and Gareth Dyke & Maria Machado as Strategic Expert Community Partners.

References

[1] Howard Bauchner, Frederick P Rivara, Use of artificial intelligence and the future of peer review, Health Affairs Scholar, Volume 2, Issue 5, May 2024, qxae058, https://doi.org/10.1093/haschl/qxae058

[2] Hanson, M. A., Gómez Barreiro, P., Crosetto, P., & Brockington, D. (2024). The strain on scientific publishing. arXiv. https://arxiv.org/abs/2309.15884v2

[3] Seghier, M.L. (2025), "AI-powered peer review needs human supervision", Journal of Information, Communication and Ethics in Society, Vol. 23 No. 1, pp. 104-116. https://doi.org/10.1108/JICES-09-2024-0132

[4] Kirtani, C., Garg, M. K., Prasad, T., Singhal, T., Mandal, M., & Kumar, D. (2025). ReviewEval: An evaluation framework for AI-generated reviews. arXiv. https://arxiv.org/abs/2502.11736v2

[5] Yang, J. (2025). Paper Copilot: The artificial intelligence and machine learning community should adopt a more transparent and regulated peer review process. arXiv. https://arxiv.org/abs/2502.00874

[6] Kuznetsov, I., Gurevych, I. et. a. (2024). What can natural language processing do for peer review? arXiv. https://arxiv.org/abs/2405.06563 GitHub Repository: Afzal, O. M. (n.d.). nlp-for-peer-review. GitHub. https://github.com/OAfzal/nlp-for-peer-review

[7] Brian, D. (2024). Incentivized vs non-incentivized open peer reviews: Dynamics, economics, and quality. ResearchHub. https://doi.org/10.55277/ResearchHub.taescjxh

8b:T40a,https://lh3.googleusercontent.com/a-/ALV-UjV75PbO1JYiNs4s5u0Tqb3lxR65SNq75XHU5HccIBIGHEawLfDhMfmrCBE-3EHcOU5rmdP7Kt6Jh4zrOr7NaWRYTiilDiA9zftNCqXgIKtnDXLNJUKwISP3cQj5M1E1a5KucCCUdszFrp12pooygqYqLbXPI6KUDF9MBxpoSE1KqiZVVHEmwjaJEovgyROZV2oGAYa9xJ3jJeFOHXS9ZeJepfr-geWM6ayVuGEEnmWzR_V0G-nV0LTSFzIC7_rKn8Qy8FBvScvYPm1KyElkUlzjsZwqwaV4odqN34y7zJF-U7KpkvaAwlf7t0EddxpGqxHUcuyHVwUt6iYTwSI2mNZc8w-VzTeaFFhbPLil2qZ6DDbBm1J_SmPi_FWp2Kfdup_CYpcGy8BQp0SKFSuA9_YuaExJNKR22nYqATOSnL0pRwlxKUsDc_E6-K2H6ZYSKO7-Pf2-prr9eDLIkvpML3V5BJzZX370dZdrA2jnpMNViGLMpcjIgJxdGtWUKKAcR0HVZqL84_vc6MBh7m817Q8uuVGHjv-BkrbUy12LfIiu747RckHXk_QIEWCOrSTVauzfkcAvExfQ9do2IfiJdALD0YJ_cFG1HSO3VtTUAWHosI9LkB0y89i38eFBZg7R2oPU6__4_AueJ3Zc5SgVFarBwOFYKox-1rtrXi_UYexOxsjEJaP5Cg3TJWdGX26gFKCkGFEzXyZ-D6AgbS7EWz8PqPIFCxaYMXyauFwxmsX2PSRpaLRD2FA8r5rpYxq_qp2M5rmDP7p5yYXG7SvCtbyakxH3-zhNtvVIkf2LkojhmpkEnbNxzaFef7VzSH1B0tW2mXuMMYb5612Swd3oCJNbLKMNRe9xiY4BufGvnK396PWWOGNOdinzBlyTntb0GJSN8hzD8RJAw5q1tqi4RInZkZAzscMds14W6OVhvSxNWCu9NZm7lDy6kfL4HXFMV36JjdRMUNWCrz_A_Dd_b3k7iF8=s96-c8c:T9d60,

ARTEMIS - Automated Review and Trustworthy Evaluation for Manuscripts in Science

Authors and Affiliations

Jeffrey Koury, ResearchHub Foundation

Dominikus Brian, ResearchHub Foundation

Abstract

Establish a Reliable Protocol to Assess Reviewer Performance: Using a standardized rubric, human editors with specific sets of expertise will evaluate AI and human reviews across methods, results, discussion, rigor, novelty, and error detection. AI reviewers are expected to outperform in technical rigor, while humans excel in contextual analysis.
Develop a Versatile Agentic Solutions for Scientific Auditing : By leveraging upon the growing number of toolkits for deploying AI Agents driven by LLM, Large multi-modal models (LMMs), long-term memories, and orchestration infrastructure, we ought to develop specialized agentic solutions honed for various aspects of scientific document auditing.

Introduction and Significance

Hypothesis and Aims

Methods and Design

Figure 1. Web-application developed by ResearchHub Foundation using AI agents (AI Editor and AI Peer Reviewers).

Figure 2. Proposed ARTEMIS system architecture and workflow design.

In Fig. 2 , we present the proposed ARTEMIS system architecture and workflow design, here we follow on with describing the key actions happening within each step of the process:

Scientific Document Assigned for Decision-Making.
Human users who make decisions based on scientific documents submit audit requests to gain further insights regarding the document and the scientific works it represents.
Document Pre-Processing for Metadata Extraction.
Depending on the metadata content, a specific full extraction protocol is performed to obtain the maximum amount of information from the document.
Full extraction results are sent to the AI Quality Control agent to ensure they meet the required standards and can be used further.
(6b, 7b, 8b.) Extracted document content that fails to satisfy the minimum requirements is sent to humans for manual decision-making, annotation of reasons for exclusion, and archiving within the Data Lakehouse.
(6a & 6a’.) Extracted documents that pass the QC requirements, either automatically or manually after human consideration, are passed on to the AI Editor.
(7a.) The AI Editor generates or orchestrates the assignment of the document to the appropriate AI reviewer(s).
(8a.) AI Peer Reviewer(s) generate their assessment of the document and send it to the AI QC agent to ensure correct structure and compliance with the required review standards.
All reviews that pass the AI QC check are curated into a raw report ready for human evaluation.
The raw report is prepared (and reformatted if necessary) to be presented to humans for Human Evaluation.
Human evaluation results are submitted to the AI QC for further checks, including verifying whether the contribution is authentically human and backed by proper login credentials.
Human feedback from verified editors or validators is requested to justify the quality of the given human evaluation results.
All data collected thus far is curated into a coherent dataset corresponding to the scientific document and the auditing task at hand.
The dataset is organized into its respective zones and/or warehouses within the Data Lakehouse.
Actionable insights are produced based on the knowledge available within the Data Lakehouse to answer the questions posed in the original audit request. This process can also be further facilitated by interactive Data Librarian/Analyst Agents that help users explore or generate additional insights beyond the original request.

Aim 1 - Strengths & Weaknesses of Human vs AI Reviewers

Classic ResearchHub Foundation prompt:
1. ResearchHub Foundation bounty prompt (italicized):
  1. Your review must be submitted during the 14-day submission window from the day this bounty was initiated.
  2. Mention your credentials and include your areas of relevant expertise and limitations in assessing this preprint.
  3. Include the version of the preprint you are reviewing (e.g. 1st, 2nd, etc.). Make sure you review the most updated version available.
  4. Use the rating system in the "Peer Reviews" tab for all 5 criteria (overall assessment, introduction, methods, results, discussion) but the content within each is flexible (in-line comments can be used instead of a block of text in each section).
  5. For each figure, including supplementary material, you must provide a detailed assessment (pros or cons).
  6. Plagiarism will not be tolerated. You must comprehensively disclose any use of artificial intelligence (AI) platforms used in the review process. Please refer to our AI Policy for additional details.
2. Currently, ResearchHub processes approximately 170 peer reviews a week (Fig 3a), with 68.7% of all preprints who receive a peer review incentive receiving at least 1 peer review that meets the bar of quality (Fig 3b). Only reviews meeting this minimum bar, as deemed by ResearchHub’s editorial team, will be leveraged for this study.
3. The fields of expertise that ResearchHub peer reviewers specialize in are namely in the biomedical and hard sciences including molecular biology, immunology, oncology, neuroscience, etc (Fig 3c, 3d).
Error Detection prompt:

Please carefully review the following academic paper and identify any errors, issues, inconsistencies, or 
questionable reasoning, with a special focus on the mathematical content. In particular, examine the 
correctness and clarity of all formulas, derivations, and calculations. Check for doubtful hypotheses, shaky
assumptions, incorrect mathematical statements, undefined terms, unjustified steps, missing references for 
key theorems, or discrepancies between the described methods and the results presented. If any part of the 
argument seems incomplete, unclear, or unconvincing, highlight these sections and explain why they might be 
problematic. Additionally, look for subtle errors such as misapplied formulas, overlooked boundary 
conditions, or unsupported assumptions in modeling steps. Make sure to check the figures, captions, and meta
data of the paper whenever relevant. Finally, suggest potential ways to strengthen or clarify the 
mathematical aspects of the paper.

{
  "overall_assessment": "<Overall Assessment and Key Takeaways>",
  "mathematical_rigor_and_correctness": {
    "equation_summary": {
      "inline_equations": <number_of_inline_equations>,
      "display_equations": <number_of_display_equations>
    },
   "equation_details" :{
      "inline_equations": <list of identified inline_equations>,
      "display_equations": <list of identified display_equations>,
    },
    "abbreviation_and_symbol_definitions": "<Comment on whether all abbreviations and mathematical symbols are well-defined. 
    Explicitly explain ill defined items.>",
    "sign_and_operation_checks": "<Comment on the correctness of signs and mathematical operations. 
    Explicitly explain what is the issue and where are the errors.>"
  },
  "issues_list": [
    {
      "issue": "<Description of Issue 1>",
      "severity_rating": <numerical_rating_from_1.0_to_10.0>,
      "severity_explanation": "<Explanation of the rating with rationale>",
      "suggestion": "<Suggestion with concrete actionable steps to address this issue>"
    },
    {
      "issue": "<Description of Issue 2>",
      "severity_rating": <numerical_rating_from_1.0_to_10.0>,
      "severity_explanation": "<Explanation of the rating with rationale>",
      "suggestion": "<Suggestion with concrete actionable steps to address this issue>"
    }
    // Repeat for issues 3 to 10
  ]
}

Standardized Scoring Approach for Severity Rating:

1.0 - 2.9: Minor typographical errors or negligible issues that do not affect understanding.
3.0 - 5.9: Moderate issues that may cause some confusion or misunderstandings but are correctable.
6.0 - 8.9: Significant errors that impact the validity of results or the clarity of the paper.
9.0 - 10.0: Critical flaws that fundamentally undermine the paper's conclusions or correctness.
In the issues_list, provide a list of the 10 most important or most glaring mistakes or issues, following the standardized 
scoring approach to maintain objectivity across papers. Be specific to the current paper.

**Ensure that your response is formatted as valid JSON only, without any additional text or commentary.
At ALL COST, NEVER INCLUDES ```json, in your valid JSON answer. a valid json should start with { and ends with }

Figure 3c. Number of Peer Reviews previously completed on ResearchHub stratified by OpenAlex Subfields

Figure 3d. Number of Peer Reviews previously completed on ResearchHub stratified by OpenAlex Topics

ResearchHub’s verification and reputation system: Using OpenAlex, an open repository for scientific data, ResearchHub has ascribed a reputation score for verified users within their various fields of expertise (Fig. 4). This will be used to value or weigh the statements made by various peer reviewers and validators.
Reviewer’s track record will be checked based on their past performances and previous editorial feedback, if any available. Additionally, each peer reviewer may need to complete a mini-quiz before receiving a tag for specific expertise, ensuring that they actually possess the required expertise. If the reputation system isn’t sufficient for properly confirming a user’s expertise, the mini-quiz will be employed and include single line domain specific questions, with each question restricted to a 5-10 second countdown to receive an answer.

Figure 4. Example of a verified user account on ResearchHub with a quantitative percentile ranking as a “reputation” based on publication history.

Aim 2 - Develop Versatile Agentic Solutions for Scientific Auditing

Rigor

Table 1. The Nine Identified factors influencing correlation between incentives on the quality of open peer reviews of scientific manuscripts

Factor Influencing Incentive-Quality Correlation	Variables
Order of Incentivization	Award First (Commissioned Task) Award Later (Rewarded) Bidding Mechanism (Stake for Chance)
Reviewer Pool Size	Small [<100] Medium [100-1000] Large [1000+]
Expertise Diversity	Ranging from 0 to 1 0: All reviewers are from the exact same research subfield 1: For each subfield there are exactly equal number of reviewer * Each reviewer can have more than 1 expertise, but maximum 3 notable expertise. Complete list of subfields, will be at most 3 x number of reviewers in the pool.
Reviewer Selection Criteria	Single Chief-Editor (or Campaign Host) make the final call First Come First Served [Based on Objective Quality Scores or Time Order] Democratic Decision [Community Votes or Expert Scoring]
Type of Incentive and Size/Intensity	Monetary Reputation Monetary + Reputation
Requested Review Structure	Simple (Yes/No answer and inline comments) Moderate (Structured Comments on Different Aspects) Rigor (Complete Assessment on All Sections, Figures, References, and more)
Difficulties of Review Task	Easy (General Knowledgeable Public) Medium (Graduate Student Level on the Topic) Hard (Subject Expert on the Topic)
Likelihood & Proportion of AI Usage	Low (<10%) Moderate (10-90%) High (>90%)
Time Restriction	Intense (<3 Days) Moderate (3-14 Days) Leisurely ( >15 Days)

Budget

Table 2. Budget description and breakdown for an estimated 12 month timeline.

Section	Need	Details	Cost
Development	Compute API Outsourced developer work for data cleaning and UI/UX design	Digital Ocean, o1/claude/gemini APIs, GPUs, observability system, auxiliary tools for AI Agents, and vector storage, etc.	$10,000
Human reviewer costs	Incentive for Peer Reviewers for ResearchHub Foundation selected papers Incentive for Peer Reviewers for YesNoError (YNE) selected papers - 50 papers Incentive for Peer Reviewers for Gareth and Maria’s selected papers - 50 papers	Preprints in the fields of biology, chemistry, neuroscience, immunology, genetics, molecular biology, biochemistry, oncology, mathematics, physics, computer science, etc	$150,000 for 1000 human reviews (assuming $150 per peer review and 1 review per preprint)
Human editor costs	Incentive for Validators (Editors or Authors) to assess AI and Human Reviewer Outputs	Validators will be asked to provide either full or partial feedback on both the original preprint and/or the AI review submissions.	$50,000 for 1000 editors (assuming $50 to assess all peer reviews per manuscript)
Administrative Cost	Marketing	The cost associated with marketing efforts to ensure campaign visibility and sustain interest for contributions from the community.	$5,000
Reserve Fund	For unexpected needs and for insurance against volatility.		$15,000, If unused, will be put toward getting a larger dataset.

Disclosure and Licensing Statement

Open Source Commitment

Content and Outputs:
- All data, algorithms, models, and other outputs derived from this study will be made available under an open-source license, allowing unrestricted access for academic, research, and non-commercial purposes.
- The specific license will be the MIT License to ensure clarity regarding permissible uses and attribution requirements.
Repository Access:
- Public repositories (e.g., on GitHub, Zenodo, HuggingFace or similar platforms) will host the content and outputs, accompanied by detailed documentation to facilitate their reuse and extension by the broader research community.

Commercialization Clause

While the content and outputs of this study will be open-sourced, we reserve the right to commercialize tools or technologies derived from these resources.

By participating in this study or utilizing its outputs, users acknowledge and agree to the terms outlined in this disclosure and licensing statement.

Declaration of Generative AI in Scientific Writing

Acknowledgements

We would like to acknowledge Matt Schlicht, founder of YesNoError for being a Strategic Data Partner and Gareth Dyke & Maria Machado as Strategic Expert Community Partners.

References

[2] Hanson, M. A., Gómez Barreiro, P., Crosetto, P., & Brockington, D. (2024). The strain on scientific publishing. arXiv. https://arxiv.org/abs/2309.15884v2

[4] Kirtani, C., Garg, M. K., Prasad, T., Singhal, T., Mandal, M., & Kumar, D. (2025). ReviewEval: An evaluation framework for AI-generated reviews. arXiv. https://arxiv.org/abs/2502.11736v2

[7] Brian, D. (2024). Incentivized vs non-incentivized open peer reviews: Dynamics, economics, and quality. ResearchHub. https://doi.org/10.55277/ResearchHub.taescjxh

8d:T40a,https://lh3.googleusercontent.com/a-/ALV-UjV75PbO1JYiNs4s5u0Tqb3lxR65SNq75XHU5HccIBIGHEawLfDhMfmrCBE-3EHcOU5rmdP7Kt6Jh4zrOr7NaWRYTiilDiA9zftNCqXgIKtnDXLNJUKwISP3cQj5M1E1a5KucCCUdszFrp12pooygqYqLbXPI6KUDF9MBxpoSE1KqiZVVHEmwjaJEovgyROZV2oGAYa9xJ3jJeFOHXS9ZeJepfr-geWM6ayVuGEEnmWzR_V0G-nV0LTSFzIC7_rKn8Qy8FBvScvYPm1KyElkUlzjsZwqwaV4odqN34y7zJF-U7KpkvaAwlf7t0EddxpGqxHUcuyHVwUt6iYTwSI2mNZc8w-VzTeaFFhbPLil2qZ6DDbBm1J_SmPi_FWp2Kfdup_CYpcGy8BQp0SKFSuA9_YuaExJNKR22nYqATOSnL0pRwlxKUsDc_E6-K2H6ZYSKO7-Pf2-prr9eDLIkvpML3V5BJzZX370dZdrA2jnpMNViGLMpcjIgJxdGtWUKKAcR0HVZqL84_vc6MBh7m817Q8uuVGHjv-BkrbUy12LfIiu747RckHXk_QIEWCOrSTVauzfkcAvExfQ9do2IfiJdALD0YJ_cFG1HSO3VtTUAWHosI9LkB0y89i38eFBZg7R2oPU6__4_AueJ3Zc5SgVFarBwOFYKox-1rtrXi_UYexOxsjEJaP5Cg3TJWdGX26gFKCkGFEzXyZ-D6AgbS7EWz8PqPIFCxaYMXyauFwxmsX2PSRpaLRD2FA8r5rpYxq_qp2M5rmDP7p5yYXG7SvCtbyakxH3-zhNtvVIkf2LkojhmpkEnbNxzaFef7VzSH1B0tW2mXuMMYb5612Swd3oCJNbLKMNRe9xiY4BufGvnK396PWWOGNOdinzBlyTntb0GJSN8hzD8RJAw5q1tqi4RInZkZAzscMds14W6OVhvSxNWCu9NZm7lDy6kfL4HXFMV36JjdRMUNWCrz_A_Dd_b3k7iF8=s96-c8e:T6f97,Authors and AffiliationsJeffrey Koury, ResearchHub FoundationDominikus Brian, ResearchHub FoundationAbstractThe peer review process, essential for maintaining scientific rigor, faces significant challenges including reviewer fatigue, bias, and insufficient rigor. The rapid increase in scientific publications has placed an unsustainable burden on a limited pool of reviewers, resulting in the inability of reviewers to detect falsified data, screen errors with high fidelity, and even establish consensus between reviewers [1]. ResearchHub Foundation aims to address these issues by 1) incentivizing reviewers and 2) integrating Agentic AI systems powered by recent breakthroughs in large language and multi-modal foundational models. Our unique approach with validated expert human reviews, and trustworthy automated tools to complement human reviewers, will enhance the quality and credibility of peer reviews to benefit both authors and the larger scientific community.We have observed from our own experiments (200+ AI peer review samples) that AI reviewers have capabilities that excel in domains such as detecting technical errors and statistical flaws, while human reviewers provide greater insight into the broader relevance and context of research. We hypothesize that incorporating human feedback into AI models of peer review can improve their performance beyond existing state-of-the-art AI systems without human feedback. The project has two aims:Establish a Reliable Protocol to Assess Reviewer Performance: Using a standardized rubric, human editors with specific sets of expertise will evaluate AI and human reviews across methods, results, discussion, rigor, novelty, and error detection. AI reviewers are expected to outperform in technical rigor, while humans excel in contextual analysis.Develop a Versatile Agentic Solutions for Scientific Auditing : By leveraging upon the growing number of toolkits for deploying AI Agents driven by LLM, Large multi-modal models (LMMs), long-term memories, and orchestration infrastructure, we ought to develop specialized agentic solutions honed for various aspects of scientific document auditing.This initiative leverages ResearchHub’s extensive reviewer database, standardized prompts, and expertise-weighted evaluations to enhance the peer review process. By addressing systemic issues and integrating AI and human expertise, this project seeks to improve the accuracy, efficiency, and scalability of scientific peer review, supporting the advancement of open and validated science.Introduction and SignificanceScientific Auditing - Peer review is a cornerstone of scientific integrity, serving as the primary mechanism for validating research and granting manuscripts the status of “verified” work. However, the process is increasingly strained due to systemic challenges. Peer reviewers face difficulties in thoroughly assessing statistical analyses, verifying references, and evaluating claims against the broader scientific literature. Despite the critical nature of their role, reviewers are not compensated for their efforts, which often results in diminished rigor, personal biases, superficial feedback, and gaps in understanding key aspects of the studies they assess. These challenges are further exacerbated by the rapid growth in scientific publications, which has significantly increased the workload on an already limited pool of reviewers. The number of scientific publications has increased an estimated 47% from 2016 to 2022 [2]. This combination of rising demand and insufficient support has created a critical bottleneck in the peer review process, highlighting the urgent need for innovative solutions to enhance the accuracy, efficiency, and sustainability of scientific auditing. With the rise of techniques resulting in large terabyte sized datasets (i.e. single-cell RNAseq, BARseq) the complexity for detecting errors and the time demand to do so for human reviewers is amplified. Additionally, this unfortunate combination has resulted in excessive burden on a small number of reviewers resulting in reviewer fatigue. The use of ResearchCoin incentives on ResearchHub has provided the impetus to overcome this apprehension of reviewers to peer review, and provides a new found incentive and motivation. Nonetheless, even with appropriate incentives, human reviewers continue to have blindspots during the review process. Agentic AI solution - With the rapid mass-adoption of LLMs and AI Agents, the ability to holistically assess a manuscript has become feasible at rapid speeds and low costs, providing immensely valuable insights into the paper that often reside within human reviewer’s blind spots. These tools can fill the gaps of human reviewers limitations and provide a more expedited turnaround time on feedback and requested edits, ultimately adding a more holistic set of feedback and context to the academic manuscript. Foundational models (o1, Claude, Gemini, LLaMA, DeepSeek, etc.) have made strides and provide a basis for this approach. With smaller amounts of data, such as peer review data on ResearchHub, it is already possible to substantially leverage these foundation models with minimal additional training data to construct a much more accurate peer review system greater than the sum of its parts. AI reviewers are not a panacea, and it’s becoming clear that human reviewers are still needed, but will have AI reviewers as a powerful supplement [3]. Hypothesis and AimsHypothesis: Published and preprinted papers have errors or issues within the manuscript that AI reviewers can catch that are often overlooked by human reviewers, and integrating human reviewer/editor reinforcement learning can enhance capabilities of the AI review system when domain specific context is essential. Aim 1: Assess the strengths and weaknesses of human reviewers and AI reviewers using a standardized rubric. Human editors will evaluate the outputs of both human reviewers and AI reviewers for methods, results, discussion, rigor, novelty and error detection rate. Anticipated result: Human peer reviewers will be superior in interpreting the implications of the methods and results that are NOT explicitly mentioned in the manuscript, while AI reviewers will be more thorough in assessing statistics, detecting errors etc based on information available explicitly in the manuscript. Aim 2: Develop versatile systems of agentic solutions driven by LLM, LMMs, and AI Agents, empowered by domain specific knowledge, in combination with human evaluation and feedback in producing thorough evaluation of scientific documents. Anticipated result: After tuning and including post-training of foundational models with the human evaluation and feedback, the resulting agentic solutions will perform better than the direct use of benchmark foundational models alone when used to a particular scientific auditing task (manuscript review, editor’s feedback, journal decision, etc.)Methods and DesignCurrently, the use of already existing foundation models equipped with simple prompt engineering has provided a viable starting point for pinpointing errors and giving additional context in scientific manuscripts. We’ve developed a minimum viable system in the form of a web app shown in Fig. 1 below. We also made several observations throughout our pilot experiment in which, holistically, using the same optimized prompt, Gemini 2.0 has been the most robust foundation model in producing consistent results human editors and readers generally consider more useful. This is in part due to the 1M+ context window for token processing, roughly 5-7 times greater than the typical 128k and 280k context windows found in other foundation models. Figure 1. Web-application developed by ResearchHub Foundation using AI agents (AI Editor and AI Peer Reviewers). Figure 2. Proposed ARTEMIS system architecture and workflow design.In Fig. 2 , we present the proposed ARTEMIS system architecture and workflow design, here we follow on with describing the key actions happening within each step of the process: Scientific Document Assigned for Decision-Making.Human users who make decisions based on scientific documents submit audit requests to gain further insights regarding the document and the scientific works it represents.Document Pre-Processing for Metadata Extraction.Depending on the metadata content, a specific full extraction protocol is performed to obtain the maximum amount of information from the document.Full extraction results are sent to the AI Quality Control agent to ensure they meet the required standards and can be used further.(6b, 7b, 8b.) Extracted document content that fails to satisfy the minimum requirements is sent to humans for manual decision-making, annotation of reasons for exclusion, and archiving within the Data Lakehouse. (6a & 6a’.) Extracted documents that pass the QC requirements, either automatically or manually after human consideration, are passed on to the AI Editor.(7a.) The AI Editor generates or orchestrates the assignment of the document to the appropriate AI reviewer(s).(8a.) AI Peer Reviewer(s) generate their assessment of the document and send it to the AI QC agent to ensure correct structure and compliance with the required review standards.All reviews that pass the AI QC check are curated into a raw report ready for human evaluation.The raw report is prepared (and reformatted if necessary) to be presented to humans for Human Evaluation.Human evaluation results are submitted to the AI QC for further checks, including verifying whether the contribution is authentically human and backed by proper login credentials.Human feedback from verified editors or validators is requested to justify the quality of the given human evaluation results.All data collected thus far is curated into a coherent dataset corresponding to the scientific document and the auditing task at hand.The dataset is organized into its respective zones and/or warehouses within the Data Lakehouse.Actionable insights are produced based on the knowledge available within the Data Lakehouse to answer the questions posed in the original audit request. This process can also be further facilitated by interactive Data Librarian/Analyst Agents that help users explore or generate additional insights beyond the original request.Aim 1 - Strengths & Weaknesses of Human vs AI ReviewersThe first stage of the proposal is focused on producing the “Human Wisdom” components described across steps 10 to 13 of the ARTEMIS architecture and work flow (Fig. 2). In the process of producing our human evaluation components of our desired dataset, we will prompt the AI Reviewers and Human Reviewers to perform assessment based on two requirements:Classic ResearchHub Foundation prompt:ResearchHub Foundation bounty prompt (italicized):Your review must be submitted during the 14-day submission window from the day this bounty was initiated.Mention your credentials and include your areas of relevant expertise and limitations in assessing this preprint.Include the version of the preprint you are reviewing (e.g. 1st, 2nd, etc.). Make sure you review the most updated version available.Use the rating system in the "Peer Reviews" tab for all 5 criteria (overall assessment, introduction, methods, results, discussion) but the content within each is flexible (in-line comments can be used instead of a block of text in each section).For each figure, including supplementary material, you must provide a detailed assessment (pros or cons).Plagiarism will not be tolerated. You must comprehensively disclose any use of artificial intelligence (AI) platforms used in the review process. Please refer to our AI Policy for additional details.Currently, ResearchHub processes approximately 170 peer reviews a week (Fig 3a), with 68.7% of all preprints who receive a peer review incentive receiving at least 1 peer review that meets the bar of quality (Fig 3b). Only reviews meeting this minimum bar, as deemed by ResearchHub’s editorial team, will be leveraged for this study.The fields of expertise that ResearchHub peer reviewers specialize in are namely in the biomedical and hard sciences including molecular biology, immunology, oncology, neuroscience, etc (Fig 3c, 3d).Error Detection prompt:Please carefully review the following academic paper and identify any errors, issues, inconsistencies, or questionable reasoning, with a special focus on the mathematical content. In particular, examine the correctness and clarity of all formulas, derivations, and calculations. Check for doubtful hypotheses, shaky assumptions, incorrect mathematical statements, undefined terms, unjustified steps, missing references for key theorems, or discrepancies between the described methods and the results presented. If any part of the argument seems incomplete, unclear, or unconvincing, highlight these sections and explain why they might be problematic. Additionally, look for subtle errors such as misapplied formulas, overlooked boundary conditions, or unsupported assumptions in modeling steps. Make sure to check the figures, captions, and meta data of the paper whenever relevant. Finally, suggest potential ways to strengthen or clarify the mathematical aspects of the paper. { "overall_assessment": "", "mathematical_rigor_and_correctness": { "equation_summary": { "inline_equations": , "display_equations": }, "equation_details" :{ "inline_equations": , "display_equations": , }, "abbreviation_and_symbol_definitions": "", "sign_and_operation_checks": "" }, "issues_list": [ { "issue": "", "severity_rating": , "severity_explanation": "", "suggestion": "" }, { "issue": "", "severity_rating": , "severity_explanation": "", "suggestion": "" } // Repeat for issues 3 to 10 ] } Standardized Scoring Approach for Severity Rating: 1.0 - 2.9: Minor typographical errors or negligible issues that do not affect understanding. 3.0 - 5.9: Moderate issues that may cause some confusion or misunderstandings but are correctable. 6.0 - 8.9: Significant errors that impact the validity of results or the clarity of the paper. 9.0 - 10.0: Critical flaws that fundamentally undermine the paper's conclusions or correctness. In the issues_list, provide a list of the 10 most important or most glaring mistakes or issues, following the standardized scoring approach to maintain objectivity across papers. Be specific to the current paper. **Ensure that your response is formatted as valid JSON only, without any additional text or commentary. At ALL COST, NEVER INCLUDES ```json, in your valid JSON answer. a valid json should start with { and ends with }Figure 3a. Total Number of Completed Peer Reviews per Week on ResearchHub from May 2022 to present. ResearchHub has housed almost 4000 peer reviews at a rate of roughly 170 peer reviews a week. Blue indicates a concerted effort to promote the peer review process on ResearchHub. Figure 3b. Total number of manuscripts separated by if ResearchHub Foundation received peer reviews where we distributed the bounty (green, 68.7%) or didn’t distribute the bounty (red, 31.3%). Green signals a manuscript which received at least 1 peer review that met a minimum threshold of quality to receive the bounty. Red signals a manuscript which either didn’t receive any peer review or received peer review but didn’t meet a minimum threshold of quality or abiding by the stipulations set by ResearchHub Foundation.Figure 3c. Number of Peer Reviews previously completed on ResearchHub stratified by OpenAlex Subfields Figure 3d. Number of Peer Reviews previously completed on ResearchHub stratified by OpenAlex Topics The Human Feedback component will be primarily driven by human editors/validators who will assess the precision and accuracy of the human and AI reviewers based on the claims made. The collection of human feedback will then be used to benchmark and improve the AI-generated reviews. An example of how this benchmarking can be performed is shown in the recent work ReviewEval by Kumar et al.[4] and the references therein. Part of which is already available within the ARTEMIS system under development. With the growth in community interest for open peer reviews, of which ResearchHub has adopted, human validators can quickly view and provide input to the reviews being produced in an open transparent marketplace.[5] The editors/validators will use a flash cards style approach for simple validation. The flash cards approach will appear in several forms, depending on the relevant context. Typical cases would include: Fact-Check, Logic-Check, Math-Check, or Expert Confidence. This alternative approach to consolidating expert feedback is designed as a "bite-sized" task that an expert in the field can answer or reason through without needing to read the entire paper. All context necessary for understanding or providing meaningful feedback on the specific excerpt will be provided on the flashcard. Peer reviewers are assumed to be knowledgeable about the particular topic and their claims will be subject to various weights depending on their background/reputation.ResearchHub’s verification and reputation system: Using OpenAlex, an open repository for scientific data, ResearchHub has ascribed a reputation score for verified users within their various fields of expertise (Fig. 4). This will be used to value or weigh the statements made by various peer reviewers and validators.Reviewer’s track record will be checked based on their past performances and previous editorial feedback, if any available. Additionally, each peer reviewer may need to complete a mini-quiz before receiving a tag for specific expertise, ensuring that they actually possess the required expertise. If the reputation system isn’t sufficient for properly confirming a user’s expertise, the mini-quiz will be employed and include single line domain specific questions, with each question restricted to a 5-10 second countdown to receive an answer.Figure 4. Example of a verified user account on ResearchHub with a quantitative percentile ranking as a “reputation” based on publication history. Alternative Approach: Assuming reviewers and/or editors will be difficult to source for the work, alternatively we can validate the AI reviews against other pre-existing datasets [6]. Additionally, in the context where the flash cards approach doesn’t prove to be a good source of validation, we can have validators assess the reviewers based on an adapted version of the ResearchHub Foundation’s standardized unified rubric (not public, but unified recommendations to reviewers can be found here) to give weighted scores on methods, results, discussion, rigor, novelty and error detection rate. Editors evaluations will be weighted contingent on their reputation score in that field according to ResearchHub’s reputation system.Aim 2 - Develop Versatile Agentic Solutions for Scientific AuditingFunctions described in steps 5 to 9 (Fig. 2) for developing AI agents for scientific auditing, we propose to develop 3 agentic roles and 5 auxiliary tooling functions. The 3 agentic roles will be composed of: Quality Control Agent, AI Reviewer Agent, AI Editor. Each of these roles may be deployed under different context and configuration to achieve the desired performance. For example, this may include an AI Editor in Neuroscience, AI Reviewer in Neuroscience, AI quality control in Neuroscience etc parallelized for a plethora of fields of expertise. The 5 Auxiliary toolings are : Long-Term Memory, Knowledge Base, Visual Understanding and Reasoning, Scientific References, Consistency of Structured Output. The design of these toolings will be made as modular as possible to allow replacement of the tech-stack when necessary to boost performance and efficiency.RigorHuman reviewers can openly review any preprints, allowing for expedited reviews. The presence of a standardized prompt for human and AI reviewers, and standardized rubrics for editors allows for apples to apples direct comparisons on specific components of the reviewers. The verification and reputation v2 system on ResearchHub will be employed to weight evaluations according to the expertise of the individual making the claims. Additionally, prior to this pre-registration, we have performed preliminary studies that attempt to understand the dynamics, economics, and quality of incentivized peer review process of which is being employed on ResearchHub [7]. All three aspects of open peer review submissions addressed above (dynamics, economics, and quality) are highly dependent with at least 9 other coupled factors beyond economic and non-economic incentives (Table 1). Therefore, any correlation found would be contextual and apply only to a certain combination of peer review configurations that involved at least the 9 factors. As it pertains to aim 1 of this pre-registration, we will ensure that the peer review context will be fixed ensuring maximal reproducibility.Table 1. The Nine Identified factors influencing correlation between incentives on the quality of open peer reviews of scientific manuscriptsFactor Influencing Incentive-Quality CorrelationVariables Order of IncentivizationAward First (Commissioned Task)Award Later (Rewarded)Bidding Mechanism (Stake for Chance)Reviewer Pool Size Small [<100]Medium [100-1000]Large [1000+]Expertise DiversityRanging from 0 to 1 0: All reviewers are from the exact same research subfield1: For each subfield there are exactly equal number of reviewer* Each reviewer can have more than 1 expertise, but maximum 3 notable expertise. Complete list of subfields, will be at most 3 x number of reviewers in the pool.Reviewer Selection CriteriaSingle Chief-Editor (or Campaign Host) make the final callFirst Come First Served [Based on Objective Quality Scores or Time Order]Democratic Decision [Community Votes or Expert Scoring]Type of Incentive and Size/IntensityMonetaryReputationMonetary + ReputationRequested Review StructureSimple (Yes/No answer and inline comments)Moderate (Structured Comments on Different Aspects)Rigor (Complete Assessment on All Sections, Figures, References, and more)Difficulties of Review TaskEasy (General Knowledgeable Public)Medium (Graduate Student Level on the Topic)Hard (Subject Expert on the Topic)Likelihood & Proportion of AI Usage Low (<10%)Moderate (10-90%)High (>90%)Time RestrictionIntense (<3 Days)Moderate (3-14 Days)Leisurely ( >15 Days) BudgetTable 2. Budget description and breakdown for an estimated 12 month timeline. SectionNeedDetailsCostDevelopmentComputeAPIOutsourced developer work for data cleaning and UI/UX designDigital Ocean, o1/claude/gemini APIs, GPUs, observability system, auxiliary tools for AI Agents, and vector storage, etc.$10,000Human reviewer costs Incentive for Peer Reviewers for ResearchHub Foundation selected papersIncentive for Peer Reviewers for YesNoError (YNE) selected papers - 50 papersIncentive for Peer Reviewers for Gareth and Maria’s selected papers - 50 papers Preprints in the fields of biology, chemistry, neuroscience, immunology, genetics, molecular biology, biochemistry, oncology, mathematics, physics, computer science, etc $150,000 for 1000 human reviews (assuming $150 per peer review and 1 review per preprint)Human editor costsIncentive for Validators (Editors or Authors) to assess AI and Human Reviewer OutputsValidators will be asked to provide either full or partial feedback on both the original preprint and/or the AI review submissions.$50,000 for 1000 editors (assuming $50 to assess all peer reviews per manuscript) Administrative CostMarketing The cost associated with marketing efforts to ensure campaign visibility and sustain interest for contributions from the community.$5,000Reserve FundFor unexpected needs and for insurance against volatility. $15,000, If unused, will be put toward getting a larger dataset. Disclosure and Licensing StatementThis document outlines the terms under which the content, outputs, and tools resulting from our study on AI reviewers for scientific manuscripts are shared and utilized. Transparency and collaboration are core to our research philosophy, and we aim to strike a balance between open science principles and the potential for practical, real-world applications.Open Source CommitmentContent and Outputs:All data, algorithms, models, and other outputs derived from this study will be made available under an open-source license, allowing unrestricted access for academic, research, and non-commercial purposes.The specific license will be the MIT License to ensure clarity regarding permissible uses and attribution requirements.Repository Access:Public repositories (e.g., on GitHub, Zenodo, HuggingFace or similar platforms) will host the content and outputs, accompanied by detailed documentation to facilitate their reuse and extension by the broader research community.Commercialization ClauseWhile the content and outputs of this study will be open-sourced, we reserve the right to commercialize tools or technologies derived from these resources. By participating in this study or utilizing its outputs, users acknowledge and agree to the terms outlined in this disclosure and licensing statement.Declaration of Generative AI in Scientific WritingDuring the preparation of this work, ChatGPT was used to enhance the readability and language of some parts of the manuscript. Additionally, it was used to consolidate the original content written in the introduction, methods, and preliminary data to construct an Abstract. Following any ChatGPT usage, content was reviewed and edited as needed.AcknowledgementsWe would like to acknowledge Matt Schlicht, founder of YesNoError for being a Strategic Data Partner and Gareth Dyke & Maria Machado as Strategic Expert Community Partners. References[1] Howard Bauchner, Frederick P Rivara, Use of artificial intelligence and the future of peer review, Health Affairs Scholar, Volume 2, Issue 5, May 2024, qxae058, https://doi.org/10.1093/haschl/qxae058[2] Hanson, M. A., Gómez Barreiro, P., Crosetto, P., & Brockington, D. (2024). The strain on scientific publishing. arXiv. https://arxiv.org/abs/2309.15884v2[3] Seghier, M.L. (2025), "AI-powered peer review needs human supervision", Journal of Information, Communication and Ethics in Society, Vol. 23 No. 1, pp. 104-116. https://doi.org/10.1108/JICES-09-2024-0132[4] Kirtani, C., Garg, M. K., Prasad, T., Singhal, T., Mandal, M., & Kumar, D. (2025). ReviewEval: An evaluation framework for AI-generated reviews. arXiv. https://arxiv.org/abs/2502.11736v2[5] Yang, J. (2025). Paper Copilot: The artificial intelligence and machine learning community should adopt a more transparent and regulated peer review process. arXiv. https://arxiv.org/abs/2502.00874[6] Kuznetsov, I., Gurevych, I. et. a. (2024). What can natural language processing do for peer review? arXiv. https://arxiv.org/abs/2405.06563 GitHub Repository: Afzal, O. M. (n.d.). nlp-for-peer-review. GitHub. https://github.com/OAfzal/nlp-for-peer-review[7] Brian, D. (2024). Incentivized vs non-incentivized open peer reviews: Dynamics, economics, and quality. ResearchHub. https://doi.org/10.55277/ResearchHub.taescjxh4f:["$","$L83",null,{"children":["$","$L2c",null,{"topBanner":["$","$L84",null,{"work":{"id":3961,"type":"$undefined","contentType":"preregistration","title":"ARTEMIS - Automated Review and Trustworthy Evaluation for Manuscripts in Science","slug":"artemis-automated-review-and-trustworthy-evaluation-for-manuscripts-in-sciencebr","createdDate":"2025-02-27T05:28:58.538424Z","createdByUserId":60018,"updatedDate":"2025-05-19T16:20:59.129767Z","publishedDate":"2025-02-27T05:28:58.538424Z","authors":[{"authorProfile":{"id":932947,"fullName":"Jeffrey Koury","firstName":"Jeffrey","lastName":"Koury","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2024/04/01/blob_Ut50nMY","headline":"President at ResearchHub Foundation","profileUrl":"/author/932947","user":{"id":0,"email":"","firstName":"","lastName":"","fullName":"Unknown User","isVerified":false,"authorProfile":"$undefined","balance":0,"lockedBalance":0,"promotionalBalance":0,"fundingCredits":0,"balances":"$undefined","totalRsc":0,"hasCompletedOnboarding":false,"createdDate":"$undefined","moderator":false,"editorOfHubs":"$undefined","isModerator":false,"isFunder":false,"referralCode":"$undefined","authProvider":"$undefined","isStakingOptedIn":false,"balanceHistory":0,"raw":34416},"description":"PhD in Biomedical Science, UC Riverside | Neuroinflammation and Neurodegeneration","createdDate":"2021-12-22T19:55:07.602698Z","education":[{"id":8197,"name":"University of California--Riverside","year":{"label":"2024","value":"2024"},"major":"Biomedical Science","degree":{"label":"Doctor of Philosophy","value":"PhD"},"summary":"Biomedical Science PhD '24, University of California--Riverside","is_public":true}],"twitter":"https://twitter.com/jeffyfish9","facebook":"https://twitter.com/home","linkedin":"https://www.linkedin.com/in/jeffreykoury/","googleScholar":"https://scholar.google.com/citations?user=TTsBcrAAAAAJ&hl=en","orcidId":"https://orcid.org/0000-0002-7368-4891","isOrcidConnected":true,"isClaimed":true,"userId":"$undefined","isVerified":true,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":932947,"is_public":true,"is_removed":false,"is_removed_date":null,"user":34416,"first_name":"Jeffrey","last_name":"Koury","created_date":"2021-12-22T19:55:07.602698Z","updated_date":"2026-08-17T07:00:29.643667Z","description":"PhD in Biomedical Science, UC Riverside | Neuroinflammation and Neurodegeneration","h_index":6,"i10_index":6,"profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2024/04/01/blob_Ut50nMY","author_score":0,"university":null,"orcid_id":"https://orcid.org/0000-0002-7368-4891","openalex_ids":["https://openalex.org/A5081987607","https://openalex.org/A5092743127"],"education":"$4f:props:children:props:topBanner:props:work:authors:0:authorProfile:education","headline":"President at ResearchHub Foundation","facebook":"https://twitter.com/home","twitter":"https://twitter.com/jeffyfish9","linkedin":"https://www.linkedin.com/in/jeffreykoury/","google_scholar":"https://scholar.google.com/citations?user=TTsBcrAAAAAJ&hl=en","claimed":true,"country_code":null,"created_source":"RESEARCHHUB","last_full_fetch_from_openalex":null,"two_year_mean_citedness":0.4666666666666667,"merged_with_author":null,"added_as_editor_date":null,"is_hub_editor_of":[],"is_hub_editor":false,"num_posts":55,"is_orcid_connected":true,"reputation":1497,"suspended_status":{"probable_spammer":false,"is_suspended":false},"total_score":null,"wallet":null,"is_verified":true}},"isCorresponding":false,"position":"middle","raw":"$4f:props:children:props:topBanner:props:work:authors:0:authorProfile:raw"},{"authorProfile":{"id":993107,"fullName":"Dominikus Brian","firstName":"Dominikus","lastName":"Brian","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/03/04/blob_pxj9rsH","headline":"Applied AI @ RHF | Founder @ OrganoBit.Bio","profileUrl":"/author/993107","user":{"id":0,"email":"","firstName":"","lastName":"","fullName":"Unknown User","isVerified":false,"authorProfile":"$undefined","balance":0,"lockedBalance":0,"promotionalBalance":0,"fundingCredits":0,"balances":"$undefined","totalRsc":0,"hasCompletedOnboarding":false,"createdDate":"$undefined","moderator":false,"editorOfHubs":"$undefined","isModerator":false,"isFunder":false,"referralCode":"$undefined","authProvider":"$undefined","isStakingOptedIn":false,"balanceHistory":0,"raw":60018},"description":"Building Backward toward the Organoid Economy.\nhttps://dominikusbrian.com","createdDate":"2024-04-19T00:03:04.727929Z","education":[{"id":4782,"name":"New York University","major":"Chemistry","summary":"Chemistry New York University","is_public":true},{"id":5903,"name":"Shanghai Jiaotong University","major":"Power Engineering & Engineering Thermophysics","summary":"Power Engineering & Engineering Thermophysics Shanghai Jiaotong University","is_public":false},{"id":5893,"name":"Shandong University of Science and Technology","major":"Chemical Engineering","summary":"Chemical Engineering Shandong University of Science and Technology","is_public":false}],"twitter":"https://x.com/dominikusbrian","facebook":"$undefined","linkedin":"https://www.linkedin.com/in/dominikus-brian-%E9%92%9F%E9%B8%BF%E7%9B%9B-82226161/","googleScholar":"https://scholar.google.com/citations?user=Ark53e0AAAAJ&hl=en","orcidId":"https://orcid.org/0000-0001-6165-0355","isOrcidConnected":true,"isClaimed":true,"userId":"$undefined","isVerified":true,"isHubEditor":true,"editorOfHubs":[{"id":32,"name":"Materials Science","slug":"materials-science","imageUrl":"https://storage.prod.researchhub.com/uploads/hub_images/2020/10/07/materials-science.jpg","description":"$undefined","namespace":"subcategory","category":"$undefined","discussionCount":"$undefined","paperCount":"$undefined","subscriberCount":"$undefined","isLocked":false,"isRemoved":false,"raw":{"editor_permission_groups":[{"id":144619,"organization":null,"user":{"id":60018,"author_profile":{"id":993107,"is_verified":true,"is_public":true,"is_removed":false,"is_removed_date":null,"first_name":"Dominikus","last_name":"Brian","created_date":"2024-04-19T00:03:04.727929Z","updated_date":"2026-07-20T17:22:07.940431Z","description":"Building Backward toward the Organoid Economy.\nhttps://dominikusbrian.com","h_index":8,"i10_index":8,"profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/03/04/blob_pxj9rsH","author_score":0,"orcid_id":"https://orcid.org/0000-0001-6165-0355","openalex_ids":["https://openalex.org/A5079564287"],"education":[{"id":4782,"name":"New York University","major":"Chemistry","summary":"Chemistry New York University","is_public":true},{"id":5903,"name":"Shanghai Jiaotong University","major":"Power Engineering & Engineering Thermophysics","summary":"Power Engineering & Engineering Thermophysics Shanghai Jiaotong University","is_public":false},{"id":5893,"name":"Shandong University of Science and Technology","major":"Chemical Engineering","summary":"Chemical Engineering Shandong University of Science and Technology","is_public":false}],"headline":"Applied AI @ RHF | Founder @ OrganoBit.Bio","facebook":null,"twitter":"https://x.com/dominikusbrian","linkedin":"https://www.linkedin.com/in/dominikus-brian-%E9%92%9F%E9%B8%BF%E7%9B%9B-82226161/","google_scholar":"https://scholar.google.com/citations?user=Ark53e0AAAAJ&hl=en","claimed":true,"country_code":null,"created_source":"RESEARCHHUB","last_full_fetch_from_openalex":null,"two_year_mean_citedness":0.9090909090909091,"user":60018,"university":null,"merged_with_author":null}},"created_date":"2024-08-23T00:40:03.600433Z","updated_date":"2024-08-23T00:40:03.600456Z","access_type":"ASSISTANT_EDITOR","object_id":32,"content_type":18}],"hub_image":"https://storage.prod.researchhub.com/uploads/hub_images/2020/10/07/materials-science.jpg","id":32,"is_locked":false,"is_removed":false,"name":"Materials Science","slug":"materials-science","namespace":"subcategory"}}],"raw":{"id":993107,"is_public":true,"is_removed":false,"is_removed_date":null,"user":60018,"first_name":"Dominikus","last_name":"Brian","created_date":"2024-04-19T00:03:04.727929Z","updated_date":"2026-07-20T17:22:07.940431Z","description":"Building Backward toward the Organoid Economy.\nhttps://dominikusbrian.com","h_index":8,"i10_index":8,"profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/03/04/blob_pxj9rsH","author_score":0,"university":null,"orcid_id":"https://orcid.org/0000-0001-6165-0355","openalex_ids":["https://openalex.org/A5079564287"],"education":"$4f:props:children:props:topBanner:props:work:authors:1:authorProfile:education","headline":"Applied AI @ RHF | Founder @ OrganoBit.Bio","facebook":null,"twitter":"https://x.com/dominikusbrian","linkedin":"https://www.linkedin.com/in/dominikus-brian-%E9%92%9F%E9%B8%BF%E7%9B%9B-82226161/","google_scholar":"https://scholar.google.com/citations?user=Ark53e0AAAAJ&hl=en","claimed":true,"country_code":null,"created_source":"RESEARCHHUB","last_full_fetch_from_openalex":null,"two_year_mean_citedness":0.9090909090909091,"merged_with_author":null,"added_as_editor_date":null,"is_hub_editor_of":["$4f:props:children:props:topBanner:props:work:authors:1:authorProfile:editorOfHubs:0:raw"],"is_hub_editor":true,"num_posts":20,"is_orcid_connected":true,"reputation":1469,"suspended_status":{"probable_spammer":false,"is_suspended":false},"total_score":null,"wallet":null,"is_verified":true}},"isCorresponding":false,"position":"middle","raw":"$4f:props:children:props:topBanner:props:work:authors:1:authorProfile:raw"}],"abstract":"$85","doi":"10.55277/ResearchHub.nj64k80a","journal":"$undefined","formats":[],"license":"$undefined","pdfCopyrightAllowsDisplay":true,"figures":[],"versions":[],"metrics":{"votes":62,"adjustedScore":"$undefined","comments":23,"saves":0,"reviewScore":0,"reviews":0,"earned":0,"views":0},"unifiedDocumentId":7345202,"registeredReportId":null,"postType":"PREREGISTRATION","fundraise":"$undefined","note":{"id":11406,"access":"WORKSPACE","organization":{"id":109221,"memberCount":2,"userPermission":"$undefined","createdDate":"2025-02-25T02:29:44.106712Z","coverImage":null,"description":"","name":"Domi and Jeff","slug":"domi-and-jeff"},"createdDate":"2025-02-25T02:30:00.038111Z","updatedDate":"2025-04-16T10:04:40.461418Z","title":"Preregistration Template","isRemoved":false,"post":{"id":3961,"slug":"artemis-automated-review-and-trustworthy-evaluation-for-manuscripts-in-sciencebr","contentType":"preregistration","documentType":"PREREGISTRATION","moderationStatus":"APPROVED","fundraise":{"id":16,"amountRaised":{"usd":0,"rsc":0},"goalAmount":{"usd":230000,"rsc":4586313.139080053},"contributors":{"numContributors":10,"topContributors":[{"id":36837,"authorProfile":{"id":956713,"fullName":"Tyler Diorio","firstName":"Tyler","lastName":"Diorio","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2022/08/09/blob_im4Dh4d","headline":"Brain Biomechanics Researcher","profileUrl":"/author/956713","user":{"id":0,"email":"","firstName":"","lastName":"","fullName":"Unknown User","isVerified":false,"authorProfile":"$undefined","balance":0,"lockedBalance":0,"promotionalBalance":0,"fundingCredits":0,"balances":"$undefined","totalRsc":0,"hasCompletedOnboarding":false,"createdDate":"$undefined","moderator":false,"editorOfHubs":"$undefined","isModerator":false,"isFunder":false,"referralCode":"$undefined","authProvider":"$undefined","isStakingOptedIn":false,"balanceHistory":0,"raw":36837},"description":"ResearchHub Chief of Staff\nPrev. PhD, Purdue University Biomedical Engineering\n\nMedical devices (ChemoFilter), neuroimaging, brain biomechanics, cerebrovascular fluid dynamics, and computational hemodynamics.","createdDate":"2022-08-08T21:44:33.446296Z","education":[{"id":5410,"name":"Purdue University--West Lafayette","year":{"label":"2023","value":"2023"},"major":"Biomedical Engineering","degree":{"label":"Doctor of Philosophy","value":"PhD"},"summary":"Biomedical Engineering PhD '23, Purdue University--West Lafayette","is_public":true},{"id":5589,"name":"Rowan University","year":{"label":"2018","value":"2018"},"major":"Chemical Engineering","degree":{"label":"Bachelor of Science","value":"BS"},"summary":"Chemical Engineering BS '18, Rowan University","is_public":false}],"twitter":"https://twitter.com/TylerDiorio","facebook":"$undefined","linkedin":"https://www.linkedin.com/in/tyler-diorio-351325aa/","googleScholar":"https://scholar.google.com/citations?user=yq2GoPUAAAAJ&hl=en","orcidId":"https://orcid.org/0000-0001-6371-9424","isOrcidConnected":false,"isClaimed":true,"userId":"$undefined","isVerified":true,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":956713,"is_verified":true,"is_public":true,"is_removed":false,"is_removed_date":null,"first_name":"Tyler","last_name":"Diorio","created_date":"2022-08-08T21:44:33.446296Z","updated_date":"2026-04-17T13:32:29.572885Z","description":"ResearchHub Chief of Staff\nPrev. PhD, Purdue University Biomedical Engineering\n\nMedical devices (ChemoFilter), neuroimaging, brain biomechanics, cerebrovascular fluid dynamics, and computational hemodynamics.","h_index":4,"i10_index":2,"profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2022/08/09/blob_im4Dh4d","author_score":0,"orcid_id":"https://orcid.org/0000-0001-6371-9424","openalex_ids":["https://openalex.org/A5026442055"],"education":"$4f:props:children:props:topBanner:props:work:note:post:fundraise:contributors:topContributors:0:authorProfile:education","headline":"Brain Biomechanics Researcher","facebook":null,"twitter":"https://twitter.com/TylerDiorio","linkedin":"https://www.linkedin.com/in/tyler-diorio-351325aa/","google_scholar":"https://scholar.google.com/citations?user=yq2GoPUAAAAJ&hl=en","claimed":true,"country_code":null,"created_source":"RESEARCHHUB","last_full_fetch_from_openalex":null,"two_year_mean_citedness":3.142857142857143,"user":36837,"university":null,"merged_with_author":null}},"totalContribution":{"rsc":2500,"usd":0},"contributions":[{"amount":2500,"currency":"RSC","date":"2025-03-03T22:40:53.972291Z"}]},{"id":14,"authorProfile":{"id":22,"fullName":"Patrick Joyce","firstName":"Patrick","lastName":"Joyce","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2019/11/19/blob","headline":"","profileUrl":"/author/22","user":{"id":0,"email":"","firstName":"","lastName":"","fullName":"Unknown User","isVerified":false,"authorProfile":"$undefined","balance":0,"lockedBalance":0,"promotionalBalance":0,"fundingCredits":0,"balances":"$undefined","totalRsc":0,"hasCompletedOnboarding":false,"createdDate":"$undefined","moderator":false,"editorOfHubs":"$undefined","isModerator":false,"isFunder":false,"referralCode":"$undefined","authProvider":"$undefined","isStakingOptedIn":false,"balanceHistory":0,"raw":14},"description":"Med student interested in neuroscience and its clinical applications","createdDate":"2019-11-19T20:10:16.840995Z","education":[{"id":765,"city":"Boston","name":"Boston University","year":{"label":"2015","value":"2015"},"state":"MA","degree":{"label":"Master's Degree","value":"Master's Degree"},"country":"United States","summary":"Master's Degree '15, Boston University","is_public":true,"created_date":"2020-11-24T21:44:27.590818Z","updated_date":"2020-11-24T21:44:27.590820Z"}],"twitter":"https://twitter.com/joycesticks","facebook":"$undefined","linkedin":"https://www.linkedin.com/in/patrick-joyce-396b953b/","googleScholar":"$undefined","orcidId":"$undefined","isOrcidConnected":false,"isClaimed":true,"userId":"$undefined","isVerified":true,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":22,"is_verified":true,"is_public":true,"is_removed":false,"is_removed_date":null,"first_name":"Patrick","last_name":"Joyce","created_date":"2019-11-19T20:10:16.840995Z","updated_date":"2025-08-01T19:12:18.661017Z","description":"Med student interested in neuroscience and its clinical applications","h_index":9,"i10_index":8,"profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2019/11/19/blob","author_score":3,"orcid_id":null,"openalex_ids":["https://openalex.org/A5051784742","https://openalex.org/A5092177346","https://openalex.org/A5066208241"],"education":"$4f:props:children:props:topBanner:props:work:note:post:fundraise:contributors:topContributors:1:authorProfile:education","headline":"","facebook":null,"twitter":"https://twitter.com/joycesticks","linkedin":"https://www.linkedin.com/in/patrick-joyce-396b953b/","google_scholar":null,"claimed":true,"country_code":null,"created_source":"RESEARCHHUB","last_full_fetch_from_openalex":null,"two_year_mean_citedness":0.9090909090909091,"user":14,"university":null,"merged_with_author":null}},"totalContribution":{"rsc":2500,"usd":0},"contributions":[{"amount":2500,"currency":"RSC","date":"2025-03-03T17:27:16.223469Z"}]},{"id":33901,"authorProfile":{"id":931964,"fullName":"Maulik Dhandha","firstName":"Maulik","lastName":"Dhandha","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2024/10/27/blob_yxyA3Mo","headline":"Editor-in-Chief - ResearchHub","profileUrl":"/author/931964","user":{"id":0,"email":"","firstName":"","lastName":"","fullName":"Unknown User","isVerified":false,"authorProfile":"$undefined","balance":0,"lockedBalance":0,"promotionalBalance":0,"fundingCredits":0,"balances":"$undefined","totalRsc":0,"hasCompletedOnboarding":false,"createdDate":"$undefined","moderator":false,"editorOfHubs":"$undefined","isModerator":false,"isFunder":false,"referralCode":"$undefined","authProvider":"$undefined","isStakingOptedIn":false,"balanceHistory":0,"raw":33901},"description":"Editor-in-Chief - ResearchHub.\nDermatologist - Pacific Skin Institute.\nFounder - Sun Protection Inc.","createdDate":"2021-12-11T12:14:42.701741Z","education":[{"id":5687,"city":"St. Louis","name":"Saint Louis University","year":{"label":"2013","value":"2013"},"state":"MO","degree":{"label":"Doctor of Medicine","value":"MD"},"country":"United States","summary":"MD '13, Saint Louis University","is_public":true,"created_date":"2020-11-24T21:44:27.614434Z","updated_date":"2020-11-24T21:44:27.614436Z"}],"twitter":"https://twitter.com/DhandhaMaulik","facebook":"$undefined","linkedin":"https://www.linkedin.com/in/maulik-dhandha-1836a5227/","googleScholar":"https://scholar.google.com/citations?user=M2JZCWMAAAAJ&hl=en&oi=ao","orcidId":"$undefined","isOrcidConnected":false,"isClaimed":true,"userId":"$undefined","isVerified":true,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":931964,"is_verified":true,"is_public":true,"is_removed":false,"is_removed_date":null,"first_name":"Maulik","last_name":"Dhandha","created_date":"2021-12-11T12:14:42.701741Z","updated_date":"2024-10-29T19:34:34.483764Z","description":"Editor-in-Chief - ResearchHub.\nDermatologist - Pacific Skin Institute.\nFounder - Sun Protection Inc.","h_index":5,"i10_index":4,"profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2024/10/27/blob_yxyA3Mo","author_score":0,"orcid_id":null,"openalex_ids":["https://openalex.org/A5015196239"],"education":"$4f:props:children:props:topBanner:props:work:note:post:fundraise:contributors:topContributors:2:authorProfile:education","headline":"Editor-in-Chief - ResearchHub","facebook":null,"twitter":"https://twitter.com/DhandhaMaulik","linkedin":"https://www.linkedin.com/in/maulik-dhandha-1836a5227/","google_scholar":"https://scholar.google.com/citations?user=M2JZCWMAAAAJ&hl=en&oi=ao","claimed":true,"country_code":null,"created_source":"RESEARCHHUB","last_full_fetch_from_openalex":null,"two_year_mean_citedness":0,"user":33901,"university":null,"merged_with_author":null}},"totalContribution":{"rsc":2000,"usd":0},"contributions":[{"amount":500,"currency":"RSC","date":"2025-03-11T18:50:24.720557Z"},{"amount":1500,"currency":"RSC","date":"2025-03-03T21:46:12.039482Z"}]},{"id":36296,"authorProfile":{"id":952195,"fullName":"Cole Delyea","firstName":"Cole","lastName":"Delyea","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2022/07/29/blob","headline":"Editor - Immunology","profileUrl":"/author/952195","user":{"id":0,"email":"","firstName":"","lastName":"","fullName":"Unknown User","isVerified":false,"authorProfile":"$undefined","balance":0,"lockedBalance":0,"promotionalBalance":0,"fundingCredits":0,"balances":"$undefined","totalRsc":0,"hasCompletedOnboarding":false,"createdDate":"$undefined","moderator":false,"editorOfHubs":"$undefined","isModerator":false,"isFunder":false,"referralCode":"$undefined","authProvider":"$undefined","isStakingOptedIn":false,"balanceHistory":0,"raw":36296},"description":"cole.editor@researchhub.foundation | Biotech scientist working with cell-derived therapies | \nCitizen scientist identifying how music affects plant growth |\nCreator of Investigate Explore Discover. Video summaries of published scientific papers.","createdDate":"2022-06-17T00:04:11.506569Z","education":[{"id":8102,"name":"University of Alberta","year":{"label":"2021","value":"2021"},"major":"Immunology","degree":{"label":"Master of Science","value":"MS"},"summary":"Immunology MS '21, University of Alberta","is_public":true}],"twitter":"$undefined","facebook":"$undefined","linkedin":"https://www.linkedin.com/in/coledelyea/","googleScholar":"https://scholar.google.com/citations?user=O99ZPj4AAAAJ&hl=en&authuser=1","orcidId":"https://orcid.org/0000-0002-5785-4985","isOrcidConnected":false,"isClaimed":true,"userId":"$undefined","isVerified":true,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":952195,"is_verified":true,"is_public":true,"is_removed":false,"is_removed_date":null,"first_name":"Cole","last_name":"Delyea","created_date":"2022-06-17T00:04:11.506569Z","updated_date":"2025-12-02T14:40:47.318656Z","description":"cole.editor@researchhub.foundation | Biotech scientist working with cell-derived therapies | \nCitizen scientist identifying how music affects plant growth |\nCreator of Investigate Explore Discover. Video summaries of published scientific papers.","h_index":4,"i10_index":4,"profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2022/07/29/blob","author_score":0,"orcid_id":"https://orcid.org/0000-0002-5785-4985","openalex_ids":["https://openalex.org/A5022534546"],"education":"$4f:props:children:props:topBanner:props:work:note:post:fundraise:contributors:topContributors:3:authorProfile:education","headline":"Editor - Immunology","facebook":null,"twitter":"","linkedin":"https://www.linkedin.com/in/coledelyea/","google_scholar":"https://scholar.google.com/citations?user=O99ZPj4AAAAJ&hl=en&authuser=1","claimed":true,"country_code":null,"created_source":"RESEARCHHUB","last_full_fetch_from_openalex":null,"two_year_mean_citedness":3.75,"user":36296,"university":null,"merged_with_author":null}},"totalContribution":{"rsc":2000,"usd":0},"contributions":[{"amount":2000,"currency":"RSC","date":"2025-03-08T02:00:08.129174Z"}]},{"id":112244,"authorProfile":{"id":7473890,"fullName":"RHF Tech Support","firstName":"RHF","lastName":"Tech Support","profileImage":"https://lh3.googleusercontent.com/a/ACg8ocKCzNCX9Nj8Acf6c5dNHWjzuAzWnrXcwjgeHkWKF-DSLD6Gfw=s96-c","headline":"","profileUrl":"/author/7473890","user":{"id":0,"email":"","firstName":"","lastName":"","fullName":"Unknown User","isVerified":false,"authorProfile":"$undefined","balance":0,"lockedBalance":0,"promotionalBalance":0,"fundingCredits":0,"balances":"$undefined","totalRsc":0,"hasCompletedOnboarding":false,"createdDate":"$undefined","moderator":false,"editorOfHubs":"$undefined","isModerator":false,"isFunder":false,"referralCode":"$undefined","authProvider":"$undefined","isStakingOptedIn":false,"balanceHistory":0,"raw":112244},"description":"A Dev account for ResearchHub Foundation R&D purposes","createdDate":"2025-02-09T03:34:48.925510Z","education":[],"twitter":"$undefined","facebook":"$undefined","linkedin":"$undefined","googleScholar":"$undefined","orcidId":"$undefined","isOrcidConnected":false,"isClaimed":true,"userId":"$undefined","isVerified":false,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":7473890,"is_verified":false,"is_public":true,"is_removed":false,"is_removed_date":null,"first_name":"RHF","last_name":"Tech Support","created_date":"2025-02-09T03:34:48.925510Z","updated_date":"2025-03-03T10:07:39.723951Z","description":"A Dev account for ResearchHub Foundation R&D purposes","h_index":0,"i10_index":0,"profile_image":"https://lh3.googleusercontent.com/a/ACg8ocKCzNCX9Nj8Acf6c5dNHWjzuAzWnrXcwjgeHkWKF-DSLD6Gfw=s96-c","author_score":0,"orcid_id":null,"openalex_ids":[],"education":"$4f:props:children:props:topBanner:props:work:note:post:fundraise:contributors:topContributors:4:authorProfile:education","headline":"","facebook":null,"twitter":"","linkedin":"","google_scholar":"","claimed":true,"country_code":null,"created_source":"RESEARCHHUB","last_full_fetch_from_openalex":null,"two_year_mean_citedness":0,"user":112244,"university":null,"merged_with_author":null}},"totalContribution":{"rsc":888,"usd":0},"contributions":[{"amount":888,"currency":"RSC","date":"2025-03-03T10:06:47.652500Z"}]},{"id":613,"authorProfile":{"id":572,"fullName":"Natalya Efremova","firstName":"Natalya","lastName":"Efremova","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/01/23/blob_iU88Om0","headline":"","profileUrl":"/author/572","user":{"id":0,"email":"","firstName":"","lastName":"","fullName":"Unknown User","isVerified":false,"authorProfile":"$undefined","balance":0,"lockedBalance":0,"promotionalBalance":0,"fundingCredits":0,"balances":"$undefined","totalRsc":0,"hasCompletedOnboarding":false,"createdDate":"$undefined","moderator":false,"editorOfHubs":"$undefined","isModerator":false,"isFunder":false,"referralCode":"$undefined","authProvider":"$undefined","isStakingOptedIn":false,"balanceHistory":0,"raw":613},"description":"$undefined","createdDate":"2020-08-03T06:16:50.639972Z","education":[],"twitter":"$undefined","facebook":"$undefined","linkedin":"$undefined","googleScholar":"$undefined","orcidId":"$undefined","isOrcidConnected":false,"isClaimed":true,"userId":"$undefined","isVerified":true,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":572,"is_verified":true,"is_public":true,"is_removed":false,"is_removed_date":null,"first_name":"Natalya","last_name":"Efremova","created_date":"2020-08-03T06:16:50.639972Z","updated_date":"2025-01-28T09:16:54.959230Z","description":"","h_index":3,"i10_index":1,"profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/01/23/blob_iU88Om0","author_score":0,"orcid_id":null,"openalex_ids":["https://openalex.org/A5021888709"],"education":"$4f:props:children:props:topBanner:props:work:note:post:fundraise:contributors:topContributors:5:authorProfile:education","headline":"","facebook":null,"twitter":"","linkedin":"","google_scholar":"","claimed":true,"country_code":null,"created_source":"RESEARCHHUB","last_full_fetch_from_openalex":null,"two_year_mean_citedness":0,"user":613,"university":null,"merged_with_author":null}},"totalContribution":{"rsc":160,"usd":0},"contributions":[{"amount":10,"currency":"RSC","date":"2025-03-11T04:36:14.292288Z"},{"amount":150,"currency":"RSC","date":"2025-03-03T15:18:02.290005Z"}]},{"id":49105,"authorProfile":{"id":982634,"fullName":"Shashikant Kotwal","firstName":"Shashikant","lastName":"Kotwal","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2024/04/27/blob_bvG0QSu","headline":"Analytical Chemist - Natural Products","profileUrl":"/author/982634","user":{"id":0,"email":"","firstName":"","lastName":"","fullName":"Unknown User","isVerified":false,"authorProfile":"$undefined","balance":0,"lockedBalance":0,"promotionalBalance":0,"fundingCredits":0,"balances":"$undefined","totalRsc":0,"hasCompletedOnboarding":false,"createdDate":"$undefined","moderator":false,"editorOfHubs":"$undefined","isModerator":false,"isFunder":false,"referralCode":"$undefined","authProvider":"$undefined","isStakingOptedIn":false,"balanceHistory":0,"raw":49105},"description":"M.S. Plant Biology - Rutgers University. Ph.D. Student. \nTeaching assistant. \nNatural Products and Bioanalysis Core facility.\nAnalytical and computational chemistry of Plants, Bacteria, Human metabolomics.\nUHPLC-QqQ-MS\nUHPLC-qTOF-MS\n\nUniversity Lab Core Facility Website: https://research.rutgers.edu/cores/npbc\n\n*All posts made on ResearchHub are not affiliated with any institution or company unless stated otherwise.*","createdDate":"2023-12-11T05:58:51.567109Z","education":[{"id":5631,"city":"Piscataway","name":"Rutgers University--New Brunswick","year":{"label":"2023","value":"2023"},"major":"Plant Biology - Natural Products & Human Health","state":"NJ","degree":{"label":"Master of Science","value":"MS"},"country":"United States","summary":"Plant Biology - Natural Products & Human Health MS '23, Rutgers University--New Brunswick","is_public":true,"created_date":"2020-11-24T21:44:27.614177Z","updated_date":"2020-11-24T21:44:27.614179Z"}],"twitter":"https://x.com/shashi_lab","facebook":"$undefined","linkedin":"https://www.linkedin.com/in/shashikant-kotwal-a23654123/","googleScholar":"https://scholar.google.com/citations?hl=en&user=1QH7ZuIAAAAJ","orcidId":"https://orcid.org/0009-0000-9196-2480","isOrcidConnected":false,"isClaimed":true,"userId":"$undefined","isVerified":true,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":982634,"is_verified":true,"is_public":true,"is_removed":false,"is_removed_date":null,"first_name":"Shashikant","last_name":"Kotwal","created_date":"2023-12-11T05:58:51.567109Z","updated_date":"2025-04-17T03:37:58.229941Z","description":"M.S. Plant Biology - Rutgers University. Ph.D. Student. \nTeaching assistant. \nNatural Products and Bioanalysis Core facility.\nAnalytical and computational chemistry of Plants, Bacteria, Human metabolomics.\nUHPLC-QqQ-MS\nUHPLC-qTOF-MS\n\nUniversity Lab Core Facility Website: https://research.rutgers.edu/cores/npbc\n\n*All posts made on ResearchHub are not affiliated with any institution or company unless stated otherwise.*","h_index":8,"i10_index":7,"profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2024/04/27/blob_bvG0QSu","author_score":0,"orcid_id":"https://orcid.org/0009-0000-9196-2480","openalex_ids":["https://openalex.org/A5077934838"],"education":"$4f:props:children:props:topBanner:props:work:note:post:fundraise:contributors:topContributors:6:authorProfile:education","headline":"Analytical Chemist - Natural Products","facebook":null,"twitter":"https://x.com/shashi_lab","linkedin":"https://www.linkedin.com/in/shashikant-kotwal-a23654123/","google_scholar":"https://scholar.google.com/citations?hl=en&user=1QH7ZuIAAAAJ","claimed":true,"country_code":null,"created_source":"RESEARCHHUB","last_full_fetch_from_openalex":null,"two_year_mean_citedness":0,"user":49105,"university":null,"merged_with_author":null}},"totalContribution":{"rsc":100,"usd":0},"contributions":[{"amount":100,"currency":"RSC","date":"2025-03-07T01:17:06.407046Z"}]},{"id":58703,"authorProfile":{"id":991858,"fullName":"M K","firstName":"M","lastName":"K","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/12/05/blob_91W20yo","headline":"","profileUrl":"/author/991858","user":{"id":0,"email":"","firstName":"","lastName":"","fullName":"Unknown User","isVerified":false,"authorProfile":"$undefined","balance":0,"lockedBalance":0,"promotionalBalance":0,"fundingCredits":0,"balances":"$undefined","totalRsc":0,"hasCompletedOnboarding":false,"createdDate":"$undefined","moderator":false,"editorOfHubs":"$undefined","isModerator":false,"isFunder":false,"referralCode":"$undefined","authProvider":"$undefined","isStakingOptedIn":false,"balanceHistory":0,"raw":58703},"description":"$undefined","createdDate":"2024-03-30T22:51:50.074340Z","education":[],"twitter":"$undefined","facebook":"$undefined","linkedin":"$undefined","googleScholar":"$undefined","orcidId":"$undefined","isOrcidConnected":false,"isClaimed":true,"userId":"$undefined","isVerified":false,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":991858,"is_verified":false,"is_public":true,"is_removed":false,"is_removed_date":null,"first_name":"M","last_name":"K","created_date":"2024-03-30T22:51:50.074340Z","updated_date":"2025-12-05T14:33:41.792683Z","description":"","h_index":0,"i10_index":0,"profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/12/05/blob_91W20yo","author_score":0,"orcid_id":null,"openalex_ids":[],"education":"$4f:props:children:props:topBanner:props:work:note:post:fundraise:contributors:topContributors:7:authorProfile:education","headline":"","facebook":null,"twitter":"","linkedin":"","google_scholar":"","claimed":true,"country_code":null,"created_source":"RESEARCHHUB","last_full_fetch_from_openalex":null,"two_year_mean_citedness":0,"user":58703,"university":null,"merged_with_author":null}},"totalContribution":{"rsc":91,"usd":0},"contributions":[{"amount":91,"currency":"RSC","date":"2025-03-10T19:47:57.736878Z"}]},{"id":113849,"authorProfile":{"id":7603504,"fullName":"Johannes Breuer","firstName":"Johannes","lastName":"Breuer","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/03/07/blob_K5CbTjU","headline":"Team leader & senior researcher at GESIS - Leibniz Institute for the Social Sciences","profileUrl":"/author/7603504","user":{"id":0,"email":"","firstName":"","lastName":"","fullName":"Unknown User","isVerified":false,"authorProfile":"$undefined","balance":0,"lockedBalance":0,"promotionalBalance":0,"fundingCredits":0,"balances":"$undefined","totalRsc":0,"hasCompletedOnboarding":false,"createdDate":"$undefined","moderator":false,"editorOfHubs":"$undefined","isModerator":false,"isFunder":false,"referralCode":"$undefined","authProvider":"$undefined","isStakingOptedIn":false,"balanceHistory":0,"raw":113849},"description":"https://www.johannesbreuer.com/","createdDate":"2025-03-07T12:30:35.376592Z","education":[],"twitter":"https://x.com/MattEagle09","facebook":"$undefined","linkedin":"$undefined","googleScholar":"https://scholar.google.com/citations?user=n6q5R2QAAAAJ","orcidId":"$undefined","isOrcidConnected":false,"isClaimed":true,"userId":"$undefined","isVerified":true,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":7603504,"is_verified":true,"is_public":true,"is_removed":false,"is_removed_date":null,"first_name":"Johannes","last_name":"Breuer","created_date":"2025-03-07T12:30:35.376592Z","updated_date":"2025-03-07T14:13:53.389630Z","description":"https://www.johannesbreuer.com/","h_index":27,"i10_index":36,"profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/03/07/blob_K5CbTjU","author_score":0,"orcid_id":null,"openalex_ids":["https://openalex.org/A5034653321"],"education":"$4f:props:children:props:topBanner:props:work:note:post:fundraise:contributors:topContributors:8:authorProfile:education","headline":"Team leader & senior researcher at GESIS - Leibniz Institute for the Social Sciences","facebook":null,"twitter":"https://x.com/MattEagle09","linkedin":"","google_scholar":"https://scholar.google.com/citations?user=n6q5R2QAAAAJ","claimed":true,"country_code":null,"created_source":"RESEARCHHUB","last_full_fetch_from_openalex":null,"two_year_mean_citedness":2.888888888888889,"user":113849,"university":null,"merged_with_author":null}},"totalContribution":{"rsc":50,"usd":0},"contributions":[{"amount":50,"currency":"RSC","date":"2025-03-07T14:53:43.711185Z"}]},{"id":110365,"authorProfile":{"id":7276819,"fullName":"Minghao Li","firstName":"Minghao","lastName":"Li","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/02/10/blob_E1IqHbP","headline":"Biomedical researcher","profileUrl":"/author/7276819","user":{"id":0,"email":"","firstName":"","lastName":"","fullName":"Unknown User","isVerified":false,"authorProfile":"$undefined","balance":0,"lockedBalance":0,"promotionalBalance":0,"fundingCredits":0,"balances":"$undefined","totalRsc":0,"hasCompletedOnboarding":false,"createdDate":"$undefined","moderator":false,"editorOfHubs":"$undefined","isModerator":false,"isFunder":false,"referralCode":"$undefined","authProvider":"$undefined","isStakingOptedIn":false,"balanceHistory":0,"raw":110365},"description":"Focusing on research in single-cell transcriptomics, cellular and molecular biology, systemic inflammation, and metabolism. I have substantial practical experience with a broad range of methodologies, including fluorescence microscopy.","createdDate":"2025-01-22T06:14:50.454686Z","education":[{"id":2226,"name":"Fujian Medical University","year":{"label":"2028","value":"2028"},"major":"","degree":{"label":"Doctor of Medicine","value":"MD"},"summary":"MD '28, Fujian Medical University","is_public":true,"university":{"id":2226,"city":"","name":"Fujian Medical University","state":"","country":"China","updated_date":"2020-11-24T21:44:27.597793Z"}}],"twitter":"$undefined","facebook":"$undefined","linkedin":"$undefined","googleScholar":"https://scholar.google.com/citations?hl=en&view_op=list_works&gmla=AOv-ny_lauTtMGYQ03CEktQz6uo-Lo5ATj6Plch9C1_F-0S3s2LSl05iGJm-NqAg2I10c62vjDpsTTiCklslTg&user=r_Zxi6UAAAAJ","orcidId":"https://orcid.org/0000-0003-4885-979X","isOrcidConnected":false,"isClaimed":true,"userId":"$undefined","isVerified":true,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":7276819,"is_verified":true,"is_public":true,"is_removed":false,"is_removed_date":null,"first_name":"Minghao","last_name":"Li","created_date":"2025-01-22T06:14:50.454686Z","updated_date":"2026-05-13T09:27:39.414908Z","description":"Focusing on research in single-cell transcriptomics, cellular and molecular biology, systemic inflammation, and metabolism. I have substantial practical experience with a broad range of methodologies, including fluorescence microscopy.","h_index":3,"i10_index":0,"profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/02/10/blob_E1IqHbP","author_score":0,"orcid_id":"https://orcid.org/0000-0003-4885-979X","openalex_ids":["https://openalex.org/A5022338646","https://openalex.org/A5100330253"],"education":"$4f:props:children:props:topBanner:props:work:note:post:fundraise:contributors:topContributors:9:authorProfile:education","headline":"Biomedical researcher","facebook":null,"twitter":"","linkedin":"","google_scholar":"https://scholar.google.com/citations?hl=en&view_op=list_works&gmla=AOv-ny_lauTtMGYQ03CEktQz6uo-Lo5ATj6Plch9C1_F-0S3s2LSl05iGJm-NqAg2I10c62vjDpsTTiCklslTg&user=r_Zxi6UAAAAJ","claimed":true,"country_code":null,"created_source":"RESEARCHHUB","last_full_fetch_from_openalex":null,"two_year_mean_citedness":1.6666666666666667,"user":110365,"university":null,"merged_with_author":null}},"totalContribution":{"rsc":50,"usd":0},"contributions":[{"amount":50,"currency":"RSC","date":"2025-03-05T12:39:14.930826Z"}]}]},"createdBy":{"id":0,"email":"","firstName":"","lastName":"","fullName":"Unknown User","isVerified":false,"authorProfile":"$undefined","balance":0,"lockedBalance":0,"totalRsc":0,"hasCompletedOnboarding":false,"moderator":false,"raw":null,"isModerator":false,"isFunder":false,"authProvider":"$undefined","isStakingOptedIn":false,"balanceHistory":0},"status":"CLOSED","goalCurrency":"USD","startDate":"2025-03-02T02:52:27.605472Z","endDate":"2025-04-01T02:52:27.605299Z","createdDate":"","updatedDate":"","postId":"$undefined","postTitle":"$undefined","postSlug":"$undefined","postImage":null,"reviewMetrics":"$undefined","nonprofit":"$undefined","raw":{"id":16,"amount_raised":{"usd":0,"rsc":0},"goal_amount":{"usd":230000,"rsc":4586313.139080053},"contributors":{"total":10,"top":[{"id":36837,"author_profile":"$4f:props:children:props:topBanner:props:work:note:post:fundraise:contributors:topContributors:0:authorProfile:raw","is_funder":false,"rsc_earned":null,"balances":null,"benefits_expire_on":null,"editor_of":[],"is_verified":true,"last_login":"2026-07-20T20:54:09.295082Z","is_superuser":false,"username":"cryptyler24@gmail.com","first_name":"Tyler","last_name":"D.","email":"cryptyler24@gmail.com","is_staff":false,"is_active":true,"date_joined":"2022-08-08T21:44:32.472634Z","is_public":true,"is_removed":false,"is_removed_date":null,"agreed_to_terms":true,"clicked_on_balance_date":"2026-08-11T15:47:05.169317Z","country_code":"US","created_date":"2022-08-08T21:44:33.395958Z","has_seen_first_coin_modal":true,"has_seen_orcid_connect_modal":true,"has_completed_onboarding":true,"complete_profile_email_sent_date":null,"is_suspended":false,"moderator":true,"probable_spammer":false,"referral_code":"48a2fe10-a111-4e57-b4e2-fe03cc790a5c","reputation":907,"should_display_rsc_balance_home":false,"is_staking_opted_in":true,"staking_opted_in_date":"2026-04-10T20:10:59.094121Z","spam_updated_date":"2026-04-11T16:19:41.448573Z","suspended_updated_date":"2026-04-11T16:19:41.471587Z","updated_date":"2026-04-11T16:22:14.774190Z","upload_tutorial_complete":false,"is_official_account":false,"invited_by":33492,"groups":[],"user_permissions":[],"total_contribution":{"rsc":2500,"rsc_usd_snapshot":819.3137467500001,"usd":0},"contributions":[{"amount":2500,"currency":"RSC","date":"2025-03-03T22:40:53.972291Z"}]},{"id":14,"author_profile":"$4f:props:children:props:topBanner:props:work:note:post:fundraise:contributors:topContributors:1:authorProfile:raw","is_funder":false,"rsc_earned":null,"balances":null,"benefits_expire_on":null,"editor_of":[],"is_verified":true,"last_login":"2026-09-22T13:38:58.720320Z","is_superuser":false,"username":"pdjoyce44@gmail.com","first_name":"Patrick","last_name":"Joyce","email":"pdjoyce44@gmail.com","is_staff":false,"is_active":true,"date_joined":"2019-11-19T20:10:16Z","is_public":true,"is_removed":false,"is_removed_date":null,"agreed_to_terms":true,"clicked_on_balance_date":"2026-08-03T15:23:55.406148Z","country_code":"US","created_date":"2019-11-19T20:10:16.834220Z","has_seen_first_coin_modal":true,"has_seen_orcid_connect_modal":true,"has_completed_onboarding":true,"complete_profile_email_sent_date":null,"is_suspended":false,"moderator":true,"probable_spammer":false,"referral_code":"6ca532d5-3128-40a2-ad6b-056f5f2c7771","reputation":2203,"should_display_rsc_balance_home":true,"is_staking_opted_in":true,"staking_opted_in_date":"2026-04-10T20:10:46.784000Z","spam_updated_date":"2021-05-29T22:48:29.527782Z","suspended_updated_date":"2020-11-05T20:56:53.923934Z","updated_date":"2026-04-08T15:34:16.382744Z","upload_tutorial_complete":true,"is_official_account":false,"invited_by":31267,"groups":[],"user_permissions":[],"total_contribution":{"rsc":2500,"rsc_usd_snapshot":927.476708,"usd":0},"contributions":[{"amount":2500,"currency":"RSC","date":"2025-03-03T17:27:16.223469Z"}]},{"id":33901,"author_profile":"$4f:props:children:props:topBanner:props:work:note:post:fundraise:contributors:topContributors:2:authorProfile:raw","is_funder":false,"rsc_earned":null,"balances":null,"benefits_expire_on":null,"editor_of":[],"is_verified":true,"last_login":"2026-07-17T21:10:09.994287Z","is_superuser":false,"username":"maulikdhandha@gmail.com","first_name":"Maulik","last_name":"Dhandha","email":"maulikdhandha@gmail.com","is_staff":false,"is_active":true,"date_joined":"2021-12-11T12:14:42.554791Z","is_public":true,"is_removed":false,"is_removed_date":null,"agreed_to_terms":true,"clicked_on_balance_date":"2025-05-30T03:20:44.574666Z","country_code":"NZ","created_date":"2021-12-11T12:14:42.686673Z","has_seen_first_coin_modal":true,"has_seen_orcid_connect_modal":false,"has_completed_onboarding":true,"complete_profile_email_sent_date":null,"is_suspended":false,"moderator":false,"probable_spammer":false,"referral_code":"6c37ca39-6dd0-4d09-bd4f-8e6e22ef104f","reputation":1446,"should_display_rsc_balance_home":false,"is_staking_opted_in":true,"staking_opted_in_date":"2026-04-16T15:44:08.328272Z","spam_updated_date":"2024-10-28T20:54:51.681815Z","suspended_updated_date":null,"updated_date":"2024-11-05T02:34:53.371912Z","upload_tutorial_complete":false,"is_official_account":false,"invited_by":34416,"groups":[],"user_permissions":[],"total_contribution":{"rsc":2000,"rsc_usd_snapshot":605.81973225,"usd":0},"contributions":[{"amount":500,"currency":"RSC","date":"2025-03-11T18:50:24.720557Z"},{"amount":1500,"currency":"RSC","date":"2025-03-03T21:46:12.039482Z"}]},{"id":36296,"author_profile":"$4f:props:children:props:topBanner:props:work:note:post:fundraise:contributors:topContributors:3:authorProfile:raw","is_funder":false,"rsc_earned":null,"balances":null,"benefits_expire_on":null,"editor_of":[],"is_verified":true,"last_login":"2026-06-02T00:10:09.678504Z","is_superuser":false,"username":"delyeacole@gmail.com","first_name":"Cole","last_name":"Delyea","email":"delyeacole@gmail.com","is_staff":false,"is_active":true,"date_joined":"2022-06-17T00:04:11.102664Z","is_public":true,"is_removed":false,"is_removed_date":null,"agreed_to_terms":true,"clicked_on_balance_date":"2026-10-19T01:25:27.030621Z","country_code":"US","created_date":"2022-06-17T00:04:11.478811Z","has_seen_first_coin_modal":true,"has_seen_orcid_connect_modal":true,"has_completed_onboarding":true,"complete_profile_email_sent_date":null,"is_suspended":false,"moderator":true,"probable_spammer":false,"referral_code":"a5bfb84c-8c51-4f55-9699-ed012901863b","reputation":2111,"should_display_rsc_balance_home":false,"is_staking_opted_in":true,"staking_opted_in_date":"2026-04-10T23:02:46.638325Z","spam_updated_date":"2024-11-23T17:29:14.387624Z","suspended_updated_date":"2025-06-16T19:20:14.331700Z","updated_date":"2026-05-26T16:12:47.221500Z","upload_tutorial_complete":false,"is_official_account":false,"invited_by":null,"groups":[],"user_permissions":[],"total_contribution":{"rsc":2000,"rsc_usd_snapshot":674.3793206,"usd":0},"contributions":[{"amount":2000,"currency":"RSC","date":"2025-03-08T02:00:08.129174Z"}]},{"id":112244,"author_profile":"$4f:props:children:props:topBanner:props:work:note:post:fundraise:contributors:topContributors:4:authorProfile:raw","is_funder":false,"rsc_earned":null,"balances":null,"benefits_expire_on":null,"editor_of":[],"is_verified":false,"last_login":"2025-06-19T07:59:50.599225Z","is_superuser":false,"username":"dev@researchhub.foundation","first_name":"Developer","last_name":"ResearchHub Foundation","email":"dev@researchhub.foundation","is_staff":false,"is_active":true,"date_joined":"2025-02-09T03:34:48.787182Z","is_public":true,"is_removed":false,"is_removed_date":null,"agreed_to_terms":false,"clicked_on_balance_date":"2025-04-09T06:34:38.432935Z","country_code":null,"created_date":"2025-02-09T03:34:48.896625Z","has_seen_first_coin_modal":false,"has_seen_orcid_connect_modal":false,"has_completed_onboarding":true,"complete_profile_email_sent_date":null,"is_suspended":false,"moderator":false,"probable_spammer":false,"referral_code":"998162e3-829c-4959-9afa-fcdfd4f1bc4e","reputation":101,"should_display_rsc_balance_home":true,"is_staking_opted_in":false,"staking_opted_in_date":null,"spam_updated_date":null,"suspended_updated_date":null,"updated_date":"2025-02-09T03:35:00.511763Z","upload_tutorial_complete":false,"is_official_account":false,"invited_by":null,"groups":[],"user_permissions":[],"total_contribution":{"rsc":888,"rsc_usd_snapshot":354.024589032,"usd":0},"contributions":[{"amount":888,"currency":"RSC","date":"2025-03-03T10:06:47.652500Z"}]},{"id":613,"author_profile":"$4f:props:children:props:topBanner:props:work:note:post:fundraise:contributors:topContributors:5:authorProfile:raw","is_funder":false,"rsc_earned":null,"balances":null,"benefits_expire_on":null,"editor_of":[],"is_verified":true,"last_login":"2026-04-24T21:37:31.725895Z","is_superuser":false,"username":"nickcryptosss@gmail.com","first_name":"Nick","last_name":"Cryptosss","email":"nickcryptosss@gmail.com","is_staff":false,"is_active":true,"date_joined":"2020-08-03T06:16:50.629114Z","is_public":true,"is_removed":false,"is_removed_date":null,"agreed_to_terms":true,"clicked_on_balance_date":"2026-04-24T21:37:41.001516Z","country_code":"RU","created_date":"2020-08-03T06:16:50.637308Z","has_seen_first_coin_modal":true,"has_seen_orcid_connect_modal":true,"has_completed_onboarding":true,"complete_profile_email_sent_date":null,"is_suspended":false,"moderator":false,"probable_spammer":true,"referral_code":"d5daacbe-22ab-4cbb-8cad-ebf2a06c1996","reputation":102,"should_display_rsc_balance_home":true,"is_staking_opted_in":true,"staking_opted_in_date":"2026-04-24T21:40:18.690118Z","spam_updated_date":"2025-01-11T01:33:50.779014Z","suspended_updated_date":"2024-01-09T23:01:26.593901Z","updated_date":"2025-01-11T01:33:43.136457Z","upload_tutorial_complete":false,"is_official_account":false,"invited_by":null,"groups":[],"user_permissions":[],"total_contribution":{"rsc":160,"rsc_usd_snapshot":60.516207886000004,"usd":0},"contributions":[{"amount":10,"currency":"RSC","date":"2025-03-11T04:36:14.292288Z"},{"amount":150,"currency":"RSC","date":"2025-03-03T15:18:02.290005Z"}]},{"id":49105,"author_profile":"$4f:props:children:props:topBanner:props:work:note:post:fundraise:contributors:topContributors:6:authorProfile:raw","is_funder":false,"rsc_earned":null,"balances":null,"benefits_expire_on":null,"editor_of":[],"is_verified":true,"last_login":"2026-01-03T17:49:34.202737Z","is_superuser":false,"username":"kotwal.shashikant7@gmail.com","first_name":"Shashikant","last_name":"Kotwal","email":"kotwal.shashikant7@gmail.com","is_staff":false,"is_active":true,"date_joined":"2023-12-11T05:58:51.371020Z","is_public":true,"is_removed":false,"is_removed_date":null,"agreed_to_terms":false,"clicked_on_balance_date":"2025-05-14T17:41:38.147719Z","country_code":null,"created_date":"2023-12-11T05:58:51.553082Z","has_seen_first_coin_modal":true,"has_seen_orcid_connect_modal":true,"has_completed_onboarding":true,"complete_profile_email_sent_date":null,"is_suspended":false,"moderator":false,"probable_spammer":false,"referral_code":"9fd57a86-b9f6-4533-8bf1-8d3256c8391e","reputation":143,"should_display_rsc_balance_home":false,"is_staking_opted_in":true,"staking_opted_in_date":"2026-04-16T15:44:08.328272Z","spam_updated_date":null,"suspended_updated_date":null,"updated_date":"2025-01-23T04:41:55.342174Z","upload_tutorial_complete":false,"is_official_account":false,"invited_by":null,"groups":[],"user_permissions":[],"total_contribution":{"rsc":100,"rsc_usd_snapshot":36.73921674,"usd":0},"contributions":[{"amount":100,"currency":"RSC","date":"2025-03-07T01:17:06.407046Z"}]},{"id":58703,"author_profile":"$4f:props:children:props:topBanner:props:work:note:post:fundraise:contributors:topContributors:7:authorProfile:raw","is_funder":false,"rsc_earned":null,"balances":null,"benefits_expire_on":null,"editor_of":[],"is_verified":false,"last_login":"2025-11-19T22:23:59.627791Z","is_superuser":false,"username":"dizzyorange@gmail.com","first_name":"dizzyorange","last_name":"","email":"dizzyorange@gmail.com","is_staff":false,"is_active":true,"date_joined":"2024-03-30T22:51:49.926026Z","is_public":true,"is_removed":false,"is_removed_date":null,"agreed_to_terms":false,"clicked_on_balance_date":"2025-03-10T19:48:33.625465Z","country_code":null,"created_date":"2024-03-30T22:51:50.062643Z","has_seen_first_coin_modal":true,"has_seen_orcid_connect_modal":false,"has_completed_onboarding":true,"complete_profile_email_sent_date":null,"is_suspended":false,"moderator":false,"probable_spammer":false,"referral_code":"6ddd691e-40a2-4aed-a144-123339cf6e1f","reputation":102,"should_display_rsc_balance_home":true,"is_staking_opted_in":true,"staking_opted_in_date":"2026-04-16T15:44:08.328272Z","spam_updated_date":null,"suspended_updated_date":null,"updated_date":"2025-03-17T17:24:36.471945Z","upload_tutorial_complete":false,"is_official_account":false,"invited_by":null,"groups":[],"user_permissions":[],"total_contribution":{"rsc":91,"rsc_usd_snapshot":22.2776894613,"usd":0},"contributions":[{"amount":91,"currency":"RSC","date":"2025-03-10T19:47:57.736878Z"}]},{"id":113849,"author_profile":"$4f:props:children:props:topBanner:props:work:note:post:fundraise:contributors:topContributors:8:authorProfile:raw","is_funder":false,"rsc_earned":null,"balances":null,"benefits_expire_on":null,"editor_of":[],"is_verified":true,"last_login":"2025-03-07T16:32:54.398194Z","is_superuser":false,"username":"jbreuer83@gmail.com","first_name":"Johannes","last_name":"Breuer","email":"jbreuer83@gmail.com","is_staff":false,"is_active":true,"date_joined":"2025-03-07T12:30:34.637319Z","is_public":true,"is_removed":false,"is_removed_date":null,"agreed_to_terms":false,"clicked_on_balance_date":"2025-03-07T16:32:57.253334Z","country_code":null,"created_date":"2025-03-07T12:30:34.747368Z","has_seen_first_coin_modal":false,"has_seen_orcid_connect_modal":false,"has_completed_onboarding":true,"complete_profile_email_sent_date":null,"is_suspended":false,"moderator":false,"probable_spammer":false,"referral_code":"d2bf3ffe-c281-49f5-ac79-10fc724d3873","reputation":103,"should_display_rsc_balance_home":false,"is_staking_opted_in":false,"staking_opted_in_date":null,"spam_updated_date":null,"suspended_updated_date":null,"updated_date":"2025-03-07T12:34:38.932813Z","upload_tutorial_complete":false,"is_official_account":false,"invited_by":null,"groups":[],"user_permissions":[],"total_contribution":{"rsc":50,"rsc_usd_snapshot":18.481236425,"usd":0},"contributions":[{"amount":50,"currency":"RSC","date":"2025-03-07T14:53:43.711185Z"}]},{"id":110365,"author_profile":"$4f:props:children:props:topBanner:props:work:note:post:fundraise:contributors:topContributors:9:authorProfile:raw","is_funder":false,"rsc_earned":null,"balances":null,"benefits_expire_on":null,"editor_of":[],"is_verified":true,"last_login":"2026-07-19T08:52:31.414259Z","is_superuser":false,"username":"liminghaodoctor@163.com","first_name":"Minghao","last_name":"li","email":"liminghaodoctor@163.com","is_staff":false,"is_active":true,"date_joined":"2025-01-22T06:14:49.959195Z","is_public":true,"is_removed":false,"is_removed_date":null,"agreed_to_terms":false,"clicked_on_balance_date":"2026-10-09T13:06:25.752048Z","country_code":null,"created_date":"2025-01-22T06:14:50.416065Z","has_seen_first_coin_modal":true,"has_seen_orcid_connect_modal":false,"has_completed_onboarding":true,"complete_profile_email_sent_date":null,"is_suspended":false,"moderator":false,"probable_spammer":false,"referral_code":"899b44b1-ddda-4fa0-bf7b-6be90787b459","reputation":127,"should_display_rsc_balance_home":true,"is_staking_opted_in":true,"staking_opted_in_date":"2026-04-24T09:39:48.620710Z","spam_updated_date":"2025-06-15T01:26:20.408143Z","suspended_updated_date":null,"updated_date":"2025-11-03T22:11:00.718841Z","upload_tutorial_complete":false,"is_official_account":false,"invited_by":null,"groups":[],"user_permissions":[],"total_contribution":{"rsc":50,"rsc_usd_snapshot":17.869392765,"usd":0},"contributions":[{"amount":50,"currency":"RSC","date":"2025-03-05T12:39:14.930826Z"}]}]},"status":"CLOSED","goal_currency":"USD","start_date":"2025-03-02T02:52:27.605472Z","end_date":"2025-04-01T02:52:27.605299Z"}},"grant":"$undefined","doi":"10.55277/ResearchHub.nj64k80a","authors":[{"authorId":932947,"userId":34416,"name":"Jeffrey Koury","raw":{"id":932947,"first_name":"Jeffrey","last_name":"Koury","user":34416}},{"authorId":993107,"userId":60018,"name":"Dominikus Brian","raw":{"id":993107,"first_name":"Dominikus","last_name":"Brian","user":60018}}],"contacts":"$undefined","image":"https://storage.prod.researchhub.com/comment_files/png/4e40a679037254672ee5f249ea491419.png","raw":{"id":3961,"status":"APPROVED","authors":["$4f:props:children:props:topBanner:props:work:note:post:authors:0:raw","$4f:props:children:props:topBanner:props:work:note:post:authors:1:raw"],"unified_document":{"fundraise":"$4f:props:children:props:topBanner:props:work:note:post:fundraise:raw","grant":null},"image_url":"https://storage.prod.researchhub.com/comment_files/png/4e40a679037254672ee5f249ea491419.png","document_type":"PREREGISTRATION","slug":"artemis-automated-review-and-trustworthy-evaluation-for-manuscripts-in-sciencebr","doi":"10.55277/ResearchHub.nj64k80a"}},"documentType":null,"proposalId":null,"registeredReportPrefill":null,"image":null,"previewImage":null,"authors":[],"grantSettings":null,"preregistrationSettings":null,"selectedGrant":null,"latestChatId":"$undefined","raw":{"id":11406,"access":"WORKSPACE","authors":[],"grant_settings":null,"latest_version":{"id":246700,"src":null,"created_date":"2025-04-11T17:08:28.256624Z","created_via":null,"plain_text":"$86","json":"$87","created_by":null,"note":11406,"parent_version":null},"organization":{"id":109221,"member_count":2,"user_permission":null,"created_date":"2025-02-25T02:29:44.106712Z","updated_date":"2025-02-25T02:30:00.479976Z","cover_image":null,"description":"","name":"Domi and Jeff","note_created":true,"org_type":"ORGANIZATION","slug":"domi-and-jeff","user":null},"post":"$4f:props:children:props:topBanner:props:work:note:post:raw","preregistration_settings":null,"registered_report_prefill":null,"selected_grant":null,"selected_grant_details":null,"unified_document":{"id":7343049,"is_removed":false},"created_date":"2025-02-25T02:30:00.038111Z","updated_date":"2025-04-16T10:04:40.461418Z","document_type":null,"image":"","preview_img":null,"title":"Preregistration Template","created_by":34416},"content":null,"versionId":246700,"versionDate":"2025-04-11T17:08:28.256624Z","plainText":"$88","contentJson":"$89","versionCreatedVia":null},"previewContent":"$8a","contentUrl":"https://storage.prod.researchhub.com/uploads/post_discussion/2025/03/07/RH-POST-DISCUSSION-USER-60018.txt","tips":[],"peerReviews":[{"id":6096,"createdBy":{"id":114138,"authorProfile":{"id":7623632,"fullName":"Scott Jacques","profileImage":"$8b","isVerified":false}},"score":4,"createdDate":"2025-03-13T15:25:56.801337Z","contentPreview":"$undefined","isAssessed":false},{"id":6099,"createdBy":{"id":114144,"authorProfile":{"id":7623638,"fullName":"Valeriia Lymishchenko","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/03/13/blob_aVoSgkJ","isVerified":false}},"score":5,"createdDate":"2025-03-13T16:23:11.572062Z","contentPreview":"$undefined","isAssessed":false},{"id":6229,"createdBy":{"id":74122,"authorProfile":{"id":5117728,"fullName":"Mario Malicki","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2024/11/20/blob_ye7YnHs","isVerified":false}},"score":4,"createdDate":"2025-03-18T05:32:56.598905Z","contentPreview":"$undefined","isAssessed":false},{"id":6274,"createdBy":{"id":114146,"authorProfile":{"id":7623663,"fullName":"Joshua Gerstenfeld","profileImage":"https://lh3.googleusercontent.com/a/ACg8ocL626qphSe8aR1IQOorXAjREY19-pqsRJu2mv2cLVWUOEI7KA=s96-c","isVerified":false}},"score":5,"createdDate":"2025-03-19T20:13:04.495020Z","contentPreview":"$undefined","isAssessed":false},{"id":6544,"createdBy":{"id":115070,"authorProfile":{"id":7658094,"fullName":"RAVI DESAI","profileImage":"https://lh3.googleusercontent.com/a/ACg8ocK3jQsp8xJqmc4UwtSkHZuNXFspAxhr76cWgNH_5Kpa8iyFPA=s96-c","isVerified":false}},"score":5,"createdDate":"2025-03-27T19:48:37.372674Z","contentPreview":"$undefined","isAssessed":false}],"image":"https://storage.prod.researchhub.com/comment_files/png/4e40a679037254672ee5f249ea491419.png","moderationStatus":"APPROVED","raw":{"score":62,"user_flag":null,"authors":["$4f:props:children:props:topBanner:props:work:authors:0:authorProfile:raw","$4f:props:children:props:topBanner:props:work:authors:1:authorProfile:raw"],"id":3961,"created_by":{"id":60018,"author_profile":{"id":993107,"is_public":true,"is_removed":false,"is_removed_date":null,"user":60018,"first_name":"Dominikus","last_name":"Brian","created_date":"2024-04-19T00:03:04.727929Z","updated_date":"2026-07-20T17:22:07.940431Z","description":"Building Backward toward the Organoid Economy.\nhttps://dominikusbrian.com","h_index":8,"i10_index":8,"profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/03/04/blob_pxj9rsH","author_score":0,"university":null,"orcid_id":"https://orcid.org/0000-0001-6165-0355","openalex_ids":["https://openalex.org/A5079564287"],"education":[{"id":4782,"name":"New York University","major":"Chemistry","summary":"Chemistry New York University","is_public":true},{"id":5903,"name":"Shanghai Jiaotong University","major":"Power Engineering & Engineering Thermophysics","summary":"Power Engineering & Engineering Thermophysics Shanghai Jiaotong University","is_public":false},{"id":5893,"name":"Shandong University of Science and Technology","major":"Chemical Engineering","summary":"Chemical Engineering Shandong University of Science and Technology","is_public":false}],"headline":"Applied AI @ RHF | Founder @ OrganoBit.Bio","facebook":null,"twitter":"https://x.com/dominikusbrian","linkedin":"https://www.linkedin.com/in/dominikus-brian-%E9%92%9F%E9%B8%BF%E7%9B%9B-82226161/","google_scholar":"https://scholar.google.com/citations?user=Ark53e0AAAAJ&hl=en","claimed":true,"country_code":null,"created_source":"RESEARCHHUB","last_full_fetch_from_openalex":null,"two_year_mean_citedness":0.9090909090909091,"merged_with_author":null,"added_as_editor_date":null,"is_hub_editor_of":[{"editor_permission_groups":[{"id":144619,"organization":null,"user":{"id":60018,"author_profile":{"id":993107,"is_verified":true,"is_public":true,"is_removed":false,"is_removed_date":null,"first_name":"Dominikus","last_name":"Brian","created_date":"2024-04-19T00:03:04.727929Z","updated_date":"2026-07-20T17:22:07.940431Z","description":"Building Backward toward the Organoid Economy.\nhttps://dominikusbrian.com","h_index":8,"i10_index":8,"profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/03/04/blob_pxj9rsH","author_score":0,"orcid_id":"https://orcid.org/0000-0001-6165-0355","openalex_ids":["https://openalex.org/A5079564287"],"education":[{"id":4782,"name":"New York University","major":"Chemistry","summary":"Chemistry New York University","is_public":true},{"id":5903,"name":"Shanghai Jiaotong University","major":"Power Engineering & Engineering Thermophysics","summary":"Power Engineering & Engineering Thermophysics Shanghai Jiaotong University","is_public":false},{"id":5893,"name":"Shandong University of Science and Technology","major":"Chemical Engineering","summary":"Chemical Engineering Shandong University of Science and Technology","is_public":false}],"headline":"Applied AI @ RHF | Founder @ OrganoBit.Bio","facebook":null,"twitter":"https://x.com/dominikusbrian","linkedin":"https://www.linkedin.com/in/dominikus-brian-%E9%92%9F%E9%B8%BF%E7%9B%9B-82226161/","google_scholar":"https://scholar.google.com/citations?user=Ark53e0AAAAJ&hl=en","claimed":true,"country_code":null,"created_source":"RESEARCHHUB","last_full_fetch_from_openalex":null,"two_year_mean_citedness":0.9090909090909091,"user":60018,"university":null,"merged_with_author":null}},"created_date":"2024-08-23T00:40:03.600433Z","updated_date":"2024-08-23T00:40:03.600456Z","access_type":"ASSISTANT_EDITOR","object_id":32,"content_type":18}],"hub_image":"https://storage.prod.researchhub.com/uploads/hub_images/2020/10/07/materials-science.jpg","id":32,"is_locked":false,"is_removed":false,"name":"Materials Science","slug":"materials-science","namespace":"subcategory"}],"is_hub_editor":true,"num_posts":20,"is_orcid_connected":true,"reputation":1469,"suspended_status":{"probable_spammer":false,"is_suspended":false},"total_score":null,"wallet":null,"is_verified":true},"balance":null,"balances":null,"created_date":"2024-04-19T00:03:04.682578Z","has_seen_first_coin_modal":true,"has_seen_orcid_connect_modal":false,"is_funder":false,"is_staking_opted_in":true,"is_suspended":false,"probable_spammer":false,"is_verified":true,"moderator":false,"reputation":1469,"updated_date":"2025-04-15T16:09:44.901057Z","upload_tutorial_complete":false,"hub_rep":null,"time_rep":null},"created_date":"2025-02-27T05:28:58.538424Z","discussion_count":23,"document_type":"PREREGISTRATION","doi":"10.55277/ResearchHub.nj64k80a","editor_type":"CK_EDITOR","full_markdown":"$8c","grants":[],"image_url":"https://storage.prod.researchhub.com/comment_files/png/4e40a679037254672ee5f249ea491419.png","is_latest_version":null,"is_removed":false,"is_root_version":true,"note":"$4f:props:children:props:topBanner:props:work:note:raw","peer_reviews":[{"id":6096,"created_by":{"id":114138,"author_profile":{"id":7623632,"first_name":"Scott","last_name":"Jacques","profile_image":"$8d"},"first_name":"Scott","last_name":"Jacques"},"created_date":"2025-03-13T15:25:56.801337Z","updated_date":"2025-03-13T15:25:56.801346Z","score":4,"is_assessed":false},{"id":6099,"created_by":{"id":114144,"author_profile":{"id":7623638,"first_name":"Valeriia","last_name":"Lymishchenko","profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/03/13/blob_aVoSgkJ"},"first_name":"Valeriia","last_name":"Lymishchenko"},"created_date":"2025-03-13T16:23:11.572062Z","updated_date":"2025-03-13T16:23:11.572074Z","score":5,"is_assessed":false},{"id":6229,"created_by":{"id":74122,"author_profile":{"id":5117728,"first_name":"Mario","last_name":"Malicki","profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2024/11/20/blob_ye7YnHs"},"first_name":"Mario","last_name":"Malicki"},"created_date":"2025-03-18T05:32:56.598905Z","updated_date":"2025-03-18T05:32:56.598917Z","score":4,"is_assessed":false},{"id":6274,"created_by":{"id":114146,"author_profile":{"id":7623663,"first_name":"Joshua","last_name":"Gerstenfeld","profile_image":"https://lh3.googleusercontent.com/a/ACg8ocL626qphSe8aR1IQOorXAjREY19-pqsRJu2mv2cLVWUOEI7KA=s96-c"},"first_name":"Joshua","last_name":"Gerstenfeld"},"created_date":"2025-03-19T20:13:04.495020Z","updated_date":"2025-03-19T20:13:04.495032Z","score":5,"is_assessed":false},{"id":6544,"created_by":{"id":115070,"author_profile":{"id":7658094,"first_name":"RAVI","last_name":"DESAI","profile_image":"https://lh3.googleusercontent.com/a/ACg8ocK3jQsp8xJqmc4UwtSkHZuNXFspAxhr76cWgNH_5Kpa8iyFPA=s96-c"},"first_name":"RAVI","last_name":"DESAI"},"created_date":"2025-03-27T19:48:37.372674Z","updated_date":"2025-03-27T19:48:37.372686Z","score":5,"is_assessed":false}],"post_src":"https://storage.prod.researchhub.com/uploads/post_discussion/2025/03/07/RH-POST-DISCUSSION-USER-60018.txt","preview_img":null,"registered_report_id":null,"renderable_text":"$8e","reviewed_by":null,"reviewed_date":null,"slug":"artemis-automated-review-and-trustworthy-evaluation-for-manuscripts-in-sciencebr","status":"APPROVED","title":"ARTEMIS - Automated Review and Trustworthy Evaluation for Manuscripts in Science
","unified_document_id":7345202,"unified_document":{"id":7345202,"documents":[{"id":3961,"title":"ARTEMIS - Automated Review and Trustworthy Evaluation for Manuscripts in Science
","slug":"artemis-automated-review-and-trustworthy-evaluation-for-manuscripts-in-sciencebr"}],"created_by":{"id":60018,"author_profile":{"id":993107,"first_name":"Dominikus","last_name":"Brian","profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/03/04/blob_pxj9rsH"}},"reviews":{"avg":0,"count":0},"is_public":true,"is_removed":false,"document_type":"PREREGISTRATION"},"version_number":1,"updated_date":"2025-05-19T16:20:59.129767Z","bounty_type":null},"aiPeerReview":null,"linkedGrant":null,"isPublic":true},"metadata":{"id":7345202,"score":49,"metrics":{"votes":49,"comments":23,"saves":0,"reviewScore":0,"conversationComments":23,"reviewComments":5,"bountyComments":1},"fundraising":{"id":16,"amountRaised":{"usd":0,"rsc":0},"goalAmount":{"usd":230000,"rsc":4586313.139080053},"contributors":{"numContributors":10,"topContributors":[{"id":36837,"authorProfile":{"id":956713,"fullName":"Tyler Diorio","firstName":"Tyler","lastName":"Diorio","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2022/08/09/blob_im4Dh4d","headline":"","profileUrl":"/author/956713","user":"$undefined","description":"$undefined","createdDate":"$undefined","education":"$undefined","twitter":"$undefined","facebook":"$undefined","linkedin":"$undefined","googleScholar":"$undefined","orcidId":"$undefined","isOrcidConnected":false,"isClaimed":false,"userId":"$undefined","isVerified":false,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":956713,"first_name":"Tyler","last_name":"Diorio","profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2022/08/09/blob_im4Dh4d"}},"totalContribution":{"rsc":2500,"usd":0},"contributions":[{"amount":2500,"currency":"RSC","date":"2025-03-03T22:40:53.972291Z"}]},{"id":14,"authorProfile":{"id":22,"fullName":"Patrick Joyce","firstName":"Patrick","lastName":"Joyce","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2019/11/19/blob","headline":"","profileUrl":"/author/22","user":"$undefined","description":"$undefined","createdDate":"$undefined","education":"$undefined","twitter":"$undefined","facebook":"$undefined","linkedin":"$undefined","googleScholar":"$undefined","orcidId":"$undefined","isOrcidConnected":false,"isClaimed":false,"userId":"$undefined","isVerified":false,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":22,"first_name":"Patrick","last_name":"Joyce","profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2019/11/19/blob"}},"totalContribution":{"rsc":2500,"usd":0},"contributions":[{"amount":2500,"currency":"RSC","date":"2025-03-03T17:27:16.223469Z"}]},{"id":33901,"authorProfile":{"id":931964,"fullName":"Maulik Dhandha","firstName":"Maulik","lastName":"Dhandha","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2024/10/27/blob_yxyA3Mo","headline":"","profileUrl":"/author/931964","user":"$undefined","description":"$undefined","createdDate":"$undefined","education":"$undefined","twitter":"$undefined","facebook":"$undefined","linkedin":"$undefined","googleScholar":"$undefined","orcidId":"$undefined","isOrcidConnected":false,"isClaimed":false,"userId":"$undefined","isVerified":false,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":931964,"first_name":"Maulik","last_name":"Dhandha","profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2024/10/27/blob_yxyA3Mo"}},"totalContribution":{"rsc":2000,"usd":0},"contributions":[{"amount":500,"currency":"RSC","date":"2025-03-11T18:50:24.720557Z"},{"amount":1500,"currency":"RSC","date":"2025-03-03T21:46:12.039482Z"}]},{"id":36296,"authorProfile":{"id":952195,"fullName":"Cole Delyea","firstName":"Cole","lastName":"Delyea","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2022/07/29/blob","headline":"","profileUrl":"/author/952195","user":"$undefined","description":"$undefined","createdDate":"$undefined","education":"$undefined","twitter":"$undefined","facebook":"$undefined","linkedin":"$undefined","googleScholar":"$undefined","orcidId":"$undefined","isOrcidConnected":false,"isClaimed":false,"userId":"$undefined","isVerified":false,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":952195,"first_name":"Cole","last_name":"Delyea","profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2022/07/29/blob"}},"totalContribution":{"rsc":2000,"usd":0},"contributions":[{"amount":2000,"currency":"RSC","date":"2025-03-08T02:00:08.129174Z"}]},{"id":112244,"authorProfile":{"id":7473890,"fullName":"RHF Tech Support","firstName":"RHF","lastName":"Tech Support","profileImage":"https://lh3.googleusercontent.com/a/ACg8ocKCzNCX9Nj8Acf6c5dNHWjzuAzWnrXcwjgeHkWKF-DSLD6Gfw=s96-c","headline":"","profileUrl":"/author/7473890","user":"$undefined","description":"$undefined","createdDate":"$undefined","education":"$undefined","twitter":"$undefined","facebook":"$undefined","linkedin":"$undefined","googleScholar":"$undefined","orcidId":"$undefined","isOrcidConnected":false,"isClaimed":false,"userId":"$undefined","isVerified":false,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":7473890,"first_name":"RHF","last_name":"Tech Support","profile_image":"https://lh3.googleusercontent.com/a/ACg8ocKCzNCX9Nj8Acf6c5dNHWjzuAzWnrXcwjgeHkWKF-DSLD6Gfw=s96-c"}},"totalContribution":{"rsc":888,"usd":0},"contributions":[{"amount":888,"currency":"RSC","date":"2025-03-03T10:06:47.652500Z"}]},{"id":613,"authorProfile":{"id":572,"fullName":"Natalya Efremova","firstName":"Natalya","lastName":"Efremova","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/01/23/blob_iU88Om0","headline":"","profileUrl":"/author/572","user":"$undefined","description":"$undefined","createdDate":"$undefined","education":"$undefined","twitter":"$undefined","facebook":"$undefined","linkedin":"$undefined","googleScholar":"$undefined","orcidId":"$undefined","isOrcidConnected":false,"isClaimed":false,"userId":"$undefined","isVerified":false,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":572,"first_name":"Natalya","last_name":"Efremova","profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/01/23/blob_iU88Om0"}},"totalContribution":{"rsc":160,"usd":0},"contributions":[{"amount":10,"currency":"RSC","date":"2025-03-11T04:36:14.292288Z"},{"amount":150,"currency":"RSC","date":"2025-03-03T15:18:02.290005Z"}]},{"id":49105,"authorProfile":{"id":982634,"fullName":"Shashikant Kotwal","firstName":"Shashikant","lastName":"Kotwal","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2024/04/27/blob_bvG0QSu","headline":"","profileUrl":"/author/982634","user":"$undefined","description":"$undefined","createdDate":"$undefined","education":"$undefined","twitter":"$undefined","facebook":"$undefined","linkedin":"$undefined","googleScholar":"$undefined","orcidId":"$undefined","isOrcidConnected":false,"isClaimed":false,"userId":"$undefined","isVerified":false,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":982634,"first_name":"Shashikant","last_name":"Kotwal","profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2024/04/27/blob_bvG0QSu"}},"totalContribution":{"rsc":100,"usd":0},"contributions":[{"amount":100,"currency":"RSC","date":"2025-03-07T01:17:06.407046Z"}]},{"id":58703,"authorProfile":{"id":991858,"fullName":"M K","firstName":"M","lastName":"K","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/12/05/blob_91W20yo","headline":"","profileUrl":"/author/991858","user":"$undefined","description":"$undefined","createdDate":"$undefined","education":"$undefined","twitter":"$undefined","facebook":"$undefined","linkedin":"$undefined","googleScholar":"$undefined","orcidId":"$undefined","isOrcidConnected":false,"isClaimed":false,"userId":"$undefined","isVerified":false,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":991858,"first_name":"M","last_name":"K","profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/12/05/blob_91W20yo"}},"totalContribution":{"rsc":91,"usd":0},"contributions":[{"amount":91,"currency":"RSC","date":"2025-03-10T19:47:57.736878Z"}]},{"id":113849,"authorProfile":{"id":7603504,"fullName":"Johannes Breuer","firstName":"Johannes","lastName":"Breuer","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/03/07/blob_K5CbTjU","headline":"","profileUrl":"/author/7603504","user":"$undefined","description":"$undefined","createdDate":"$undefined","education":"$undefined","twitter":"$undefined","facebook":"$undefined","linkedin":"$undefined","googleScholar":"$undefined","orcidId":"$undefined","isOrcidConnected":false,"isClaimed":false,"userId":"$undefined","isVerified":false,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":7603504,"first_name":"Johannes","last_name":"Breuer","profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/03/07/blob_K5CbTjU"}},"totalContribution":{"rsc":50,"usd":0},"contributions":[{"amount":50,"currency":"RSC","date":"2025-03-07T14:53:43.711185Z"}]},{"id":110365,"authorProfile":{"id":7276819,"fullName":"Minghao Li","firstName":"Minghao","lastName":"Li","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/02/10/blob_E1IqHbP","headline":"","profileUrl":"/author/7276819","user":"$undefined","description":"$undefined","createdDate":"$undefined","education":"$undefined","twitter":"$undefined","facebook":"$undefined","linkedin":"$undefined","googleScholar":"$undefined","orcidId":"$undefined","isOrcidConnected":false,"isClaimed":false,"userId":"$undefined","isVerified":false,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":7276819,"first_name":"Minghao","last_name":"Li","profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/02/10/blob_E1IqHbP"}},"totalContribution":{"rsc":50,"usd":0},"contributions":[{"amount":50,"currency":"RSC","date":"2025-03-05T12:39:14.930826Z"}]}]},"createdBy":{"id":0,"email":"","firstName":"","lastName":"","fullName":"Unknown User","isVerified":false,"authorProfile":"$undefined","balance":0,"lockedBalance":0,"totalRsc":0,"hasCompletedOnboarding":false,"moderator":false,"raw":null,"isModerator":false,"isFunder":false,"authProvider":"$undefined","isStakingOptedIn":false,"balanceHistory":0},"status":"CLOSED","goalCurrency":"USD","startDate":"2025-03-02T02:52:27.605472Z","endDate":"2025-04-01T02:52:27.605299Z","createdDate":"","updatedDate":"","postId":"$undefined","postTitle":"$undefined","postSlug":"$undefined","postImage":null,"reviewMetrics":"$undefined","nonprofit":"$undefined","raw":{"id":16,"amount_raised":{"usd":0,"rsc":0},"goal_amount":{"usd":230000,"rsc":4586313.139080053},"contributors":{"total":10,"top":[{"id":36837,"author_profile":"$4f:props:children:props:topBanner:props:metadata:fundraising:contributors:topContributors:0:authorProfile:raw","first_name":"Tyler","last_name":"D.","total_contribution":{"rsc":2500,"rsc_usd_snapshot":819.3137467500001,"usd":0},"contributions":[{"amount":2500,"currency":"RSC","date":"2025-03-03T22:40:53.972291Z"}]},{"id":14,"author_profile":"$4f:props:children:props:topBanner:props:metadata:fundraising:contributors:topContributors:1:authorProfile:raw","first_name":"Patrick","last_name":"Joyce","total_contribution":{"rsc":2500,"rsc_usd_snapshot":927.476708,"usd":0},"contributions":[{"amount":2500,"currency":"RSC","date":"2025-03-03T17:27:16.223469Z"}]},{"id":33901,"author_profile":"$4f:props:children:props:topBanner:props:metadata:fundraising:contributors:topContributors:2:authorProfile:raw","first_name":"Maulik","last_name":"Dhandha","total_contribution":{"rsc":2000,"rsc_usd_snapshot":605.81973225,"usd":0},"contributions":[{"amount":500,"currency":"RSC","date":"2025-03-11T18:50:24.720557Z"},{"amount":1500,"currency":"RSC","date":"2025-03-03T21:46:12.039482Z"}]},{"id":36296,"author_profile":"$4f:props:children:props:topBanner:props:metadata:fundraising:contributors:topContributors:3:authorProfile:raw","first_name":"Cole","last_name":"Delyea","total_contribution":{"rsc":2000,"rsc_usd_snapshot":674.3793206,"usd":0},"contributions":[{"amount":2000,"currency":"RSC","date":"2025-03-08T02:00:08.129174Z"}]},{"id":112244,"author_profile":"$4f:props:children:props:topBanner:props:metadata:fundraising:contributors:topContributors:4:authorProfile:raw","first_name":"Developer","last_name":"ResearchHub Foundation","total_contribution":{"rsc":888,"rsc_usd_snapshot":354.024589032,"usd":0},"contributions":[{"amount":888,"currency":"RSC","date":"2025-03-03T10:06:47.652500Z"}]},{"id":613,"author_profile":"$4f:props:children:props:topBanner:props:metadata:fundraising:contributors:topContributors:5:authorProfile:raw","first_name":"Nick","last_name":"Cryptosss","total_contribution":{"rsc":160,"rsc_usd_snapshot":60.516207886000004,"usd":0},"contributions":[{"amount":10,"currency":"RSC","date":"2025-03-11T04:36:14.292288Z"},{"amount":150,"currency":"RSC","date":"2025-03-03T15:18:02.290005Z"}]},{"id":49105,"author_profile":"$4f:props:children:props:topBanner:props:metadata:fundraising:contributors:topContributors:6:authorProfile:raw","first_name":"Shashikant","last_name":"Kotwal","total_contribution":{"rsc":100,"rsc_usd_snapshot":36.73921674,"usd":0},"contributions":[{"amount":100,"currency":"RSC","date":"2025-03-07T01:17:06.407046Z"}]},{"id":58703,"author_profile":"$4f:props:children:props:topBanner:props:metadata:fundraising:contributors:topContributors:7:authorProfile:raw","first_name":"dizzyorange","last_name":"","total_contribution":{"rsc":91,"rsc_usd_snapshot":22.2776894613,"usd":0},"contributions":[{"amount":91,"currency":"RSC","date":"2025-03-10T19:47:57.736878Z"}]},{"id":113849,"author_profile":"$4f:props:children:props:topBanner:props:metadata:fundraising:contributors:topContributors:8:authorProfile:raw","first_name":"Johannes","last_name":"Breuer","total_contribution":{"rsc":50,"rsc_usd_snapshot":18.481236425,"usd":0},"contributions":[{"amount":50,"currency":"RSC","date":"2025-03-07T14:53:43.711185Z"}]},{"id":110365,"author_profile":"$4f:props:children:props:topBanner:props:metadata:fundraising:contributors:topContributors:9:authorProfile:raw","first_name":"Minghao","last_name":"li","total_contribution":{"rsc":50,"rsc_usd_snapshot":17.869392765,"usd":0},"contributions":[{"amount":50,"currency":"RSC","date":"2025-03-05T12:39:14.930826Z"}]}]},"status":"CLOSED","goal_currency":"USD","start_date":"2025-03-02T02:52:27.605472Z","end_date":"2025-04-01T02:52:27.605299Z"}},"bounties":[{"id":3950,"amount":"410.0000000000","status":"EXPIRED","expirationDate":"2026-10-24T01:00:53.443Z","bountyType":"BOUNTY","createdBy":{"id":34416,"email":"","firstName":"","lastName":"","fullName":"Unknown User","isVerified":false,"authorProfile":{"id":932947,"fullName":"Jeffrey Koury","firstName":"Jeffrey","lastName":"Koury","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2024/04/01/blob_Ut50nMY","headline":"","profileUrl":"/author/932947","user":"$undefined","description":"$undefined","createdDate":"$undefined","education":"$undefined","twitter":"$undefined","facebook":"$undefined","linkedin":"$undefined","googleScholar":"$undefined","orcidId":"$undefined","isOrcidConnected":false,"isClaimed":false,"userId":"$undefined","isVerified":false,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":932947,"first_name":"Jeffrey","last_name":"Koury","profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2024/04/01/blob_Ut50nMY"}},"balance":0,"lockedBalance":0,"promotionalBalance":0,"fundingCredits":0,"balances":"$undefined","totalRsc":0,"hasCompletedOnboarding":false,"createdDate":"$undefined","moderator":false,"editorOfHubs":"$undefined","isModerator":false,"isFunder":false,"referralCode":"$undefined","authProvider":"$undefined","isStakingOptedIn":false,"balanceHistory":0,"raw":{"id":34416,"author_profile":"$4f:props:children:props:topBanner:props:metadata:bounties:0:createdBy:authorProfile:raw"}},"solutions":[],"contributions":[],"totalAmount":"410","comment":"$undefined","raw":{"id":3950,"created_by":"$4f:props:children:props:topBanner:props:metadata:bounties:0:createdBy:raw","amount":"410.0000000000","status":"EXPIRED"}},{"id":3951,"amount":"390.0000000000","status":"EXPIRED","expirationDate":"2026-10-24T01:00:53.443Z","bountyType":"BOUNTY","createdBy":{"id":60018,"email":"","firstName":"","lastName":"","fullName":"Unknown User","isVerified":false,"authorProfile":{"id":993107,"fullName":"Dominikus Brian","firstName":"Dominikus","lastName":"Brian","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/03/04/blob_pxj9rsH","headline":"","profileUrl":"/author/993107","user":"$undefined","description":"$undefined","createdDate":"$undefined","education":"$undefined","twitter":"$undefined","facebook":"$undefined","linkedin":"$undefined","googleScholar":"$undefined","orcidId":"$undefined","isOrcidConnected":false,"isClaimed":false,"userId":"$undefined","isVerified":false,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":993107,"first_name":"Dominikus","last_name":"Brian","profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/03/04/blob_pxj9rsH"}},"balance":0,"lockedBalance":0,"promotionalBalance":0,"fundingCredits":0,"balances":"$undefined","totalRsc":0,"hasCompletedOnboarding":false,"createdDate":"$undefined","moderator":false,"editorOfHubs":"$undefined","isModerator":false,"isFunder":false,"referralCode":"$undefined","authProvider":"$undefined","isStakingOptedIn":false,"balanceHistory":0,"raw":{"id":60018,"author_profile":"$4f:props:children:props:topBanner:props:metadata:bounties:1:createdBy:authorProfile:raw"}},"solutions":[],"contributions":[],"totalAmount":"390","comment":"$undefined","raw":{"id":3951,"created_by":"$4f:props:children:props:topBanner:props:metadata:bounties:1:createdBy:raw","amount":"390.0000000000","status":"EXPIRED"}},{"id":3953,"amount":"500.0000000000","status":"EXPIRED","expirationDate":"2026-10-24T01:00:53.443Z","bountyType":"BOUNTY","createdBy":{"id":14,"email":"","firstName":"","lastName":"","fullName":"Unknown User","isVerified":false,"authorProfile":{"id":22,"fullName":"Patrick Joyce","firstName":"Patrick","lastName":"Joyce","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2019/11/19/blob","headline":"","profileUrl":"/author/22","user":"$undefined","description":"$undefined","createdDate":"$undefined","education":"$undefined","twitter":"$undefined","facebook":"$undefined","linkedin":"$undefined","googleScholar":"$undefined","orcidId":"$undefined","isOrcidConnected":false,"isClaimed":false,"userId":"$undefined","isVerified":false,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":22,"first_name":"Patrick","last_name":"Joyce","profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2019/11/19/blob"}},"balance":0,"lockedBalance":0,"promotionalBalance":0,"fundingCredits":0,"balances":"$undefined","totalRsc":0,"hasCompletedOnboarding":false,"createdDate":"$undefined","moderator":false,"editorOfHubs":"$undefined","isModerator":false,"isFunder":false,"referralCode":"$undefined","authProvider":"$undefined","isStakingOptedIn":false,"balanceHistory":0,"raw":{"id":14,"author_profile":"$4f:props:children:props:topBanner:props:metadata:bounties:2:createdBy:authorProfile:raw"}},"solutions":[],"contributions":[],"totalAmount":"500","comment":"$undefined","raw":{"id":3953,"created_by":"$4f:props:children:props:topBanner:props:metadata:bounties:2:createdBy:raw","amount":"500.0000000000","status":"EXPIRED"}},{"id":3952,"amount":"390.0000000000","status":"EXPIRED","expirationDate":"2026-10-24T01:00:53.443Z","bountyType":"BOUNTY","createdBy":{"id":60018,"email":"","firstName":"","lastName":"","fullName":"Unknown User","isVerified":false,"authorProfile":{"id":993107,"fullName":"Dominikus Brian","firstName":"Dominikus","lastName":"Brian","profileImage":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/03/04/blob_pxj9rsH","headline":"","profileUrl":"/author/993107","user":"$undefined","description":"$undefined","createdDate":"$undefined","education":"$undefined","twitter":"$undefined","facebook":"$undefined","linkedin":"$undefined","googleScholar":"$undefined","orcidId":"$undefined","isOrcidConnected":false,"isClaimed":false,"userId":"$undefined","isVerified":false,"isHubEditor":false,"editorOfHubs":[],"raw":{"id":993107,"first_name":"Dominikus","last_name":"Brian","profile_image":"https://storage.prod.researchhub.com/uploads/author_profile_images/2025/03/04/blob_pxj9rsH"}},"balance":0,"lockedBalance":0,"promotionalBalance":0,"fundingCredits":0,"balances":"$undefined","totalRsc":0,"hasCompletedOnboarding":false,"createdDate":"$undefined","moderator":false,"editorOfHubs":"$undefined","isModerator":false,"isFunder":false,"referralCode":"$undefined","authProvider":"$undefined","isStakingOptedIn":false,"balanceHistory":0,"raw":{"id":60018,"author_profile":"$4f:props:children:props:topBanner:props:metadata:bounties:3:createdBy:authorProfile:raw"}},"solutions":[],"contributions":[],"totalAmount":"390","comment":"$undefined","raw":{"id":3952,"created_by":"$4f:props:children:props:topBanner:props:metadata:bounties:3:createdBy:raw","amount":"390.0000000000","status":"EXPIRED"}}],"activeBounties":0,"closedBounties":0}}],"rightSidebar":"$L8f","children":"$L90"}]}]

Drafts

ARTEMIS - Automated Review and Trustworthy Evaluation for Manuscripts in Science

Drafts

Drafts

ARTEMIS - Automated Review and Trustworthy Evaluation for Manuscripts in Science