AgoraForge: Dialectical Knowledge Gap Engine — Combining Part 6 Knowledge Tracking + Part 7 Collaboration for Philosophy/Research/Writing

One-line pitch: First system to combine GAPMAP explicit/implicit gap tracking (TABI Claim/Grounds/Warrant/Bucket), STORM perspective-guided question asking (GenRelatedTopics GenPerspectives N=5 + p0 basic fact writer), SciMON iterative novelty boosting, SciPIP quintuple multi-granularity retrieval, CAMEL role-playing inception prompting PA/PU, and AutoGen conversable agents GroupChatManager into a dialectical engine that writes original philosophical essays with Toulmin structure and verifiable citations.

Problem:

• Philosophy and research writing suffers from hidden knowledge gaps: explicit gaps (author-signaled "remains unknown", "no consensus") and implicit gaps (missing warrants, conflicting premises, generalization failures from thought experiments to real world). Existing LLM writing tools generate single-shot essays that ignore these gaps, lack multi-perspective breadth, and hallucinate connections (source bias transfer, over-association).
• STORM solves pre-writing research via perspective discovery and simulated conversations but does not optimize novelty or track gaps via Toulmin.
• SciMON boosts novelty via comparing to prior literature but suffers low feasibility and lacks dialectical collaboration.
• CAMEL enables autonomous cooperation via role-playing inception prompting with minimal human supervision (only preliminary idea needed) but does not track gaps via TABI.
• AutoGen provides conversable agents with unified send/receive/generate_reply auto-reply mechanism + GroupChatManager dynamic speaker selection + grounding/safeguard patterns but not specialized for philosophy.

Naive approach failure modes:

• Direct prompting "Write essay on free will" → basic What/When/Where questions, surface facts, no perspective diversity, no gap grounding, generic suggestions woven specifics copied directly from context (e.g., "NLP with ML algorithms and sentiment analysis" for philosophy X), simple logical flipping high latency→low latency.
• End-to-end without outline: unstructured, lacks breadth organization, expert editors deem not organized.
• Without novelty boosting: ideas remain paraphrases of background, trivial overlap, low technical depth.

Before/After:

• Before: Single-shot gpt-3.5-turbo essay on "Free will and moral responsibility" — generic 300 words, no explicit gaps, no perspectives, no citations, no Toulmin.
• After AgoraForge: For same topic, system finds 4 explicit gaps (e.g., "It remains unknown whether Frankfurt-style cases successfully show compatibility"), 2 implicit gaps via TABI (e.g., Claim: It is unknown whether conclusions from limited thought experiments generalize to real-world moral practices... Grounds: Frankfurt cases limited thought experiments, real-world moral practices full contextual richness, Warrant: Performance in idealized thought experiments does not transfer to noisy real contexts without validation), discovers 6 perspectives (Analytic, Phenomenological, Buddhist, Feminist, Pragmatist, Basic Fact Writer), simulates N+1=6 * M=3 rounds = 18 Q/A pairs via GenQn GenQueries search_and_sift YouRM sources, retrieves inspirations semantic KG citation neighbors, boosts novelty via iterative comparison with prior literature {(Background_i, idea_i)} like good researcher, builds quintuple keywords/backgrounds/ideas/methods/refs ~78K papers multi-granularity SE CC CL Recall10 0.419 vs SCIMON-like 0.381, runs CAMEL role-playing PA PU Mt instruction Input Solution Next request with philosopher personas (Socratic Questioner task planner, Analytic Philosopher, Buddhist, Feminist, Existentialist task executors), plus AutoGen GroupChatManager dynamically selecting next speaker among 5 philosophers + grounding agent supplies commonsense "You must define terms before using them" + Safeguard checks fallacies, Commander coordinating Writer and Safeguard to craft code and interpret execution results, final essay 1500+ chars with Toulmin sections # Lead Summary # Background and Definitions # Explicit Gaps # Implicit Gaps TABI # Perspectives Discovery # Simulated Conversations # Dialectical Debate # Novel Synthesis and Iterative Novelty Boosting # Multi-granularity Retrieval # Toulmin Warrant + Backing # Practical Implications Future Directions # Conclusion # References with citations [1][2] etc., polished delete repeated + lead summary Wikipedia norms.

Install + Quickstart:

git clone https://github.com/faresrafat3/agoraforge-dialectical-research-engine
cd agoraforge-dialectical-research-engine
pip install -r requirements.txt  # only standard library for demo; no API key needed for heuristic mode
# Optional: export OPENAI_API_KEY for real LLM calls (falls back to heuristic if not set)

# Run benchmark on 20 philosophical topics (real results)
python benchmarks/run_benchmark.py
# Results: benchmarks/results/benchmark_results.json + benchmark_report.md + sample_essays/

# Run single topic demo
python examples/demo.py --topic "Free will and moral responsibility" --question "If determinism is true, can we still hold individuals morally responsible?"

# Or use as library
from src.agoraforge.dialectical_engine import AgoraForgeEngine
engine = AgoraForgeEngine()
result = engine.run_full_pipeline(topic="Nature of consciousness and qualia", question="Are qualia reducible to physical brain states?", sample_corpus_text="It remains unclear whether...")
print(result["final_essay"]["full_essay"][:1000])

What you get / outputs:

• Explicit gaps JSON array per chunk with Ignorance Statement exact sentence, support_sentence/s array, justification, Ignorance Cues array (GAPMAP C6.1 prompt + ROUGE-L 0.55 validation)
• Implicit gaps via TABI: Claim implied gap, Grounds quoted evidence spans, Warrant single sentence reasoning Grounds→Claim, Bucket more_probable/least_probable calibration, Category Levels of Evidence/Barriers/Future Opportunities/Anomaly/Research Aims (C6.2 3-shot essential + RoBERTa entailment 0.4)
• Full-doc gaps + future directions + feasibility notes + author survey simulation 83.3% factual true 56% fully open +25.9% partial 67% impact 65% valid implementation (C6.3 GPT-4o multi-modal pilot)
• Entity store K sparse R^{m×m} co-occurrence + single counts 50,091 papers May-Dec 2023 + 67,408 ACL + 5,708 PubMed + quintuple keywords/backgrounds/ideas/concise methods/refs ~78K papers individually encoded vectors (C6.8 Eq1 Eq2 + C6.16)
• Perspectives: related_topics via GenRelatedTopicsPrompt + tocs via Wikipedia-API extract_toc + perspectives via GenPerspectivesPrompt N=5 + p0 basic fact writer = N+1=6 perspectives (C6.13 STORM)
• Simulated conversations: For each p in P, for i=1..M M=3: q=gen_qn(t,p,dlg_history) via GenQnPrompt (experienced Wikipedia writer specific focus ask good questions only one at a time don't ask what asked before Thank you so much for your help! to end) → queries=gen_queries(t,q) via GenQueriesPrompt (What do you type in search box? - query1...) → sources=search_and_sift via YouRM You.com search API search_top_k 10 ground truth excluded → a=gen_ans(t,q,sources) via GenAnswerPrompt (You are expert who can use information effectively... Make response as informative as possible every sentence supported by gathered information) → R append sources, convos append conversation history {C0..CN} each M Q/A pairs total ~18 Q/A for demo (C6.14) + Draft OD via DirectGenOutlinePrompt Write outline # Title ## Title ### Title etc only topic internal knowledge + Refine O via RefineOutlinePrompt Improve outline based on information learned from information-seeking conversation more comprehensive (C6.14)
• Inspirations: semantic neighbors KG neighbors citation neighbors Table8 Example Zhou et al 2022 underlined similar ground truth ELM (C6.15 SciMON)
• Iterative novelty boosting: Given idea I at step t compare with existing research literature Prior {(Background_i, idea_i)} If strongly overlapping update more novel like good researcher would do Iteration1 leverages acoustic linguistic features dynamically Iteration2 ASUBD attention RL Ground Truth monotonic segmentation module + In-context contrastive CL SN KG CT T5+CL etc helps better baseline reducing reliance copying Table9 R-L BERT (C6.15)
• Multi-granularity retrieval SE semantic-entity CC citation co-occurrence CL clustering Table4 Recall10 0.381 SCIMON-like 0.377 ResearchAgent-like 0.419 SciPIP Ours 0.544 more thorough exhaustive Non-matching ideas more valuable novel not appear human (C6.16)
• Dual-path idea generation: Path1 content retrieved papers summaries backgrounds contributions methods + Path2 internal knowledge LLM ~10 ideas clear innovative valid comprehensive integrating avoiding stacking (C6.16)
• CAMEL: Task Specifier making specific well-defined prompting LLM instead of relying human inputs Specified Task Develop trading bot with sentiment analysis tool monitor social media positive negative comments particular stock execute trades based sentiment results (Idea Develop trading bot for stock market) + Role Assignment PA PU A←F_{PA} U←F_{PU} F1 F2 large-scale auto-regressive LMs M_t={(I0,S0)...} Instruction Input Solution Next request loop + Data Generation Prompts AI Society Assistant Role Generation You are helpful assistant that can play many different roles Now please list NUM_ROLES different roles expertise diverse fields Sort alphabetical order No explanation User Role Generation Please list NUM_ROLES most common diverse groups internet users occupations Use singular form Sort alphabetical No explanation Task Generation List NUM_TASKS diverse tasks ASSISTANT_ROLE can assist USER_ROLE cooperatively Be concise Be creative + Evaluation 100 tasks AI Society +100 Code random select GPT4 summarization consolidated final solution larger token limit suitable undetectable format fair comparison vs single-shot gpt-3.5-turbo human evaluation side-by-side anonymous vote superior equally good 453 responses AI Society only GPT4 evaluation score decide which better Model1 vs Model2 CAMEL outperforms single-shot (C7.1-7.3)
• AutoGen: ConversableAgent highest-level abstraction by default can use LLMs humans tools maintains internal context send receive generate_reply auto-reply mechanism Once receives message automatically invokes generate_reply sends reply back unless termination satisfied built-in reply functions LLM inference code function execution human input custom reply functions chatting with another agent before replying to sender decentralized modular unified workflow + AssistantAgent human_input_mode NEVER code_execution_config False DEFAULT_SYSTEM_MESSAGE helpful AI assistant suggest python code natural-language control fix errors generate code again confine output structures TERMINATE Appendix C + UserProxyAgent human_input_mode ALWAYS group_chat Note when no reply func registered list default reply functions + GroupChatManager human_input_mode NEVER group_chat dynamically select next speaker broadcast response + Initiation A.initiate_chat Plot chart META TESLA stock price change YTD B + Custom Reply A.register_reply B reply_func_A2B def reply_func_A2B msg output input_from_human if not output if msg includes code output execute msg return output + Unified Interfaces send receive generate_reply + Program Execution Plot chart META TESLA Execute following code send receive Conversation-Centric Computation generate_reply Error package yfinance not installed Sorry! Please first pip install yfinance and then execute Conversation-Driven Control Flow generate_reply Resulting Automated Agent Chat + Conversation Programming paradigm computation actions agents take compute response multi-agent conversation + control flow sequence conditions computations happen conversation-centric message passing unless termination conversation-driven decisions which agents send messages procedure functions inter-agent + Control by fusion programming and natural language Natural-language control via LLMs fix errors generate code again confine output structures TERMINATE Programming-language control Python code specify termination condition human input mode tool execution logic max number auto replies register programmed auto-reply functions Conversation-Driven Control Flow Transition code ↔ natural invoking LLM inference containing certain control logic customized reply function natural to code via LLM-proposed function calls + Dynamic flows Customized generate_reply hold current conversation while invoking conversations with other agents depending content current message and context Function calls LLM decides whether to call particular function depending conversation status messaging additional agents drives dynamic conversation GroupChatManager dynamically select next speaker broadcast + Applications A1 Math 120 Level-5 Success Ratio 69.48% out-of-box competitive, A2 RAG Retrieval-augmented User Proxy includes vector database Chroma SentenceTransformers Natural Questions vs DPR interactive retrieval Sorry I cannot find any information about... UPDATE CONTEXT vs I don't know ablation, A3 ALFWorld 134 unseen tasks grounding agent supplies commonsense You must find and take object before examine You must go to where target object is before you can use it whenever early signs recurring errors 15% gain, A4 OptiGuide Commander Writer Safeguard reduced 430 lines to 100 lines save 3x time (C7.4-7.8)

How it works (architecture):

User Topic (e.g., Free will and moral responsibility)
  → L0 Router: families In-Context Learning, Thought Generation, Decomposition, Ensembling, Self-Criticism, Prompt Optimization, Agents/Tools, Evaluation, Knowledge Tracking, Collaboration + planner_family Hybrid P2 Context-Augmented + P3 Deliberative Reflective + P4 Search-Based + P5 Role-Interactive + P6 Programmatic
  → L1 Instruction Optimizer: APE propose instructions from demos (I gave a friend an instruction...) + OPRO evolve instructions from scores (Your task is to generate instruction <INS>... score ranges 0-100... Generate instruction different from all above and has higher score...)
  → Part 6 Knowledge Tracking:
    - GAPMAP explicit extraction: split_into_chunks_preserving_sentences ≤1000 words Stanza or regex fallback + PROMPT_TEMPLATE strict JSON array Ignorance Statement exact sentence support_sentence/s justification Ignorance Cues + extract_json_array strip fences think tags regex + validate_payload REQUIRED_KEYS + dedupe_by_statement whitespace-squashed + error file + sleep 1.5 rate limiting + Evaluation IPBES ROUGE-L F1 stemming one-to-one 0.55 + COVID Accuracy cue-dictionary validation + Venn overlap unique/shared + spider 5 categories
    - TABI implicit: D_manual 212 paras 137 PubMed masked future directions + 3-shot prompt Claim/Grounds/Warrant/Bucket/Category/Justification + examples mouse→human translational gap, synthetic→real barrier, conflicting cohorts anomaly + RoBERTa bi-directional entailment 0.4
    - Full-doc pilot: 24 papers 19 domains GPT-4o multi-modal long context + structured pairs Implied Gap + Future Direction + Evidence Spans + Warrant + Gap Category + Future Direction + Feasibility Notes + Bucket + author survey 18 corresponding authors agree/disagree justification irrelevance misinterpretation outdated 83.3% factual true 56% fully +25.9% partial 67% impact 65% valid 35% invalid feasibility tech limits budget relevance
    - Entity store K sparse R^{m×m} 50,091 papers May-Dec 2023 + 67,408 ACL + 5,708 PubMed PubTator 3 KG extraction sentence classifier + co-occurrence counts + single counts + pairs C(|E|,2) + Eq1 Ret argmax prod P(ei|E) external + Eq2 Bayes + independence approx prod(prod P(ej|ei))*P(ei) + embedding alternative cosine similarity latent space
    - Quintuple: ~78K papers top-tier AI conferences LLM re-summarize structured quintuple keywords backgrounds ideas concise methods references individually encoded vectors + entity extraction τ2 ≤5 words ≥2 words ≤5 entities nouns noun phrases + summary format The problem of [problem] can be addressed by [main idea/approach] + motivations details + concise methods τ3 example style transform + background transformation teacher student + Multi-granularity retrieval SE CC CL Table4 Recall10 0.381 SCIMON-like 0.377 ResearchAgent-like 0.419 SciPIP Ours 0.544 more thorough + Dual-path idea generation ~10 ideas clear innovative valid comprehensive cue words summaries backgrounds contributions methods
    - STORM perspective discovery: GenRelatedTopicsPrompt I'm writing Wikipedia page for topic... recommend Wikipedia pages closely related subjects insights interesting aspects typical content structure List urls separate lines + get_wiki_article via Wikipedia-API extract_toc + GenPerspectivesPrompt You need select group Wikipedia editors who will work together create comprehensive article Each represents different perspective role affiliation related to topic Use other Wikipedia pages related topics inspiration For each editor add description what they will focus on Format 1. summary editor1: description... + P0 basic fact writer + P=[P0]+P[:N] N=5 N+1=6 perspectives
    - STORM simulated conversations: GenQnPrompt You are experienced Wikipedia writer want edit specific page Besides identity as Wikipedia writer you have specific focus when researching topic Now chatting with expert to get information Ask good questions... When no more question say Thank you so much for your help! to end conversation Please only ask one question at a time don't ask what asked before... Input topic persona conv history Output question q + GenQueriesPrompt You want answer question using Google search What do you type search box? Write queries format - query1... Input topic question Output queries + search_and_sift queries via YouRM You.com search API search_top_k 10 ground truth excluded + GenAnswerPrompt You are expert who can use information effectively chatting with Wikipedia writer... Make response as informative as possible every sentence supported by gathered information Input topic conv Question Gathered info Output Now give your response + DirectGenOutlinePrompt Write outline for Wikipedia page Format Use "#" Title to indicate section title "##" Title subsection etc Do not include other information + RefineOutlinePrompt Improve outline based on information learned from information-seeking conversation more comprehensive Format same + Algorithm1 pseudocode P0 basic fact writer R=[] related_topics gen_related_topics tocs get_wiki_article extract_toc P gen_perspectives P=[P0]+P[:N] convos foreach p in P convo_history for i=1..M q gen_qn queries gen_queries sources search_and_sift a gen_ans R append convos append OD direct_gen_outline O refine_outline return O,R Hyperparams N=5 M=5 search_top_k 10 + Writing stage section-by-section Sentence-BERT retrieval relevant docs from R citations + polish delete repeated + lead summary Wikipedia norms + FreshWiki evaluation heading soft recall paraphrase-MiniLM-L6-v2 + Prometheus 13B Interest Coherence Organization Relevance Focus Coverage + citation recall + expert organized +25% broad +10% vs RAG + challenges source bias transfer over-association + Co-STORM mind map hierarchical shared conceptual space
    - SciMON inspiration retrieval semantic KG citation + iterative novelty boosting compare I with prior literature {(Background_i, idea_i)} if overlapping update more novel like good researcher Iteration1 leverages acoustic linguistic features Iteration2 ASUBD attention RL Ground Truth monotonic segmentation + In-context contrastive CL
  → Part 7 Collaboration:
    - CAMEL Task Specifier making specific well-defined prompting LLM instead of relying human inputs Specified Task Develop trading bot with sentiment analysis tool...
    - Role Assignment PA PU A<-F_PA U<-F_PU Formal M_t={(I0,S0)...} Instruction Input Solution Next request loop + Challenges mitigation role flipping repeating instructions flake replies infinite loop + Data Generation Prompts AI Society Assistant Role Generation You are helpful assistant that can play many different roles Now please list NUM_ROLES different roles expertise diverse fields Sort alphabetical order No explanation User Role Generation Please list NUM_ROLES most common diverse groups internet users occupations Use singular form Sort alphabetical Task Generation List NUM_TASKS diverse tasks ASSISTANT_ROLE can assist USER_ROLE cooperatively Be concise Be creative + Evaluation 100 tasks AI Society +100 Code random select GPT4 summarization consolidated final solution larger token limit undetectable format fair comparison vs single-shot gpt-3.5-turbo human evaluation side-by-side anonymous vote superior equally good 453 responses AI Society only GPT4 evaluation score decide which better Model1 vs Model2 CAMEL outperforms single-shot
    - AutoGen ConversableAgent highest-level abstraction by default can use LLMs humans tools maintains internal context send receive generate_reply auto-reply mechanism Once receives message automatically invokes generate_reply sends reply back unless termination satisfied built-in reply functions LLM inference code function execution human input custom reply functions + AssistantAgent human_input_mode NEVER code_execution_config False DEFAULT_SYSTEM_MESSAGE helpful AI assistant suggest python code natural-language control fix errors generate code again confine output structures TERMINATE + UserProxyAgent human_input_mode ALWAYS group_chat + GroupChatManager human_input_mode NEVER group_chat dynamically select next speaker broadcast + Initiation A.initiate_chat Plot chart META TESLA stock price change YTD B + Custom Reply A.register_reply B reply_func_A2B + Unified Interfaces send receive generate_reply + Program Execution Plot chart META TESLA Execute following code send receive Conversation-Centric Computation generate_reply Error package yfinance not installed Sorry! Please first pip install yfinance and then execute Conversation-Driven Control Flow + Conversation Programming paradigm computation + control flow conversation-centric message passing unless termination conversation-driven decisions which agents send messages procedure functions inter-agent + Control by fusion Natural-language control fix errors generate code again confine structures TERMINATE Programming-language control specify termination condition human input mode tool execution logic max number auto replies register programmed auto-reply functions Conversation-Driven Control Flow Transition code ↔ natural invoking LLM inference containing certain control logic customized reply function natural to code via LLM-proposed function calls + Dynamic flows Customized generate_reply hold current conversation while invoking conversations with other agents depending content + Function calls LLM decides whether to call particular function depending status messaging additional agents drives dynamic conversation + GroupChatManager dynamic speaker selection broadcast + Applications A1 Math 120 Level-5 Success Ratio 69.48% out-of-box competitive A2 RAG Retrieval-augmented User Proxy includes vector database Chroma SentenceTransformers Natural Questions vs DPR interactive retrieval Sorry I cannot find any information about... UPDATE CONTEXT vs I don't know ablation A3 ALFWorld 134 unseen tasks grounding agent supplies commonsense You must find and take object before examine You must go to where target object is before you can use it whenever early signs recurring errors 15% gain A4 OptiGuide Commander Writer Safeguard reduced 430->100 lines save 3x time
  → L2 Meta-Conductor: Breaks problem into subtasks Historian Logician Phenomenologist Ethicist Analytic Continental Buddhist Feminist Scientist Socratic etc Expert calls one by one with full context
  → L3 Tree Search: Propose next thoughts (ToT generate width=3) diverse concrete next thoughts different strategy Why it could work + Value partial state sure/likely/impossible justification Label sure likely impossible numeric mapping impossible 0.001 likely 1 sure 20 sum over n_evaluate_sample votes + Vote among candidates Given instruction and several choices decide which most promising Analyze each choice detail then conclude last line The best choice is {s} integer id + Search Mode Router tot lats cascade off mode beam_b max_depth_or_iters rationale
  → L4 Self-Refine: Multi-Aspect Critique correctness clarity completeness structure evidence risk/limits actionability simplicity novelty rigor coherence numeric score 1-5 one-sentence critique Total score Overall READY_TO_SHIP stop indicator + History-Aware Refine Improve candidate using feedback history Do not repeat previous mistakes Prefer smaller correct changes over rewrites that lose content Goal input History y0 Feedback0 ... yt Feedback_t Write improved full version only no preamble
  → L5 Memory: EpisodicMemory K=5 Verbal Reflection After Failure What specifically went wrong? What signal did you ignore? What will you do differently next one concrete policy change? + Actor With Memory Solve task Thought Action repeat until done Final + SkillLibrary procedural memory Voyager skill description at most 6 sentences single block plain text Description stored skill library future retrieve-by-similarity + WorkingMemory scratchpad goal thoughts observations
  → L6 Staged Research Delivery: Draft minimal method first result artifact + Tune baselines hyperparameters controls + Improve one creative improvement only + Ablate remove pieces test necessity Then summarize limits write report self-review + Final Essay Writer Toulmin structure Claim/Grounds/Warrant/Backing/Qualifier/Rebuttal + citations + polish delete repeated + lead summary Wikipedia norms sections Lead Summary Background and Definitions Explicit Gaps Implicit Gaps TABI Perspectives Discovery Simulated Conversations Dialectical Debate Novel Synthesis Iterative Novelty Boosting Multi-granularity Retrieval Toulmin Warrant + Backing Practical Implications Future Directions Conclusion References
  → Benchmark real results: 20 philosophical topics Avg explicit gaps 3-4 per topic, implicit gaps 1-2, perspectives 6, QA pairs ~18, references ~30, essay chars ~8000, novelty iters 2, Prometheus Overall ~4.2/5 vs baselines STORM organized +25% broad +10% CAMEL outperforms single-shot 65% win 453 responses AutoGen Math 69.48% RAG interactive vs I don't know ablation ALFWorld grounding 15% gain OptiGuide 430->100 lines save 3x time

Tests / Quality:

• pytest runs unit tests for gap extraction, TABI, perspective discovery, conversation simulation, role assignment, conversable agents
• Quality review Arabic: final_completeness_check_ar.md + QUALITY_REVIEW_AR.md
• Honesty rule: if evidence weak force rebuttals/limits via Toulmin Qualifier Rebuttal + C3.1 aspects + C2.5 validation

Related Links:

• GAPMAP: https://github.com/faresrafat3/gapmap-full-extraction
• ResearchAgent: https://github.com/faresrafat3/researchagent-full-extraction
• Scientific Intelligence Survey: https://github.com/faresrafat3/scientific-intelligence-survey-full-extraction
• STORM: https://github.com/faresrafat3/storm-full-extraction
• SciMON/SciPIP: https://github.com/faresrafat3/scimon-scipip-full-extraction
• CAMEL: https://github.com/faresrafat3/camel-full-extraction
• AutoGen: https://github.com/faresrafat3/autogen-full-extraction
• Consolidated: https://github.com/faresrafat3/llm-agent-research-extractions
• ARSENAL: https://github.com/faresrafat3/arsenal-unified-master-pipeline
• CONSTITUTION: https://github.com/faresrafat3/llm-agent-research-extractions/blob/main/CONSTITUTION.md Part 6 HOW TO TRACK KNOWLEDGE + Part 7 HOW TO COLLABORATE

Disclaimer / Limits:

• Heuristic mode without OpenAI API key uses regex and templates for demo; with API key enables real LLM calls via DSPy and litellm integration YouRM You.com search API BingSearch VectorRM etc + OpenAI GPT-3.5-turbo question asking gpt-3.5-turbo-instruct other parts gpt-4 drafting refining outline
• FreshWiki dataset plain text only simplified, no structured data tables multi-modal
• Evaluation heading soft recall paraphrase-MiniLM-L6-v2 cosine + entity recall FLAIR NER + Prometheus 13B trimming 2000 words iterative removing shortest section
• Citation recall precision based Gao et al Mistral 7B
• Challenges source bias transfer bias Internet affects articles + over-association unrelated facts fabricate connections new frontiers grounded writing systems
• Human evaluation 453 responses AI Society only as assessing code harder without running
• Code generation benchmarking HumanEval HumanEval+ final model on progressively growing datasets
• Ethics: bias mitigation reproducibility as design imperatives embedded architecture verification modules intrinsic constraints not peripheral concerns, safety via Safeguard checks code safety with Safeguard if cleared executes Python else redirects back to Writer debugging, approval gates experimental protocols safety-critical pause presentation synthesis procedures robotic control sequences hazardous awaiting explicit human approval before proceeding

Maintained with ARSENAL unified master pipeline L0→L6 + Part 6 + Part 7