Beyond Single-Framework Architectures: A Systematic Evaluation and Hybrid Design for Scalable Multi-Agent Coordination

• Published in: IEEE Access
• Main Authors: Muhammad Saleem Khan, Muhammad Janas Khan, Muhammad Sharif, Sadaf Yasmin
• Format: Article
• Language: English
• Published: IEEE 2026-01-01
• Subjects: Multi-agent systems; large language models; agentic AI; framework comparison; hybrid architectures; scalability
• Online Access: https://ieeexplore.ieee.org/document/11481053/

Large language model (LLM)-based multi-agent frameworks have emerged as powerful tools for complex coordination tasks, yet systematic comparisons across realistic, large-scale scenarios remain limited. This paper presents a comprehensive evaluation of three prominent frameworks, i.e., CrewAI, AutoGen, and LangChain, alongside a novel hybrid architecture that strategically combines their complementary strengths. Using CREW-WILDFIRE, a realistic wildfire response benchmark featuring heterogeneous agents and partial observability, we evaluated the framework&#x2019;s performance across 17 task levels and 7 behavioral competencies. Our results reveal distinct architectural trade-offs: CrewAI achieves 34% higher task designation scores through hierarchical coordination but lacks adaptability; AutoGen demonstrates 28% superior plan adaptation via conversational feedback, yet incurs high token costs; LangChain offers maximum flexibility at the expense of a 42% increase in computational overhead. Based on these insights, we propose and validate a hybrid LangGraph-CrewAI architecture that achieves a 96.1% success rate while reducing token consumption by 76.2% and decision latency by <inline-formula> <tex-math notation="LaTeX">$14.5\times $ </tex-math></inline-formula> compared to pure CrewAI implementations. Our comprehensive validation across 51 episodes (17 scenarios <inline-formula> <tex-math notation="LaTeX">$\times 3$ </tex-math></inline-formula> seeds) demonstrates perfect scores in task designation and agent capitalization competencies, outperforming state-of-the-art baselines. This work provides evidence-based guidance for framework selection, highlights fundamental scalability challenges in current systems, and demonstrates that intelligent hybrid designs can overcome individual framework limitations while maintaining cost-efficiency.