Real-time data streaming systems increasingly operate under highly variable workloads, heterogeneous data sources, latency constraints, and frequent service disruptions. Conventional stream-processing architectures generally depend on predefined routing, static resource allocation, and centralized coordination, which can limit their ability to adapt when event rates, computational requirements, or infrastructure conditions change rapidly. This paper proposes an Adaptive Multi-Agent AI Framework for Real-Time Data Streaming with Enhanced Scalability and Resilience, in which autonomous AI agents collaboratively perform stream monitoring, workload classification, task allocation, resource adaptation, anomaly detection, and recovery. The theoretical foundation combines multi-agent coordination with contextual representation, long-document processing, memory management, and adaptive decision-making. Prior work on aspect-controllable summarization demonstrates the value of controlling computational objectives according to task requirements, while studies of coreference, lexical chains, and entity-based coherence emphasize the importance of preserving relationships across distributed information units (Amplayo, Angelidis, & Lapata, 2021; Baldwin & Morton, 1998; Barzilay & Elhadad, 1997; Barzilay & Lapata, 2005). Long-context language modeling further motivates mechanisms capable of retaining relevant information over extended streaming windows (Beltagy, Peters, & Cohan, 2020). The proposed framework extends these principles to adaptive streaming environments and aligns with recent multi-agent event-streaming research emphasizing resiliency and scalability (Reddy et al., 2026). Analytical findings indicate that decentralized agent specialization, shared contextual state, adaptive workload redistribution, and failure-aware coordination can provide a stronger basis for resilient streaming than static pipelines. The paper also identifies trade-offs involving coordination overhead, state consistency, model complexity, and resource consumption.
Adaptive Multi-Agent AI Framework for Real-Time Data Streaming with Enhanced Scalability and Resilience
DOI:
Abstract
References
Amplayo, R. K., Angelidis, S., & Lapata, M. (2021). Aspect-Controllable Opinion Sum-marization. InProceedings of the 2021 Conference on Empirical Methods in NaturalLanguage Processing(pp. 6578–6593). Online and Punta Cana, Dominican Republic:Association for Computational Linguistics.
Baldwin, B., & Morton, T. S. (1998). Dynamic coreference-based summarization. InProceedings of the Third Conference on Empirical Methods for Natural Language Pro-cessing(pp. 1–6). Granada, Spain: Association for Computational Linguistics.
Barzilay, R., & Elhadad, M. (1997). Using Lexical Chains for Text Summarization. InIntelligent Scalable Text Summarization.
Barzilay, R., & Lapata, M. (2005). Modeling Local Coherence: An Entity-Based Approach.InProceedings of the 43rd Annual Meeting of the Association for Computational Lin-guistics (ACL’05)(pp. 141–148). Michigan, USA: Association for ComputationalLinguistics.
Beltagy, I., Lo, K., & Cohan, A. (2019). SciBERT: A Pretrained Language Model for Sci-entific Text. InProceedings of the 2019 Conference on Empirical Methods in NaturalLanguage Processing and the 9th International Joint Conference on Natural LanguageProcessing (EMNLP-IJCNLP)(pp. 3615–3620). Hong Kong, China: Association forComputational Linguistics.
Beltagy, I., Peters, M. E., & Cohan, A. (2020). Longformer: The Long-Document Trans-former.ArXiv preprint,abs/2004.05150.
Bengio, Y. (2017). The consciousness prior.ArXiv preprint,abs/1709.08568.
Berman, M. G., Jonides, J., & Lewis, R. L. (2009). In search of decay in verbal short-termmemory.Journal of Experimental Psychology: Learning, Memory, and Cognition,35(2), 317.
R. Reddy, P. Udayaraju, K. K. Goyal, S. C. R. Vudem, R. Sayana and V. Gummadi, "Creating an AI-based Multi-Agent Model for Optimized Data Streaming with Improved Resiliency and Scalability for Event Streaming," 2026 6th International Conference on Image Processing and Capsule Networks (ICIPCN), Dhulikhel, Nepal, 2026, pp. 1372-1379, doi: 10.1109/ICIPCN67432.2026.11438445.
R. Reddy, P. Udayaraju, K. K. Goyal, S. C. R. Vudem, R. Sayana and V. Gummadi, "Creating an AI-based Multi-Agent Model for Optimized Data Streaming with Improved Resiliency and Scalability for Event Streaming," 2026 6th International Conference on Image Processing and Capsule Networks (ICIPCN), Dhulikhel, Nepal, 2026, pp. 1372-1379, doi: 10.1109/ICIPCN67432.2026.11438445.
K. Ramamurthy, N. Bellamkonda and N. Amanmadov, "ScalePulse: Combinatorial Llm Framework for Scalability Constraint," SoutheastCon 2026, Huntsville, AL, USA, 2026, pp. 1-6, doi: 10.1109/SoutheastCon63549.2026.11476075
Geo Philip, Paulson, Artificial Intelligence and Machine Learning Applications in Project Schedule Forecasting: A Predictive Framework for Time-Control in Complex Building Projects (April 16, 2026). Available at SSRN: https://ssrn.com/abstract=6588119 or http://dx.doi.org/10.2139/ssrn.6588119
Vollem, S., Mulla, F. M., Shah, A. K., Kodela, S., Kaur, M., & Kumar, V. (2026, February). Multi-Model Time-Series Forecasting Framework for Stock Price Prediction Using Statistical and Deep Learning Techniques. In 2026 2nd International Conference on Big Data & Machine Learning (ICBDML) (pp. 1-6). IEEE.