Abstract :

The Advent Of Big Data Has Transformed The Way Businesses And Organizations Collect, Store, Process, And Derive Insights From Massive Volumes Of Data, And Building Such Applications Requires Specialized Development Frameworks Able To Handle The Scale, Variety, And Complexity Of Data Generated From Diverse Sources. Traditional Data-processing Tools Are Not Equipped For These Workloads, And The Primary Challenges—data Ingestion, Storage, Processing, And Analysis, Together With Real-time Streaming—necessitate Purpose-built Technology Stacks. This Paper Examines The Key Considerations In Selecting A Development Framework For Big Data Applications. We First Present The Organizational Big Data Framework, A Vendor-independent Structure Of Six Capabilities—strategy, Architecture, Algorithms, Processes, Functions, And Artificial Intelligence—that An Enterprise Must Develop To Embed A Successful Big Data Practice. We Then Survey The Dominant Open-source Processing Frameworks Along The Batch/stream Axis, Covering Hadoop MapReduce, Apache Spark, Flink, Storm, And Samza, And Consolidate Recent Comparative Evidence On Their Performance, Cost, Ease Of Use, Fault Tolerance, And Suitability For Machinelearning And Real-time Analytics Workloads. Reported Benchmark Studies Show That Engine Choice, Data-storage Format, And Workload Type Interact Strongly, So No Single Framework Dominates All Conditions. Finally, We Map Framework Capabilities To Application Classes—reliable Platform Engineering, Governance, Interactive Exploration Of Unstructured Data, And Domain Deployments In Health, Insurance, And Environmental Prediction—and Identify Open Challenges In Throughput, Storage Efficiency, And Streaming Program Development.

Published:

20-7-2023

Issue:

Vol. 23 No. 7 (2023)

Page Nos:

385-391

Section:

Articles

License:

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

How to Cite