Olga Segou, INTRASOFT International
9 November 2020
Over the past years, the increased digitization and proliferation of digital devices and sensors has led to an explosive growth of data in almost every industry and area of human activity. At the same time, high-performance computing facilities have become more accessible, due to factors like the cloud adoption and Moore’s Law. This has led to the development of Big Data  as a field that refers to the technologies used to handle and systematically analyse extremely large data sets, to reveal patterns, trends and insights that might be otherwise unobtainable. The difference with traditional data approaches hinges on the large scale of basic data characteristics, also known as “the 5 Vs” namely the Volume, Velocity, Variety, Veracity and Value. A large variety of technologies and heterogeneous architectures have since been applied in the implementation of Big Data use cases. Reports illustrating the value of Big Data approaches have also been published by major corporations such as LinkedIn , Netflix  and others, focusing on the use of open source solutions used as the basis for complex recommendation systems, mining association rules and other uses.
Within the Pledger project, Intrasoft International SA is adapting and deploying its Streamhandler Platform, which provides the hooks for interconnecting, storing, transforming and processing big data, with the additional capacity to integrate and train machine learning and Deep Learning algorithms, resulting to a fully-fledged and AI-ready Big Data solution. Streamhandler is a high-performance distributed streaming platform for handling real-time data based on Apache Kafka with demonstrated low latency and high throughput. It can efficiently ingest and handle massive amounts of data into processing pipelines, for both real-time and batch processing. The platform and its underlying technologies can support any type of data-intensive IT services (Artificial Intelligence, Business Intelligence, etc.) from cloud to edge.
The key capabilities and features offered by the platform are:
- Real-time monitoring and event-processing
- Interoperability with all modern data storage technologies and popular data sources
- Distributed messaging system
- High fault-tolerance - Resiliency to node failures and support of automatic recovery
- Elasticity - High scalability
- Security (encryption, authentication, authorization)
The relevance of this technology, both for the project and for edge deployments globally, is further highlighted by its fast growth. According to MarketsAndMarkets , the Big Data streaming analytics market size is expected to grow from USD 10.3 billion in 2019 to USD 35.5 billion by 2024, at a Compound Annual Growth Rate (CAGR) of 28.2% during this period. Driven by the emergence of Edge Computing and the increasing IoT, smartphone and internet penetration, this expected increase in market size uncovers a strategic shift toward real-time provisioning and analysis of data for faster decision making. Additional sources further support the same conclusions. For example, Mordor Intelligence estimates market size at USD 7.08 billion in 2019 and expects to reach a value of USD 38.53 billion by 2025 at a CAGR of 32.67%, during the forecast period of 2020-2025. Digitization and cloud Computing are considered vital catalysts in the Big Data digital transformation, and Pledger is in the position to greatly benefit from a Big Data approach.
 D. Boyd and K. Crawford, "Six Provocations for Big Data," in Social Science Research Network: A Decade in Internet time: Sumposium on the Dynamics of the Internet and Society, 2011.
 R. Sumbaly, J. Kreps and S. Shah, "The “Big Data” Ecosystem at LinkedIn," in 2013 ACM SIGMOD International Conference on Management of Data, New York, USA, 2013.
 X. Amatriain, "Big & Personal: data and models behind Netflix recommendations," in The 2nd International Workshop on Big Data, Streams and Heterogeneous Source Mining: Algorithms, Systems, Programming Models and Applications, Chicago, Illinois, USA, 2013.
 MarketsAndMarkets, Streaming Analytics Market by Component, Application (Predictive Asset Management, Risk Management, Location Intelligence, Sales and Marketing, Supply Chain Management), Industry Vertical, Deployment Model, and Region - Global Forecast to 2024.