Alex Varshavsky, CEO, Talksum
Last week, we had the pleasure of leading a workshop for the New Mexico Consortium (NMC) and the Los Alamos National Laboratory (LANL) about Big Data management at the data center level.
At the workshop, we discussed how Big Data initiatives need more than just new storage platforms and BI solutions, and how they need new approaches to data architectures and management strategies. The reason for these needs can be broken down into following:
- Operate in real time to keep pace with velocity.
- Adaptive to meet changing requirements.
- Simple to use to avoid specialized skills and custom code.
- Low overhead in terms of people, time, and infrastructure.
Enterprises are overloaded with data. In fact, as highlighted in a Marcia Conner blog post, every day, more than 2.5 quintillion bytes of data (1 followed by 18 zeros) are created, with 90 percent of the world’s data created in the last two years alone. As a society, we are producing and capturing more data every day than was seen by everyone since the beginning of earth.
Most of this data filters through data centers, and along with the data come Big Data problems, such as:
- How do you reasonably ingest, transform, route, and analyze data in real time?
- How can you apply more logic earlier in the pipeline, while minimizing ingest performance impact?
- How can you begin to create a holistic view of the information in the data so that you can correlate events from multiple domains?
These and other problems were covered with use cases, and we followed up with solutions to those use cases. We were excited to show how the Talksum Data Stream Router™ (TDSR™) handles the above and showed examples using their own data. The TDSR can ingest multiple transport and application protocols, as well as multiple formats, and convert the incoming data into parallel data streams, or events messages, that are aggregated, filtered, and then contextually routed to their respective end points (summarized data, aggregated data, dynamic streams, data stores, data warehouses, log storage, and so on).
It was a fruitful workshop and we hope to present at many more.