Gartner Blog Network

Apache Flink Offers a Challenge to Spark

by Nick Heudecker  |  September 13, 2015  |  Submit a Comment

While Apache Spark has been hogging most of the data processing and analytics (DPA) spotlight over the last year, Apache Flink has managed to turn a few heads for real-time use cases. If you’re unfamiliar with Flink, it’s a memory-centric stream processing engine that can also do batch processing. Spark, on the other hand, is a memory-centric batch processing engine that simulates stream processing through micro batches. (I’m oversimplifying, but feel the rough comparison holds up.)

Based on a few user reports, Flink’s advantages relative to Spark are:

  • True event-based processing
  • Better memory management
  • Less configuration

Despite these potential advantages, information management and operational intelligence vendors haven’t embraced Flink they way the have Spark. (Flink is only commercially supported by data Artisans, the startup formed to offer support and training for Flink.) There are a few possible reasons for the lack of support: geography, attention and incumbents.

First, Flink originated at the Technical University of Berlin, not in Silicon Valley. It’s possible a culture of “not invented here” is keeping attention off Flink. Frankly, I think this is the least probable but it merits a mention. More probable is the finite amount of attention available in the space. Flink became a top-level project in January of 2015, after Spark had already captured a substantial amount of developer and vendor attention. Finally, there are several incumbent open source stream processing frameworks already available. Apache Storm and Samza are probably the best known, but there’s also the newly incubating Apache Apex. And if you want your stream processing in the cloud, Google, Microsoft and others have increasingly robust stories to tell.

I don’t expect the lack of broad commercial support to last. Early adopter customers will demand it from their information management and operational intelligence vendors, and the vendors will have little choice but to react. Whether Flink makes a difference to mainstream customers, or is simply another windmill to tilt at, remains to be seen.

Additional Resources

Predicts 2019: Data and Analytics Strategy

Data and analytics are the key accelerants of digitalization, transformation and “ContinuousNext” efforts. As a result, data and analytics leaders will be counted upon to affect corporate strategy and value, change management, business ethics, and execution performance.

Read Free Gartner Research

Category: data-and-analytics-strategies  

Tags: analytics  in-memory  real-time  

Nick Heudecker
Research Vice President
5 years at Gartner
19 years IT Industry

Nick Heudecker is an Analyst in Gartner's Research and Advisory Data Management group. Read Full Bio

Leave a Reply

Your email address will not be published. Required fields are marked *

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.