Gartner Blog Network

Hadoop’s Achilles Heel in 2015

by Nick Heudecker  |  December 26, 2014  |  2 Comments

Recently my colleague Merv Adrian and I spoke with a top four financial services firm about its upcoming Hadoop deployment. After several months of testing, the ten-node pilot was moving to production in a few weeks. We asked if any consulting firms or systems integrators helped with the project and the response was that Hadoop skills were simply unavailable.

If a top-tier finserv company can’t find Hadoop skills, how will other enterprises?

Hadoop is a complicated beast. Commercial distributions frequently comprise more than two dozen projects, each with its own pace of development, version incompatibilities and bugs. Just getting Hadoop running – let alone processing data – is a substantial achievement requiring a variety of skills. Actually doing something with the data requires another set of uncommon, if not rare, skills.

Enterprise adoption of Hadoop will be significantly impacted in 2015 unless Hadoop becomes drastically simpler to deploy and manage or the pool of available talent increases exponentially. With new components being added almost daily (Spark, Flink, Hama, etc.), the former doesn’t look possible. The availability of Hadoop skills must increase, and increase rapidly, for meaningful adoption to occur.

What are you doing to develop Hadoop skills at your company? Let me know in the comments.


Additional Resources

Predicts 2019: Data and Analytics Strategy

Data and analytics are the key accelerants of digitalization, transformation and “ContinuousNext” efforts. As a result, data and analytics leaders will be counted upon to affect corporate strategy and value, change management, business ethics, and execution performance.

Read Free Gartner Research

Category: data-and-analytics-strategies  

Tags: hadoop  

Nick Heudecker
Research Vice President
5 years at Gartner
19 years IT Industry

Nick Heudecker is an Analyst in Gartner's Research and Advisory Data Management group. Read Full Bio

Thoughts on Hadoop’s Achilles Heel in 2015

  1. Alan Grogan says:

    If you look at any Big Consulting or IT firm’s recruitment site you will see many Big Data roles open (and for many months). The problem I see is that the excellent Data Scientists I coach or have the pleasure to manage either refuse to work with/ for them due to corporate processes, or they’ve interviewed but been turned down at interview stage for no technical reason, despite the roles needing them >90% technical. I also add that many Data Scientists will also not work directly for a Financial Services firm due to the sector’s perceived weakness in IT support and lack of credible Proof-Of-Concept environments.

    I feel the issue is not that the support is not available, but that large organisations can fall into the trap to think Big Data needs a large IT or Consulting firm to help. I don’t blame them as the large Big Data ‘suppliers’ probably spend as much on Marketing in this space as capability.

    Successful Big Data proving and deployment is rather straight forward, the issue is once a client comes across a successful Big Data consulting firm (usually by word of mouth), is (1) getting access to the necessary client internal data, (2) Obtaining the needed suppliers on the PSL and (3) confirming preferred support model in BAU (Inhouse/ Managed/ Offshore/ Nearshore/ Onshore).

  2. Shil Desai says:

    I believelarge organisations can not do this due to multiple reasons. World is moving towards custom. Niche companies/skills are needed to address specific problems. So we see lot of startups rising in these niche space. I recommend all companies working in this space must invest and do below to keep their foot with dynamic needs of big data.

    1. Create/ think of all possible domain specific use cases and prototype on basic infrastructure
    2. Knowledge share with your team and create workforce
    3. Do above regularly

    For clients
    1. Choose the partner carefully
    2. Trust the implementation partner
    3. Go for pragmatic approach
    4. Do regular health checkups for such projects

Leave a Reply

Your email address will not be published. Required fields are marked *

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.