Merv Adrian

A member of the Gartner Blog Network





Coverage Areas:

BYOH – Hadoop’s a Platform. Get Used To It.

by Merv Adrian  |  October 9, 2013  |  9 Comments

When is a technology offering a platform? Arguably, when people build products assuming it will be there. Or extend their existing products to support it, or add versions designed to run on it. Hadoop is there. The age of Bring Your Own Hadoop (BYOH) is clearly upon us.  Specific support for components such as Pig and Hive vary, as do capabilities and levels of partnership in development, integration and co-marketing. Some vendors are in many categories – for example, Pentaho and IBM at opposite ends of the size spectrum interact with Hadoop in development tools, data integration, BI, and other ways. A few category examples, by no means exhaustive:

Analytic Platforms: Kognitio - an analytics-focused in-memory database with SQL 2011 support – offers significant Hadoop support. Jethrodata adds indexes for SQL and stores them in HDFS to accelerate BI tools, while Splunk’s offering called – gotta love it – Hunk has a similar approach. SAS recently added a partnership wkith Hortonworks, extending existing capabilities. For  integrated marketing, RedPoint Global offers a true YARN-enabled engine with a rich array of capabilities that many tool vendors would envy.

Application Performance Management – longtime stalwart Compuware is bringing its portfolio to both Hadoop and leading NoSQL offerings.

BI Tools: specialists like Alpine Data LabsDatameerKarmasphere and Platfora position themselves as targeted for Hadoop environments. Traditional players like SAP Business Objects may represent themselves as connecting via Hive, and increasingly we will see some, like Alteryx,  Qlikview  and Tableau, are partnering with emerging distribution-specific stack components like Cloudera’s Impala.

Database: some vendors, like IBM and Teradata offer their own distributions, and even appliances. Others like Actian, CalpontOracle and Microsoft partner with pureplay vendors. All provide connectors, management interfaces to their own management tools, etc. MarkLogic adds an “enterprise NoSQL” flavor; Rainstor adds an archiving solution for a highly compressed Hadoop environment.

Data Integration: Informatica and Talend both support HDFS and even have specific offerings for ETL, data quality, etc. Revelytix Loom offers data prep and metadata creation capabilities to shorten time to use cycles.

Development platforms: Continuuity is out to an early lead here, but is hardly alone and won’t be the last. For example, SQL Server development tool player Red Gate will enter the market soon.

Hadoop as a Service: Altiscale, Amazon, QuboleRackspace,  Savvis, and Xplenty (who mask Hadoop development complexity) offer varying degrees of control and surrounding capabilities – and marketing, as the links demonstrate.

In-memory data grid (IMDG) engine: Gridgain offers GGFS, one of several HDFS substitutes, and like  ScaleOut hServer offers an in-memory grid for execution of MapReduce code. Longtime IMDG player Terracotta has added a Hadoop connector.

Lifecycle Management: WANdisco is offering ALM and support for highly available distributed network deployments, and has recently partnered with Hortonworks.

Platform Performance Management: Appfluent, already providing visibility and performance optimization for Oracle, Teradata and IBM DB2 and PureData for Analytics (aka Netezza) platforms, has now added a Hadoop offering as well.

Search: numerous approaches and players here. One of the more interesting is LucidWorks, leveraging Lucene and Solr for search based use cases on a Hadoop infrastructure.

Security: Dataguise,  GazzangProtegrity, and Zettaset offer varying components of a full-stack security hardening for a Hadoop deployment.

Stream Processing: DataTorrentTibco Streambase, Vitria and Zoomdata are among the early players here.

Workload Automation: BMC’s Control-M is already in place for a need that will become more significant as adoption rises and efficiency becomes more of an issue.

This is just a bare smattering of the evolving ecosystem, and I’d be delighted to have your additions, recommendations and comments. I will  update this post to include them. Please jump in.

9 Comments »

Category: Big Data BigInsights Cloudera DBMS Hadoop HDFS Hive Hortonworks IBM Lucene MapR MapReduce Microsoft Oracle Pig Rainstor RDBMS Security Solr SQL Server SQLstream Talend Teradata YARN     Tags: , , , , , , , , , , , , , , , , , , , , , , ,

9 responses so far ↓

  • 1 BYOH – Hadoop’s a Platform. Get Used To It. | Merv Adrian's IT Market Strategy   October 9, 2013 at 8:21 pm

    [...] —more— [...]

  • 2 Yves de Montcheuil   October 10, 2013 at 6:28 am

    @Merv – great market landscape, good to see how far reaching it is. There are clearly many tools connecting to various constituents of Hadoop – HDFS, Hive, etc. – but Hadoop really becomes a platform when tools run inside/on top of it.

    In the data integration/data quality space, that means running the transformation & cleansing jobs inside Hadoop, through MapReduce code generation for example (a bit like ELT jobs run inside a RDBMS). In the BI space that would mean crunching these queries in MapReduce, not just extract the raw data to load it in a distinct cube.

    I know this is the direction the market is taking, but not all legacy architectures will be able to transition to that concept.

  • 3 Merv Adrian   October 10, 2013 at 12:03 pm

    Thanks, Yves. It is really just a quick sketch; doing it right would take many pages. I’m hoping folks weigh in and say “Add this” – and I will, keeping this as a living post for a while.
    There’s a good discussion to be had here about how the architecture will evolve, starting with “which pieces have to be in the stack to consider it Hadoop?” – and that will change as YARN proliferates and people use other, non-MR engines in their tacks. But certainly MR will remain the vehicle of choice for DI work in the HDFS-and-alternatives data layer.

  • 4 Immo Salo   October 10, 2013 at 2:46 pm

    Good post about what Hadoop is all about: the ecosystem! Far too often Hadoop is only seen as a cheap alternative to store massive amounts of data or batch processing it. Seeing it from platform or ecosystem perspective emphasises the potential it has to utilise the stored (or streamed) data.

    I believe that the wide range of appliances and other so called big data solutions that are providing integration to Hadoop or are “powered by Hadoop” make it more and more obvious that this platform is here to stay (HP’s HAVEn, IBM’s InfoSphere BigInsights, Microsoft’s Parallel Data Warehouse etc.).

  • 5 Pierce Lamb   October 10, 2013 at 9:22 pm

    Great overview. One category you might consider adding is ‘In-Memory Hadoop’ a la ScaleOut Software, Grid Gain, Terracotta, Pivotal etc. Basically the vendors that are combining in-memory computing technology with Hadoop to enable developers to run MapReduce on live, fast-changing data and speed up job execution.

    For reference

    ScaleOut hServer (disclaimer, I’m a ScaleOut employee)
    http://www.scaleoutsoftware.com/hserver

    Grid Gain accelerator for Hadoop
    http://www.gridgain.com/products/in-memory-hadoop-accelerator/

    Terracotta BigMemory Hadoop Connector
    http://blog.terracotta.org/2013/04/02/hadoop-bigmemory-run-elephant-run/

    Pivotal (not yet launched)
    http://blog.gopivotal.com/products/in-memory-data-grid-hadoop-integrated-real-time-big-data-platform-previewed-at-springone-2gx-2013

  • 6 Merv Adrian   October 11, 2013 at 10:04 pm

    I won’t hold your vendor affiliation against you. Thanks for the additions!

  • 7 BYOH – Hadoop’s a Platform. Get Use...   October 12, 2013 at 3:04 pm

    [...]   [...]

  • 8 Platfora Vizboards: Instantly turn raw data into actionable results - Analyst One   October 13, 2013 at 3:01 am

    [...] BYOH – Hadoop’s a Platform. Get Used To It. (blogs.gartner.com) [...]

  • 9 Robert Eisses   October 17, 2013 at 10:01 pm

    Nice overview, Don’t forget the ODBC Connection plumbing offered by Simba. We have the most of the Major Hadoop players (Cloudera, MapR, Hortonworks, Intel, Microsoft, etc) offering our ODBC drivers as part of their distributions, as well as solutions for MongoDB, Cassandra, Salesforce Google BigQuery and others.

    We enable the ecosystem to talk to each other in a standardized and reliable way, via the most common interface.

    http://www.simba.com/connectors