Big data finally reached Wall Street. Not for small science experiments, but seriously, in grand style, with exorbitant salaries, for production and DevOps. A couple of years ago, Silicon Valley companies were bragging to each other about hiring Wall Street quants for data scientists. Now Wall Street is happy to grab — whom? No, not data scientists (New York has plenty of its own quants) — Data Architects, the breed who can come up with new architecture to combine structured and unstructured data where architecture for one use case usually (still) does not apply to another.
Dice.com returned just 34 data scientist jobs vs. 645 data architects in the New York area tonight. Even my search of “data architect + Hadoop” returned twice as many data architecture jobs compared to data scientists. Sorry, data scientists: data architects are sexy! My client inquiries shifted lately to no-nonsense big data architecture, management and real-time use cases. Big data vendors hinting left and right about “Wall Street customers in alpha,” e.g. newly signed contracts. My friend, a Wall Street recruiter, had to cut his vacation short – Wall Street is in a hurry to get data architects right now. And also, performance engineers, and developers, and administrators. And did I say, DevOps? Yes, for agility, my friends.
This hiring means that Wall Street is ready to use big data strategically. The picture below shows typical stages of big data adoption described in my research note The Road Map for Successful Big Data Adoption. The red dot on this picture — a stabilized infrastructure — is the most prominent milestone. After the infrastructure has been built, a capability to derive value from big data technologies leaps to a new level. Nonbelievers turn into believers.
At the red dot, big data becomes the new normal. It eventually gets related to other information sources. (When using the term “big data” analytics and information management professionals (across the Globe!) first say that they don’t like it.) At the red dot, companies substantially expand the number of nodes or totally rebuild the earlier small layouts.
A widespread myth is that Hadoop is inexpensive to implement. Really? With Wall Street salaries? An initial implementation is usually more expensive than expected. It involves a lot of unanticipated technical and nontechnical difficulties. By the way, another myth is that big data infrastructure usually takes advantage of commodity hardware. Maybe, but not on Wall Street. Enterprises buy high-end hardware.
I gave my friend a Wall Street recruiter a t-shirt stating “My data is bigger than yours”— he wears it to work (on Fridays). I should make one for myself, with the text Data is what used to be big data. Wall Street is getting there.
Follow Svetlana on Twitter @Sve_Sic
Read Complimentary Relevant Research
Organizing for Big Data Through Better Process and Governance
With big data past the Peak of Inflated Expectations on the Hype Cycle, organizations are addressing next-level challenges and asking,...
View Relevant Webinars
Hadoop and Spark: Understanding Open Source Opportunities and Risks
As companies build foundational data and analytics infrastructure with Spark and Hadoop, the market continues to shift and evolve in...
Category: data-scientist big-data big-data-market crossing-the-chasm data hadoop market-analysis
Tags: big-data big-data-adoption crossing-the-chasm data data-paprazzi end-users hadoop hiring market-analysis
Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.