Big data finally reached Wall Street. Not for small science experiments, but seriously, in grand style, with exorbitant salaries, for production and DevOps. A couple of years ago, Silicon Valley companies were bragging to each other about hiring Wall Street quants for data scientists. Now Wall Street is happy to grab — whom? No, not data scientists (New York has plenty of its own quants) — Data Architects, the breed who can come up with new architecture to combine structured and unstructured data where architecture for one use case usually (still) does not apply to another.
Dice.com returned just 34 data scientist jobs vs. 645 data architects in the New York area tonight. Even my search of “data architect + Hadoop” returned twice as many data architecture jobs compared to data scientists. Sorry, data scientists: data architects are sexy! My client inquiries shifted lately to no-nonsense big data architecture, management and real-time use cases. Big data vendors hinting left and right about “Wall Street customers in alpha,” e.g. newly signed contracts. My friend, a Wall Street recruiter, had to cut his vacation short – Wall Street is in a hurry to get data architects right now. And also, performance engineers, and developers, and administrators. And did I say, DevOps? Yes, for agility, my friends.
This hiring means that Wall Street is ready to use big data strategically. The picture below shows typical stages of big data adoption described in my research note The Road Map for Successful Big Data Adoption. The red dot on this picture — a stabilized infrastructure — is the most prominent milestone. After the infrastructure has been built, a capability to derive value from big data technologies leaps to a new level. Nonbelievers turn into believers.
At the red dot, big data becomes the new normal. It eventually gets related to other information sources. (When using the term “big data” analytics and information management professionals (across the Globe!) first say that they don’t like it.) At the red dot, companies substantially expand the number of nodes or totally rebuild the earlier small layouts.
A widespread myth is that Hadoop is inexpensive to implement. Really? With Wall Street salaries? An initial implementation is usually more expensive than expected. It involves a lot of unanticipated technical and nontechnical difficulties. By the way, another myth is that big data infrastructure usually takes advantage of commodity hardware. Maybe, but not on Wall Street. Enterprises buy high-end hardware.
I gave my friend a Wall Street recruiter a t-shirt stating “My data is bigger than yours”— he wears it to work (on Fridays). I should make one for myself, with the text Data is what used to be big data. Wall Street is getting there.
Follow Svetlana on Twitter @Sve_Sic
Category: "Data Scientist" Big Data big data market Crossing the Chasm data Hadoop market analysis Uncategorized Tags: big data, big data adoption, crossing the chasm, data, data paprazzi, end users, hadoop, hiring, market analysis