Hadoop Tipping Point

2 05 2013

If you’ve ever been to a Hadoop or Big Data meetup, typically it’s full of developers in jeans and sneakers indulging in pizza and beer while engaging with one or more presenters (also engineers in jeans and sneakers). Lots of demos with lines and lines of command outputs and code scrolling across screens more quickly than my eye can capture, never mind my mind grok. And everyone is happy, cuz it really is all about pizza, beer and code.

From Morton Grodzins, UChicago poly sci professor “in sociology, a tipping point is a point in time when a group–-or a large number of group members–rapidly and dramatically changes its behavior by widely adopting a previously rare practice.” This past week I witnessed what I believe is a tipping point in the world of Hadoop and Big Data.

Cloudera hosted a session in Boston on “Getting the Most from Hadoop”. The Boston Hadoop User Group was among the invitees. I snagged a latte after driving into rainy Boston for the all-day event, registered, and then found myself in a hotel event room, with scrambled eggs and BACON (possibly better than pizza and beer in my book). I navigated the room, chatting up some peeps, and found that almost no one there had used Hadoop at all. Few had downloaded it, experimented with it, and many weren’t really certain what to do with it. I took a closer look, and saw more khakis than jeans (even though I was still sporting mine) and many more shoes than sneakers (well, I wear heels to most meetups, so I have little to comment on there). It was weird.

Then the event started. Kicked off by a team of Cloudera marketing and sales peeps – all in suits! A couple formal presos – one from a sponsor – followed. All powerpoints, not code scrolling rapidly across screens. I started getting really nervous – surely this was not appropriate for the Hadoop audience.

Yet no one left the room. Everyone was intently listening. The guy next to me was taking pictures and had a stack of business cards that he had collected from anyone he could. The guy on the other side of me asked if my company used Hadoop, and wanted to know how we got started. I spent a bunch of time talking to people about how Hadoop and the traditional data warehouse can play complementary roles, which seemed to be a huge concern from the audience.

And that’s when I realized Cloudera had helped ‘tip’ Hadoop from the original group of early adopters to a broader audience. The room was full of enterprise IT peeps. Wow.


I like Clouds

22 12 2010

OK, I like clouds. Nah, not those big ones that have been dumping lots of white stuff on us this week. But I do like the ones that prevented Punxsutawney Phil from seeing his shadow this morning – maybe once I dig out from this storm we’ll see that early spring! And I also really like the clouds that are enabling our new way of working, living, communicating. A couple months back I re-entered the workforce after a pleasant sabbatical – I had been considering opportunities in four different spaces – all places that have piqued my interest over the past few years.

  • I wanted to work in mobile, because we all are mobile these days, and as IDC predicts “shipments of app-capable, non-PC mobile devices will outnumber PC shipments within the next 18 months”.
  • OR I wanted to work in BigData. Back when I was in Sun storage I was awed by the challenge of storing data efficiently – as Eric Schmidt recently said “Every two days now we create as much information as we did from the dawn of civilization up until  2003” and the challenges around storing, retrieving and anayzing data have increased exponentially.
  • OR I hoped for more work with social media. After spending most of my sabbatical designing and implementing social media strategies for non-profits, I had a taste of where the world is going from a communications perspective. Social media is turning it inside out.
  • BUT REALLY I wished for something interesting  in the cloud world – what a fascinating paradigm shift for the compute and data worlds. It simply doesn’t make sense for so many people in the world to be managing servers and storage systems – cloud answers a basic competitive advantage question.

And then I came across an intriguing opportunity at Nokia: to run product management for Nokia’s cloud services. A cloud that stores mobile app data… A cloud that delivers analytics… A cloud that enables social networking… So, I do like clouds. Just not those bad ones outside right now. D*mn gotta go shovel again. D*mn clouds.

The black line is the top of my patio table

Front sidewalk - where can I put the snow?

Mother’s Little Helper? Data

24 06 2010

I’m fully supportive when my loved ones go off-the-grid on some new adventure. But the worrier inside me usually starts roaring around 3am… “Is everything ok?” “Where are they now?” Until recently it was pretty difficult to calm my worrier. But now, like the lyrics of the Stones, I’ve discovered my own mother’s little helper: Data. Yup, data.

Ed's Kilimanjaro RouteIn March my adventurer photographer husband spent 6 weeks in various parts of Africa working with children’s charities, exploring and mountain climbing. And attached to his backpack was a GPS sensor, which sent data about Ed’s whereabouts to a centralized service. I could go to the service website to see his location mapped, like his footsteps up Kilimanjara shown here. And I could post these satellite images to Facebook so friends and family could keep track as well. No more lost sleep. Saved by the data.

I was at an interesting conference this week where much was discussed about data. FindMeSpot SatelliteStephen Baker, of Num3rati fame, delved into the next big wave of data coming from sensors in the world around us. Kinda creepy in many ways, but as with every new innovation, there’s a cost-benefit analysis to be done. In the simple case of the FindMeSpot, I’m all for it. We’re gonna have some interesting discussions, though, over the next few years as sensors become more prevalent around us. And we’re certainly not gonna suffer from a lack of data.

Where data center meets the cloud

6 07 2009

I’ve seen lots of data centers. Big ones, small ones, pod-designs, diesel-powered, environmentally controlled. But I can’t say I’ve ever seen a data center like the one I encountered this week in Bermuda. We had climbed seven flights of winding stairs inside Gibbs Lighthouse – with only one flight to go – when we ran across a miniature data center about 100 feet above ground. It’s the data center that keeps the Gibbs lantern shining, so ships know they’re close to the lovely yet treacherous Bermuda shores.
185 Winding StepsThe Tiny Data CenterThe Huge Lantern

Here’s what I learned from the placards inside the lighthouse: From it’s start in 1846, the Gibbs lighthouse keeper wound a 1200 pound weight by hand every 30 minutes to revolve the lighthouse lens. The lantern itself was originally kerosene. In 1964 electrical equipment was installed, and today the whole lighthouse process works automatically: computers maintain the light, and the APC equipment and a diesel generator make sure it keeps shining even during terrible storms that bring power outages. Which is pretty important given that 39 ships were wrecked off the Western end of Bermuda in the decade before the lighthouse was constructed. Now that’s a mission-critical data center!

Additionally the lighthouse stands on a hill that is 245 feet, and the lighthouse itself is 117 feet tall, which is why it can be seen from 40 miles away. And I can personally attest to the fact that most of Bermuda can be seen from the top of the lighthouse. Gibbs Lighthouse – where data center truly meets the cloud.
My view from top, facing west

When expertise is important…

26 11 2008

When I picked up my four Thanksgiving pies at Stonybrook Farm this afternoon, the baker mentioned that he made 500 pies for the holiday. I trust his expertise much better than my own, admittedly poor, pie-making skills.

Which brings me to our announcement yesterday. Keeping data for long periods of time is important these days. Of equal importance is truly destroying data as required by internal corporate erasure or regulatory policy. That’s why we developed a new on-site service to help customers with this challenge. Our experts delivering the new Sun Data Protection Data Erasure service will work with customers to ensure their erasure policies are compliant.

Now back to Thanksgiving: tomorrow we will give thanks for our baker’s pie-making expertise while we erase those four pies.

Mini Me

31 03 2008

At Sun we’re all about “The Network is the Computer”, which really means instant access to information, anytime, anywhere. And the pace at work continues to speed up as development happens more and more rapidly on the Internet. Each and every day it’s a pace that races in the fast lane. A good friend recently articulated his strategy for maintaining a balance to our overdrive work environment: “work hard play hard”. Well said. To that end I matched my work pace while at play this past weekend – under the tutelage of Phil Wicks (he stunt drove the red Mini in the original Italian Job). That’s me driving my yellow Mini…

Brad Pitt & George Clooney: Tape is not Dead

20 06 2007

When I was at the movies last week watching Ocean’s Thirteen, it dawned on me that our SL8500 tape library got closer to Brad Pitt and George Clooney than I ever will. As Nigel mentioned, we lent one of these beauties to Ocean’s Thirteen for the ‘Bank Casino Operations Center’. So yeah, tape is not dead. Tape is alive, well, and hanging out with the likes of George/Danny Ocean and Brad/Rusty Ryan.

But I wonder if George and Brad know just how awesome the SL8500 actually is. Do they know, for example, an SL8500 can hold a petabyte of data – about two hundred thousand copies of their Oceans Thirteen movie? Do they know that if the Bank Casino used 1000 cameras to gather their surveillance data and stored that data for 30 days, they would fill the tapes in an SL8500? And in interests of saving the planet, do they know tape is about 25 times less expensive to power and cool than disk because it uses that much less energy? All great news for the IT budget and the planet.

I spent some time this week with our Media and Entertainment sales team – to say the data in M&E is exploding is a complete understatement. One customer digitizing TV shows is expecting to have 50 petabytes of metadata to enable all the searches they need to handle – never mind the raw entertainment itself! And the M&E industry is heading full steam ahead into complete digitization, consumer mashups, affiliate communities… Data, data, and more data. No wonder why many cool web sites are using SL8500s to help store that data.

So, sure Brad and George were cast for the Ocean’s movies because they’re so hip, but our SL8500 certainly fit right in with the Ocean gang on their latest caper.