UPDATED 16:17 EDT / JULY 20 2017

BIG DATA

How Spark helped build a community inside a distributed company

It’s hard to appreciate just how large organizations can be until someone has to get all those departments and splintered businesses talking to each other. They need a common platform to share data. Hewlett Packard Enterprise Co., a company currently in a downsizing transition of its own, found that common platform in the open-source data management platform, Apache Spark.

“There’s a lot of organizational dynamics as part of our migration to Spark,” said John Cavanaugh (pictured), research and development master architect at HPE.

Cavanaugh was speaking with David Goad (@davidgoad) and George Gilbert (@ggilbert41), co-hosts of theCUBE, SiliconANGLE’s mobile livestreaming studio, at the Spark Summit 2017 conference in San Francisco. He discussed the benefits of open-source data management frameworks in addressing the fragmented nature of early data systems. (* Disclosure below.)

Herding cats and building platforms

While the benefits of Spark are great, moving to the new technology was not easy for industry giant HPE, according to Cavanaugh. It is a very distributed company, with much autonomy in the various parts, so trying to implement a centralized activity was a challenge. It took some effort to get past the immediate “shields up” response and soothe fears about change.

During Spark Summit, Cavanaugh gave a talk that mentioned the Wild West era of big data. Every department was standing up their own big data systems, he explained. Soon, people realized they had to pull these fragmented setups together. They needed a common platform, and Spark caught on quickly.

In a distributed company, leadership can’t dictate to the businesses. Because of this, HPE made a big push toward cloud-hosted and business-owned. The company knew big data had to be built into their products and services, and at HPE, those were managed by the businesses. “Part of the whole strategy was to empower the businesses,” Cavanaugh said.

Even so, this was not a model where the businesses were set loose to do all sorts of things. HPE shepherded its managed companies to create a virtual community within the company. Through this community, it has been able to share knowledge and strategy. The result has been a more organic movement up through the developers rather than a series of dictates from above, Cavanaugh concluded.

Watch the complete video interview below, and be sure to check out more of SiliconANGLE’s and theCUBE’s coverage of Spark Summit 2017(* Disclosure: DataBricks Inc. sponsored this Spark Summit 2017 segment on SiliconANGLE Media’s theCUBE. Neither DataBricks nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE

A message from John Furrier, co-founder of SiliconANGLE:

Your vote of support is important to us and it helps us keep the content FREE.

One click below supports our mission to provide free, deep, and relevant content.  

Join our community on YouTube

Join the community that includes more than 15,000 #CubeAlumni experts, including Amazon.com CEO Andy Jassy, Dell Technologies founder and CEO Michael Dell, Intel CEO Pat Gelsinger, and many more luminaries and experts.

“TheCUBE is an important partner to the industry. You guys really are a part of our events and we really appreciate you coming and I know people appreciate the content you create as well” – Andy Jassy

THANK YOU