freshidea - Fotolia

TDWI finds big data approaches that move jobs from POC to production

Big data approaches outlined in a TDWI maturity model help bridge the chasm between proof of concept and production. One finding: Stick-to-itiveness is a key to success.

Analyst Fern Halper is not alone in seeing a chasm between the promise and the reality of big data. Many have seen the difficult path to successful big data analytics first hand.

But Halper, vice president and senior research director for advanced analytics at TDWI, goes a step further, as her research looks for ways to narrow the chasm. The best big data approaches focus on skills, culture and persistence, in Halper's estimation.

Different factors in different organizations can stall big data technologies like Hadoop from moving from proof of concept (POC) to actual business operations.

"Sometimes, it's about [a] lack of skills. Sometimes, it is about an organization's political culture," Halper said. "The fact is that succeeding with big data analytics can take a while -- people have a hard time getting beyond early adoption."

Crossing the chasm is hard

Three years ago, TDWI forged a big data maturity model to help organizations compare their big data efforts with their peers'. The maturity model comprises nascent adoption, preadoption, early adoption, corporate adoption and mature/visionary stages.

As of early this year, TDWI found a large majority of more than 600 respondents (86%) to be in the preadoption and early adoption stages of big data maturity. Just a bit less than 10% had gotten beyond what TDWI describes as "the chasm that separates early adoption from corporate adoption."

It matters whether or not the proof of concept that you build shows value.
Fern Halperresearcher, TDWI

Best practices that emerge from interviews with those that have crossed the big data chasm show what some people are doing right, and what others can learn from, according to Halper. Even among those with more big data analytics success, a lack of skilled analytics professionals remains a challenge. Here, organizational issues become important.

"Folks need to build a data sense, which means not taking data at face value," Halper said. "They also need domain expertise."

This is most helpful when people with such talents are encouraged to share them with others, expanding the knowledge base.

Halper said building special big data deployment teams and creating centers of excellence for big data analytics are ways people are trying to go about this alone. Sharing skills is crucial because it's hard to find all the necessary skills in one person.

What the maturity model tells us

In making the leap from experimenting and POC to maturity and production, the choice of pilot project is very important, according to Halper.

"The people that are successful do proofs of concept that matter -- something the business can get behind," she said.

"It matters whether or not the proof of concept that you build shows value," Halper continued. "New big data applications should have measureable impact."

To do this may require that companies elect projects that address a type of work that the organization has measured before.

Today, big data approaches most often remain works in progress. The vendor hyperbole has been beat like a drum, but production success with big data analytics remains challenging.

Halper's research points to big data approaches teams can take to vault the chasm. Part of that, she said, is a good share of persistence.

"There is no silver bullet, but there are strategies that are helpful," Halper said. "Don't give up."

Next Steps

Discover seven best practices for data management

Learn about big data approaches to master data management

Check out warehousing best practices for big data

Dig Deeper on Hadoop framework