This content is part of the Essential Guide: Cloud data warehouse guide: Using Redshift, rival platforms

Agile data warehousing casts business light on dark process

At the TDWI Executive Summit in Boston, users talked about the benefits and challenges of incorporating Agile development methodologies into data warehouse and business intelligence projects.

An unfortunate part of data warehousing is this: There are lots of disappointments. CEOs and other business executives sign on to fund projects aimed at empowering better decision making, but too often they instead get data warehouses that take too long to build, that no longer meet business requirements and are hard to update once they're live.

Experts and users at last week's 2014 TDWI Executive Summit in Boston discussed those issues and the potential benefits of Agile data warehousing and business intelligence(BI) -- the use of Agile development methods to accelerate and improve the outcomes of data warehousing and closely-related BI initiatives.

"Usually, data warehouse development cycles are long -- they could be 18 months," said David Stodder, director of BI research at The Data Warehousing Institute (TDWI), which organized the summit. "There is frustration on the business side that we are not getting value fast enough."

Agile techniques that deliver functionality in short iterations have long been used in software development circles and organizations began trying to apply their tenets to data warehousing and BI several years ago. But Philip Russom, who heads data management research at TDWI, said that Agile BI and data warehousing are "still a minority practice" compared to traditional waterfall development approaches.

Several users who spoke at the summit shared their experiences adapting aspects of Agile development to data warehouse and BI efforts.

"The reputation of data warehouse projects is 'slow progress' and 'big expense,'" said Ryan Fenner, vice president and enterprise data solutions architect at Union Bank in San Francisco. In Fenner's view, business leaders "say 'here are my requirements,' people go into a dark room for six months and then they come out. The business people say: 'It's not what I wanted. What took you so long? Why did it cost so much?'"

Agile goal: Smaller projects, bigger benefits

So first in a shadow IT unit and now as part of a data warehousing initiative within IT, Fenner has worked to adopt Agile practices at Union Bank, partitioning projects into smaller chunks -- or Scrums -- that are reviewed regularly by involved business users and the stakeholders who are funding the efforts.

"Agile is a way to bring visibility into the process -- to overcome hurdles with some iterative or rapid delivery," he said. "You are going to get a better product -- you're going to get use out of the data warehouse in a quicker fashion."

But Agile data warehouse design isn't easy, Fenner said, as it requires major changes in the way things are normally done. He told attendees that Agile processes require a strong team with good communication skills, as well as executive-level buy-in. In addition, the team must include business users, according to Fenner. He said he was fortunate in being able to source several Scrum team members from the business side, which helped create software releases that better reflected real needs.

In Agile data warehousing and BI efforts at Canadian National Railway Co., a key was to assign a leader from the business side of the organization, someone who was "willing to succeed or fail with the rest of team," said Mark Giesbrecht, senior manager for BI at the Montreal-based organization. But that can be a challenge as well, added Giesbrecht, who also spoke at the TDWI event.

"Such people are hard to find," he said. "It's easier to find the project money than to find a person to join the project with intensity for the duration."

Big data adds more reasons to be Agile

The difficulties of data warehousing are driving some of the interest in new data management technologies such as Hadoop and NoSQL software that requires less up-front planning and design of database schemas -- one of the elements that contributes to data warehouse development slowness. Meanwhile, the growing focus on big data processing affects data warehousing project management as well.

"There are added pressures as users are looking to do more with their data," Stodder said. He noted that not all parts of the Agile way work well with data-centric development, "It doesn't align with everything people are trying to do." Still, iterative approaches can help business managers and users sharpen their requirements, Stodder said, adding that it's often difficult for them to say right off the bat exactly what they want.

In a presentation at the summit, Forrester Research Inc. analyst Boris Evelson recommended no more than a two-week gap between tangible BI and data warehouse deliverables. But increased business agility requires "a lot more than just Agile software development," Evelson said, pointing to the need to also focus on organizational structures, analytical processes and the overall BI technology infrastructure.

Conference attendee Irma Murillo, manager of the BI group at medical insurer and healthcare provider Fallon Health, said she plans to adopt Agile development processes in updating a new data warehouse that the Worcester, Massachusetts, organization put into production in June. The warehouse, based on Microsoft's SQL Server database, took 2.5 years to develop; now Murillo is looking to do updates in four-week increments.

"The mechanics of it are the tough part -- making it happen," she said. "I definitely have a challenge ahead of me. We'll trip and fall before we figure it out." But data warehouse development has to stop being so "tectonic," Murillo added. "We can't keep using the excuse that data warehousing is difficult. That's just not acceptable anymore."

Jack Vaughan is SearchDataManagement's news and site editor. Email him at [email protected], and follow us on Twitter: @sDataManagement.

Executive editor Craig Stedman contributed to this story.

Next Steps

Read about Agile's role in the rise of a NoSQL database

Find out what went on at a 2013 TDWI summit

Learn about the Agile techniques of retrospective design review

Dig Deeper on Data warehouse project management