It's time to put virtualization software on your business intelligence (BI) and data warehouse technologies shopping list. Software developers use it and so should BI and data warehousing teams.
Admittedly, my years working in software engineering may have slanted my views a bit, but I think those of us in BI and data warehousing have a lot to learn from software development best practices. Although software firms have typically followed these best practices rigorously, in my experience, these same recommendations often seem superfluous to BI or IT teams that are not used to them. Thankfully, this is starting to change. The tools are getting better, the best practices provide tangible benefits, and the teams are more receptive.
Business intelligence best practice: Use features for source code, version and release management
One set of software engineering best practices involves source code, version and release management. These practices help teams effectively work together on code during the development process and enable them to maintain a detailed tracking of the changes in each version as they are released.
When I first started recommending that my consulting clients follow these practices, most of the BI and data warehouse software tools did not provide embedded source code management, version control or metadata management. So, in order to implement these processes, BI teams had to exert a considerable, disciplined manual effort. And, it was often difficult to get IT's attention when everyone was fixated on the latest and greatest BI software features -- rather than some of the important, but less flashy, underlying processes that made for a more stable release. And since business users always want their BI and data warehouse projects completed yesterday, the manual effort was an extremely tough sell.
But now, fortunately, most BI and data warehouse tools have these capabilities, and it's more a question of discipline and following best practices because teams no longer have to create these tools from scratch. And, quite frankly, many people have been burned for not following these practices, so they are much more receptive to these ideas now. I'm happy to see that source code management and formal release schedules have become a BI and data warehousing best practice and hope to see more companies adopting the idea. And with compliance top of mind for many business people, it's easier to sell the idea that these best practices and repeatable processes should be followed for every project and release.
Business intelligence best practice: Maintain separate environments for development, maintenance and production
A second set of software engineering best practices that developers used to rely on was to create multiple environments for development, maintenance and production. The production environment was then further divided into coding, testing (unit and system), quality assurance and maybe a few other smaller environments depending on the complexity of the project. In "the old days," most companies doing software development saw the value of these separate environments. By having separate environments, new code could be tested in a non-production environment, so that no bugs were introduced into a live system. Essentially, it allowed companies to test fixes or features more thoroughly before releasing new production software.
So software teams used to invest in separate development and production environments, but it was usually too expensive and complex to have more than two separate systems. They needed to buy, set up and maintain separate servers, disks and other components -- and invest in software to support separate development and production environments. As if that weren't expensive enough, when they added other development and maintenance environments, costs skyrocketed. And they used these environments only occasionally. It was a tough sell to suggest buying so much hardware and software that was not used most of the time.
But times have changed.
Now, thanks to virtualization software, the cost of those additional environments for development, maintenance and production has dropped considerably -- and this approach can make a big difference in BI and data warehousing projects. Virtualization also means there isn't any hardware sitting around idle anymore. You do need more disk space (maybe that's why EMC bought VMware) than you used to, but now you can set up multiple virtual environments to run on the same hardware infrastructure.
BI and data warehousing projects can now benefit from multiple development and testing environments as they fully test and debug data integration and business intelligence code. In addition, you can now fix bugs in a maintenance environment that mimics production, avoiding the problem of introducing bugs in the development environment.
Virtualization makes adopting software development best practices a no-brainer. It is already helping BI and data warehouse teams meet tough deadlines and high quality standards. Is it on your shopping list yet?
About the author
Rick Sherman is the founder of Athena IT Solutions, a Boston-based consulting firm that provides data warehouse and business intelligence consulting, training and vendor services. In addition to over 20 years in the business, Sherman is also a published author of more than 50 articles, an industry speaker, a DM Review World Class Solution Awards judge and a data management expert at SearchDataManagement.com. Sherman can be found blogging at The Data Doghouse and can be reached at rsherman@athena-solutions.com.
