michelangelus - Fotolia

Manage Learn to apply best practices and optimize your operations.

Machine learning, IoT bring big changes to data management systems

Companies are using big data systems, deep learning and machine learning techniques to drive software advances. To go even further, their data management systems must also evolve.

This article can also be found in the Premium Editorial Download: Business Information: Managing the big data ecosystem requires agility amid disruptions:

On a recent commute to TechTarget's Newton, Mass., office, I passed by an Apple Maps vehicle -- a white SUV topped with cameras and spinning light detection and ranging equipment that collects vast amounts of data about streets all over the world.

I have a terrible sense of direction and rely on map apps more often than I'd care to admit -- so much so that Google Maps knows my routine inside and out. It proactively updates me on the time it will take to get from home to work or lets me know that I'm only 10 minutes from a Target store. I don't take for granted what a technological feat this level of personalization is; it requires massive amounts of data to be collected in real time, stored and validated before it can be put to use in a handy iPhone app.

Commanding lakes of data

Yet, with so many connected devices and widely available sensor technologies, access to data isn't the problem. Indeed, most companies are swimming in data. The real challenge is managing all that data and putting it to use. But once the hard part's figured out, companies are able to improve products, services and their business operations by leaps and bounds.

As we learn in our October feature story, inexpensive sensors and internet of things connectivity capable of collecting big data have improved supply chain visibility. Manufacturers can see not only where their goods are, but also precisely when their goods will arrive -- and that's just the beginning.

In our cover story, we explain how big data systems have allowed vendors to make significant strides in software development. One example is iPass, which launched SmartConnect software about two years ago. The software uses algorithms to identify Wi-Fi access points and rank performance so that mobile users can connect to the fastest, most reliable hotspots nearby. Previously, iPass was only able to provide static lists of hotspots. The software advances in SmartConnect were developed, thanks to a data management system built around the Spark data processing platform, which crunches data in real time.

There are many other examples in this month's issue of how big data systems underpin technology advances. But to make further strides, industry experts say data management systems must also evolve.

Managing data in the AI era

Once viewed as platforms for data controls, big data management systems also need to be viewed as delivery systems.

The demand for instant data access, whether by mobile applications or back-end machine learning systems, means data management systems must be agile. Once viewed as platforms for data controls, big data management systems also need to be viewed as delivery systems, and the data they deliver must be valid for the models to work. That requires data engineers to spend a significant portion of their time analyzing raw data before feeding it to machine learning algorithms. And that, too, is expected to change.

The same data-hungry algorithms that will transform the data management game are being applied to more quickly find data in the first place. Smart data management systems that integrate machine learning are used to move data from management platforms to their destinations faster than ever.

As exciting as all that is, I'm fairly certain just the thought of an automated, hands-off approach to data management is enough to give data management and governance professionals an all-out anxiety attack. With so much riding on data, companies need to ensure that the data they collect and analyze meets a specific level of quality and reliability for it to be trustworthy.

My prediction -- based on instinct, not data -- is that many companies will maintain their traditional data management practices for years to come, while making compromises for access and speed where it's safe to do so. But like so many predictions these days, this one could be wrong.

Reach out to me at bbotelho@techtarget.com and let me know what you think about that as well as this month's issue. Thanks for reading.  

Next Steps

Big data platforms for data management and analytics

Data lakes muddy the waters for data managers

Mistakes companies make when managing big data

This was last published in October 2017

Dig Deeper on Big data management

PRO+

Content

Find more PRO+ content and other member only offers, here.

Join the conversation

1 comment

Send me notifications when other members comment.

By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Please create a username to comment.

What changes in data management techniques is your company undergoing?
Cancel

-ADS BY GOOGLE

SearchBusinessAnalytics

SearchAWS

SearchContentManagement

SearchOracle

SearchSAP

SearchSQLServer

Close