An in-memory database management system (IMDBMS) is a database management system (DBMS) that predominantly relies on main memory for data storage, management and manipulation. This eliminates the latency and overhead of hard disk storage and reduces the instruction set that’s required to access data. To enable more efficient storage and access, the data may be stored in a compressed format.
Traditional DBMSes move data from disk to memory in a cache or buffer pool when it is accessed. Moving the data to memory makes re-accessing the data more efficient, but the constant need to move data can cause performance issues. Because data in an IMDBMS already resides in memory and doesn’t have to be moved, application and query performance can be significantly improved.
To ensure the durability of the data in an IMDBMS, it must be moved from memory to persistent, non-volatile storage periodically. This is important because data stored in memory will not survive an outage. There are various ways to achieve data persistence. One option is transaction logging, in which periodic snapshots of the in-memory database get written to non-volatile storage media. If the system fails and must be restarted, the database can then be rolled back or forward to the last completed transaction.
Another option for maintaining data persistence is to create additional copies of the database on non-volatile media. Yet another option is to utilize non-volatile RAM (NVRAM), such as battery RAM that is backed up by a battery or ferroelectric RAM (FeRAM) that can maintain data when the power is turned off. Hybrid IMDBMSes, which store data on hard disk drives as well as on memory chips, are also an option.
In-memory database systems have broad uses but are primarily used for real-time applications that require high performance. Use cases for IMDBMSes include applications with real-time data management requirements such as telecom, finance, defense and intelligence. Applications that require real-time data access, including call center apps, travel and reservations apps and streaming apps are also good candidates for IMDBMS.
Today, in-memory capabilities are one of the hallmarks of most NewSQL DBMS offerings, although IMDBMS products can also manage other types of databases, including NoSQL, columnar, graph, and relational. An IMDBMS may also be called a main memory database system or memory resident DBMS.
Database expert Craig S. Mullins says that In-memory data processing has a long history and over the years there have been many different ways that programmers have used the concept to improve performance. One of the earliest ways involved creating in-memory tables to store data that could be accessed multiple times as a data processing program ran on a mainframe computer.