Powerful!!! NVMe SSD turns into memory!

NVM Express completed the formulation of NVMe 1.4 specification in 2019. The new NVMe protocol brings a large number of new features, especially in error correction, performance enhancement, and optimization for special and enterprise areas, including PMR, a new capability provided by NVMe 1.4 for high-end enterprise SSDs.

The so-called PMR is a piece of storage, and using this feature will create and control a storage area called persistent memory area (PMR), which can be mapped to the address space on the PCI Express bus and can be accessed by hosts and other devices.

The main feature of PMR is that the data written to PMR is also retained after the power outage of the (power cycle), the controller resets and the PMR enables/disables switching.
In other words, this feature enables SSD to provide another non-volatile storage area in addition to the storage area accessed through the logical block address (LBA), and this storage area is assumed to be accessed by memory access rather than block access.
PMR requires high performance. For example, the write bandwidth in a steady state is much larger than that of PCIe. If this is not possible, there will be options such as elastic buffers on the protocol to fill the bandwidth gap.

In general, PMR space can provide a memory-level read and write speed, a storage area where data will not be lost after a power outage.
PMR is functionally similar to NVDIMM, although the performance and capacity of PMR are far less than that of NVDIMM, PMR has the same advantages as NVDIMM. Compared with reading and writing a piece of data through the NVMe IO command and waiting for the command to complete, the operation of reading and writing PMR is much easier and faster.
PMR has the characteristics of non-volatile, low latency, and Bytes addressing, which makes data management more flexible.
It is ideal for environments that require frequent access to complex data sets, as well as sensitive environments where downtime is caused by power failure or system crash.

The main advantages of PMR include:

• The access latency is less than that of NAND flash memory and is close to that of DRAM.

• Compared with NAND flash memory, the throughput is greatly increased.

• Cheaper than DRAM.

• Bytes addressable, real-time access to data, allowing ultra-fast access to large datasets

• The data remains in memory after a power outage (just like using flash memory).

So, how on earth can we achieve this powerful function?
With the advent of PCIe Gen4, the bandwidth of PCIe increases rapidly, the common non-volatile memory device NAND flash memory is difficult to meet the high-speed performance required by PMR, and originally NAND flash memory is more suitable for block access, not suitable for memory pointed to by PMR, so SSD will not directly use NAND flash memory as a memory device to achieve PMR.

In fact, the specific implementation method of PMR is not specified in the NVMe protocol, but from the existing information, the so-called new memory SCM (Storage Class Memory) can be used to implement PMR, such as using Intel’s Optane memory.
There is also a more mainstream way to achieve PMR is to allocate the DRAM (part of the SSD) to this area. Generally, enterprise NVMe solid-state drives have a large capacity DRAM cache, and the whole solid-state disk is protected by a power outage protection design. Combined with these two characteristics, plus a certain amount of conventional NAND flash memory, PMR can be realized.

NVMe has been actively exploring other uses of DRAM in solid-state drives, and PMR is a potential application.
Most enterprise SSDs have a certain amount of DRAM memory. Cache buffer, SSDs that store FTL entries can map logical addresses and flash physical addresses through these FTL entries.
In addition, the NVMe 1.2 protocol defines the feature of CMB (Controller Memory Buffer) in the controller, which aims to make the DRAM space in part of the SSD accessible directly through the PCI address space. This feature allows the SQ, and CQ needed for NVMe to transmit IO commands to be directly stored in the DRAM memory of the SSD instead of in the host memory, which can reduce the delay of command interaction.
In addition, it can eliminate the unnecessary replication operation in the DMA transmission between SSD end-to-end in the case of NVMe over Fabrics, and make the transmitted data bypass the DRAM of the host completely.

The feature of NVMe 1.4The PMR works like CMB, where the host system can read and write to this area of memory directly using the underlying PCIe transport without any command queue overhead.
In practice, CMB is usually expected to be used to support normal NVMe operations (such as placing SQ/CQ/PRP, etc.) and to be used as a piece of DRAM buffer, but PMR is different. Although PMR is also a DRAM area within the SSD, it is mainly used as a large piece of general-purpose non-volatile storage for the host. A typical enterprise SSD has a special power-off protection capacitor.
These capacitors enable data in the PMR to be safely flushed to flash memory in the event of an accidental power outage.
When the SSD is powered off, the contents of the PMR are automatically written to the flash memory, and when the host system is powered on, the host can ask the SSD to reload the contents of the PMR.

A typical application scenario for the PMR function implemented in this way is a scenario that receives a large number of (coverage) writes, in which PMR does not consume any flash memory other than PMR capacity, because SSD will only brush and save PMR data in the event of a power outage, so it is very suitable for logging database or system logs, which are constantly written in large numbers.
And the operation of writing logs can easily become a performance bottleneck in the system, causing congestion. PMR can provide DRAM-level read and write speed and DRAM overwrite features.
Another potential application scenario is the in-place execution technology XIP (execute-in-place), which is widely talked about to greatly improve the performance of applications, and may become popular with the advent of PMR.

At present, PMR is relatively simple in terms of protocol. NVME 1.4 specification only defines some registers that control PMR, including the detailed setting items such as the write bandwidth in steady-state, the size of the elastic buffer, the state of PMR, the timeout time for hosts to wait for PMR to ready, and so on.
Therefore, it is necessary to effectively use the corresponding drivers on both the PMR, device (driver) side, and host (OS and library) side. At present, the related drivers of Linux are still in the planning stage, and there is still a long way to go to support PMR driver software in the future.

NVMe 1.4 was just released last year, and the applications of PMR in the industry are mainly concentrated in the field of enterprise storage. Most of them are still in the stage of research and discussion, so there is still a lot of room for operation about this new feature of PMR, and its application potential and prospects are expected to be further explored.

References

[1] NVM Express, “NVM ExpressTM Base Specification”, Revision 1.4, June 10, 2019

[2] N. Adams, “NVMeTM Base Spec 1.4 Features Overview”[PPT], Flash Memory Summit 2019, August 6, 2019

[3] Ken-yossy, NVMe Revision 1.4: New commands and features (except for NVM Sets and its related components),https://qiita.com/ken-yossy/items/ae42c06d8c9face76934,,updated at 2019-09-12.

[4] Billy Tallis, NVMe 1.4 Specification Published: Further Optimizing Performance and Reliability, https://www.anandtech.com/show/14543/nvme-14-specification-published, June 14, 2019.

[5] NVM Express, NVMe Specification Readiness, https://nvmexpress.org/nvme-readiness-part-three/, June 10, 2019

[6] David Allen, “NVMe™: What you need to know for next year”[PPT],2018.