Intel® Optane™ media is a unique memory technology with a number of important advantages over other memory options in production. One of these important advantages is endurance: The number of times the memory can be written before it is considered worn-out and unsuitable for continued, reliable use. Intel® Optane™ media allows for a much larger number of lifetime write cycles, and thus a higher endurance than NAND memory. Furthermore, Intel® Optane™ media enables write-in-place updates, avoiding the extra writes, at the cost of endurance, needed in NAND-based systems. (See the technology brief “Intel® Optane™ technology: Memory or Storage? Both.”) For these two reasons, Intel® Optane™ DC Solid State Drives (SSDs), built on Intel® Optane™ media, feature much higher endurance than NAND SSDs. In addition, the higher endurance of Intel® Optane™ media enables its use as system persistent memory, which is currently available as Intel® Optane™ DC persistent memory.
This paper discusses endurance issues and properties of NAND SSDs and traditional memory, and it shows how Intel® Optane™ media can overcome those issues to provide much higher endurance when used in either Intel® Optane™ DC SSDs or Intel® Optane™ DC persistent memory modules.
What Is Endurance?
Simply put, endurance is the total number of data writes a system can accept and reliably remember in its lifetime. Endurance should not to be confused with capacity: The total number of those writes that can be in storage at any one time. Endurance for rewritable storage technologies is much higher than capacity, and it is sometimes specified in terms of the number of times that capacity can be written.
To make an analogy, think of endurance as being similar to wear and tear on your car tires. A set of tires is useful only for a set number of miles driven. Once the tires become too worn, they can no longer be counted on to provide safe transportation. How long your tires last depends on the quality of the tires (their designed-in endurance) and how often you use them (the number of miles driven per day). With light use, even medium-quality tires might last years. But under heavy use, those tires could run out of tread in a year. Non-volatile memories are similar to tires in that they can accept only a limited number of data writes before they “lose useful tread” and are unable to support additional writes. And like tires, different media technologies have different limitations on lifetime writes.
In general, non-volatile memories enter computing systems as part of a larger subsystem that is often an SSD. SSD endurance describes the number of times the entire capacity of a drive can be overwritten before it becomes unreliable. Fundamentally, SSD endurance limits result from the limits of the non-volatile memory cells used to create NAND SSDs. Architectural choices made in the design of the NAND flash memory, in addition to the design of the SSD, also impact SSD endurance limits. Endurance is commonly expressed either as average drive writes per day (DWPD) or as total lifetime writes. The two metrics are equivalent over the SSD lifetime—typically five years—factoring in the capacity of the SSD.
The endurance required by the workloads and the longevity expected in the data center are critical factors to consider when placing SSDs in a data center. Endurance, after all, determines the usable lifetime of SSDs, making it an important factor in the cost of operating a reliable data center. To select an endurance level, operators must understand the write load required by their expected workloads, then multiply that load by the lifetime expected before the SSDs are to be retired. Selecting drives that fall short of the expected load could incur extra expense and potentially impact system uptime as SSDs are replaced.
How NAND SSD Architecture Impacts Endurance
NAND SSD endurance is a complex topic that requires an understanding of how NAND memory itself functions. With NAND memory, a write operation can only flip a 1 to 0, but not the other way around. Before any write can be performed, data currently existing in the flash memory cell must first be erased, setting all the bits to 1. While the unit of erase in NAND media varies, in all cases it is large relative to the minimum unit of write or read. Erase blocks are multi-megabyte in size, whereas write pages are 10s of kilobytes, and read sizes of just 4 kilobytes are currently common. An erase operation deletes the entire block, including many write pages. This adds complexity to SSD operations.
NAND SSDs are designed specifically to manage this underlying structure while providing sufficient performance and endurance. Reads from the system can simply access the right page. Writes to the SSD are always performed to an already erased block. Multiple SSD writes are collected until enough 4 KB sectors have been grouped to fill a write page, at which point the data is written. The indirection table of the SSD (usually held in DRAM) is then updated to keep track of the location of each sector so that the sector can be found on subsequent accesses. This process continues until the entire block is written.
At the SSD level, the situation gets more complex when one of the sectors is rewritten with new data. NAND memory will not allow that data to simply be updated; instead, it is written to a different, already erased block. Now the original block holds stale data. But the rest of the data in the erase block might still be valid, so the block can’t be erased. The page holding stale data has temporarily become stranded—it holds no useful data, but it cannot be rewritten. Eventually, this space must be reclaimed with a process often called “garbage collection.” In this process, the remaining valid data pages are copied by internal SSD logic to an already erased block. Because it now holds no necessary data, the original block can be erased and made ready for writing. In this manner, the invalid space is reclaimed.
Writes matter. Endurance equals the total number of data writes a system can accept and reliably remember in its lifetime. Why only writes and not reads? Because reading from media is non-destructive, and therefore is not as much of a concern for system architects.
Notice that the process described in the previous paragraph generates writes within the SSD. These writes are not present on the input to the SSD; these are extra writes. The term used to describe this phenomenon is write amplification (WA). WA equals the amount of data written to the NAND media, divided by the amount of data written by the system to the SSD. Endurance is specified at the SSD level, so these writes are not included in that number. However, they must still be accounted for because media endurance is the fundamental reliability limit that must not be exceeded.
You can see that the characteristics of the write workload from the host can have a negative impact on WA. A workload that requires largely random writes—particularly small-sized writes, such as 4 KB—is a worst-case scenario. That’s because random writes tend to strand invalid 4 KB chunks of data across many different erase blocks, generating a higher garbage-collection overhead. WA equal to 3 or higher is not unusual. And for a WA of 3, only a third of the underlying NAND memory endurance is accessible to the system.
To enable garbage collection, the NAND SSD has to maintain some spare space, not visible to the user, to which data can be copied. The ratio of total drive space to that seen by the user is called over-provisioning (OP) space. The larger the OP space, the more likely the SSD will be able to find blocks with little valid data, reducing the overhead of WA. A larger OP space increases SSD cost. For this reason, higher-endurance and higher-write-performance SSDs will result in larger OP, and therefore tend to cost more.
So endurance is driven by the capabilities of the underlying NAND memory, by the OP of the SSD, by the efficiency of the SSD’s algorithms that identify blocks to erase, and finally by the workloads themselves. For this reason, the endurance of NAND SSDs varies from just tenths of DWPD for the lowest cost SSDs to 10 DWPD for the most expensive SSD.1 As you will see in the next paragraph, things are much simpler with Intel® Optane™ media enabling much higher endurance.
How Intel® Optane™ Media Is Different
When it was released, Intel® Optane™ technology offered the first new memory technology in volume production in more than two decades. Intel® Optane™ technology is built with entirely unique memory media, featuring new materials with different enhanced endurance properties. Moreover Intel® Optane™ media and the SSD or platform memory subsystem surrounding that technology has been purpose-designed with high endurance in mind. Intel® Optane™ DC SSDs and memory modules offer substantially higher performance and endurance than NAND SSDs.
Intel® Optane™ media has deep endurance-related advantages that set it apart from NAND media. The first, and by far the most important, advantage is the cells themselves. Intel® Optane™ media cells have been created with materials and manner of operation specifically designed to enable much higher endurance. Intel® Optane™ media is built from many individual cells featuring a selector and memory material. Cells store “1” or ”0” in the bulk properties of a memory material. The process used is resilient across decades more cycles than the trapped charge process used in NAND media.
Intel® Optane™ Technology Breaks Through the NAND Barrier
Intel® Optane™ technology is built on a completely new memory media that is byte addressable like DRAM, non-volatile like NAND, and has a read/write latency between the two.
Intel® Optane™ technology combines Intel® Optane™ media with Intel controllers, software, and system interconnects that can be deployed as memory or storage.
In fact, this endurance advantage is so great that, unlike NAND media, Intel® Optane™ media has sufficient endurance to act as system memory. Secondly, Intel® Optane™ media is capable of byte accessibility and write in place. This memory has no erase-before-write requirement, just read and write. This greatly simplifies system design and means that all the endurance of the underlying media is available at the system level, as described next.
Intel® Optane™ DC SSDs Skip the NAND Overhead
Intel® Optane™ DC SSDs maintain data in 4 KB units, matching the most common quanta used by existing operating systems’ storage stacks. With the write-in-place capability of the media, the relocate-on-write process described for NAND SSDs is not necessary. When an existing sector of storage is written again, it can simply be updated in place. This is a significant difference because it means:
- No garbage collection is required
- No extra user non-visible space needs to be allocated
- No extra writes are generated
So, for an Intel® Optane™ DC SSD, WA equals 1 (none) and OP equals 1 (none). There are no background writes using up memory endurance for garbage-collection operations, which improves the endurance that the SSD provides. There are also no garbage-collection-driven writes to get in the way of host-generated accesses, delivering more media performance to the system.
Intel® Optane™ DC SSDs implement a proprietary wear-leveling algorithm to ensure that the wear is distributed evenly across all cells of the Intel® Optane™ media. This indirection ensures that repetitive writes to a particular address will not wear out a fraction of the SSD prematurely. Writes occurring disproportionately to one region cause those writes to be mapped to a different physical location. The SSD handles this invisibly with a simple swap between the data at the old and new locations—again, with no garbage collection required.
With these memory-level and SSD architectural-level innovations, the whole high-endurance capability of Intel® Optane™ media is made available to the system. Intel® Optane™ DC SSDs are currently available at up to 60 DWPD,2 far exceeding those available with NAND SSDs.
Endurance Over a Lifetime Matters
SSD lifetimes are frequently measured by calculating the projected average number of DWPD. Usage that exceeds that projected DWPD will decrease the useful life of an SSD. That’s why it is important to calculate the average DWPD that will commonly be incurred with the workloads imposed by your data center.
A usage such as a storage cache, for example—which is closer to the CPU and has a smaller capacity than extended storage—will service writes more frequently. Other applications might require far fewer write operations.
The maximum number of writes that a drive can sustain is determined by multiplying DWPD by the capacity of the SSD and again by the lifespan (in years) of the drive, as specified in the manufacturer’s warranty. Applying that simple formula to both a NAND SSD and an Intel® Optane™ DC SSD clearly illustrates the impact of the architectural differences of the two memory media:
- A ½ TB NAND drive warrantied for three DWPD over five years supports a maximum of 3 PB of total writes over the five-year term (1 TB = 1012 bytes and 1 PB = 1015 bytes).
- A ½ TB Intel® Optane™ DC SSD is warrantied at 60 DWPD for five years, supporting a maximum of 55 PBs of total writes over the five-year term.2
Comparing the two SSDs highlights the large endurance advantage provided by the Intel® Optane™ media. To match the projected lifetime capacity of a single Intel® Optane™ DC SSD would require the equivalent of more than 18 NAND SSDs.
For many data center operations, the best balance of performance and cost can be attained by deploying both Intel® Optane™ DC SSDs and Intel® NAND SSDs. For example, a workload with sufficient data locality might best be supported by a data center with a small number of Intel® Optane™ DC SSDs dedicated to servicing 90 percent of the write operations, with the remaining 10 percent going to NAND SSDs. This might result in a smaller total number of SSDs in the system, potentially reducing the number of systems needed to host SSDs and further improving total cost of ownership (TCO). A well-planned combination of Intel® Optane™ DC SSDs and Intel® NAND SSDs might improve cost efficiency while ensuring that warrantied lifetime DWPD is not exceeded for either.
Taking Endurance Further: Intel® Optane™ DC Persistent Memory
From the beginning, Intel® Optane™ media was conceived for use in both storage systems (SSDs) and memory systems. System memory is accessed by CPU load and stores instructions across the system memory bus instead of the system input/output (I/O) bus. While the byte addressability and write-in-place attributes of Intel® Optane™ media match use as system memory, endurance must also be satisfied. The memory bus is capable of much higher read and write rates than the I/O bus. Those higher write rates translate to a significantly higher endurance requirement than SSDs. So Intel® Optane™ DC persistent memory needs significantly higher endurance than an SSD to remain useful over a five-year period.
Intel® Optane™ DC persistent memory’s endurance is specified specifically to enable the system to write to the memory module over its lifetime without worry of premature wear-out. Endurance is specified at the maximum power rating of the memory module, where the write bandwidth is at its maximum. To provide numbers, Intel® Optane™ DC persistent memory is specified to last 5 years, 24/7, for a total amount of data of more than 350 PB (for a 256 GB module) lifetime writes. This is significantly higher than the 55 PB lifetime of Intel® Optane™ DC SSDs. Like SSDs, the memory module counts progress toward this lifetime limit and provides the write count to system-management software in response to a smart command request. With these high levels of endurance, this unique technology can be used as memory, now persistent, revolutionizing the system memory and storage hierarchy.
What Intel® Optane™ Media Can Do for Your Data Center
Intel® Optane™ media provides data centers with the ability to increase endurance while simultaneously reducing costs and boosting performance. And endurance directly impacts operational costs: Improved endurance can translate directly into lower TCO without sacrificing performance.
The number of writes that an SSD can accept in its lifetime—its endurance—is a key determiner to system usage and, in the long run, system cost. Through media advances and SSD architecture optimizations, Intel® Optane™ DC SSDs can deliver more than 10x the endurance of many NAND SSDs.1 2
Intel® Optane™ DC SSDs Deliver › 10x the Endurance of Many NAND SSDs1 2
Finally, Intel® Optane™ DC persistent memory allows access of Intel® Optane™ media as system memory and again delivers new, higher levels of system endurance.3 At the system level, these new levels of endurance enable new uses and new ways to reduce cost and increase system performance.