Category Archives: Reliability

Flash Memory Summit 2014 Spreadsheet

I am posting the spreadsheet that accompanies the Flash Memory Summit 2014 Seminar A chart deck. This is a LibreOffice 4 spreadsheet. Note that it includes a macro that extends the range of the cumulative binomial distribution. Interestingly, LibreOffice suffers from the same lack of precision that Excel does. The macro is provided as-is, and is not guaranteed to work in all cases. [Translation: I haven’t had time to debug it and the error handling in LibreOffice basic is abysmal.]

The spreadsheet shows how to compute reliability for flash-based storage systems, and compares various RAID  architectures when using a DNR ECC approach (described in the chart deck).

Note: I have seen a lot of talk in the industry about how new systems use erasure codes instead of RAID. Technically, all RAID designs use erasure codes – even mirroring (that’s called a replication code). I use the term RAID loosely to describe a system utilizing an erasure code  to protect against data loss.

The PMDS codes described here are erasure codes designed to protect against simultaneous device loss and sector loss with high data efficiency. You can read about them here, here and here.

You can download the RAID Reliability Spreadsheet.

The spreadsheet is provide under GPLv2.

Series SSD: 6. The test methodology

A philosophical point

When testing devices, it is usually best to test beyond the specifications.
This is especially important when architecting systems.
This allows for a better understanding of behavior in the region of the failure limits.
I also find it better to interpolate behavior than to extrapolate it.
Finally, we may learn something interesting!

The test methodology

I have measured the error rate behavior for a set of MLC NAND SSDs with real-time aging. I have looked at 5xnm, 4xnm and 3xnm class consumer-grade devices. I haven’t had the chance to measure 2xnm devices yet. I chose consumer-grade flash as this is the dominant flash on the market, and one of the goals is to learn how to use such devices in enterprise systems.

I have created a test suite to measure the error rate surface in multiple dimensions. It can measure the bit error rate as a function of the data age (retention), the P-E cycle count (endurance), the number of reads since the last write (read disturb) and temperature. I’m not looking at other effects such as write disturb here. Off the shelf SSDs with a special micro code load are used. The special features include: turning off wear leveling, providing direct read, write and erase, and raw (no ECC) read. The latter allows a bit-for-bit compare to be used to determine the bit error rate. The SSD controller still determines how the write, read and erase operations are performed. All the SSDs were purchased through retail channels, thus should be representative of what is available.

In the test procedure, the cycling and aging are all performed at the same temperature, as is likely to be the case in an enterprise application. This is distinct from many other test methods, where the device is cycled at room temperature, and then the retention is measured at high temperature. For example, the JEDEC JESD218 specifies this type of test. I don’t feel that this approach adequately reflects the operational environment. Enterprise SSDs are are typically installed in rack mount systems in a data center, where the temperature is controlled. I think the test method should replicate the operating conditions as closely as practicable. A further concern wit the JESD218 is the assumption that accelerated testing is valid, and that the Arrhenius model is valid with a 1.1eV activation energy. You can tell from the surface equation of chapter 5 that the observed temperature dependence is not Arrhenius. I will cover this in detail in a later chapter.

In enterprise applications, SSDs aren’t usually “rode hard and put up wet”. That is to say, they are aren’t used heavily and then left idle. This might be more true of a light IT workload, such as a laptop, where a device might be used for a short period,with long idle periods. However, if the laptop was used hard, then shut down, it might be that the SSD is “put up wet” in that it has no time to perform clean up post activity once the power is removed. It might also heat up for a while as the cooling fans are turned off.

When I presented some of this data at the Non-volatile Memories Workshop in March 2012, I had a couple of people from flash vendors offer the opinion that my test methodology was flawed. I won’t identify them or their companies, but they said measuring non-recoverable read errors (sector failures) was not a valid method for determining SSD reliability. It is clear to me that measuring sector failures is a valid measure of system reliability. NRRE tests of storage media reliability in systems has been used for decades (hard disk, tape, etc.). (If a retention failure doesn’t result in a sector loss, then what exactly is the error?) I bring this anecdote up because I think it will help my readers understand why there may be large differences between how the SSDs behave and how they are specified. If the manufacturers don’t believe measuring non-recoverable read errors is valid, then perhaps they themselves don’t do it. This by itself could explain how the measured behavior deviates from the specification.

JEDEC Solid State Technology Association “Solid-State Drive (SSD) Requirements and Endurance Test Method”, JESD218 Sep. 2010.

Series SDD 5: The error rate surface for MLC NAND flash

Update July 2014: modified bit-error equation

I have modified the MLC equation based on examination of further data.

Expected behavior of the  MLC error rate surface

NAND flash has rather complex bit error rate behavior compared with magnetic recording. In the case of hard disks, the bit error rate behavior tends to be a constant, without strong dependence on other factors. For a given bit, the error rate doesn’t depend on the number of write cycles or the age of the data. Unfortunately, the same can’t be said for flash. Flash has a complex multi-dimensional error rate surface. Continue reading

Series SSD: 4. SSD Wear Leveling and Reliability

SSD Wear Leveling and Reliability

In the prior post, we explored issues with SSD NRRE specifications and testing. In this post, we’ll take a look at how the internal architecture of SSDs affects reliability and the ability of integrators to test flash.

NAND flash has a few limitations that must be overcome to create a useful storage device. The memory cells have finite write endurance and finite data retention, and don’t support direct overwrite. SSD vendors use a technique called wear leveling to address these issues. The goal is to increase the useful lifetime of the device. However, according to the 2nd law of thermodynamics, there is no such thing as a free lunch. (OK, that’s not precisely what is says, but this result can probably be derived from the 2nd law.) Thus, we should expect side effects! We shall examine these here in detail.

Continue reading

Series SSD: 3. The Problem with SSD Specifications

Common oversights with SSD specifications

It is commonly assumed that solid state disks are more reliable than hard disks because they have no moving parts. While the evidence for this is lacking, we’ll look at the impact of non-recoverable read errors on reliability. Continue reading

I have seen failure analysis reports on many hard disk programs over  the years. Typically, mechanical related failures account for less than half the total. Electronic and microcode failures are quite common (ever had a cell phone break?). So, the non-mechanical argument needs to be backed up with field data.

Series SSD: Solid State Disks in Storage Systems

I will cover a number of topics relating to solid state disks (SSDs) in storage systems in a series of posts. My hope is to stimulate discussion in the industry, and to encourage the open exchange of reliability information for solid state disks.

Disclosure: I previously worked in the HDD business, and have a background in solid state physics. However, I have been working on storage systems for at least a dozen years and have no vested interest in one technology over another.

Solid state storage is beginning to be widely deployed in IT storage systems. NAND flash has largely displaced hard disk drives (HDD) in mobile consumer applications, effectively eliminating the use of HDDs in the sub 1.8” form factors. Based on these events, it is commonly assumed that solid state storage is poised to make substantial inroads in the IT storage space. However, there are both technological and business constraints that dictate how this process will proceed. In this series, I will focus on the technological constraints as they relate to storage systems.


Unfortunately, most of the industry seems to be more concerned with the performance of solid state disks, taking the reliability for granted. There is a dearth of published information on SSD reliability, and I aim to rectify that here. In this series, I will examine the technological impacts of solid state storage on systems. I will show how to determine which parameters are important for system reliability. Actual device data will be presented and the impact on reliability analyzed. We will find out if the reliability of SSDs is as great as widely assumed.


Another topic for exploration is the testability of SSDs. The current crop of devices leave much to be desired in this regard. In fact, I am amazed that system vendors who would never  accept a hard disk that they could not be tested, would willing accept such SSDs. Perhaps this is due to the small volumes of SSDs. However, I do not expect this situation to last, and believe there should be an industry wide requirement for fully testable SSDs.


Over the course of the series, I will share a wealth of data that I have gathered from SSDs. Much of this data will be raw, and I haven’t had time to analyze all of it. Thus, I encourage the community to help in this regard. It is my hope that this presentation will be valuable to the industry, and I encourage SSD vendors and flash manufacturers to openly share their data.

I will not identify particular component vendors here – so don’t ask me to. It isn’t my intention to cast aspersions on any manufacturer. I would much prefer them to be more open with such data. I have looked at a number of vendors’ parts, so if you recognize yours, you are welcome to  share this information.


I plan to post a new section each week. Here is the current planned outline for this series, although this is subject to change:

  1. Hard errors and reliability – things that go bump in the drive
  2. System level reliability targets – it’s not what you think
  3. The problem with SSD specifications – speed kills
  4. SSD wear leveling and reliability – in flash errors never sleep
  5. The error rate surface for multi-level cell flash – and an equation in 4 dimensions
  6. The test methodology – how we get all the data
  7. Bit error rate: cycling data (endurance) – we see error fountains
  8. Flash temperature data – differences from the Arrhenius model
  9. Bit error rate: aging data (retention) – ski jumps
  10. Bit error rate: dwell time – into the fifth dimension
  11. Error uniformity – is the the mean meaningless?
  12. Error fountains –  wronging a few writes
  13. Error uniformity – population effects
  14. Caching with MLC flash – playing to your strengths
  15. A proposal for a testable flash device

I will continue to to run reliability test on SSDs during the course of this series, and am open to suggestions for modifications to the tests. I have much more data than I can possibly share on this site. If anyone in academia would like to help analyze the data, please contact me and we’ll see if we can arrange something.

Some of this information has been previously presented in my tutorials at  FAST ’11 and NVMW 2012 .