NetApps

In 1519, Hernando Cortez began his conquest of Mexico. He “motivated” his soldiers to fight by burning all his ships, making retreat impossible.

Here’s the thing: It’s tough being a big boss. You can try ordering people around, but it’s hard to make hundreds or thousands of people do what you want. Sometimes they don’t pay attention or don’t understand. Sometimes they hear you clearly but decide to do something different anyway. Cortez’s management technique left no escape: fight or die.

Several years ago, our CEO Dan Warmenhoven applied this technique to force NetApp managers to hire more people outside of Sunnyvale. We had development centers in Research Triangle Park in North Carolina, in Pittsburgh, in Bangalore India, but most of our employees were in Sunnyvale and their natural inclination was to hire more people in Sunnyvale. I remember the executive staff meeting where our VP of facilities was explaining why we needed to start construction on a new building in Sunnyvale: “It will take 18 to 24 months to complete construction, and at the rate we’re growing, we must start now to be ready.” Dan replied that he didn’t want to continue the same growth rate in Sunnyvale, but the head of facilities pointed out that Dan had been saying this for years, and the hiring rate in Sunnyvale was faster than ever. “We really need to start those buildings now.”

Dan has this little vein in his forehead that pops up when he is frustrated: it pulsed pinkly. “If we build more buildings, the hiring will get faster still. I don’t want to hear another proposal for buildings in Sunnyvale for at least 12 months. If managers hire more people in Sunnyvale, they’ll have to work standing in hallways.” Cortez management in action. We eventually did do more construction in Sunnyvale, years later, but not until the hiring trajectory had changed dramatically.

In many companies, the tightest control systems are for large capital purchases. Perhaps bosses can’t control what workers do on a day-to-day basis, but they can control what gets purchased.

This was a common tool in the transition from Mainframes to UNIX servers in the eighties and nineties. CIOs said, “It’s not that I don’t like mainframes, but they’re too expensive, and I won’t buy another one.” The boss didn’t need to specify details about how many UNIX systems to purchase or what applications to move over. One top-level decision forced the transition, and the IT staff handled the details on their own.

A popular use of Cortez management these days is for CIOs to announce that they aren’t going to build another data center: “I know the current data center is almost full and has no more power, but I’m not building another one. Figure it out.” This is the best way I know for a CIO to drive faster adoption of Server Virtualization.

The physical version of my book is now available for pre-order on Amazon, but they don’t have the Kindle edition. Being the Kindle-loving geek/nerd that I am, I’m trying to light a fire under my publisher, so I’d love it if people would help me by going to the Amazon page for How To Castrate A Bull: Unexpected Lessons on Risk, Growth, and Success in Business and clicking on the link that says, “Please tell the publisher I’d like to read this book on Kindle.” It’s on the left side, just below the picture of the front cover.

There’s even a link to read a random page. I wonder if anyone has written a script to click a bazillion times and download whole books for free? You can also do searches. (I recommend “thud” or “testicles”.)

Many people think WAFL is a filesystem. I certainly thought so fifteen years ago when I wrote it, but folks like Kostadis Roussos are now claiming that I was wrong. (A NetApp employee no less!)

To understand why, you have to understand how WAFL is structured.

WAFL has a “top-half” that deals with files and folders—“My Documents” and “QuarterlyEarnings.ppt”. It keeps track of who created the file, when they created it, who can look at the file, who can modify it, and so on. This certainly sounds like what a filesystem would do. The top-half actually supports multiple filesystem protocols. We started with NFS for UNIX, but right from the beginning we knew we’d be adding more. Originally we expected that Novell Netware would be next, but Windows CIFS gained momentum so quickly that we did that instead and never got around to Netware.

WAFL also has a “bottom-half” that manages the physical disks in the system. It organizes the disks into separately managed pools, it keeps track of which disks are part of which RAID array, and it arranges data on the disk to maximize read and write performance. The “bottom half” also handles data management functions like snapshots, remote mirrors, cloning, de-duplication, thin provisioning, and so on. These capabilities would traditionally be part of a volume manager or a block virtualization layer.

One of the things that was unique about WAFL when we shipped it was the way we integrated the two layers. There are many opportunities for new features and clever optimization if you design the layers to work well together.

When we decided to support iSCSI and Fibre Channel SAN, WAFL’s bottom-half data management capabilities were a perfect fit. In fact, one of the things that helped convince me that NetApp should support block-based storage was realizing how valuable our data management features would be in that environment.

This top-half/bottom-half structure explains the confusion about WAFL. My current view is that WAFL contains a filesystem, multiple filesystems actually, but that’s different from being a filesystem.

Someone asked me the other day whether our patent infringement claims against Sun (see here, here, and here) include Sun’s “Amber Road” storage system. I said "Of course!" Commercial products that use ZFS are exactly the point of our lawsuit. Sun likes to portray this as a dispute about open source, and ZFS being open source does complicate things, but the reality is that this lawsuit is about Sun building and shipping products based on our intellectual property. At this point, we have sixteen patents that—we believe—ZFS infringes on. So naturally, the case we are making applies to any product from Sun (or any other company) that is based on ZFS, including Amber Road.

Almost every measure of computer performance increases exponentially. Here is one important exception: disk drives keep getting bigger, but they are not getting much faster. As a result, the number of seeks-per-second available for each gigabyte of data (seeks/second/GB) is plummeting.

Let me give a concrete example. Fifteen years ago, a typical one-gigabyte disk drive had a seek time of ten-milliseconds or so. Do that math, and that’s 100 seeks-per-second for that one gigabyte. Today, one-terabyte drives have maybe five-millisecond seek time. Do the math, and that’s 200 seeks-per-second for the whole terabyte, or just 0.2 seeks/second per gigabyte. Over the past fifteen years, your ability to access the data you own has gone down by a factor of 500.

That’s bad enough, but if you consider how many CPU instructions you can execute while you wait for the disk drive, things are even worse. Fifteen years ago, the Intel 486DX could do 54 MIPS (million instructions per second). Today Intel’s QX9770 can do 59,455 MIPS. For every millisecond you wait, today’s chips execute 1000 times more instructions.

Consider these facts together: From a human perspective, seeks/second/GB has gone down by a factor of a five-hundred, but from the CPU’s perspective, it is five-hundred-thousand times slower! Remember those old mainframe computers with tape drives on the front, jerking back and forth? To the CPU, tape drives back then seemed faster than disk drives today.

Disks are the new tape—worse than tape used to be—so we obviously need to find a new disk. That’s where flash comes in: ten times more expensive than disk, but for random access, a hundred times faster. For now, flash is mostly used in small portable devices, or for very high performance, but if you look forward five or ten years, I predict that people will think of flash the way they think of disks today—as the fast-access storage for everyday use. And they’ll think of disks as being more like tape—a slow-access media that’s useful for stuff you might want to look at someday, but not particularly useful for data you use all the time.

Flash is too expensive to replace disk right away, so first we’ll see a new generation of storage systems that combine the two: flash for performance and disk for capacity. I hesitate to compare this with the old HSM (hierarchical storage management) solutions that combined disk with tape, because those were so sucky—things that would take seconds on disk could sometimes take hours. They only worked for a limited subset of applications, and only with intricate and painful management. Fortunately, the performance ratio between flash and disk is much better than for disk and tape, so we will be able to automate the management and still get good performance. HSM without the M, if you will.

(This article is mostly about the industry in general. For more on what NetApp is doing, check here and here.)

I don’t intend to blog on every little step in the Sun/NetApp lawsuit, but a ruling this week really supports the point that I made in my last blog entry: pre-trial rulings aren’t usually a good way to judge the final outcome of a case.

Mike Dillon, Sun’s general counsel, posted a couple of blogs (here and here), which implied that Sun was winning because the patent office issued preliminary rejections on a handful of NetApp's sixteen different patents in the case. Sun requested that we delay until the patent office had a chance to do a final review.

In this week's ruling, the judge denied Sun’s request, noting that the patent office “almost always grants initial rejections” so “the Court gets only limited guidance” from these rejections. Waiting for the patent office makes no sense because software patents like this “take longer than other groups to result in a final action” and “it is unclear at best whether the PTO will keep pace with the increase in reexamination requests.”

My point is not to prove who is winning or losing. Quite the opposite: My point is that you generally shouldn’t pay too much attention to pre-trial posturing by participants in a court case.

Judging a trial based on these early skirmishes is like judging a football game based on the coin-toss.

I hadn’t planned to blog any more on the Sun/NetApp patent litigation, but Mike Dillon, Sun’s General Counsel, has recently made some very optimistic posts (here and here), and I need to respond. Dillon shared the results of some pre-trial wrangling—Markman hearings and preliminary Patent Office decisions—but sometimes when you focus on intricate details, you miss the big picture. (For background on the lawsuit, see here, here, and here.)

To me, the best indicator of strength is to look at which party wants to get on with the case (the one with a strong position), and which party consistently drags its feet and tries to delay (the one with the weak position).

Sun is requesting a “stay,” which is a request to put our claims on hold and delay the trial, because the Patent Office has issued a preliminary rejection of claims in 3 of our patents (out of 16). Such a ruling is not unusual for patents being tried for the first time, and there are two ways to resolve the issue. Waiting for the Patent Office, which is what Sun wants to do, is the slow way. The patent office currently has a backlog of 730,000 patents, and they can’t hire fast enough to close the gap. Waiting could take years. The legal system isn’t always fast, but it can be. When NetApp agreed to relocate the case to California, we did it on the condition that we’d get to trial relatively quickly. Dillon mentioned issues with three patents, but NetApp currently has 16 WAFL patents that we believe apply to ZFS, with more on the way. We believe that we have a strong case, and we want to get it resolved.

As I have said in earlier blogs, I like open source. In fact, I have personally written and contributed open source code! NetApp has also made many corporate contributions. However, I believe that open source contributors must always follow one critical guideline: Only give away things that belong to you. This is where we think that Sun isn’t playing by the rules.

Here’s the big picture. If you were Sun, and if you were confident in your case, wouldn’t you want to clear the name of ZFS as quickly as possible, to reassure your customers and partners? By contrast, if you were NetApp, and you had no confidence in your patents, wouldn’t you try to slow things down to maintain the cloud of doubt as long as possible? I believe we have a strong case, but whether we are right or wrong, isn’t it best for everyone involved to get the answer as quickly as possible?

In the past two weeks, I've had lots of people ask me how I think the financial meltdown will affect things. I don't have a crystal ball, but I thought it might be interesting to look back to the tech crash in 2000/2001.

I remember one of our executive staff meetings in particular, where it became clear how bad things were getting. One of the topics of the meeting was how sales were going, and Rob Salmon, who ran world-wide sales at the time, described an ugly picture. We were still winning deals, at least according to the lower level decision makers, but when it came time to collect the purchase order, we would find out that the CFO, or even the CEO, had frozen the funds at the last minute. It wasn't that anyone else was taking the business: the business was simply disappearing, or at least being delayed.

The executive staff meeting continued, and an hour or so later we had a status update on a big business software project that we were working on. I can't remember exactly what it was—ERP or CRM or something like that. Anyway, about twenty minutes into the conversation, Dan, our CEO, interrupted and said, "I don't think it makes sense to do this right now. It's just too expensive, and given the economic situation, too risky." The IT people giving the presentation said, "But it’s been approved. We already committed to our vendor!" Dan's response was, "It's not too late to cancel." That was that.

From the other side of the room, Rob Salmon groaned. He said, "I bet this is exactly the same conversation that is going on at our customers, right before they tell us that the deal we thought we had won just disappeared.

That was in 2001, but I have a hunch that the same conversation is going on in boardrooms all around the country. All around the world, for that matter. Remember, it's only two weeks ago Monday that Lehman collapsed. It seems pretty likely that CEOs and CFOs will reconsider, or at least delay, any big decisions that they can, even if they haven’t told their staff yet.

I’m curious about peoples’ experiences at different companies. What has the boss said so far?

When we were first starting NetApp, back in 1992, we thought that we could create a product and a company on an unusually small amount of money. We also thought our low cost would give us an advantage in trying to raise Venture Capital (VC) money. We were half right.

It took us just over a year from when we incorporated to ship product, and we had raised only $1.6 million dollars. To put that in context, some start-ups have raised over $100 million to enter the storage market and still not gone public. We shipped our first product with just eight full time employees and only three full time development engineers. The eight were a CEO, head of sales (the only sales person), head of marketing (the only marketing person), head of customer support (the only support person), an office manager, and three engineers including both James and me. We also had three part-time consultants—one on manufacturing and two more engineers.

We were right about building a product with surprisingly little cash, but we were wrong in thinking that VCs would find this attractive. VCs don’t particularly like small investments. In fact, we had to raise that first $1.6 million entirely through “angel investors”—rich individuals who invest their own money. The VCs wouldn’t invest until after we’d shipped product. It took another $10 million or so for us to go public.

In business, it’s always helpful to understand the other person’s perspective, so let me describe how the world appears if you are a VC. A VC fund is a lot like a mutual fund, except that instead of getting money from regular people, you get it from university endowments, pension funds, really rich people, and so on. And instead of buying stock in public companies, you invest in start-ups. Your job, as a VC, is to invest all that money you’ve raised, and the funds are sometimes very large. Some funds are a few hundred million dollars, some multi-billion dollars.

The problem is, it’s just not easy to invest that much money, and it’s almost impossible to invest that much money in small chunks. I recently had lunch with the CEO of a small start-up, and he told me that he was looking to raise $7 million, but one investor said, “Couldn’t you take $13 million?”

It boggles most people’s minds to imagine investing a hundred million dollars, much less a billion, so let’s imagine that you have $100 to invest, and you are trying to figure out what stocks to buy. (Me personally—I recommend index funds, but let’s just assume that you have your heart set on stocks.) It wouldn’t make sense to invest $1 each in a hundred different companies. That would take way too much research and would be a pain to manage. You’d probably be completely uninterested in a fifty-cent investment, no matter how good it sounded.

That’s exactly how a VC feels when a start-up comes in and says, “We’ve got this great idea, and best of all, we’ll only need a million or two.” VCs typically go on the boards of all the companies they invest in, and the last thing they want is to be on a hundred boards, or even fifty. After all, that might distract from the summer in Europe and the winter in the Caribbean. It’s much, much easier to invest larger chunks of money in a smaller number of companies.

A friend of mine goaded me into buying a Kindle—Amazon’s electronic book reader. “You are supposed to be some kind of high-tech visionary. Electronic books are the future! How can you not own a Kindle? You ought to be ashamed of yourself.”

Thus chastened, I raced to my Amazon account and 1-clicked—$359. Ouch. I wonder how long till it’s below $100.

I have to say, I was surprised by how good it is. I’d read reviews describing it as “clunky and ugly”, but I found it to be light, easy to use, and—most important—the electronic ink is very easy to read, even in bright sunlight. That was critical during my beach vacation earlier this summer. Unlike LCD, it’s not backlit and has no funny polarizing effects with sunglasses; it feels just like ink on paper.

Another friend told me, “My Kindle completely changed the way I read.” That surprised me, so I asked him to explain. He said he used to read magazines in bed, because they are light and easy to hold, but books, especially hardcover books, are heavier and a bit awkward. The Kindle converted him from magazines (candy), to books (meat), and he’s happier for it.

One of the knocks on e-books is that people like the feel of a real book with real pages to turn. People worry that an electronic gadget will get in the way of reading. Perhaps that’s why I hadn’t bought an e-book. I can report that after the first couple books, the Kindle “disappeared”—it vanished from my conscious awareness. For people who like flashy gadgets, that may be a disadvantage, but as an avid reader, I thought that’s how it should be. Interestingly, the Kindle doesn’t have the same addictive attraction for me as a smart phone or a web browser. Maybe because the book just sits there, waiting for me to turn the page—no new-message alerts or latest news flashes.

Part of what appeals to me is the Kindle’s appliance nature. The Kindle sucks if you think of it as a small computer, PDA, or web browser: the keyboard is clunky, the display updates oddly, and browsing speed is slow. But for reading a book, the Kindle is absolutely perfect. An appliance does just one thing, and it does it well.

What’s most innovative about the Kindle is not the device itself, but the way it integrates so cleanly with the cellular network and with my Amazon account. Even though it downloads books using cell-phone technology, Amazon pays the bandwidth cost—pennies per book—directly to the phone company, so you don’t have to deal with yet another cell phone account. There was no setup at all: it said “Dave’s Kindle” as soon as I turned it on. (It knew because I had ordered it with my Amazon account.) You can order books from the Kindle itself, via a somewhat clunky web-like interface, or you can use your regular Amazon account. If you buy the “Kindle edition”, listed right next to “hardcopy”, “paperback”, and “audiobook”, it downloads automatically. And Amazon remembers what you’ve already purchased, so no backup is necessary.

This kind of tight integration can radically change the rules of an industry. It’s what Gary Hamel calls “strategic innovation” in his book The Future of Management. He contrasts it with “product innovation”. Folks who have tried the Sony Reader tell me that it’s absolutely beautiful—much sexier than the Kindle—but to download books you have to cable it up to your PC and go to Sony’s special web page. Great product innovation, not great strategic innovation. It’s like the difference between a generic MP3 player and how Apple has integrated the iPod with the iTunes store.

My favorite feature is that there is an email address where I can send JPEGs or Word documents to my Kindle. Whatever I send to dave_hitz@kindle.com shows up on the Kindle in seconds. The book I just finished writing won’t be published until January, but I sent the manuscript to my Kindle, and it made me feel all official and important seeing it there with the other “real” books!