Solaris

• Propaganda war: trusting what we see?
  Interesting, insightful and informative article examining the propaganda war around the events in Gaza.
• A prediction that's a safe bet
  "Excess wealth is gone like the codpiece. The free market will continue but any respect for the idea of free money is all over."
• Amtrak photo contestant arrested by Amtrak police in NYC’s Penn Station
  What for? Why, for trying to enter the competition by taking a photograph, of course! Personally I think the competition is a cunning scheme by Amtrak to flush out all the subversives so the police can arrest them and snuff out this disgusting train photography hobby for once and for all.
• OpenTable: So Web 1.0 It Hurts.
  Mirrors my experience. OpenTable is definitely ripe for a competitor that is about gathering an epicurean community and delivering it to deserving restaurants.
• Cornyn promises filibuster on Franken
  Any system contains within it the games that will be used to play it. Corollary: The more complex the system, the easier it is to find ways to game it. Corollary: Complex systems get gamed for longer than simple systems.

If you haven't seen it, DTrace is now shipping in Mac OS X Leopard. This is very exciting for us here at Sun, but one could be forgiven for asking an obvious question: why? How is having our technology in Leopard (which, if Ars Technica is to be believed, is "perhaps the most significant change in the Leopard kernel") helping Sun? More bluntly, haven't we in Sun handed Apple a piece of technology that we could have sold them instead? The answer to these questions -- which are very similar in spirit to the questions that were asked over and over again internally as we prepared to open source Solaris -- is that they are simply the wrong questions.

The thrust of the business questions around open source should not be "how does this directly help me?" or "can't I sell them something instead?" but rather "how much does it cost me?" and "does it hurt me?" Why must one shift the bias of the questions? Because open source often helps in much more profound (and unanticipated) ways than just this quarter's numbers; one must look at open source as long term strategy rather than short term tactics. And as for the intense (and natural) desire to sell a technology instead of giving away the source code, one has to understand that the choice is not between "I give a customer my technology" and "I sell a customer my technology", but rather between "a customer that I never would have had uses my technology" and "a customer that I never would have had uses someone else's technology." When one thinks of open source in this way, the business case becomes much clearer -- but this still may be a bit abstract, so let's apply these questions to the specific case of DTrace in Leopard...

The first question is "how much did it cost Sun to get DTrace on Leopard?" The answer to this first question is that it cost Sun just about nothing. And not metaphorical nothing -- I'm talking actual, literal nothing: Adam, Mike and I had essentially one meeting with the Apple folks, answering some questions that we would have answered for anyone anyway. But answering questions doesn't ship product; how could the presence of our software in another product cost us nothing? This is possible because of that most beautiful property of software: it has no variable cost; the only meaningful costs associated with software are fixed costs, and those costs were all borne by Apple. Indeed, it has cost Sun more money in terms of my time to blog how this didn't cost anything to Sun than it did in fact cost Sun in the first place...

With that question answered, the second question is "does the presence of DTrace on Leopard hurt Sun?" The answer is that it's very hard to come up with a situation whereby this hurts Sun: one would have to invent a fictitious customer who is happily buying Sun servers and support -- but only because they can't get DTrace on their beloved Mac OS X. In fact, this mythical customer apparently hates Sun (but paradoxically loves DTrace?) so much that they're willing to throw out all of their Sun and Solaris investment over a single technology -- and one that is present in both systems no less. Even leaving aside that Solaris and Mac OS X are not direct competitors, this just doesn't add up -- or at least, it adds up to such a strange, irrational customer that you'll find them in the ones and twos, not the thousands or millions.

But haven't we lost some competitive advantage to Apple? Doesn't that hurt Sun? The answer, again, is no. If you love DTrace (and again, that must be presupposed in the question -- if DTrace means nothing to you, then its presence in Mac OS X also means nothing to you), then you are that much more likely to look at (and embrace) other perhaps less ballyhooed Solaris
technologies like SMF, FMA, Zones, least-privilege, etc. That is, the kind of technologist who appreciates DTrace is also likely to appreciate the competitive advantages of Solaris that run far, far beyond merely DTrace -- and that appreciation is not likely to be changed by the presence of DTrace in another system.

Okay, so this doesn't cost Sun anything, and it doesn't hurt Sun. Once one accepts that, one is open to a much more interesting and meaningful question: namely, does this help Sun? Does it help Sun to have our technology -- especially a revolutionary one -- present in other systems? The answer is "you bet!" There are of course some general, abstract ways that it helps -- it grows our DTrace community, it creates larger markets for our partners and ISVs that wish to offer DTrace-based solutions and services, etc. But there are also more specific, concrete ways: for example, how could it not help Solaris to have Ruby developers (the vast majority of whom develop on Mac OS X) become accustomed to using DTrace to debug their Rails app? Today, Rails apps are generally developed on Mac OS X and deployed on Linux -- but one can make a very, very plausible argument that getting Rails developers hooked on DTrace on the development side could well the change the dynamics on the deployment side. (After all, DTrace + Leopard + Ruby-on-Rails is crazy delicious!) This all serves as an object lesson of how unanticipatable the benefits of open source can be: despite extensive war-gaming, no one at Sun anticipated that open sourcing DTrace would allow it to be used to Sun's advantage on a hot web development platform running on a hip development system, neither of which originated at Sun.

And the DTrace/Leopard/Ruby triumvirate points to a more profound change: the presence of DTrace in other systems assures that it transcends a company or its products -- that it moves beyond a mere a feature, and becomes a technological advance. As such, you can be sure that systems that lack DTrace will become increasingly unacceptable over time. DTrace's shift from product to technological advance -- just like the shifts in NFS or Java before it -- is decidedly and indisputably in Sun's interest, and indeed it embodies the value proposition of the open systems vision that started our shop in the first place. So here's to DTrace on Leopard, long may it reign!

Vacation over, left the mountains, back to work tomorrow. Good to get away for a bit, but it was too short. I have a million emails to delete. Better get started ...

For once, "Tokyo" refers in fact to a physical place, not some code project. Shocking but true.

Just prior to Christmas, I took a week-long trip to Tokyo Japan on Joyent business. This was interesting for me because it was both my first time to Japan and in fact first time to leave the country. Given that I am a California native, I've had little reason to leave. We commonly say here "Your within a 4 hour drive of almost any environment on earth". California is just a great place and I figured if I ever did leave the country, it should be some place particularly interesting, not just Mexico or Canada.

The first thing about traveling to Japan is that jet lag sucks and travel is painful. Sure, like everone has a tragic "I took a 36 hour flight" story, but a 12 hour flight in coach just sucks. The flight is 12 hours there and 9 hours back, thanks to trade winds... shocking that they take a full 3 hours off, but its true. When you take all travel concerns into account (including, in my case, a connection through LAX) I lost about 3 days to travel. I wanted to go to the Tokyo OpenSolaris Users Group OpenSolaris 2008.11 release event, and it was funny that I was scrambling Wed morning (PST) to make it in time for a Thursday evening (JST) event.

Once you get there, "jet lag" takes on a new meaning. Typically I think of "jet lag" as a minor diviation of your sleep schedule, like going coast-to-coast. But in Japan the time is so off, that you get hit hard about 5PM (JST) and then get a second wind around 7PM and then have trouble sleeping till 3-4AM. The first morning I was there I woke up at 5AM and by 6 gave up on trying to sleep.

While I can't talk much about my work there, I was in a data center for 2 days straight, then did a hand off to our other staff at home while we were on standby for another 2 days. We used that time for customer meetings and taking in as much of Tokyo as possible. Lesson to my fellow administrators, when your in a strange place and up against a deadline... pre-stage, pre-stage, pre-stage. I actually took a 2.5" USB powered drive with ZFS Datasets ready for mount and use. ZFS rules.

Anyway... I thought I'd share some miscellaneous thoughts in general about Tokyo for those who've never ventured to Japan:

1. They say that going to Japan is like going to another planet. Not true. It was very much like any other large metropolis... people just don't speak English.
2. I was told, that in a city like Tokyo which does a lot of international business that most people know english pretty well. Bullshit. In the large hotels in Shinjuku, ya, but everywhere else they don't know english. Due to the ammount that Japanese culture has integrated english words, they might know a couple of words, but it really comes down to hand jestures. If you walk into McDonalds and say "how are you today?" you get a blank smile. In my hotel (a really nice one actually, in Ariake, even the front desk barely knew any english.)
3. "Large Coffee" in Japanese is "Oh-key ko-he"... life was difficult before this.
4. You always see Japanese crowded into packed areas in the media, so you think Japanese like being crowded. Wrong. They like space too... but when you need to rely on public transportation to get anywhere and you can squeeze into a train, you bear it and cram.
5. Japanese don't look at anyone else. At least, young people don't. In America we're constantly sizing up everyone around us, looking, thinking, perhaps even commenting.... not in Japan. In America if you walk past someone that is alone, you commonly say something like "hey", "yo", "hows it going?", nod, or otherwise acknowledge their existence. In Japan you can be around hundreds of people and feel absolutely isolated and alone. Consequently, its a really depressing and lonely place if your alone.
6. ...unless you wear a kilt. I wore a kilt one day when there and people couldn't believe what they were seeing, women especially. After 3 days of feeling like I didn't exist this was a welcome reaffirmation of my humanity. :)
7. Elderly Japanese (70+?) are much more friendly... they'll commonly give you a smile or say something back if you say hello (in Japanese obviously).
8. The American understanding of "Hello" in Japanese is "Konichiwa"... but in fact, that means "Good Afternoon". There are variations for morning, afternoon and evening. Commonly this is followed with the word "gozaimasu", which adds some formality, like saying "Good morning sir" instead of "Morning" ("Ohayoo gozaimasu").
9. Japanese pronunciation is more important than even the words themselves. I asked the from desk where I could find a "Key-mo-noh" (Kimono)... this turned into a confusing number of jestures and ultimately a dash for a Casio pocket translator. The word was right but due to my bad pronunciation we could not connect.
10. In America we give people a hard time about "butchering our language"... if felt somehow redeeming to have people giving me a look of dispare and amusement as I butchered theirs.
11. Learning Japanese is really tough. Pronunciation is the key to spoken Japanese... but writing is a whole seperate problem, as they have 3 seperate major writing systems Kanji (iconic, drawn from Chinese), Katakana (syllabic, meaning characters that you can sound out), and Hiragana (the American equivalent is cursive). The kick in the teeth is that commonly in Japanese they will use all 3 in a single sentence.
12. Tokyo is huge. Taxi's are expensive, especially if your traveling more than a couple miles. Supposedly a taxi ride from the Narita airport on the edge of town (feels way out of town actually) to the heart of the city will run you US$500 and takes about an hour.
13. Navigating trains in Tokyo is really complex. There are hundreds of stops and the kicker is that unlike most places there is not a single central train authority that runs all the trains.... there are several different train companies with their own lines, so you commonly cross over from one to another. As a result there were many people who have lived there for 5+ years and had considerable trouble navigating the train system unless they were familiar with that particular route.
14. Tokyo is clean. Super clean. And, ironically, finding a trash can is hard to do. All the taxi's and buses have clean white doily things on the head-rests, and people just don't litter. You see the occasional cigarette butt, but thats about it.
15. Bathrooms are fun in Japan. They use electric dryers exclusively, commonly a "toaster" like contraption in which you insert your hands, and a stream of high-pressure air blows across your hands as you slowly pull them up... bone dry hands, totally awesome. Even bathrooms in Japan don't have trash-cans.
16. Toto toilets are scary and wonderful things. You know, you've seen those images of Japanese toilets with an instrument panel right? I could write a whole series just on those things, but needless to say the first time you sit down on a toilet seat thats warm, it freaks you out.
17. The ability to order Sushi like a pro in the US doesn't mean jack sh*t in Japan.
18. All Japanese are short. Totally wrong. I'm 6'4", everyone wanted pictures of me towering over the little Japanese. Just plain wrong, I didn't notice any difference between California and Japan in terms of variation in height. In fact, there were several Japanese construction workers that were massive and definitely not to me messed with.
19. If the Toyota released all their japanese cars in the US, GM and Ford would be out of business. I saw several Toyota's that put Mercedes to shame. You have to see it to believe it.
20. Japanese quality is awesome. If I traveled there regularly I'd probly buy all my clothes in Japan.
21. Adjusting to coinage is odd. The smallest Japanese bill is 1,000 yen (round it to US$10; less due to conversion, but ballpark). $5 and down is all coinage. In the US we tend to discard change (collected in jars, or whatever)... but there, you have to adjust to using coinage frequently or you walk around with a bulging pocket all the time.
22. Mint... apparently mint isn't big in Japan, you don't hardly see it. If its green its almost certainly green tea flavored. Strawberry, however, is very popular.
23. Japanese aren't big on candybars or chocolate in general. At least, not like we are in the US. In a mini-mart in the US we have one or more isles dedicated just to chocolate, commonly in candy-bar form. Over there you find only a couple varieties. Kit Kat and Snickers are the only US bars I saw.
24. Yes, Hentai is as common as they say. Also, Japanese Manga is telephone book sized, not little things like we read in the US.
25. Strange observation... I was hard pressed to find a Japanese magazine about business or computers. I found one magazine about PC's, but most were about TV or culture. I wanted to pick up some economic/news magazines but couldn't find 'em.
26. Vending Machines. You hear that they are everywhere. This is true, there is almost always one within eye shot... however the notion that you can "buy anything in a vending machine" is overblown. Most of the vending machines were just drinks and maybe a can of nuts or something. I didn't see any vending machines for portable electronics, or books, or all the wierd stuff you hear about. I'm sure they exist, but some people make it sounds like you can buy a Sony Walkman in a vending machine in the middle of a park.
27. Dress. Dress varies based on what area ("Ward") of Tokyo you are in, but in general they dress much nicer than in the US. Men most commonly wear a 2 button suit. Young women wear short skirts with knee or thigh high tights and either leg-warmers or tall boots. Teenage boys tend toward jeans and a tshirt.
28. Video Games. If you walk into an arcade, all the arcades are played sitting down! What we commonly consider an "up-right" game, has a little bench. The "crane-pickup" games are really popular and have kool prizes. One arcade had these games filled with food items like ice-cream bars and such.
29. Couples. I was really amazed at how many couples I saw! In the US its generally difficult to tell who is a couple because we've lost the tradition of holding hands. A man and women in San Francisco exiting a restaurants may be a couple, or brother-sister, or friends, or co-workers... its hard to tell. In Tokyo there were tons of couples holding hands and cuddling on trains.
30. Gambling. Gambling is big in Tokyo. Commonly in the form of slot machines and a game called "Pachinko". They don't have card games, and thus most people didn't seem to think of it as gambling, but these things are eveywhere!
31. Mini-marts. Mini-marts are big there, particularly 7-11 and Circle K. People buy lunch, breakfast, and dinner at these places, typically before or after getting on a train. They sell a lot of Ramen (yes, they do sell "Cup o' Noodle" in Japan) and provide hot-water to fill it up before leaving. Other meal items include every variation of rice and seafood you can think of, including sushi.
32. Sushi. I wondered how much better sushi was there than here. I wasn't shocked, the sushi in Japan is unlike anything you've had in the US. I've eaten at some of the high-end places in San Francisco and they don't come close to your average box-lunch sushi there.

    I could go on for a while but will leave it there. I commonly reflected on the movie "Lost in Transation" while in Tokyo. I even got to quickly venture into Shinjuku to the Tokyo Hyatt where it was largely filmed (the "bar" that he hangs out in has a 1,000 Yen cover charge JUST to sit there. A Guinness in a pub can cost me 1800 yen. But man oh man it was a beautiful lounge.) The theme of being disconnected and alone in Tokyo rings true from the film.

    I didn't get to see as much of the city as I wanted to. I especially wish I'd had time to see the legendary Akihabara (Japanese Geek Central), but time didn't permit. None-the-less I'm happy with what I was able to take in. We spent one day without a guide just taking the train some place and exploring around the station, the other day with a guide in between customer meetings.

    I'm absolutely indebted to Alain Hoang who helped guide us and answer our questions. He's an amazing sysadmin and one of the nicest guys I've ever met. If it weren't for his help we would have probly never ventured further than we can walk. Besides that, he deserves a metal for helping me stumble through some basic Japanese and better understand the culture.

    I don't know if I'll ever have reason to return to Tokyo. I certainly would enjoy being able to, especially if it weren't so close to Christmas (I returned the day before Christmas Eve), but given the cost I doubt I would ever return to vacation. Never the less, I've picked up an odd desire to continue learning Japanese and katakana... I've got an odd feeling I'll be back again one day. Who knows.

    So, in short, if you ever have the opportunity to visit Tokyo I encourage you to take it, but make sure you pad the trip with at least 5 days to take in as much as possible.

... and other tails. Back in the distant past I used to host a personal blog on journalspace, so I've watched the stories coming out about the loss of all of journalspace's [slashdot] user data[journalspace] with more than a passing interest. It has really brought home to me that no matter how cheap storage gets, and with the Sun Storage 7000 Unified Storage Systems we are radically redefining storage economics (and allowing some Australians to vent their cricket frustrations at some jbods, just a theory mind...), you do sometimes need some form of offsite backup. For most businesses that is still tape.

Of course tape by itself is not a disaster recovery plan, and here at Sun we offer a whole range of Business Continuity & Disaster Recovery Services to meet your needs, with specific services within our Managed Services portfolio to address storage, including remote back up.

As an aside I wrote a small script to migrate journalspace content at the time I moved blogs, its available on my personal site if someone wants to grab it. A quick look at Andrews entry on how to recover journalspace entries with google cache suggests it would be a pretty straight forward set of changes to get the script to drag data from google instead. ~

I`ve always wanted to try this side-by-side. I think I like black and white better. Yes? No?

• Unusual disk latency
  Don't shout at your hard disks; it upsets them. (Watch the video - yes, it's been everywhere, but it was on my Twitter feed first)
• The 2008 Chutzpah in IT Marketing Awards
  Not sure I really understand these, but the Sun vs IBM one is priceless.
• Free Endangered Species Ringtones - Center for Biological Diversity
  Some great and unusual sounds here that no-one else will be using. Well, apart from everyone that reads Webmink!

• Profile: Terry Pratchett
  That's SIR Terry Pratchett from now on. I wonder which town he'll pick as his "of"? Great choice for a knighthood, combining genius, philanthropy and an adoring public.
• Pioneer of cyberspace honoured
  Congratulations to Wendy. There ain't nothin' like a dame.
• Predictions for 2009 - Paul Murphy
  Some interesting insights here - see especially Sun and IBM. I really hope we'll be able to prove him wrong.
• The 10 Coolest Open Source Products Of 2008
  Top of the list? OpenOffice.org 3.0 (the first page of the sequence is the title slide).

• OOoCon 2009 - Call for Location
  Want to host the next OpenOffice.org conference? the 2008 event was in Beijing so I'd have to guess a location in Europe would stand a good chance this year...
• Dan Gilbert researches happiness
  Just watched this TED talk from 2005 and it's a brilliant and entertaining explanation - which seems so obvious in retrospect - of what influences our choices and why we are so often wrong. Unusually for a TED video there's also a Q & A at the end that's worth watching.
• On Christmas Day in the morning
  25 million downloads of OpenOffice.org 3.0. Regardless of the self-interested whining from some quarters, it's clear something about this release gets it right for the people who really matter, the users of the software.
• FUD from the Linux Foundation or: Mr Zemlin again
  "Using FUD is a good fear detector in my daily business. The amount of spreaded FUD is proportional to the amount of fear. Thus i have to assume that Mr. Zemlin of the Linux Foundation is really afraid of Solaris."
• Matthew Alexander on Torture
  "I learned in Iraq that the No. 1 reason foreign fighters flocked there to fight were the abuses carried out at Abu Ghraib and Guantanamo." (the book looks interesting too).
• Blog Response guide
  Excellent flow chart by the USAF PR team for responding to commentary on blogs that captures what many of us already learned and try to practice (even if we do end up feeding the odd troll).
• Ring flash
  Fascinating idea - a ring-light that works by redirecting and diffusing a normal flash.

Part of Shogatsu in Japan involves going to temples and shrines to pray. Last night at midnight we went down in the valley in a lovely light snowfall to this temple to pray, ring the bell that you can hear for miles around, and meet others in the community. People pray at three locations at this temple and then walk down the hill to a shrine to pray there as well. Shrines and temples mix in Japan totally freely ... 

Today, Brendan made a very interesting discovery about the potential sources of disk latency in the datacenter. Here's a video we made of Brendan explaining (and demonstrating) his discovery:

This may seem silly, but it's not farfetched: Brendan actually made this discovery while exploring drive latency that he had seen in a lab machine due to a missing screw on a drive bracket. (!) Brendan has more details on the discovery, demonstrating how he used the Fishworks analytics to understand and visualize it.

If this has piqued your curiosity about the nature of disk mechanics, I encourage you to read Jon Elerath's excellent ACM Queue article, Hard disk drives: the good, the bad and the ugly! As Jon notes, noise is a known cause of what is called a non-repeatable runout (NRRO) -- though it's unclear if Brendan's shouting is exactly the kind of noise-induced NRRO that Jon had in mind...

If you missed this years excellent Systems Administration Advent Calendar Blog you missed some great content. But do not despair! Its all there for your reading pleasure. Articles on scripting, new technology, primers, and workflow are there to help you into the new year. I even contributed an entry: Day 17 - Time Management.

A warm round of applause goes to Jordan Sissel for organizing it and rallying various bloggers to participate.