It only took about ten years, but I finally got a story on Slashdot, about Nikon's encryption of the white balance information in the NEF raw data from their DSLRs. Much inflamed commentary ensued in typical Slashdot "RTFA" style, which also spilled over on to the normally placid waters of PhotographyBlog (sorry, Mark). For what it's worth now, assuming this is still of interest, here's my take on it, including the reason for the "butt out" comment.

As a distinctly amateur photographer, I own three Nikon film SLRs, both modern and classic, and several lenses, most of which are secondhand because, being Nikons, I know they'll probably outlast me. I'd buy a D2X tomorrow if I had that much money, purely because it's one of the few Nikon DSLRs that can meter with the manual focus lenses that I'm not about to give up. The NEF encryption issue wouldn't stop me, not because I don't care or it wouldn't affect me (I use Linux for all my post-processing), but because I know from past experience that almost all proprietary encrypted formats will either be reverse-engineered and broken or publically leaked within days if the urge is there - as indeed has happened again in this case. (Those people who claim they are now going to sell their $50,000 Nikon collections in favour of Canon are either admirably principled or flouncing drama queens. It's not like Canon are the poster child for openness and consumer freedom - try getting one of their peripherals to work under Linux.)

But Nikon are compounding this foolishness with the arrogant, condescending tone of their press statement. It's very rare for the company to comment publically like this on a perceived flaw with their products. While they must perform some excellent market research, judging by their product designs (mostly), they do not generally make themselves amenable to feedback and discussion; you won't find an official Nikon blog anywhere, for example. Even a recognised pro like Thom Hogan grumbles frequently about how his (very well-informed) opinions are never solicited.

But reading their statement, the prevailing ethos at Nikon and the reason for their reluctance ever to come down from their tower is clear. In fact, it's probably for the best that they don't explain themselves more often, because it would only put more backs up and help to drive away their remaining customers. Superficially, the statement says: "This is not a problem, the data is accessible to the people who need it, move along, there's nothing to see." But reading between the lines, what I find is an ill-concealed contempt for the concept of openness and the people who promote such a stance, namely the open source community. Snide references to "bona fide" software companies that are "approved" by Nikon and repeated emphasis on the "proprietary" and "confidential" nature of their raw data format add up to a concerted raised digit against everyone who complained about not having full access to "their" image data.

Yes, it's their invention and they may have the right to keep it a commercial secret (although given that it's customer data, then again maybe not), but that doesn't necessarily equal "smart move". Their stated position is almost entirely bogus:

• The SDK is not widely available. It's not even easily available, given the old-fashioned written application process. Neither are the terms available in advance.
• The SDK is still a proprietary, closed piece of software, even if you have the right to bundle the required libraries with your own application. NEF in part remains encrypted and undocumented, and access to it is strictly via the limited set of mechanisms implemented by the SDK. This closes down the possibility of innovation from outside (it's not like Nikon's own offerings are universally applauded as the best applications in their class).
• According to reports, the SDK is only supplied as a C++ API for Windows and Mac; other platforms are apparently irrelevant.
• Nikon ultimately decides who is qualified to receive the SDK, and the prerequisites for this honour are not made public. "Bona fide" is quite a loaded term. Even if the bar is actually quite low, the attitude on display here would suggest otherwise.
• The mentions of the "choices", "benefit" and "security" provided for photographers by this situation is at best disingenuous and at worst knowingly deceptive. Behind this smokescreen, Nikon remains the gatekeeper on data that forms the images taken by their customers.
• If their concern was truly to protect their customers from poor quality image processing by third party applications, this would not be best served by attempting to hide implementation details. That only encourages developers, particularly those who cannot use the provided mechanisms, to experiment and create their own solutions, which may or may not be optimal (not that anyone can challenge Nikon's own implementation on this score). In comparison, an open, well-documented format limits the potential for broken or incorrect implementations. It also helps to promulgate that format.

Many major software companies have evolved beyond this narrow-minded, arms-length style of business to embrace the growth of Linux and open source with good reason (even the other big player in this story, Adobe, has released several well-known open, documented content formats), so it's tragic to see Nikon starting again from scratch and making the same mistakes without learning from history. Do they really believe there is some long term advantage to be gained, that there is a commercially successful software strategy, in attempting to lock-in their customers in this fashion?

No, they've strayed into unfamiliar territory now and fully deserve the resulting shitstorm. Whether it will help them gain a Clue, I'm not sure, because their approach so far is typical of many Japanese technology companies (e.g. Sony). We may be witnessing a clash between a Western consumer perspective that is gradually demanding more control over its purchases and the traditional but previously very successful Japanese determination to compete without either conceding to outsiders or cooperating with them.

Other bubbles

• Thread from the Nikonians D1/2 forum.
• Thom's D2X review has one huge reservation about the camera; guess what it is.
• The Raw Flaw, a call to arms ("The Raw Deal" would have worked too).

Whenever there's an application problem, project managers and analysts immediately turn to two groups of people to find out what's wrong: sysadmins and developers.

So once more, let's play....IT'S NOT MY PROBLEM!!

A good sysadmin will quickly view the logs and perform a quick sanity check on the system config to confirm that all is well.

A developer, often, in my experience, subject to sampling error and other qualifications to avoid too much offence, will simply say, "It's a configuration problem." They may follow that with one of two qualifiers:

• "It works on my box."
• "The code doesn't do anything to cause that."

...And guess who now has to back up their assertion?

Every.
Single.
Time.

It doesn't matter that the config looks good, or that the logs suggest it's not a problem with the system (e.g. they contain huge APPLICATION errors). It doesn't matter how many times you've been right in the past. It doesn't even matter that the developer hasn't produced any firm corresponding evidence the other way. It's "your" system, therefore it must be "your" issue.

Programming is a complex, intellectual activity and, faced with several thousand lines of code spread across a hundred files that interact in subtle ways, without much of a clue where to dive in, it's hardly surprising that many developers will forego the opportunity to examine it in detail. Besides, it "works on their box", therefore it must be something on the other system, right? This has become the cop-out du jour since the advent of programming models like J2EE, where applications are developed and tested on individual workstations then deployed, unaltered, on distributed multi-host production environments. Yes, configurations do tend to very different between the two. For example, the production environment probably has performance tweaks, extra features enabled, live backend data feeds and vastly increased load and demands. Small wonder that configuration errors are more likely. But it's also quite likely that the application code will behave differently in the wider world, and that subtle bugs or incompatibilities can materialise. (The usual way to catch these before deployment is to utilise a load-testing simulation tool, but such packages are often difficult to use effectively and anyway, even the best simulation falls short of the random twists of fate that can befall the average Internet application. But in this case, we're dealing with a problem that has appeared subsequent to testing.)

At this point, the sysadmin has to go out of their way to prove that the fault lies in the application, without necessarily possessing the actual code or the skills to understand it. Assuming you've triple-checked the configuration again, forensic examination of the logs is often the best avenue here, tracing sequences of events, following error trails and searching for a sign of the app saying "oops!". Sometimes it's possible to enable extra levels of debugging info (gird loins, hassle developer to release the magic incantation). Failing that, it might come down to process tracing (as we've said before, "when your only tool is a hammer..."). A hefty dose of logic applied in root-cause analysis may pinpoint the fault, but translating this into management-speak is a challenge in itself. In the worst case, you may have to set up a complete facsimile test environment and try to recreate the problem (which often fails for similar reasons to the development/production dichotomy). And all the while, you have to fend off distracting conversations like this:

"Are we sure a reboot won't fix it?"
"Positive. It's an application issue."
"Well, can we try it anyway?"
("What, and break my uptime world record attempt??!")

It can be absorbing fun, but more frequently it's frustrating because you find yourself going through this again and again. "Why is it They get to disclaim all responsibility and no one calls Them on it, but I have to move heaven and earth to defend My position? I've heard of being a system advocate, but I didn't realise it meant in the legal sense."

So if you're a manager or a developer, please consider the following:

• If your sysadmin has had a good hit rate in the past, take a little more on trust next time.
• If you have less or poorer evidence for your position than the other guy, your's is the weaker case. Work on it.
• Many bugs can be found and fixed in the time it takes a sysadmin to demonstrate them via indirect means.

Because there's one thing we really enjoy, that spurs us on, that makes it all worthwhile, that nearly compensates for the time, effort and trouble we've been forced to go to:

Wiping the smug grin off your face.

PostScript: Of course, there are rules for sysadmins too: Don't Gloat; and When It's Your Fault, Admit Blame And Say Sorry. This sometimes makes the developers gloat, but take comfort in your moral superiority. Besides, they're bound to fuck up again sooner or later. We're all human.

Since I mentioned that I would write further about IBM WebSphere Application Server (WAS), I've had several requests. Well, one. But I'm going to write about it anyway.

Working with WAS for several years now (since v4.0), reading about it and now writing about it makes me realise something important: I need to work less on WAS. Continued exposure proves detrimental to one's joie de vivre, much like administering database systems.

True, it's not as awful as BroadVision One-To-One 4.0, a pre-standards middleware product based on (I kid you not) server-side JavaScript. (On second thoughts, at least JavaScript on the server afflicts fewer computers than in widespread use on the client. Too late now though.) And WebSphere V5 is much more stable than V4 ever was.

Unfortunately, a day with any version of WebSphere reminds you of the fleeting transience of human mortality. Here's why:

Java

OK, I realise Java is a fundamental component of a J2EE application server. But it's the kind of language and framework you get when academic "software engineers" are running the asylum. EJBs, JNDI, JSSE, JMX, JMS, JCA, JAF, JSF, JSP, JARs, WARs and EARs - is there no end to the madness? Why should I care?? The answer is: you shouldn't, or you'll never get it running. Abstraction is usually considered a good thing because it "hides the details of the underlying implementation", but in this case too many levels of abstraction serve only to obscure clarity and obfuscate understanding. Write once, run away.

It's not just me. Even an extremely competent Java developer and colleague opined that much of the J2EE standard was the emperor's new clothes writ large. After a concerted attempt at using XML/XSL transforms to generate web pages, separating the presentation logic from the business logic like a good little buzzword-compliant coder, he threw away the lot with all its grim performance problems in favour of plain old JSPs.
WAS is a product that feels like it was developed by people who've been smoking the J2EE crack pipe for far too long. For overall integration, it scores highly - too bad that it's integrated around this dog's dinner.

Security

Out of the box, WAS is completely unsecured. Anyone can log on to the administration console via a web browser and do what they want. It even installs with global read permission on every file (although that is admittedly an improvement on V4, which had global write permission everywhere). You can firewall it from the Internet (oh, except the admin URI is automatically added to the web server plugin config so you better remember to remove it), but that won't stop people on your company network - and the odd lucky hacker - from reconfiguring your entire setup. Heck, it's so complicated, you might as well let someone else have a go.

(Un)Fortunately, WAS has a large, extensive security component that can be enabled with pretty much one click. And then it's so secure, even you can't login to it. Although it can authenticate against the local OS, an LDAP server or a custom registry, the former isn't supported on multi-node setups (precisely where it would be most useful) and the second doesn't support standard LDAP resilience features like replicated servers - which means another heap of hardware and/or software to make your LDAP server highly available. (IBM claim that the local OS can't be used in distributed cells because it would require the passwd files to be synchronised on each host, but that's a lot easier to do than running a wide-open e-commerce platform. In fact, it would let you use the native LDAP/NSS backend support in the local OS, which would actually be a huge improvement on their own inadequate hack.) LDAP implementation isn't exactly a mature skill yet and, much as their somewhat limited support of it is laudable, it's a mistake to assume such an infrastructure exists at every site. (Of course, IBM would love to sell you a resilient LDAP solution too.)

Naturally, WAS Security flavours the Java acronym soup - CSI, SASL, LTPA, WTF is all this crap?? Hiding at the bottom of all this is plain ol' SSL, for which IBM give you another icky Java management utility called IKeyman. Don't forget to replace the default ('dummy') certificates, by the way. Oh, and copy them manually to every node in the cell. Including the web server.

Assuming you have a suitable registry, you can set up WAS users with defined roles and passwords. Once enabled, an authorised account is required for all administrative actions, including running the command scripts - and if you want to run those scripts unattended (e.g. to start WAS during boot), you'll have to specify the appropriate "secret" password on the command line. So this vast and baroque security edifice is completely undermined by the kind of no-no that all sysadmin neophytes have beaten out of them in their first year.

It would be interesting to know how many WebSphere sites run with Security disabled, either because they don't know it's there, nobody had time to figure it out or because they decided it's a waste of space. At the very least, a simple username/password lock on the admin console and an auto-generated self-signed SSL cert, both created at install time, would stop every Tom, Dick and Harriet from reconfiguring your entire live cluster. (Interestingly, Sun's JES Application Server, supposedly an inferior product, does this.)

Administration console

IBM replaced the dedicated Java administration client in WAS V4 with a smart new web-based console in V5, which was probably the right move (although installation of the relevant client software was previously at least one barrier to accessing an unsecured WAS cluster). Unfortunately, the interface is clunkier than ever and you'll expend many mouse miles expanding menus, scrolling long frames of dialogue, clicking on sub-options, then saving your changes and navigating back to the same place again. Also, there's no way to make an identical change to every server in a cell or cluster simultaneously. V4 at least had some settings that were shared across every cloned server, but V5 removes even this partial support. If you want to tweak the JVM heap size for every app server (a parameter that is buried about three or four levels deep - depending on which route through the menus you take), IBM makes you click and navigate all those dialogues for every single one. The only alternative is to script your changes using the WSADMIN client.

Scripting

WAS admin actions can be scripted in JACL, a Java implementation of TCL based around Managed Beans (MBeans). TCL itself is a simple scripting language with only a few keywords. But the MBean malarky is bolted on like the head of Frankenstein's monster, only not as prettily.

To perform an operation on some aspect of WAS using JACL, you first figure out what object you need and then fetch a configuration ID for it. Then you call the relevant method with the appropriate arguments. Don't know the object name, the ID syntax, the method name or the argument usage? Well, tough. There's a help facility that gives basic one line summaries of each item for those who enjoy cryptic puzzles. There are various commands to list out all the types, attributes and objects in the cell, although the WSADMIN client doesn't let you page back through the reams of output these produce. Nothing is properly documented and the adventurous WAS administrator is reduced to scratching around for the few inadequate examples tantalisingly revealed by IBM. Throw me a frickin' bone here, people.

If you don't like JACL, you can also script in Jython but it's the same shit in a different syntactic sugar.

IBM

My main struggle with any IBM product is the fact that it was made by IBM. Because there's everyone else's way, and there's the IBM way - and the IBM way is...well, weird frankly. Look at AIX for example. Even their suggested Best Practices for WebSphere leave you thinking, "Hmm, not sure I'd do it that way."

Add to that, the extra support you constantly need demands navigating and searching the IBM web site, which is the kind of arduous quest that would have even Henry Stanley saying, "Let's just write 'Livingstone presumed dead' and go to the beach instead." Consistency and effectiveness fly out the window, through the hole made by your monitor. The search function appears to be some kind of advanced anti-Google engine that returns only irrelevant results. Every link takes you somewhere you didn't quite expect, even the ones that ostensibly lead to the same place: if you go directly to the IBM Redbooks site, there's a special "WebSphere domain", but if you click "Redbooks" in the WAS support zone, it returns a badly sorted search result for tangentially related documents. Many of the bug reports (APARs) make no sense in English and searches on symptoms, such as error message texts, usually return bugger all. When you finally cave under the weight of inadequate, badly organised documentation that tells you nothing you want to know, and pay for three days of IBM consultancy, they send someone who repeats exactly what it says in the documentation.

These are just a few of the reasons why WebSphere administration lends your days a Bergmanesque quality (and I don't mean Ingrid) - thanks, IBM. That's why a thirty page "quick start" document written by someone who's been there is necessary. (If it's any consolation, I've heard that BEA WebLogic is just as bad.)

Update, 2006-06-14:

Well, don't say IBM never listen (um, they don't but they don't want to hear anyone saying that, capish?). You can now search their support web site via Google, which seems to be a tacit acknowledgement that their own search engine sucks. They've had a go at simplifying scripting in WAS 6.0 by adding some higher level objects, although whether this is sufficient is a moot point (you can't polish a turd). And some of the security & LDAP madness has been addressed in WAS 6.1, specifically the lack of security out-of-the-box.

Oh, and there's finally going to be support for Solaris x86. (Don't laugh - without this, you'll end up running Linux if you migrate off SPARC.)

Now the downside: WAS 5.1 doesn't support Solaris 10 and WAS 6.1 doesn't support Solaris 8. So to go from one extreme to the other in any kind of staged manner, you must perform a completely pointless WAS 6.0 installation inbetween. This simply isn't sufficient breadth of support for a rapidly changing application, particularly on an OS that carries backwards compatibility guarantees. Hopeless...

I thought I'd write a nice, concise guide to using WebSphere on UNIX. Something more straightforward than the IBM documentation, which appears designed to bring on a mental collapse. Almost thirty pages later ...

Download Using WebSphere Application Server on UNIX in PDF.

I hope to say more about WebSphere and about LaTeX, which I used to write this paper, later.

Update, 2009-02-02: Fixed the link.