(Un)safe English locales

I recently spent a frustrating amount of time troubleshooting a Java application which had stopped working properly after an upgrade: Certain features of the product were broken due to the application running on a Linux server which had been configured with the "en_DK" locale.

"en_DK" had been chosen expecting to have system with messages in English, but with currency symbols etc. suitable for Denmark. This makes sense, because it's not uncommon for IT systems to provide more precise (error) messages in English, compared to a small language like Danish.

Unfortunately, there is not universal agreement about locales. Even within Linux distributions, there is not 100% consistency.

The GNU C Library (Glibc) seems to have the longst list of recognized locales, including 19 English ones. Java's list of supported locales is somewhat shorter and includes only 11 English locales. Interestingly, Java has an English locale for Malta (en_MT) which glibc does not have.

I haven't been able to find MacOS's list of locales, but forum posts suggest it has only 6 English locales: en_AU, en_CA, en_GB, en_IE, en_NZ, en_US.

Ignoring Mac for a moment, these are unsafe English locales, i.e. not supported by both glibc and Java:

Locale	Country
en_AG	Antigua and Barbuda
en_BW	Botswana
en_DK	Denmark
en_HK	Hong Kong
en_IL	Israel
en_MT	Malta
en_NG	Nigeria
en_SC	Seychelles
en_ZM	Zambia
en_ZW	Zimbabwe

On the other hand, the following locales should be safe:

Locale	Country	Safe even om Mac
en_AU	Australia	🍎
en_CA	Canada	🍎
en_GB	Great Britain	🍎
en_IE	Ireland	🍎
en_IN	India
en_NZ	New Zealand	🍎
en_PH	Philippines
en_SG	Singapore
en_US	USA	🍎
en_ZA	South Africa

Looking beyond unix/POSIX-like systems: Windows recognizes more than 100 English locale identifiers. Windows' list does invalidate the above safe-list.

For Danes, I suggest using one of the following locales:

• da_DK (and sometimes accept poor error messages)
• en_IE, as the Irish are sane enough to use a 24-hour date format, etc
• C, which is the fall-back "POSIX system default" locale

Unbootable RHEL 9 when using BP-028

When installing Red Hat Enterprise Linux 9, you may choose to apply a security profile, such as ANSSI-BP-028 High.

I've recently seen two VMware-virtualized RHEL 9.2 servers not being able to boot properly when installed with the ANSSI-BP-028 High profile. Instead, they booted into emergency mode.

The way to fix it:

Add file /etc/modules-load.d/for_uefi.conf containing a single line:

vfat 

Then run the following two commands:

dracut -f /boot/initramfs-$(uname -r).img $(uname -r)
reboot

parallel_ntp_scan

NTP-based DDoS attacks are fashionable, currently.

I've coded a little application which quickly scans a network for NTP servers. For those found, it rates them according to their susceptibility to being implicated in an amplification DDoS attack.

What a dying SFP looks like

Fibre channel (FC) storage is handy, and generally very reliable, in my experience. I certainly do not miss the days of messing around with disks in a server-room. And I like the fact that RAIDs may be cut up into slices (LUNs) which may be shared by many servers, resulting in very efficient use of the disks (if so wanted).

One part about FC that I dislike (in addition to the price tags): SFPs. Why on earth are transceivers not an integral part of a Fibre Channel switch? Having the transceivers be separate units means more electrical contact points, and a potential support mess (it's not hard to imagine a situation where the support contract of an SFP has run out, while the switch itself is still covered).

Anyway: Today, I experienced an defunct SFP, for the first time. The following observations may give a hint of how to discover that an SFP is starting to malfunction. The setup is an IBM DS4800 storage system where port 2 on controller B is connected to port 0 on an IBM TotalStorage SAN32B FC switch (which is an IBM-branded Brocade 5100 switch).

Friday morning at 07:49, in syslog: A few messages like this from the FC switch:

raslogd: 2011/09/30-07:49:07, [SNMP-1008], 2113, WWN 10:... | FID 128, INFO, IBM_2005_B32_B,  The last device change happened at : Fri Sep 30 07:49:01 2011

At the same time the storage system started complaining about "Drive not on preferred path due to ADT/RDAC failover", meaning that at least one server had started using a non-optimal path, most likely due to I/O timeouts on the preferred path. And a first spike in the bad_os count occurred for the FC switch port:

bad_os is a counter which exists in Brocade switches, and possibly others. Brocade describes it as the number of invalid ordered sets received.

At 10:55, in syslog:

raslogd: 2011/09/30-10:55:02, [FW-1424], 2118, WWN 10:... | FID 128, WARNING, IBM_2005_B32_B, Switch status changed from HEALTHY to MARGINAL

At the same time, there was a slightly larger spike in the bad_os graph.
Coinciding: The storage system sent a mail warning about "Data rate negotiation failed" for the port.

At 17:00: The count for bit-traffic flat-lined (graph not shown). I.e.: All traffic had ceased.

At no point did the graphs for C3 discards, encoding errors or CRC errors show any spikes.

The next morning, the involved optical cable was switched; that didn't help. Inserting another SFP helped, leading to the conclusion that the old SFP had started to malfunction.

Morale: Make sure to not just keep spare cables around. A spare SFP should also be kept in stock.
And monitor your systems: A centralized and regularly inspected syslog is invaluable. Generating graphs for key counters is also mandatory for mature systems operation; one way to collect and display key counts for Brocade switches is to use Munin and a Munin plugin which I wrote.

PS: Brocade documentation states that SFP problems might result in the combination of rises in CRC errors and encoding/disparity errors. This did not happen in this situation.

Separation or co-location of database and application

In a classical three-tier architecture (database, application-server, client), a choice will always have to be made: Should the database and the application-server reside on separate servers, or co-located on a shared server?

Often, I see recommendations for separation, with vague claims about performance improvements. But I assert that the main argument for separation is bureaucratic (license-related), and that separation may well hurt performance. Sure, if you separate the application and the database on separate servers, you gain an easy scale-out effect, but you also end up with a less efficient communication path.

If a database and an application is running on the same operating system instance, you may use very efficient communication channels. With DB2, for example, you may use shared-memory based inter-process communication (IPC) when the application and the database are co-located. If they are on separate servers, TCP must be used. TCP is a nice protocol, offering reliable delivery, congestion control, etc, but it also entails several protocol layers, each contributing overhead.

But let's try to quantify the difference, focusing as closely on the query round-trips as possible.

I wrote a little Java program, LatencyTester, which I used to measure differences between a number of database<>application setups. Java was chosen because it's a compiled, statically typed language; that way, the test-program should have as little internal execution overhead as possible. This can be important: I have sometimes written database benchmark programs in Python (which is a much nicer programming experience), but as Python can be rather inefficient, I ended up benchmarking Python, instead of the database.

The program connects to a DB2 database in one of two ways:

• If you specify a servername and a username, it will communicate using "DRDA" over TCP
• If you leave out servername and username it will use a local, shared memory based channel.

After connecting to the database, the program issues 10000 queries which shouldn't result in I/Os, because no data in the database system is referenced. The timer starts after connection setup, just before the first query; it stops immediately after the last query.

The application issues statements like VALUES(...) where ... is a value set by the application. Note the lack of a SELECT and a FROM in the statement. When invoking the program, you must choose between short or long statements. If you choose short, statements like this will be issued:

VALUES(? + 1)

where ? is a randomly chosen host variable.
If you choose long,  statements like

VALUES('lksjflkvjw...pvoiwepwvk')

will be issued, where lksjflkvjw...pvoiwepwvk is a 2000-character pseudo-randomly composed string.
In other words: The program may run in a mode where very short or very long units are sent back and forth between the application and the database. The short queries are effectively measuring latency, while the long queries may be viewed as measuring throughput.

I used the program to benchmark four different setups:

• Application and database on the same server, using a local connection
• Application and database on the same server, using a TCP connection
• Application on a virtual server hosted by the same server as the database, using TCP
• Application and database on different physical servers, but on the same local area network (LAN), using TCP

The results are displayed in the following graph:

Click on figure to enlarge; opens in new window

Clearly, the the highest number of queries per second is seen when the database and the application are co-located. This is especially true for the short queries: Here, more than four times as many queries may be executed per second when co-locating on the same server, compared to separating over a LAN. When using TCP on the same server, short-query round-trips run at around half the speed.

The results need to be put in perspective, though: While there are clear differences, the absolute numbers may not matter much in the Real World. The average query-time for short local queries were 0.1ms, compared to 0.8ms for short queries over the LAN. Let's assume that we are dealing with a web application where each page-view results in ten short queries. In this case, the round-trip overhead for a location connection is 10x0.1ms=1ms, whereas round-trip overhead for the LAN-connected setup is 10x0.8ms=8ms. Other factors (like query disk-I/O, and browser-to-server roundtrips) will most likely dominate, and the user will hardly notice a difference.

Even though queries over a LAN will normally not be noticeably slower, LANs may sometimes exhibit congestion. And all else being equal, the more servers and the more equipment being involved, the more things can go wrong.

Having established that application/database server-separation will not improve query performance (neither latency, nor throughput), what other factors are involved in the decision between co-location and separation?

• Software licensing terms may make it cheaper to put the database on its own, dedicated hardware: The less CPUs beneath on the database system, the less licensing costs. The same goes for the application server: If it is priced per CPU, it may be very expensive to pay for CPUs which are primarily used for other parts of the solution.
• Organizational aspects may dictate that the the DBA and the application server administrator each have their "own boxes". Or conversely, the organization may put an effort into operating as few servers as possible to keep administration work down.
• The optimal operating system for the database may not be the optimal operating system for the application server.
• The database server may need to run on hardware with special, high-performance storage system attachments. - While the application server (which probably doesn't perform much disk I/O) may be better off running in a virtual server, taking advantage of the flexible administration advantages of virtualization.
• Buying one very powerful piece of server hardware is sometimes more expensive than buying two servers which add up to the same horsepower. But it may also be the other way around, especially if cooling, electricity, service agreements, and rack space is taken into account.
• Handling authentication and group memberships may be easier when there is only one server. E.g., DB2 and PostgreSQL allows the operating system to handle authentication if an application connects locally, meaning that no authentication configuration needs to be set up in the application. (Don't you just hate it when passwords sneak into the version control system?)
• A mis-behaving (e.g. memory leaking) application may disturb the database if the two are running on the same system. Operating systems generally provide mechanisms for resource-constraining processes, but that can often be tricky to setup.

Summarized:

Pro separation	Con separation
Misbehaving application will not be able to disturb the database as much.	Slightly higher latency.
May provide for more tailored installations (the database gets its perfect environment, and so does the application).	Less predictable connections on shared networks.
If the application server is split into two servers in combination with a load balancing system, each application server may be patched individually without affecting the database server, and with little of no visible down-time for the users.	More hardware/software/systems to maintain, monitor, backup, and document.
May save large amounts of money if the database and/or the application is CPU-licensed.	Potentially more base software licensing fees (operating systems, supporting software).
Potentially cheaper to buy two modest servers than to buy one top-of-the-market server.	Potentially more expensive to buy two servers if cooling and electricity is taken into account.
Allows for advanced scale-out/application-clustering scenarios.	Prevents easy packaging of a complete database+application product, such as a turn-key solution in a single VMWare or Amazon EC2 image.

Am I overlooking something?
Is the situation markedly different with other DBMSes?

PostgreSQL innovation: Exclusion constraints

It seems like the next generation of PostgreSQL will have a new, rather innovative feature: Exclusion constraints. The feature is explained in a video from a presentation at a recent San Francisco PostgreSQL Users' Group; the presentation couples the new feature with the time period data type.

In a perfect World where databases implement the full ISO SQL standard (including non-core features), exclusion constraints could be nicely expressed as SQL assertions, but the perfect World hasn't happened yet. And I think that I know the reason for this: SQL assertions seem like a strong cocktail of NP-hard problems - they are probably very hard to implement in an efficient way.

In our less than perfect World, it's nice that PostgreSQL will soon offer a way to specify exclusion constraints other than the UNIQUE constraint.

The presenter, Jeff Davis, has an interesting blog, by the way. An on the subject of video, Vimeo has a little collection of PostgreSQL video clips that looks interesting.

SQL comparison update: PostgreSQL 8.4

I finally got around to adjusting my SQL comparison page, so that the improvements in PostgreSQL 8.4 are taken into account. PostgreSQL is now standards-compliant in the sections about limiting result sets; actually, PostgreSQL is the first DBMS that I know of which supports SQL:2008's OFFSET + FETCH FIRST construct for pagination. PostgreSQL's new support for window functions is also very nice.

My page doesn't cover common table expressions (CTEs) yet, but it's certainly nice to see more and more DBMSes (including PostgreSQL, since version 8.4) supporting them. Even non-recursive CTEs are important, because they can really clean up SQL queries and make them more readable.

SQL comparison updates: Oracle 11R2, diagnostic logs

I finally found some time to update my page which compares SQL implementations. I performed a general update regarding Oracle, now that Oracle 11R2 has been released. And I added a new (incomplete) section which describes where the different DBMSes store diagnostic logs.

I turned out that very little has changed in Oracle since generation 10. The only remarkable new SQL feature in Oracle 11 is support for CTEs and proper CTE-based recursive SQL (introduced in version 11, release 2) -- but as I don't cover this topic on my page, updating the Oracle items was mostly a question of updating documentation links. Oracle still doesn't support the standard by having a CHARACTER_LENGTH and a SUBSTRING function, for example. This is simple, low-hanging fruit, Oracle(!) Sigh. It seems like Oracle's market position has made them (arrogantly) ignore the SQL standard.

VLDB2009: Non-cloud storage

Not all of VLDB2009 was related to cloud storage. Luckily, local and SAN storage is still being explored. Here are some notes from selected presentations.

Mustafa Canim: An Object Placement Advisor for DB2 Using Solid State Storage: What data should be put on disks, and what data on solid state disk storage? Solid state drives (SSDs) shine at random read I/O, but in most situations, there's limited funds, so only part of a database will be eligible for SSD placement. It turns out that if a naïve/simplified strategy like let's place all indexes on SSD is used, only a small overall performance gain is measurable, and it's hardly a justification of the expensive SSD storage. But if placement of database objects (tables/indexes) is based on measurements from a representative workload, then data which is often seeked to can be placed on the SSDs. Canim has created a prototype application which uses DB2 profiling information to give advice on the most optimal use for xGB of SSD storage, and he demonstrated very convincing price/performance gains from his tool. The principle should be easy to apply to other DBMSes.

Devesh Agrawal: Lazy-Adaptive Tree: An Optimized Index Structure for Flash Devices: Since SSDs do not shine at random writes, an SSD-optimized index structure would be highly welcome. The Lazy-Adaptive (LA) Tree index is a (heavily) modified B+ tree which buffers writes in a special way, yielding significant performance improvements.

Nedyalko Borisov and Shivnath Babu had a poster and a demonstration about a prototype application they have created: DIADS. DIADS integrates information from DB2 query access plans with knowledge about the storage infrastructure (all the way from files on the disk, through the LVM and SAN layers, to individual disks) to help diagnosing performance bottlenecks. This would certainly be useful for DBAs, and it could probably bridge the worlds of DBAs and storage people. They are considering making it an open source project. By the way: I believe that I've heard Hitachi claim to be selling a tool with a similar objective: Hitachi Tuning Manager.

Finally, a little rule of thumb from Patrick O'Neil's The Star Schema Benchmark and Augmented Fact Table Indexing (part of the TPC Workshop): Throughput of magnetic disks has grown much more than seek latency, so at the moment, 1MB of scanning can justify 1 seek.

VLDB2009: Sloppy by choice

One of the recurring themes at the VLDB2009 conference was how to create massively scalable database systems, by decreasing expressiveness, transaction coverage, data integrity guarantees—or any combination of these. The theoretical justification for this is partly explained by the CAP Theorem. Much has already been written about database key-value/object stores, cloud databases, etc. But there were still a few surprises.

Ramakrishnan's keynote
Yahoo!'s Raghu Ramakrishnan (whose textbook many readers of this article will know) gave a keynote on the subject. It was actually new to me that Yahoo! is also entering the cloud business; it's getting crowded in the sky, for sure. As we know, the business idea is this: A large operation like Amazon already has a massive, distributed IT infrastructure; so letting other companies in isn't that much of an extra burden. And the more users of the hardware, the easier it is to build a cost-efficient setup which can still handle spikes in performance demands (with many users it's very unlikely that their systems run at peak performance at the same time). Nice idea, but it may take a long while before users convert to using software which runs in the cloud, and it remains to be seen how many cloud vendors which can survive that long.

Anyway, Raghu Ramakrishnan presented a much-needed comparison of cloud database solutions (pages 55-60 in this presentation). The consistency model of Yahoo!'s cloud database system, PNUTS, does not provide ACID, but nor is it 'sloppy' to the degree of BASE. Another nice aspect of Yahoo!'s cloud systems is that much of it is based on open source software. Yay Yahoo!

When to be sloppy
At the conference, some claimed that cloud storage can also be used for important data, but no one gave a plausible example. In cloudy times, we should not forget that not all data are about tweets, status updates, weblog postings, etc. There are actually data which is actually important. That's why I liked the Consistency Rationing in the Cloud: Pay only when it matters presentation: It provided at framework for categorizing data in degrees of acceptable sloppiness, based on the cost associated with potential inconsistencies, versus the savings gained from the lower transaction overhead.

Panel on cloud storage
On Wednesday, there was a panel discussion on How Best to Build Web-Scale Data Managers, moderated by Philip Bernstein. Random notes from the discussion:

• A Surprising, and somewhat strange viewpoint from Bernstein: We should not ditch ACID (not surprising coming from Mr Transaction himself), but we should give hierarchical DBMSes a new chance. According to Bernstein: The reason why it will not fail this time is that we have become so good at handling materialized views, and they allow us to make sure that fast queries are not restricted to restricting/scanning one one dimension. Bernstein failed to alleviate my fear of the return of another major drawback of hierarchical databases: navigational queries.
• While there's much not-so-important data out there, the phenomenon of moderate amounts of important data hasn't gone away (not every business is a twitter.com). So although the non-ACID, non-relational database systems may have a lot of attention, it doesn't matter for the makers of "traditional" DBMSes, because RDBMS business is doing great.
• Sub-question: Why do web start-ups seem to make use of key-value stores, and not use Oracle's DBMS (for example) when they need to scale to beyond a single data server? There wasn't much opposition to the view that—in addition to being administration labor intensive—the cost of an Oracle cluster is way out of budget in many businesses. So: If database researchers want to help prevent relational+transactional research from becoming increasingly irrelevant, it's time to help the open source database projects. While I agree that researchers can make a difference in the open source world, but I's skeptical to the perception of RDBMSes being abandoned; whatever numbers I've seen actually indicate the opposite. And while Facebook—for example—has a key-value store, their non-clickstream-data is actually in sharded MySQL databases, as far as I've heard; sharded MySQLs will never win relational database beauty contests, but at least it's tabular data, accessible with SQL, and with node-local transactions.
• Interesting point: SAP, an application with undisputed business significance, is well known for using nothing but the most basic RDBMS features. With that in mind, one should be cautious to denounce cloud databases for lack of expressiveness.

Finally, a couple of pictures from the nearby Tête d'Or Park:

VLDB2009: TPC Workshop

Monday, I attended the Transaction Processing Council (TPC) Workshop about the latest developments in database benchmarks. The workshop was kicked off with a keynote from a true database hero, Michael Stonebraker. Mr. Stonebraker has not only published significant research papers—he is also initiated a number of projects and startups: Postgres (the precursor to the great PostgreSQL DBMS), Vertica (one of the pioneers within the "column-oriented"/"column-store" database realm), StreamBase, and others. Stonebraker held it that benchmarks have been instrumental in increasing competition among vendors, but there are aspects to be aware of: As the benchmarks allow the vendors to tune the DBMS (and sometimes create setups not resembling most setups, like using five figure disk counts), this doesn't improve the out-of-box experience—an experience which all too often needs to be significantly tweaked. Stonebraker also criticized the TPC from being too vendor focused, instead of having focus on users and (simple) applications. And he urged the TPC to speed up the development of new benchmark types (I'm thinking: geospatial, recovery, ...), partly by cutting down on organizational politics.

Personally, I'm astonished that some (most?) of the big DBMS vendors prohibit their users from publishing performance findings. This curbs discussion among practitioners, and it decreases reproducibility and detail of research papers ("product A performed like this, product B performed like that"). I doubt that this actually holds in a court of law, but it would certainly take guts (and a big wallet) to challenge it. I'm also annoyed that the vendors don't really support the TPC much: The TPC-E benchmark (OLTP-benchmark, sort-of modernized version of TPC-C) is two years old, yet only one vendor (Microsoft) has published any results yet.

Nevertheless, references to TPC benchmarks were prevalent at the conference, being referred to in several papers.

I'm planning to try running TPC-H on our most important database systems, to see if it is feasible to use it for regular performance measurements—in order to become aware of performance pro-/re-gressions. By the way: It would be of great if IBM (and others) published a table of reference DB2 TPC findings for a variety of normal hardware configurations. That way, a DBA like me could get an indication of whether I've set up our systems properly.

Other speakers had various ideas for new benchmarks, including benchmarks which measure performance per kWh, and benchmarks which expose how well databases handle error situations.

A researcher from VMWare pledged for benchmarks of databases running in virtual environments. He presented some numbers of a TPC-C-like workload running on ESX guests, showing that a DBMS (MSSQL, I believe) can be set up to run at 85%-92% of the native speed. Certainly acceptable. And much in like with what I'm experiencing. I hope that figures like this can gradually kill the myth that DBMSes shouldn't run in virtual hosts—a myth which results in a situation where many organizations don't realize the full potentials of virtualization (increased overall hardware utilization/lower need for manpower/less late-night service windows, as workloads can be switched to other hosts when a server needs hardware service).

I forgot to take a picture from the workshop, so here's a picture of me being sceptical about traditional DBMS vendors—next to the Saôme river.

At VLDB2009

I'm attending the 35th VLDB conference in Lyon: VLDB2009. The conference portrays itself as the premier international forum for database researchers, vendors, practitioners, application developers, and users. Of the 700 people (from 44 countries), I'm one of the few practitioners at the conference; and though there's a risk that the conference will be too research oriented, I've signed up, especially hoping to get the latest updates and thoughts on

• probabalistic databases
• column-oriented databases
• performance quantification
• cloud databases

During the next couple of days, I'll share my experiences here.

I—and others—have tweeted a bit from the conference, as well.

MTU adjustment needed on Orange wifi

I'm on the road. On the hotel, there's wifi, provided by Orange (France Télécom). The connection has been strange: Most of the time, I couldn't connect to Facebook and MSN Messenger (XMPP/Jabber connections worked fine, though); SSH connections were also unstable. Finally found out that it helped changing my network settings such that the MTU parameter was decreased to 1200.

Storage system experiences

Background
Some people dislike Storage Area Networks (SANs). This may be due to complexity; and it may be because SAN management is often delegated to a special group of administrators which may then become an organizational bottleneck.

Personally, I very much like SANs. I do not miss messing around with disks and cables and enclosures in the server room. In my opinion, the complexity of a SAN is easily balanced out by easy and timely allocation of storage chunks to servers. And as a database administrator, I'm very fond of being able to share RAIDs between several servers, resulting in storage backed by a large amount of spindles. Also, it's nice to be able to monitor I/O activity across a large set of systems.

At work, we use two different FC SANs. One SAN connects Windows/ESX/Linux Dell servers to a Hitachi AMS500 storage system, while another SAN connects IBM servers to IBM DS4800 controlled storage.

I've had 2½ years of experience with the IBM storage system which keeps 24TB of data. We've had the AMS500 for ½ a year; it stores around 20TB of data. Both storage systems are configured with RAID5/6 logical drives with varying speed profiles, for different kinds of use patterns.

GUI management interface
The GUI management interface for DS4800, Storage Manager (SM), is vastly superior to AMS500's Storage Manager Modular (SMM). SM lets you assign text lables to your LUs, while SMM only works with numeric LUNs; this means that SM lets you get away with less separately maintained systems documentation. Also, it's much easier to view snapshots of I/O activity in IBMs Storage Manager. And IBMs Storage Manager was very easy to install while Hitachi's Storage Manager Modular required a fair amount of tweaks during installation, due to Java runtime issues (seemingly because Hitachi's software is bundled with an ancient JRE).

Command line interface
For both storage systems, the command line tools use rather awkward syntax and calling conventions. I wish that the programmers of these tools would lean more towards modern Linux/unix command line conventions.

Health monitoring
For both storage systems, you are encouraged to install software which regularly checks storage system health; in case of trouble, it alerts you by e-mail and "phones home" to the respective support organizations. Again, the IBM software is easy to install, while we had problems getting the Hitachi software to install on contemporary server software.

The AMS500 lets you turn on SNMP, so that you can easily poll for the health of the system from your central monitoring system; nice. DS4800 doesn't seem to offer this.

Stability
During my 2½ years with the DS4800, we have had three serious breakdowns: Two related to firmware trouble, and one related to a controller hardware defect. This is too much, I think.
We haven't had experienced instability with the AMS500 during the ½ year that we've had it in production.

UPDATE Dec 2009: We got our first stability problem with AMS500 :-(
A misconfigured ESX-cluster generated a large number of I/O requests for a LUN which had been deleted, and the AMS500 started getting periodic absence seizures. In AMS500's defence: Had we been up-to-date with regard to AMS500 firmware, this wouldn't have caused trouble.

Performance
Benchmarking storage systems is hard, especially because you normally don't have the luxury of being able to shut down all other I/O than that generated by the benchmark. So although I've conducted an extensive set of performance measurements, I can only say that the systems seem to perform equally well.

One difference, though: Hitachi recommends that you stick with rather narrow RAIDs; if needed, these may then be joined into larger storage areas using a special LUSE feature, or using logical volume management at the hosts. This is somewhat annoying: We would generally like to have wide RAIDs comprising a large number of spindles; AMS500 makes it a bit more complicated to meet this goal.

Price
At the time of our AMS500 acquisition, Hitachi's storage system was substantially less expensive than IBMs comparable offerings. The comparison may be a bit unfair, though, because our procurement is tightly controlled by government procurement contracts.

Support
We haven't used IBMs support offerings much, because we have historically used consultants from an IBM partner for support of our DS4800. But, regarding software and documentation, IBMs support is mostly very good: It's easy to download updates for both firmware, management software, and documentation. IBMs documentation of how to use DS4800 multipathing with Linux is inconsistent and confusing, though.

Hitachi's support is a mixed experience. They have appointed a technician from their Danish office to us, and this works very well: He is easy to get in touch with, and he provides good answers. On the other hand, Hitachi's distribution of software and documentation is miserable: The software is hidden behind a confusing extranet, and even after months of mail/phone correspondance with the extranet support (and other parts of Hitachi's organization), we haven't been able to log in and download software or documentation. So we have resorted to ask our tech contact to send us CDs via snail mail once in a while. Another Hitachi annoyance is licence keys: Why on Earth do we need to enter licence keys when installing multipath driver software (on Windows); as if there would ever be a black market for that kind of software. And when the going gets tough in operations (e.g. if a new server needs to be quickly installed after a server breakdown), it's frustrating to have to spend time trying to dig out that CD with licence keys. Argh! What are they thinking?

Multipathing
Multipathing with the DS4800 works well with Windows and PowerVM servers, but we never got it to work perfect with Linux on Intel. It's strange that IBM puts a significant amount of work into Linux, but still can't make it simple and easy to integrate an important storage product with Linux.

Multipathing with AMS500 works well (and out of the box) on Linux and ESX if you follow certain conventions (Red Hat knowlegebase article; Hitachi ESX configuration document). However, Hitachi's Windows HDLM drivers have trouble discovering new LUs, resulting in the need for several reboots when a new LU is mapped to a Windows host; when the discovery is up and running, things work fine with Windows, too.

Overall experiences
Pro IBM: Good management software, simple distribution of software and documentation.
Con IBM: Hich price. More breakdowns than should be expected, I think.

Pro Hitachi: Good price, good stability so far.
Con Hitachi: Bad management software, very bad distribution of software and documentation.

SQL comparison updates: Informix, JDBC

A reader of my SQL implementation comparison page suggested that I keep a change log for the page. Currently, I don't want to keep a detailed log, but I'll add blog posts here when I've made changes which are somewhat significant, tagging the posts 'sqlpage'.

Today, I added a new section related to JDBC, because I'm tired of having to spend time looking up details like JDBC driver names and connection URLs in manuals. (I don't work much with Java, so I keep forgetting where to find this stuff when I need it.) The section is far from complete.

I also started coverage of the Informix database, as I've recently become responsible for an important Informix database at work. Informix was once a very important player in the database game, but the story goes that the Informix company suddenly started getting managed very badly -- and at some point IBM bought Informix. Since then, IBM's strategy for Informix has been rather unclear. I have a personal, very weakly founded theory: I think IBM is positioning Informix as the place where the exciting new developments happen first; and when the new features are known to work well, they are introduced to DB2. This may not be a bad role for Informix.

Flash 10

Adobe released generation 10 of its Flash player software today.

Good:
Adobe provides a "yum" channel for the Flash player, so my PCs were automatically updated. This is nice and responsible: Many security problems on home computers are actually due to 3rd party software which isn't automatically patched when the user performs a base system update. After fixing a problem (see below), Flash 10 seemed to run fine.

Bad:
Unfortunately, Adobe still doesn't seem to provide x86_64 versions of the player. And consequently, 64-bit PCs need to have a bunch of -- otherwise irrelevant -- compatibility software installed. The need to maintain parallel 32 and 64 software worlds can also be tricky.

I wonder what's keeping Adobe from making life a little bit easier for users by adding a 64-bit download option. Which reminds me that it's probably time to take a closer look at the The Lively Kernel and Moonlight soon.

Flash 10 systematically crashed on my home PC, until I added a 32-bit curl package to the system (Flash 10 seems to link to libcurl -- something that earlier versions didn't). Strangely, the Flash RPM package doesn't carry a dependency for libcurl, although it's clearly needed.

DB2 lets you drop a parent table

DB2 is usually a rather strict database system: It doesn't allow you to drop a procedure which is being used by a function. It uses pessimistic locking. It typically forces you to back up a tablespace if you aggressively load data into a table if the database isn't using circular logging. Etc.

So I found it surprising that it allows you to drop a table which is being referred to in a foreign key. DB2 doesn't even warn you about the fact that the child table(s) have lost a potentially important constraint. That's evil.

I know of no other DBMS which lets you do drop a parent table: PostgreSQL refuses it (unless you add a CASCADE option to the DROP statement), MSSQL refuses it. Not even MySQL lets you do it.

As always, MySQL has little surprises:

CREATE TABLE child (
   child_id INT NOT NULL,
   parent_id INT NOT NULL,
   whatever VARCHAR(50) NOT NULL,
   PRIMARY KEY(child_id,parent_id),
   CONSTRAINT child_fk FOREIGN KEY (parent_id) REFERENCES parent
);
ERROR 1005 (HY000): Can't create table './test/child.frm' (errno: 150)

So what is the reason behind that confusing error message (which also qualifies as evil)? -- "REFERENCES parent" must be explicit: "REFERENCES parent(parent_id)"...

Recursive SQL, soon also in PostgreSQL

SQL:1999-style recursive SQL has been added to the development version of PostgreSQL. Consequently, soon, three DBMSes (DB2, MSSQL, PostgreSQL) will be supporting the "WITH [RECURSIVE]" construct. Recursive SQL is becoming mainstream. (Oracle also has recursive SQL, but implemented in a limited and non-standard way.)

Open Source Days 2008, Saturday

Today was actually a work day for me, as we carried out a major service window. Fortunately, my duties during the service window were limited, so I managed to sneak over and attend a few sessions at Open Source Days while waiting for some SAN and operating system patches to be applied by co-workers.

I attended the last half of Jan Wieck's talk about Slony-I (not "Slony-l", as written in the conference agenda). Slony-I is an asynchronous master-slave replication program for the PostgreSQL database management system. Unfortunately, I don't work much with PostgreSQL in my current job, but if I did, I'd certainly try out Slony-I. It can be useful for scalability (think: read-only database slaves in a CMS cluster) and continuous backup to an offline location. It can also be used when upgrading PostgreSQL, resulting in close to zero down time, because Slony-I can (to a certain degree) replicate between different PostgreSQL versions. Slony-I has some rather impressive replication routing features, so that you have have master->slave->slave->slave.

This talk was an example of why I like participating in conferences with open source focus: Jan was very clear about Slony-I's limitations and weaknesses -- contrary to some corporate guy who might not be lying, but who might be suppressing unfortunate facts. Slony-I has a number of weak points: It's rather complex to install, configure, and manage. And the current version 1 does some dirty tricks with the system schema (will be cleaned up in version 2).

Jan once had a plan for multi-master replication in Slony-I, but that idea has been dropped for now. Fine with me: Although it sounds cool, I would have a hard time trusting such a feature anyway, thinking about the implementation complexity it would entail.

Next, Magnus Hagander spoke about Hidden gems of PostgreSQL. Magnus works at Redpill who provides 24x7 PostgreSQL support (among a number of other services). As far as I know, Redpill has recently opened an office in Denmark -- which means that it's now possible to sign up for local PostgreSQL support in our little pond.

Magnus went through a few selected PostgreSQL features, meaning that he had the time to explain them properly:

• DDL operations (such as dropping a table) are transactional PostgreSQL. Magnus presented this as a rather exclusive feature which few other DBMSes have. Actually, DB2 has the feature, and it's a mixed blessing: Transactions are a tremendously handy and time-saving feature, including transactional DDL. But if DDLs are transactional, it also means that a user with very low privileges can lock the system catalog by performing a DDL and not committing -- meaning that other users (potentially equipped with loads of high permissions) are blocked from completing DDL operations. I assume that PostgreSQL's transactional DDL suffers from the same drawback(?) By the way, Magnus pointed out a serious drawback with performing DDLs in some other DBMSes that don't have transactional DDL: They may carry out an implicit commit when a DDL statement is executed; this leaves potential for rather uncomfortable situations.
  Update, Monday, Oct 6: PostgreSQL doesn't suffer from the problem described above for DB2.
  
• PostgreSQL now has built-in full text indexing (FTI), based on a somewhat cleaned up version of "Tsearch2" which used to be an add-on to PostgreSQL. The FTI can be been used in a simple way, but you can also configure it in very specific and powerful ways, using language specific dictionaries and/or specialized parsers and stemmers.
• Finally, Magnus when through a few of the packages in PostgreSQL's "contrib" add-on. The crypto add-on is something, I'd much like to have in DB2.

After the talks, I went to the SSLUG booth to have a look at the extremely small PC which was on display there. Fascinating stuff. I really like the trend towards down-scaled and cheaper PCs, exemplified also by the EEE (which were everywhere at the conference). At the booth, I had a chat with Phillip S. Bøgh who told me that for a typical desktop PC, 81% of its energy consumption actually happens during production, long before it's sold to the customer. The corollary is that there is value in keeping old hardware alive, instead of buying new equipment whenever some large software company decides to try to force us to buy new products featuring new heights of bloat.

Open Source Days 2008, Friday

For several years, there has been an annual two-day open source conference in Denmark. It has had different names in the past ("Linux 98", "Open Networks 99", "Linuxforum 200{0,2,3,4,5,6,7}"), but nowadays, it's called "Open Source Days".

I've attended the conference almost every year. This year is no exception, although I may miss out on most of the Saturday talks.

Here are my notes from Friday.

OpenID, by Simon Josefsson
Users of authentication-requiring web applications normally have an unfortunate choice: Use one or two passwords at all web sites, or store passwords in the local browser or a local password "wallet". The first option is clearly not attractive, because a rogue web site administrator could be using your credentials to log in as you on other web sites. The second option is troublesome if you use several PCs, or if your PC is stolen (workstations are often not regularly backed up). OpenID brings a good solution to this dilemma: Create an account at an OpenID provider which you choose to trust (I use myOpenId, currently). Then, you can use that account at all sites supporting OpenID logins (several weblog sites, Plaxo, Stack Overflow, etc). OpenID can also make life easier for web site developers.

Simon Josefsson went through the OpenID protocol, superficially (time was limited). In a comparison with other authentication systems, he noted that OpenID is based on a voluntary trust relationship between website and authenticator, in contrast with--e.g.--SAML. OpenID can only be used in a web context. All in all, OpenID is a rather simple and light-weight protocol.

The main potential security problem with OpenID is phishing, but Simon noted that this is a problem with other systems as well: Even though the system may use non-web-browser password dialogs, such dialogs can be rather closely mimicked using Flash. The most effective solution to the phishing threat is to avoid relying (exclusively) on passwords, through SMS-based one-time codes, one-time code dongles, etc. Simon's company produces an elegant, small USB device which emulates USB keyboards; when you press a button on the device, a long password is emitted. In combination with an encryption system, this results in very secure authentication.

Where I work, we face an identity handling challenge: We need to have the authenticator convey a list of group memberships for an account to the web application. OpenID has deliberately been kept simple, so there is no dedicated solution for that. But Simon noted that OpenID 2 includes an assertion mechanism which can--in priciple--be used to communicate any kind of attribute about a user to a web-site.
Unfortunately, we can't really use OpenID for the before mentioned challenge, but I would certainly look at OpenID if I were to implement an authentication system elsewhere.

Using Puppet to manage Linux and Mac platforms in an enterprise environment, by Nigel Kersten
Ever since I heard a recent interview with Luke Kanies, I've wanted to know more about Puppet. Luke has an interestering statement about system administrators: Sysadmins need to move to a higher level, by adopting some of the methology used in software development. This relates to version control, traceability, abstraction, and code reuse. I very much agree on this.

Without having personally tried Puppet yet, I think it's somewhat fair to characterize as a modern cfengine, and as the unix-world's version of the Microsoft-world's SMS-tool (SMS having better reporting facilities, while Puppet probably has better scripting features). Puppet has gained a lot of attention in Linux and Mac sysadm circles, lately. Kersten is part of a team managing more than 10000 Linux and Mac internal workstations at Google.

Puppet is written in the Ruby programming language. So it was reassuring to hear that Nigel Kersten is "a Python guy": Puppet is not just being hyped as an example of an Ruby implementation.

Random notes from the talk: I learned that Puppet can actually work offline: Many rules will work without network dependencies. And it seems that Puppet can be a good way to implement local adjustments to software packages without having to mess around with local re-packaging. Puppet goes out of its way to avoid adjusting configuration files if there is no need (nice: that way, file mtimes don't mysteriously change without file content changes). Unfortunately, it sounds like there are issues to be worked out regarding Puppet installations on workstations where SELinux is in enforcing mode.

Nigel has heard from no one with personal experiences getting Puppet running on AIX. And as we are (for better or for worse) using AIX on the majority of unix installations where I work, I probably can't justify fiddling with Puppet, currently.

By the way: RedMonk has a podcast where Nigel Kersten is interviewed. (RedMonk's podcasts generally have too much random chit-chat for my taste, but this interview is actually good, as far as I remember).

PostgreSQL
During a lunch break, I had a talk with Magnus Hagander at Redpill's excibition booth. Magnus Hagander is one of the developers of my favorite database system, PostgreSQL. PostgreSQL is generally being very conservative/unaggressive by default, in order to be a good citizen on any server installation. But often, the administrator of a PostgreSQL installation actually wants it to be very aggressive. I asked Magnus Hagander for a list of top-three PostgreSQL parameters he generally adjusts in PostgreSQL installations. His answer: shared buffers, work mem, checkpoint segment (and effective cache size).

Open Source Virtualization, an Overview, by Kris Buytaert
I've been using Xen virtualization for a while, both at home and at work. And I regularly touch IBM's Advanced POWER Virtualization as well as VMWare ESX. In other words, I'm interested in virtualization, not just from a theoretical perspective.

So I went to Kris Buytaert for an update of the status of open source virtualization technologies. Kris Boytaert went through the history of open source virtualization. He listed three virtualization products which he currently recommends: Xen for servers, VirtualBox for desktops, and Linux-VServer for mass hosting of web-servers. And he mentioned the openQRM solution which can be used for setting up a local cloud, as far as I understood. He had some surprising statements: If you have the choice between full, VT-based virtualization and paravirtualization, then go for paravirtualization, for performance reasons. Live migration is of little practical use (contrary to experiences where I work). It sounded like Kris is somewhat skeptical with regard to KVM; on the other hand, Kris described how Xen has been moving further and further away from the open source world, ever since it was bought by Citrix (Citrix: How can you let this slip away?)

Best practices, by Poul-Henning Kamp
The best practice concept is starting to annoy me. I've often heard obviously stupid solutions being motivated by "but that's best practice!"; the statement is often heard from someone with highly superficial knowledge about the related field. Recently, Mogens Nørgaard had some good comments about the phenomenon in his video column (in Danish only).

In his talk, Poul-Henning was also skeptical about the best practice term. He joked about people asking for operations to be done in a way which is "best practice, or better!". Apart from that, Poul-Henning went through various recommendations for programmers, C-programmers in particular: Do use assertions, and don't compile them away. Use code generation when possible. Print out your code, zoomed out to fit in few pages; surprisingly, that can reveal patterns in your code which you didn't realize. Use lints and other code checkers, and try compiling your code with different compilers on different platforms. Certainly good advice, but the talk left me wondering: How about changing to a programming language with better type safety, instead of all the band-aids? (I believe that Poul-Henning once touched upon this in another context, basically stating that C is the only realistic language for a systems programmer, for various reasons.)

Many people have high regard for Poul-Henning, the coder. At this talk, however, the loudness of the applauses were in the guru-admiration league. --Which was a bit out of proportion for the talk, in my opinion.