JDK 8, Lucene and SSD based Xeon Servers

Recently, I've been working on an app that involves near real-time ingestion and indexing of full twitter fire-hose on cutting-edge physical hardware. This mostly involves Lucene and home grown secret sauce analytical algorithms in pure Java.

As an enthusiastic Java developer, I decided to switch run-time of this app to newly released JDK 8, because why not?

This worked great with Linux on multitudes of hardware (32-bit AMD/K2 based CPU on my laptop, raspberry-pi/ARM, 64-bit Intel-I7 Desktop/Intel, 2 generations old 64-bit 2U-Xeon/Intel based server - all with spinning rust), but when I deployed the same app to our latest Xeon servers with SSD based storage I started seeing problems. The first problem I observed was on index optimization phase with insignificant index size for SSD storage (~12G). In optimization phase, the machine becomes irresponsive with ~70% system utilization with no more than 2% io-utilization. The second problem was with zero hits (i.e. no search results) on near real time Index reader, this was unique to SSD based server because when I copy the same index on other machine it would work just fine.

Despite many JDK8 bugs reported by Lucene committers, it looks like there are still some hidden JIT compiler bugs lurking in corner cases when your indices are very large even with the latest JDK 8.

The workaround that has worked (great) for me so far is to switch back to latest JDK 7.

When your app is deployed on SSD based storage infrastructure, the bottleneck of a search app shifts from IO to CPU and you start to see these type of issues. Even after a couple of days of googling, I haven't found anyone on internet seeing this problem, so I'll conclude that this may be specific to my use-case but hopefully someone will find switching back to JDK7 useful.

MySQL Insert performance and random primary key values

It's been almost a year since I've been putting up with MySQL and its bizarre performance characteristics and lack of choices on index data structures. If it wasn't for the volume and velocity of data we're storing in it, I would have proposed to ditch it by now.

We use MySQL purely as a stable high performance key-value store so many might not share the frustrations. I've been bitten by poor performance of InnoDB tables with semi-random values in primary key more than once now and hopefully this post will help someone who is in similar situation.

MySQL with InnoDB struggles with random primary key values, probably because it only supports BTree indices on it and page split cost on a BTree can be very IO intensive, I've only experienced this with numerical columns but it may be the true for other data types as well.

The only solution that has worked satisfactorily for me so far is to add an AUTO INCREMENT primary key and index the other column as unique. This is a complete waste of storage space but has offered the best insert performance so far. AUTO INCREMENT with upsert (ON DUPLICATE UPDATE) is probably the only reliable and performant way of using MySQL as super fast ingestion engine.