Recently, I've been working on an app that involves near real-time ingestion and indexing of full twitter fire-hose on cutting-edge physical hardware. This mostly involves Lucene and home grown secret sauce analytical algorithms in pure Java.
As an enthusiastic Java developer, I decided to switch run-time of this app to newly released JDK 8, because why not?
This worked great with Linux on multitudes of hardware (32-bit AMD/K2 based CPU on my laptop, raspberry-pi/ARM, 64-bit Intel-I7 Desktop/Intel, 2 generations old 64-bit 2U-Xeon/Intel based server - all with spinning rust), but when I deployed the same app to our latest Xeon servers with SSD based storage I started seeing problems. The first problem I observed was on index optimization phase with insignificant index size for SSD storage (~12G). In optimization phase, the machine becomes irresponsive with ~70% system utilization with no more than 2% io-utilization. The second problem was with zero hits (i.e. no search results) on near real time Index reader, this was unique to SSD based server because when I copy the same index on other machine it would work just fine.
Despite many JDK8 bugs reported by Lucene committers, it looks like there are still some hidden JIT compiler bugs lurking in corner cases when your indices are very large even with the latest JDK 8.
The workaround that has worked (great) for me so far is to switch back to latest JDK 7.
When your app is deployed on SSD based storage infrastructure, the bottleneck of a search app shifts from IO to CPU and you start to see these type of issues. Even after a couple of days of googling, I haven't found anyone on internet seeing this problem, so I'll conclude that this may be specific to my use-case but hopefully someone will find switching back to JDK7 useful.
No comments:
Post a Comment