Wednesday, April 03, 2013

JRuby on Heroku? Bah, humbug.

My experiments with JRuby on Heroku have mostly been a bust.

I migrated a fairly simple 2.0.0 application over to JRuby, which didn't involve much other than switching to Kramdown from RedCarpet 2, and spending some time fucking around with my `database.yml` so the `activerecord-jdbc-postgresql` gem would actually connect.

But goddamn, the problems.

I let the JRuby 1.7.3 version of the application run for a little less than 24 hours, and in that time I got:

* about 6 hours of downtime spread sporadically throughout the period
* four or five "memory quota exceeded" at peak time, which didn't seem to do anything
* several random dyno crashes I wasn't seeing under Ruby 2.0.0, which were followed by...
* ...many, many "boot timeout" errors, which I never saw under 2.0.0 unless I had a bad configuration / initializer that was causing the app to crash

And those are just the critical errors. Performance across the board was crap: the New Relic comparisons from last week versus the switch to JRuby were pretty bad looking. Lots of request timeouts. Lots of request timeouts. And request throughput was apparently crap.

That last thing -- throughput -- bothers me almost as much as the random crits above. The setup is running puma, a threadsafe web server for Rack, so I was expecting to see much better request throughput than what I actually got. For those of you not in the know, JRuby has "real" threading, where-as Ruby MRI has a GIL and green threads, which means threading performance is not very good.

Switching to JRuby should have given me a real boost in throughput, but, unexpectedly, performance was piss-poor and there was a significant increase of request timeouts. The site's not getting _that_ much traffic, and the assets are stored on Cloudfront, so...

I don't have any idea what to make of this. Apparently the setup is working well for others (I don't see any complaints on Google, at least). But with such a simple setup on Heroku going sideways, I'm not sure what to make of it. Is it a bug in JRuby 1.7.3? A Heroku-specific problem?

Meh. I've redeployed on MRI, and the numbers are turning back to normal.

That concludes that experiment.

UPDATE 4/4/2013:

For curiosity's sake, I ended up chasing the source of the downtime to a bug in Kramdown not being able to deal with some erratically formatted content. The code doesn't throw an exception, it just... hangs, I guess. New Relic reports a ridiculously long runtime of 1.5 million milliseconds, but I'm not sure if that's because it actually takes that long to render, or something (probably Herkou) killed the process, and that's just how long it took. RedCarpet has no problems with the markdown in question, which explains why I wasn't seeing any problems before the switch to JRuby.

No comments: