Unicorn on Heroku is throwing weird bugs that cause H13 server errors. What is killing my unicorns?
I have a Rails 4 application running on Heroku that suddenly started throwing a lot of H13 errors. I followed it through the logs and saw many errors with a unicorn complaining about
memoized object is not a string
. The processes are then handled by SIGIOT. Here's the exact error:
unicorn_http.rl:295: write_value: Assertion `rb_type((VALUE)(f)) == RUBY_T_STRING && "memoized object is not a string"' failed.
Errors can also look like this:
unicorn_http.rl:296: write_value: Assertion `(!!((!(((VALUE)(f) & RUBY_IMMEDIATE_MASK) || !!(((VALUE)(f) & ~((VALUE)RUBY_Qnil)) == 0)) && (int)(((struct RBasic*)(f))->flags & RUBY_T_MASK) != RUBY_T_NODE)?(((struct RBasic*)(f))->flags&((((VALUE)1)<<11))):((((int)(long)(f))&RUBY_FIXNUM_FLAG)||((((int)(long)(f))&RUBY_FLONUM_MASK) == RUBY_FLONUM_FLAG)))) && "unfrozen object"' failed.
This is happening on different paths and I can't figure out what's going on. I ran the unicorn config with a big headroom:
worker_processes Integer(ENV["WEB_CONCURRENCY"] || 2) timeout 29 preload_app true before_fork do |server, worker| Signal.trap 'TERM' do puts 'Unicorn master intercepting TERM and sending myself QUIT instead' Process.kill 'QUIT', Process.pid end defined?(ActiveRecord::Base) and ActiveRecord::Base.connection.disconnect! end after_fork do |server, worker| Signal.trap 'TERM' do puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to send QUIT' end defined?(ActiveRecord::Base) and ActiveRecord::Base.establish_connection end
And here is a snippet of Heroku logs with error:
Sep 15 15:38:42 Sep 15 15:38:42 Sep 15 15:38:42 Sep 15 15:38:42 ... Sep 15 15:26:52 Sep 15 15:26:52 Sep 15 15:26:52
A typical request queue is usually <10ms in this app and it seems to have just started recently, but I haven't changed the age unicorn setting. The app is a Heroku app with Rails 4.1.1 with Ruby 2.0.0 in Unicorn 4.8.2. It uses memecache with dalli for minimal caching. Requests don't work with error H13, so I don't find the 29th timeout set in the unicorn.rb config.
source to share
No one has answered this question yet
Check out similar questions: