Ruby Kafka Producer BufferUnderflow
I wanted to use the ruby kafka client library to generate events, but ran into a problem that I don't know how to solve. Any help would be greatly appreciated.
I've tried using kafka-rb (acrosa, mheffner and bpot forks). The problem is that no matter what I post to it via a library for example.
require 'kafka'
host = 'localhost'
port = 9092
producer = Kafka::Producer.new(
:topic => 'login',
:host => host,
:port => port
)
producer.send([Kafka::Message.new("aaaaa")])
I get:
java.nio.BufferUnderflowException
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
at java.nio.ByteBuffer.get(ByteBuffer.java:675)
at kafka.api.ApiUtils$.readShortString(ApiUtils.scala:22)
at kafka.api.ProducerRequest$.readFrom(ProducerRequest.scala:34)
at kafka.api.RequestKeys$$anonfun$1.apply(RequestKeys.scala:34)
at kafka.api.RequestKeys$$anonfun$1.apply(RequestKeys.scala:34)
at kafka.network.RequestChannel$Request.<init>(RequestChannel.scala:48)
at kafka.network.Processor.read(SocketServer.scala:321)
at kafka.network.Processor.run(SocketServer.scala:231)
at java.lang.Thread.run(Thread.java:680)
on server. On the same server, I can send text through the provided console manufacturer without any problem.
If you've seen this before, I would appreciate some help. Since I am not very familiar with Scala, I am not sure what the problem is, but it seems to me that the line where this exception is thrown is related to reading the clientId from the socket, and it also seems that there is no such thing sent from the ruby client ...
When I look at posts generated on tcpdump form kafka-rb and provided by the manufacturer. Rubies seem to be shorter. Also, it doesn't matter if I use kafka-0.7 or 0.8, I get the same behavior.
source to share
I can reproduce your error using Kafka 0.8, but when I try this implementation:
require 'rubygems'
require 'kafka'
producer = Kafka::Producer.new({ :host => "localhost", :port => 9092, :topic => "test" , :compression => 0 })
message = Kafka::Message.new("some random message content")
producer.send(message)
it works with Kafka 0.7:
According to the Kafka documentation, the mentioned client is only supported for 0.7.x ( https://cwiki.apache.org/KAFKA/clients.html#Clients-Ruby ). Section 0.8 says:
Release 0.8 changes the protocol significantly. This version has not yet been released. The new protocol is described here. There are a number of clients in progress and we will update them when completed.
So I think https://github.com/acrosa/kafka-rb won't work in this case: - (.
The best
preliminarily
source to share