Clever Geek Handbook

Field constellation for cavern

Is it possible to group fields on tuples emitted by a kafka kapka? If so, how does Storm recognize the fields in Kafka's record?

+3


source to share


2 answers


Kafka Spout declared its output fields like any other component. My explanation is based on the current KafkaSpout implementation .

In the KafkaSpout.java class, we see the declareOutputFields method, which calls the getOutputFields () method of the KafkaConfig schema.

@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
    declarer.declare(_spoutConfig.scheme.getOutputFields());
}


By default, KafkaConfig uses RawMultiScheme , which implements this method this way.

  @Override
  public Fields getOutputFields() {
    return new Fields("bytes");
  }

So what does this mean ?, if you declared a bolt that reads tuples from KafkaSpout with fieldGrouping, you know that every tuple that contains equal fields, bytes, will do the same task. If you want to emit any field, you must implement a new circuit for your needs.

+1


source


Field grouping (and grouping in general) in Storm is for bolts, not spouts. This is done using a class InputDeclarer

.
When you call setBolt()

on TopologyBuilder

, it returns InputDeclarer

.



+2


source






More articles:


All Articles