How to register instances of cryo serializer?
I am desperately trying to set up serializer instances for use in my storm topology.
There are two ways to register serializers in the storm documentation:
1. The name of a class to register. In this case, Storm will use Kryo’s FieldsSerializer to serialize the class. This may or may not be optimal for the class – see the Kryo docs for more details.
2. A map from the name of a class to register to an implementation of com.esotericsoftware.kryo.Serializer.
I want to use 2. →
Map<String, Object> serializerConfig = new HashMap<String, Object>();
serializerConfig.put(Record.class.getName(), new AvroSerializer(params));
conf.put(Config.TOPOLOGY_KRYO_REGISTER, serializerConfig);
Unfortunately, this leads to
Exception in thread "main" java.lang.IllegalArgumentException: Storm conf is not valid. Must be json-serializable
on topology representation.
Does anyone know how to do this (register serializer instances)?
Many thanks
source to share
There is a method in the backtype.storm.Config class to register a custom class derived from Serializer.
As an example, place this in the main method that creates and presents the topology:
// Read in Storm Configuration
Config conf = new Config();
conf.registerSerialization(Record.class, AvroSerializer.class);
source to share
As Stephen Magana-Zuck said above, you want to register the class in the config as he did. This apparently prevents you from passing parameters, but if you look into SerializationFactory.java in the source of the storm, you can see that it allows various possible constructors for your serializer class, including several that contain the storm Config. You can enter your parameters there.
Not exactly what you hoped for, but you must reach the same end.
source to share