Apache Ignite stores JSON map in distributed cache

I wrote below code to read data from JSON file and insert it into Ignite distributed cache, this code works fine, however the requirement to create a ContainerAgg class is a problem for me. Our data structure is not predefined. The extract is generated dynamically based on user settings.

I tried to use BinaryObject, however, when I use BinaryObject I cannot run the SQL query, do you have any samples that use BinaryObject and don't use precompiled Java class for schema.

This "StreamVisitorExample" exists, however it uses the Precompiled Java Class (Instrument.class)

  public static void main(String[] args) {
    ObjectMapper mapper = new ObjectMapper();
    try (Ignite ignite = Ignition.start("examples/config/example-ignite.xml")) {
      if (!ExamplesUtils.hasServerNodes(ignite))
        return;

      CacheConfiguration<String, ContainerAgg> config = new CacheConfiguration<>(MASSIVE_CACHE);

      config.setIndexedTypes(String.class, ContainerAgg.class);

      ignite.getOrCreateCache(config);


      try (BufferedReader br = new BufferedReader(new FileReader(args[0]))) {

        IgniteCache<String, ContainerAgg> cache = Ignition.ignite().cache(MASSIVE_CACHE);

        String line = null;
        Long cnt = 0L;
        while ((line = br.readLine()) != null) {
          ContainerAgg inst = mapper.readValue(line, ContainerAgg.class);
          cache.put(cnt.toString(), inst);
          cnt++;
        }


        long startTime = System.currentTimeMillis();
        String sql =
            "SELECT SFID, LABEL, PARTID, PROVIDERID, SUM(TOTALCNT) "
            + "FROM CONTAINERAGG GROUP BY SFID, LABEL, PARTID, PROVIDERID";

        QueryCursor<List<?>> cursor = cache.query(new SqlFieldsQuery(sql));
        long endTime = System.currentTimeMillis();

        for (List<?> row : cursor.getAll()){
          System.out.println(row);
        }
        System.out.println("Total Time: " + (endTime - startTime));

      } catch (FileNotFoundException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
      } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
      }
    }
  }

}

      

To give you more information on what I did with BinaryObject

. I converted JSON to map

and added each record to BinaryObjectBuilder

and instantiated BinaryObject

and saved it toIgniteCache<String, BinaryObject>

+3


source to share


1 answer


BinaryObject

- way. To run SQL queries, you need to configure indexed fields with CacheConfiguration.QueryEntities

(see https://apacheignite.readme.io/docs/indexes#queryentity-based-configuration ).

However, you can only customize the request objects once per cache. Therefore, when your schema changes, you must destroy

cache and create a new one with the updated configuration QueryEntity

.



There is no Java example in this use case. However, you can take a look at the C # example, the APIs are very similar: https://github.com/apache/ignite/blob/master/modules/platforms/dotnet/examples/Apache.Ignite.Examples/Datagrid/BinaryModeExample. cs

+2


source







All Articles