What's a good design pattern for implementing a dynamic data importer tool?
We are planning to create a dynamic data import tool. Basically, getting information at one end in a specific format (access, excel, csv) and uploading it to a web service.
The situation is that we do not know the names of the export fields, so the application will need to be able to see the definition of wsdl and match the valid entries on the other end.
In the import section, we can define most of the fields, but usually they have a few that are common. Which I don't see any problem with this.
I am just wondering if there is a design pattern that will suit this type of application or help in its development.
source to share
I'm not sure where the complexity is in your application, so I'll just give an example of how I used templates to import data in different formats. I created a factory that takes a file format as an argument and returns a parser for a specific file format. Then I use the build template. The parser is provided with a constructor that invokes the parser when it parses the file to create the desired data objects in the application.
// In this example file format describes a house (complex data object)
AbstractReader reader = factory.createReader("name of file format");
AbstractBuilder builder = new HouseBuilder(list_of_houses);
reader.import(text_stream, builder);
// now the list_of_houses should contain an extra house
// as defined in the text_stream
source to share
I would say adapter pattern since you are "adapting" the data from the file to the object, for example SqlDataDataAdapter makes it from Sql table to DataTable
is there a different adapter for each file type / format? For example, SqlDataAdptor, MySqlDataAdapter, they process the same commands but different data sources to get the same DataTable output
NTN
Bones
source to share
You will probably also need an abstract Factory and team patterns.
If the data doesn't match the format of the input, you may have to convert it in some way. This is when the command pattern appears. Since the formats are dynamic, you will need to base the commands you generate outside of the input. That Abstract Factory is helpful.
source to share
Our situation is that we need to import parametric shapes from competitors' files. The layout of their screen and data fields is similar, but so different that there is a conversion process. Plus, we have over a dozen competitors and maintenance would be a nightmare if it was done with code alone. Since most of them use tables to store their parameters for their shapes, we wrote a collection of shared objects to convert X to Y.
In my CAD / CAM application, importing a file is a command. However, the transformation magic is done by a set of rules through the following steps.
- Import data into a table. Field names are also pulled in, depending on the format.
- We are passing the table to RuleSet. I'll explain the structure of the ruleset in a minute.
- The Ruleset transforms the data into a new set of objects (or tables) that we retrieve
- We pass the result on to the rest of the software.
A RuleSet contains a set of rules. The Rule can contain another Rule. The rule has a CONDITION that it checks and a MAP table.
MAP TABLE maps the input field to the field (or property) in the result. There can be one display or many. Matching shouldn't involve just popping an input value into an output field. We have syntax for calculating and concatenating strings.
This syntax is also used in the Condition and can include multiple files such as ([INFIELD1] and "-" and [INFIELD2]) = "AB" or [DIM1] + [DIM2]> 10. Anything between the brackets , is replaced with the input field.
Rules may contain other Rules. The way that this works is that for the ruleset matching to be applied to both its condition and the parents (or parents), it must be true. If the subRule has a mapping that conflicts with the parent display, the subRule binding is applied.
If two rules at the same level have a condition that is true and has an inconsistent match, then the rule with the higher index (or lower in the list if you are looking at the tree view) will be relevant to it.
Nested rules are equivalent to AND, while rules at the same level are equivalent to OR.
The result is a mapping table that is applied to the input to convert it to the desired output.
It looks like it is showing up in the UI. Namely a Treeview showing the rule hierarchy and a sidebar showing the mapping table and rule conditions. It is also important to create wizards that automate the general rule structures.
source to share