Data modeling in FORTRAN or C (or managed code)?

We are planning to develop a data processing package for windows. The core program / calculation engine will be developed in F # using GUI files / DB bindings, etc., done in C # and F #.

However, we have not yet decided on the implementation of the model. Since we want high performance, we probably cannot use managed code here (any objections here?). The question is, is it smart to design models in FORTRAN or should we stick with C (or perhaps C ++). We are looking into using OpenCL at some point for suitable models - you find it funny to go from managed code -> FORTRAN -> C -> calling OpenCL for these situations.

Any recommendations?

+2


source to share


2 answers


F # is compiled to the CLR, which has a compiler at exactly the right time. It is a dialect of ML that is strongly typed, allowing all the nice optimizations that come with this type of architecture; this means that you are likely to get reasonable performance from F #. For comparison, you can also try porting your code to OCaml (IIRC compiles this to native code) and see if it affects the material.

If this is really too slow, then see how far you get to the scaling hardware. With the performance available through a modern PC or server, it seems unlikely that you will need to go for something exotic unless you are working with truly brobdinagian kits. Users with smaller datasets can work fine on a regular PC.

Workstations give you perhaps an order of magnitude more storage capacity than a standard desktop PC. A high-performance workstation such as the HP Z800 or XW9400 (a similar kit is available from several other manufacturers) can accept two 4 or 6 core processor chips, tens of gigabytes of RAM (up to 192 GB in some cases) and has various options for high-speed input drives - SAS type outputs, external disk arrays or SSD.This type of equipment is expensive but can be cheaper than most of the programmer's time. Your existing desktop support infrastructure shouldn't be in such a set. The most likely issue is compatibility issues with 32-bit software on 64-bit O / S. In this case, you have various options such as virtual machines or KVM switches to solve compatibility issues.

The next step is socket server 4 or 8. Fairly common wintel servers go up to 8 sockets (32-48 cores) and maybe 512GB of RAM - without having to move from the Wintel platform. This gives you a fairly wide range of options on your platform of choice before you need to jump to anything exotic 1 .

Finally, if you can't get it up and running quickly in F #, check out the F # prototype and do a C implementation using the F # prototype as the control. If it's not fast enough yet, you're in trouble.



If your application can be structured in a way that suits the platform, you might want to look at a more exotic platform. Depending on what will work with your application, you can host it in a cluster, a cloud provider, or build a core GPU engine , Cell processor, or FPGA. However, doing this you get (quite significant) additional costs and exotic dependencies that can cause support issues. You may also need to bring in a third party consultant who knows how to program the platform.

In the end, the best advice is to suck it and watch it. If you're comfortable with F #, you can quickly prototype your application. See how fast it runs and doesn't worry too much about performance until you have a clear indication that it will indeed be a problem. Remember Knuth said premature optimization is the root of all evil people 97% of the time. Watch out for problems and reevaluate your strategy if you think performance will actually cause problems.

Edit: If you want to build a packaged application, you are likely to be more performance sensitive than you would otherwise. In this case, performance is likely to become an issue sooner than when using a standalone system. However, this does not affect the basic principle of "suck and see".


  • For example, if you run the risk of starting a game with a bingword, if your application can be parallelized and made to run in a non-shared architecture, you can see if one of the [duck] cloud server vendors can be called to accept it. The corresponding interface can be created to run locally or via a browser.

    However, in this type of architecture, the connection to the data source becomes a bottleneck. If you have large datasets, loading this data to the service provider becomes a problem. It may be faster to process a large dataset locally than download it over an Internet connection.
+4


source


I would advise not to do optimization yet. Try to get a working prototype first and then figure out where the computation time is going. You can probably port the biggest bottlenecks to C or Fortran when and when needed - then see what the difference is.



As the saying goes, often 90% of the computation is spent on 10% of the code.

+3


source







All Articles