Fast and scalable RPC between C # and CPython
I need some feedback on C # - cPython integration for a scientific application.
The script looks like this: C # for data collection and visualization - CPython for data processing using multiple and modifying third part library to implement domain specific tasks.
General use case:
- C # code get real time data from device
- C # code feeds data to cPython code and asks for clarification.
- Cpython code does magic
- Cpython code feeds the result back to C # code
- C # render data in WPF application
Both C # and cPython codes (item 1), 3) and 5)) are already optimized for speed. The data to be transferred in both directions (item 2) and 4)) are essentially 10E6 double arrays (no complicated data structure).
I would like to find a conjugate solution with good performance (transition speed 1 to 5) and decoupling capabilities that also minimize development on the client (C # code), sideways. In fact both sides are running on the same machine, but I would like the solution to scale.
The solutions I've tried are problems:
- a) C # + porting python algorithm to .NET - too much code to write / rewrite on the domain side lack of specialized cPython library provides almost everything on the scientific side.
- b) Embedding in C # ironpython + porting / porting the C ++ extension to managed code (with and without ironclad ) - marginal performance caused by marshaling of structures used when executing algorithms - dll hell when updating the version of subcomponents.
- c) C # nesting ironpython + rpc between ironpython and python - maintainability issues, not satisfied with the need to "transfer" data between three C # / ironpython / python.
- d) XML-RPC, JSON-RPC - performance (speed) not satisfying.
Solutions that have not tried to evaluate already, I plan to evaluate:
- e) Apache Thrift
- f) ZeroRPC
Thanks in advance for any criticism, comments and suggestions.
source to share
I have a similar requirement (C # ↔ Python RPC). I just experimented with Apache Thrift and I am very impressed. It's very fast: 50% slower than Pyro4 / pickle, 5x faster than RPyC. This is for sending numpy arrays as binary data. The rest of the overhead with Thrift is probably copying array data to / from strings. If Thrift could be refactored to take buffer objects instead of strings, I would expect it to be as fast as Pyro.
The code generation seems to be very clean and the API is good. The main thing it lacks is the documentation. However, I was able to figure it out from the main tutorial, check and view the source.
As far as I can see, ZeroRPC doesn't have a C # implementation yet.
source to share