Pyomo performance to create model with huge number of constraints
I am interested in the performance of Pyomo for creating an OR model with a huge amount of constraints and variables (about 10e6). I am currently using GAMS to run optimizations, but I would like to use various python functions and hence use Pyomo to generate the model.
I did some tests and apparently when I write the model the python methods used to define the constraints are called every time the constraint is executed. Before I go any further in my implementation, I would like to know if there is a way to create a constraint block directly based on the numpy array data? From my point of view, building block constraints can be more efficient for large models.
Do you think it is possible to get performance comparable to GAMS or other AMLs using the pyomo library or another python modeling library?
Thanks in advance for your help!
source to share
I was also curious when I came across this piece of code from Jonas Hörsch and Tom Brown and it was very helpful to me:
https://github.com/FRESNA/PyPSA/blob/master/pypsa/opt.py
They define classes to define constraints more efficiently than the original Pyomo parser does. I ran some tests on a large model that I have and this reduced the generation time significantly.
source to share
While you can use NumPy data when creating Pyomo constraints, you cannot create constraint blocks in a single NumPy style command with Pyomo. Anyway, I don't believe you can in languages like AMPL or GAMS. While Pyomo may eventually support users defining constraints using matrix and vector operations, this interface is unlikely to avoid creating separate constraints, since the solver interfaces (eg NL, LP, MPS files) are "flat" representations that explicitly represent are separate restrictions. This is because Pyomo must explicitly generate algebra representations (i.e. Expressions) to send to solutions. In contrast, NumPy only has to compute the result: it gets its efficiency by creating data in the C / C ++ backend (i.e. not in Python).relying on low-level BLAS operations for efficient computation of results and only casts Back to Python.
In terms of performance and scalability, I created original models with over 13e6 variables and 21e6 constraints. However, Pyomo was designed to be flexible and scalable for speed. Runtimes in Pyomo can be an order of magnitude slower than AMPL when using cPython (although this can be reduced by a factor of 4 or 5 using pypy). At least historically AMPL has been faster than GAMS, so the gap between Pyomo and GAMS should be smaller.
source to share