How to use parameters in Python Luigi
How do I pass parameters to Luigi? if I have a python file named FileFinder.py with a class named getFIles:
class getFiles(luigi.Task):
and I want to pass to a directory to this class, for example:
C://Documents//fileName
and then use this parameter in my run method
def run(self):
how can I run this on the command line and add a parameter to use in my code? I used to run this file on the command line like this:
python FileFinder.py getFiles --local-scheduler
What should I add to my code to use the parameter and how do I add this parameter to the command line argument?
Also, as an extension of this question, how to use multiple arguments? or arguments of different data types like strings or lists?
source to share
As you already figured out, you can pass luigi arguments via
--param-name param-value
at the command line. Within your code, you must declare these variables by instantiating the class Parameter
or one of its subclasses. Subclasses are used to indicate luigi if the variable is of a data type that is not a string. Here's an example that uses two command line arguments, one Int
and one List
:
import luigi
class testClass(luigi.Task):
int_var = luigi.IntParameter()
list_var = luigi.ListParameter()
def run(self):
print('Integer Param + 1 = %i' % (self.int_var + 1))
list_var = list(self.list_var)
list_var.append('new_elem')
print('List Param with added element: ' + str(list_var))
Note that ListParams are actually converted to luigi tuples, so if you want to perform operations on them, you need to translate them first (this is a known issue , but doesn't look like it will be fixed anytime soon).
You can call the above module from the command line as follows (I saved the code as a file named "testmodule.py" and made the call from the same directory):
luigi --module testmodule testClass --int-var 3 --list-var '[1,2,3]' --local-scheduler
Note that for variables containing a _
, this should be replaced with -
. The call raises (along with many status messages):
Integer Param + 1 = 4
List Param with added element: [1, 2, 3, 'new_elem']
source to share
So, I think it works, in the code I added:
fileName = luigi.Parameter()
if i run this on the command line:
python FileFinder.py getFiles --local-scheduler --getFiles-fileName C://Documents//fileName
but if anyone has any advice on the different types of parameters and their use, especially numbers and lists, please let me know.
source to share