Switch to GAE
What is the best way to transfer MySQL tables to Google Data Warehouse and create python models for them?
I have a PHP + MySQL project that I want to migrate to a Python + GAE project. The big hurdle so far is the migration of tables and the creation of the corresponding models. Each table is about 110 columns wide. Creating the model for the table by hand is a bit tedious, not to mention creating a loader and importing the generated csv table view.
Is there a more efficient way to migrate?
source to share
In general, creating your models automatically shouldn't be too difficult. Suppose you have a csv file for each table, with lines consisting of (field name, datatype), then something like this will do the job:
# Maps MySQL types to Datastore property classes
type_map = {
'char': 'StringProperty',
'text': 'TextProperty',
'int': 'IntegerProperty',
# ...
}
def generate_model_class(classname, definition_file):
ret = []
ret.append("class %s(db.Model):" % (classname,))
for fieldname, type in csv.reader(open(definition_file)):
ret.append(" %s = db.%s()" % (fieldname, type_map[type]))
return "\n".join(ret)
Once you've defined your schema, you can bulk load directly from the database - no need for intermediate CSV files. See my blog post on the subject.
source to share
In your shoes, I would write a one-line Python script to read the existing MySQL schema (with MySQLdb ) generating model.py to match (then do some manual checks and changes in the generated code, just in case). This assumes that a data model with "about 110" properties for an object is something that you are happy with and want to keep, of course; it might be worth taking the opportunity to dig a little bit (indeed, you might need to if your current approach also relies on joins or other SQL functions that GAE doesn't give you), but this of course requires more manual work.
Once the data model is in place, bulk loading can occur , usually via intermediate CSV files (there are several ways to create these).
source to share
You can port them first to django models
In particular, use
python manage.py inspectdb > models.py
And edit models.py to completion. You may need to include ForeignKeys, adjust the CharFields length, etc.
I have successfully converted several old databases to django.
Django models, however, are different from GAE models (which I'm not very familiar with), so it might not be very useful, I don't know!
source to share