Performing a large migration exercises a Drupal installation in different ways than normal production usage. There are several things you can do to improve the performance of your migration processes.
If you are going to upload large files through Table Wizard, you should increase the following parameters, as well as increasing Drupal's maximum file upload values:
php_value max_input_time 600 php_value upload_max_filesize 255M php_value post_max_size 255M
Bulk creation and deleting of nodes and users can strain memory usage - you should consider significantly boosting memory_limit:
php_value memory_limit 256M
When performing a migration process interactively, the Migrate module monitors execution time and exits a few seconds before max_execution_time or max_input_time is exhausted, to avoid partial updates. It is recommended to set them to a few minutes:
php_value max_execution_time 240 php_value max_input_time 240
In normal Drupal usage, nodes and users are created and deleted one at a time - thus, the core is optimized for that kind of load and is not well optimized for the bulk operations that the Migrate module performs. The following patches are recommended to help your performance when performing migrations.
node_delete() unnecessarily caches the node being deleted - when hundreds or thousands of nodes are deleted in a single request, you can very quickly run out of memory. Go to issue #287063 and apply the patch in comment #8 to prevent this caching.
Deleting a user updates the comments and node_comment_statistics tables rows referencing that user's uid, but those columns are not indexed, making deletion of users in bulk very slow. The patch at http://drupal.org/node/289504 addresses this.
Note: These patches have already been incorporated into Drupal 7.
Migrate can import data from another database into the default Drupal database. To set up a connection to an external database, see the Table Wizard help. Make sure that both database connections use the same user names and passwords; otherwise, MySQL will produce errors on certain queries. If the default database user does not have access to the external database, grant the 'select' privilege:
GRANT SELECT ON `external_db`.* TO 'default_db_user'@'localhost';