Large MySQL database support - Error 2013 - PATCH attached.

[rosskevin]

Our production database has outgrown the standard method. I started experimenting with parameters on the mysqldump command, but had no luck. I decided to go back to command line and see if I had any issues dumping the db with the standard parameters and had no issues whatsoever. I believe the crux of this issue is that mysqldump is holding open the mysql connection while attemping to dump to gzip, and in tern utilize jS3tream to upload. Essentially, we receieved a "Lost Connection" to the database. While I could start manipulating the db to get around this, our production values are good, and inline with best practices. I decided to fix the problem by streaming to a temp file, then triggering jS3tream, followed by a cleanup of the file. The sendToS3 function has an optional third parameter that could generally be used for an interim temp file. I've tested this method, and it works well in our production system. I believe it is the correct way to solve the problem and handles larger databases with realistic timeouts/packet sizes etc.

[rosskevin]

I just noticed that my formatting of the sendToS3 method (needed because braces were not correct) may have formatted the entire doc. This is ok, but some of the comments are out.

I would recommend going with standard brace placement anyway, as the way you indented the braces was a cause for great confusion.

Again, sorry, everything may turn up as changed in the diff.

[soergel]

I'll be able to look at this in detail when I get back from vacation in a couple of weeks. Briefly, I vaguely remember getting around this by increasing the appropriate timeouts in mysqld. However, your method may be preferable because it finishes the dump faster (especially if there are table-locking issues, etc.)

[soergel]

Okay, yes, I had overcome this by setting long mysqld timeouts in my.cnf:

net_read_timeout=3600
net_write_timeout=3600

That may still be the right solution for some circumstances, e.g. if the databases are larger than the available scratch disk. I agree that the temp file approach will be better some of the time too, so I'll see about making that a configurable option. Thanks fot the patch!

[soergel]

As of v1.04, any block that contains the "UseTempFile?" keyword will be dumped to a local file before being streamed to S3. This works for regular filesystem backups and Subversion backups as well as for MySQL backups.