From time to time, WebFaction (like all ISPs) requires that we migrate our accounts from one server to another.
The process of migration can, if we do not pay attention to the details, create major issues including data loss, data corruption, lost/duplicated posts, customer orders, etc.
The problems are particularly likely to happen for sites that use scripts that access shared data (e.g. in mysql databases, flat files, etc.). That includes all of you who are using pre-configured scripts like WordPress!
Fortunately, WebFaction helps us avoid problems in two main ways:
- The way in which WebFaction has carefully designed their automatic migration process will avoid most or all of these problems, but with a cost: your website will appear to go down for viewers out there on the internet for a period up to a few hours (viewers will see a WebFaction message in their browser saying the server is being migrated).
- When WebFaction notifies you of an upcoming required migration, WebFaction gives you the option to do a manual migration before that date. Manual migration is an involved process that is only for more technical users, but WebFaction documents it pretty thoroughly on this page.
But there is one missing piece.
Currently WebFaction does not document the exact steps that WebFaction takes during their automatic migrations.
By knowing WebFaction's migration plan, WebFaction customers like me who have scripts+databases can determine ahead of time whether WebFaction's automatic migration will work for us, or whether we need to do a manual migration to avoid data corruption or unnecessary downtime.
Today I have been working with the kind people at WebFaction support to answer this question and I'd like to share the answer with all WebFaction customers.
[WebFaction folks: if anything needs correction/updating, please let me know]
"WebFaction customer:" you!
"viewer" or "site viewer:" your customer: the person viewing your website in a web browser
WebFaction Automatic Migration Key Facts
- What data is being migrated: Your server stores your web files, crontabs, WebFaction-specific settings (WebFaction "domains," "websites," and "applications") and emails. It also stores the data of your databases (e.g. mysql databases you created using the WebFaction control panel) unless you have taken explicit steps in your scripts to access a database server other than "localhost." So, during the migration process, there will for a brief time be two copies of all this data, one on each server. It is important to make sure the data can never fall out of sync (i.e. two servers trying to modify what they think is "the" data), otherwise you will have duplicate or lost information (e.g. blog posts, comments, customer orders, etc.)
WebFaction Automatic Migration Steps
- 14 days before the migration date, WebFaction
- notifies WebFaction customers of upcoming migration by email. This email includes the IP address that will be used for the new server.
- 3 days before the migration date, WebFaction
- changes the DNS entries for your domains so that they have a "time-to-live" (TTL) of 5 minutes instead of the WebFaction's normal DNS TTL of 60 minutes. This setting is important because it will decrease the average amount of downtime seen by your site viewers during the migration. This setting must be done 3 days ahead of time because any DNS TTL setting takes up to 3 days to propagate across the internet.
- Note: WebFaction customers who host their own DNS entries (meaning that you do not point your domains on your domain registrar to WebFaction's standard DNS servers) should make a similar setting at most 3 days before the migration. Failing to do so may greatly increase the downtime seen by your site viewers.
- On the migration date at the specified time, WebFaction
- disables access to all your pages on the HTTP server on OldServer. From now on, any site viewer attempting to reach your web pages and scripts at OldServer (all HTTP requests) will get a generic message in their browser that the server is being migrated.
- kills all processes belonging to your linux user on OldServer (so this abruptly stops any of your scripts that are currently running: prepare for this by disabling your scripts before migration if it may cause data corruption)
- locks your linux user, preventing any new processes of yours from starting on OldServer (for example, cron jobs)
- Note: at this point, it should not be possible for anything to modify your database or any of your files on OldServer, since both HTTP and cron access has been stopped, as well as ssh and ftp access (which can no longer function because your linux user has been locked). The only way trouble could happen is if you are in phpMyAdmin at that moment making changes to your database, so don't do that. And you should make sure, before migration, that all your scripts are using "localhost" to find database servers. Your scripts should never hard-code the IP address or host name of any particular WebFaction server when opening the database, unless you are purposely running script across servers (in which case you need to think about a bigger migration problem).
- copies all data from OldServer to NewServer (including files and databases and other data: see above). Nitpick: actually the data copy starts days before the migration, but at this stage the automatic migration script makes sure all the latest versions of all data makes it to NewServer, so the effect is the same.
- brings up all servers on NewServer (HTTP, mysql, FTP, ssh, ...)
- switches the DNS entries for your domains so that they point your domains to the IP address of NewServer instead of the IP address of OldServer. WebFaction customers who host their own DNS server should do the same.
- enables an HTTP proxy server on OldServer that will proxy any HTTP requests that go to OldServer's IP address to the HTTP server running on NewServer. Note that OldServer will not use HTTP 30x redirects---it will relay the request to NewServer internally and then it will relay the response from NewServer back through OldServer and respond to the request from the viewer. This distinction may be important for some scripts that expect the viewer and the script to see the same server IP address.
- Note: there is no mysql proxy. You should not be making any mysql requests of OldServer, as explained above.
- emails a notification to WebFaction customers that the migration is complete
- Note: the average time from step 1 to step 10 is approximately 3 hours, during which your site viewers will continue to see the "migrating" message in their browser when they access your site. If WebFaction encounters issues, this time may be longer. After WebFaction completes through step 10, the servers are ready to serve your content through NewServer. But there is still a certain period when site viewer's web browsers will still head to OldServer because they have not yet refreshed the mapping from hostname to IP; the proxy server set up in step 8 will help paper over that period, and the DNS settings made 3 days ago will help reduce the duration of that period.
- 3 days after the day of migration, WebFaction
- changes the DNS entries for your domains to restore the "time-to-live" (TTL) from the temporary 5 minutes to the normal 60 minutes.
- 5 days after the day of migration, WebFaction
- disables the HTTP proxy server on OldServer that we set up in step 8. Now any HTTP requests to OldServer will fail.
- 7 days after the day of migration, WebFaction
- deletes all your data from OldServer (though WebFaction keeps a copy somewhere else). Remember that the data on OldServer has definitely remained unchanged since step 3 on the migration date, as explained above.
So the key insight here is that while there is a period when there are two copies of your data (both files and databases), there is no period when your scripts can be writing to those two copies at the same time from two servers.
The cost paid for this benefit is that your scripts will be killed abruptly at step 2 of the migration (which you can prepare for by disabling them ahead of the migration) and that your website will appear to go down for customers for a few hours (which you can only avoid using more complex, manual migration).
I hope this guide has been helpful to others, like me, who have a lot of scripts to migrate.
30 Dec '16, 17:38