Today we’re very excited to announce the next step in our datacenter operations expansion. We’ll be switching primary operations from our current datacenter to the Amazon Web Services cloud in order to allow us to continue to expand our service reliability and capacity. We have been successfully running AWS as a backup site for two months and are comfortable that we’ve built a resilient new architecture that will allow us to greatly increase our quality of service and reduce outages.
What to expect
We will be executing this switch on Saturday Dec 1st from 06:00-10:00 EST. During this window, we expect all of Chargify’s services to CONTINUE TO RUN and are working to reduce any impact on availability to an absolute minimum. If everything goes as planned, you’ll experience no interruption in service. However, Chargify has a large number of moving parts, so if we run into any issues, there is the possibility for a small period where the site is unavailable to you. If this occurs, our aim is to keep this interruption to less than 15 minutes. We’ll be keeping you updated on the entire process at http://status.chargify.com/ and @chargifystatus
In addition to serving web and API requests, Chargify is also constantly running renewals for your customers in the background. For safety during the switch, these jobs will be paused. This is a very regular part of our operation: We often pause job processing during any kind of maintenance and we’ve got a battle-tested system for ensuring that nothing is double-billed and nothing is missed. This switchover will be no different and we expect to resume renewal processing immediately afterward. This is a well tested process on our end and we’re confident that there is no long-term negative impact on our renewal handling.
The technical details
We’ve developed our new infrastructure to handle simultaneous operation across multiple datacenters. However, one region must remain the ‘primary’. The primary region is where our DNS is pointed and all requests to our service begin. Because the DNS will be changing, you’ll be routed to new IP addresses whenever you make a request to Chargify.
Our DNS TTL is set at 5 minutes, so for a short period, you may still be routed to the old datacenter. The old datacenter will CONTINUE TO RUN, and will seamlessly forward your request behind-the-scenes to the new datacenter. You may experience increased latency as your request bounces from one region to another, but your requests won’t be lost or rejected.
We’ll be operating in all three availability zones in us-west2 (Oregon) region. Every piece of our infrastructure will be running simultaneously in 2a, 2b, & 2c. We’ve also done significant testing to ensure we can handle an outage of an entire availability zone. Finally, we’re also bringing online our THIRD regional backup in us-east1 that will be ready to take over on short notice if there is an extended outage of the entire us-west2 region.
New IP Addresses
Adding a new datacenter also means that the source we use for outgoing payment processing requests and webhooks will be coming from new locations. We have set up several new IP addresses as potential sources for traffic coming from Chargify. The latest set of IPs can always be referenced here: http://help.chargify.com/entries/20559136-chargify-ip-addresses We know six looks like a lot, but that is ONE per datacenter, so six IPs allows us to expand into six more zones without bugging you again. We’ll be notifying our merchants separately about this change, especially those using QuickPay.