Oops - Bad DNS Update 2

Posted by JD 03/03/2010 at 11:59

Today I was migrating a few core services from one server to another as part of the new server build project. Basically, I need to wipe the 1st physical server and reload it with a new, different OS as part of this project. Before that can be accomplished, there are a number of services that I need to migrate to a new VM running on a different server.

Along the way, a bad DNS update was made to the DNS service for *.jdpfu.com. Basically, my error took the jdpfu.com DNS resolution, website and email servers off the internet for about 90 minutes (10:34a-11:49a) according to the logs. That’s the difference between using the public and internal IPs for the services.

Ooops.

This points out why any server or network changes should only be made during approved maintenance windows. I didn’t intend to change the DNS at all, but I did intend to migrate the management services from one server to another. BTW, the two servers involved were not considered production servers. The 1st server only needs to be available overnight for backups.

The website was never actually down, just the public DNS entry was bad, so nobody could get to it.

There is good news. I use two DNS management services and only messed up one. Previously, those programs where compiled from source code and custom installed. Now they are Ubuntu package installations, so any updates will be handled automatically. That’s a step forward in my book provided I never screw up the settings for either of the tools again.

I knew better, but didn’t think there could be an issue. Oops.

Trackbacks

Use the following link to trackback from your own site:
https://blog.jdpfu.com/trackbacks?article_id=533

  1. urban_mari 07/02/2010 at 16:31

    Hi John,
    It sounds like you are an experienced admin. I had an oops moment while (amateurishly) adminning a change to DNS registrars in that I forgot to tell the original host/server (terapad.com) about the change. They allow communications through the interface but otherwise have little way to communicate to a human. Would you have any suggestion about how to regain control of the site that is still hosted there but not visible?

    If not, thanks anyway.

  2. JD 07/03/2010 at 07:58

    Urban Mari – thanks for the feedback.

    Sure, just access the site via IP address, not DNS name. If you are sharing the server with other virtual web domains, I think you are out of luck without asking for support to provide a temporary virtual domain for you to access it. It really comes down to whether you have your own IP or are sharing a single IP with 150 other users. Cheap isn’t always better.

    For some of my domains, we have multiple DNS domains that can be used to access the same server. (.biz and .com), so perhaps that is another option for you, assuming the IP address access doesn’t work.

    Finally, I tend to avoid using cPanel interfaces and like to manually configure my systems via an ssh login. With that, there is not web virtual domain, so you could go in that way too.

    I always want 2 ways to admin my servers – ssh is preferred but some web GUI is fine, provided I have another way to admin it too. Then there’s the main rules for setting up a web server.

    • You need 4 different companies
      1. Registrar for the DNS name – e.g. GoDaddy
      2. DNS Service for the IP to DNS name resolution – e.g. DynDNS
      3. Hosting provider – e.g. NearlyFreeSpeech, BlueHost, …
      4. SSL Cert provider – e.g. Verisign, startssl, etc.

    Yes it is easier to have all those in 1 company AND that single company loves it when you do, but when you want to make a change in any 1 of them, you will probably be screwed. Further, a mistake by a single person can bring down your website and you only have 1 company to beg for help from. Obviously, if you know what you are doing, 99.9999% of the time, not thing bad happens. I’ve never had any issues with my DNS service or Registrar in about 15 yrs of using them that wasn’t self inflicted. Fortunately, the self inflicted issues have only happened 2-3 times.

    I know you’ve heard this before, but a good daily backup of your site stored off that server probably would have made this a non-issue. You could cancel that old domain and create a new domain, then restore from the backup and be mostly happy. Even if you didn’t have a way to backup everything, a quick wget -R http://your_site.com/ could have grabbed all the content recursively (before the DNS issue). I prefer to use a code version control system for my changes and store them on a different machine that gets backed up. Restoring will be a pain, but it will be possible. Also, if I make a change that doesn’t work out so well, I can roll back to the older version fairly easily.

    Not all of these ideas are workable for everyone. Hopefully, 1 of them will help you the next time. Good luck.