Adding & Removing Domain Controllers – the good, the bad and everything in between!

So, as some of you may be aware, I recently ran a project to upgrade all the Domain Controllers (DC’s) in my organisations Active Directory (AD) environment from a mixture of 2008 R2 & 2012 to 2012 R2. Considering the number of domains/domain controllers involved, this was no simple feat – especially considering I did the entire piece of work myself!

Because I’m a bit of a documentation nut, I thought that it would make sense to document this process from beginning to end – building up a new domain controller and all the things involved; through to decommissioning a domain controller and getting rid of all of its traces. This is good for a few reasons:
1) so I don’t have to do it each and every time;
2) so that in a disaster situation, we’re well prepared; and
3) because lists make me happy ^_^

I have kept in place the order in which this process was performed – I’m a big fan of adding in domain controllers before removing old ones, especially if your plan is to overhaul your entire DC fleet (which is what I did and what generated these instructions!).

So, without further ado, here are my steps for both
a) commissioning a new DC; and
b) decommissioning a DC.

Now, I’ve tried to keep this documentation as organisational-neutral as possible, but some tweaks may slip in. I’ve also included a list of tips and tricks that I gained while doing this long and painful fun exercise.

I am also a huge fan of constructive criticism – so if you think I’ve missed something, let me know!


Adding a Domain Controller

So, when you’re adding a domain controller, there’s a bit of prep work involved to start:

  • Work out how you want your new structure to be once the new DC/DC’s are in place – You could be lucky and just be adding in an extra DC because you’ve been given the green-light. Or you could be in my situation – replacing all your DC’s in one hit. No matter what the situation, you’ll need to plan it out. Work out new server names (keeping in mind naming conventions), where all your Flexible Single Master Operations (FSMO) roles are going to sit, what replication links are going to work, what sites your new DC/DC’s are going to sit in. Put it all down on paper. I’m a big Visio fan and I like seeing things visually represented to get my head around it.
  • Check the health of your AD – before making any big changes, checking what state your domain is in should be a no-brainer. Seeing what errors come up and what is considered “normal” for your environment before you go about changing anything. I’d suggest going through these AD maintenance tasks prior to this kind of change. (Shout out to /u/pandiculator from Reddit for suggesting I add in this step!)
  • Request the new Domain Controller – Be it physical or virtual, through whichever change process you’re using – you are using a change process, right?
  • Basic tasks – Once the physical hardware/virtual machine is provisioned, there are basic things that will need to be done (which would need to be done to every new server, I would think)
    • Install the operating system and any drivers that are required
    • Configure the server using any standard settings/tools that are needed (e.g. monitoring agents, AV installs)
    • Join it to the domain (if the domain already exists) and the OCD side of me wants to make sure you give it a proper details description!
    • Perform all Windows Updates

Your shiny new server is ready to be a domain controller. Wooo!

Promote the server to be a domain controller – This can be done either via the GUI or via PowerShell. I’m personally a fan of doing the first one via GUI (to ensure I’ve got all the right settings because I will inevitably forget something) and then grab the PowerShell and use that for any that I have to add after. Server 2012/2012 R2 no longer uses the old ‘dcpromo’ to promote a domain controller & install Active Directory Directory Service (AD DS). Instead, it’s a two step process:

  • Add the AD DS Role – This is the easy part, pretty much a “Next, Next, Next, Finish” style of install
  • Configure the AD DS Role – This is the promotion of the domain controller (pretty much the dcpromo part) and involves all the extra configuration you’d expect

Few things I’d recommend when configuring your domain controller:

  • Make your DC a Global Catalogue (GC) – The only big downsides to this are size of the database (might need a bit more HDD space) and increased replication. Truthfully, this is worth it, if it means your users/computers can always access a GC when they need to
  • Ensure your Directory Services Restore Mode (DSRM) password is documented somewhere – This should be a no-brainer, but it pays to mention it. You really want to write this thing down, just in case.
  • Check the location of new folder creation – I’m personally a fan of putting all three of the database, log, & sysvol folders on a separate drive. You don’t have to do this if you’re happy keeping it all on C: (and your C: is big enough!).
  • Export the Powershell! – This is the big one to pay attention to. When you’ve configured it the way you want to, export the PowerShell command that the GUI lets you view, so you don’t have to go through this long-winded GUI process again. If you’re only doing one, you may not want to bother, but if you were building 20 (like I was!) it becomes a long process to do manually.

Once the DC has been promoted, I usually check a few places to be sure that it’s promoted successfully:

  • Check within AD DS itself – You want to be sure the DC is now showing up within the Domain Controller OU. If it’s not, something didn’t quite go to plan!
  • DNS – Do an nslookup of the domain, you should now see the IP address of the new domain controller in this list. Again, if it’s not showing up, something hasn’t gone quite right
  • Sites & Services – Check Sites & Services, just to make sure your new DC is showing up under whichever site you added it to and has generated replication links.
  • Run through your maintenance checks again – You know those checks you ran at the beginning? Yeah, you get to run through them again now. This is just to make sure that things are behaving as they should be.

Once you’ve done that, you’ll probably want to add in any extra settings & tools that you would normally have on a DC. The following is a list of what I recommend you have on a DC (perhaps not on all, but on at least one per domain!):

  • FSMO roles – If the new servers are going to be replacing old ones (like what I did), you’ll want to shift the FSMO roles across. How you do this is entirely up to you – I like to split my FSMO roles between two servers, down the “Domain” and “Enterprise” lines: PDCE, RID & Infrastructure on one; Schema & Domain Naming on another.
  • Time synchronisation – Make sure the time settings are set correctly: if the new server now hosts the PDCE role, make sure the NTP settings are correct; if it’s a standard DC, make sure the time settings are pointing to whatever is the current PDCE (see here for more info on time settings!)
  • Log monitoring – This can be whatever you want it to be. We currently use two different products: ELK (see here for an awesome guide on setting it up) & the sekret-skwirrel product our Security team uses. Make sure you have your forwarders set up as soon as practicably possible after promotion so you don’t miss any events.
  • Backups – As much as I love my backup team, we do like having our own backups of both AD & GPO’s. We do this via a system state backup that’s run on two different DC’s (though it can be less or more depending on your environment) and also run a script to backup our GPO repository (and AGPM!).

Removing a Domain Controller

Removing a domain controller doesn’t require as much technical work as adding one does luckily!

  • Work out how you want your new structure to be once the old DC/DC’s are removed – As with adding a DC, you want to make sure all your bases are covered when you’re removing a DC. Check if the server has FSMO roles and if they need to be shuffled, if it’s the last DC in a site and site links may need to be changed etc.
  • Check the health of your AD – Again, as with adding a DC, checking what state your domain is in should be a no-brainer prior to removing a DC. Seeing what errors come up and what is considered “normal” for your environment before you go about changing anything. I’d suggest going through these AD maintenance tasks prior to this kind of change. (Shout out to /u/pandiculator from Reddit for suggesting I add in this step!)
  • Request the removal of the old Domain Controller – through whichever change process you’re using – again, you are using a change process, right?! Even if you’re not, some form of communication to let people know what’s going on (I go into this in more detail in the “Tips and Tricks” section).

Once you’ve got approval through your change process, you can proceed with the removal!

First step in decommissioning a DC? Shut it down. Yup, you read me right. Turn it off. Leave it off. This is a way of catching all those lovely people who’ve used this specific DC’s name somewhere in their server/service/configuration/script. There’s more detail about this in the “Tips and Tricks” section below.

Once a DC has been turned off and you’ve ascertained what’s talking to it directly and have fixed those up (again, see below in the “Tips and Tricks” section for that!) you can then proceed to doing the actual demotion.

Removing the DC – I run this very similarly to the promotion in that I do the first one via the GUI, export the PowerShell and use that on all the others I’m planning on removing.

  • Demote the domain controller – when you do this, it’ll politely remind you that the server is a domain controller so you’ll need to ‘Demote this domain controller’ before continuing. Go ahead with the demotion. This is a fairly straight forward “Next, Next, Next, Demote” process.
  • Remove the AD DS role – Once the DC has been demoted, the server will restart and you’ll need to login to it as you would any other server (so, using your Server Administrator password rather than you Domain Admin password, right? If not, you need to brush up on your AD Security…) and remove the role. Again, another straight forward “Next, Next, Next, Done” procedure.

As with adding a domain controller, there are a few things I recommend doing after removing one:

  • Document the local administrative password – Even if this includes just writing it down somewhere while you’re doing the demotion, just in case you need to get back into the server and there is network connectivity issues.
  • Remember to remove it from monitoring/backup schedules – DC’s are special beings and usually have special routines. They are monitored differently, backed up differently, have log monitoring/forwarders on them. Make sure to remove all of this – and not just from the server itself, but from all the systems that were looking at it for information (e.g. SCOM, Nagios).
  • Don’t continue to use the server (decommission it completely!) – Hopefully, when this server is no longer a DC, you’re planning on getting rid of it because it’s no longer required (and, fingers crossed, it shouldn’t hold any other roles at all!) Don’t reuse the server, especially if you’re not planning on changing its name or IP address. This can just cause confusion if there any systems out there that are trying to talk to the decommissioned DC. I highly recommend removing the system altogether once the AD DS role has been removed
  • Run through your maintenance checks again – You know those checks you ran at the beginning? Yeah, you get to run through them again now. This is just to make sure that things are behaving as they should be.

Now, despite Microsoft being fabulous in the way they do demotions, sometimes it just isn’t clean. You may have to do a little manual clean-up yourself, so I highly recommend checking the following to be sure the server doesn’t exist any more:

  • Domain Controllers OU – This should automatically happen, and I have yet to see this fail. The server should no longer be showing in this OU. If it is, something has gone wrong in the demotion and you will have hopefully received some form of error message as to why.
  • DNS – I had to do this for every single DC I removed. For some reason, DNS cleanup just didn’t happen (maybe I’m just bad at DNS, that’s not out of the realm of possibility!). So, check your DNS for entries that point to the DC you’ve just removed. And I mean properly check! There are all sorts of places DC names/IP’s are stored, be sure that it’s completely gone.
  • Sites & Services – When I performed mine, most of the Sites & Services did clean up themselves, but I did have a few manual removals that were required out in my DMZ domain. Just do a quick check to make sure that a) the server has gone and b) none of your existing servers are using the decommissioned server as a replication link.

Raising Domain/Forest Functional Level

So, now that you’ve done that, you may want to raise the domain or forest functional level. Exciting!

Few things you’ll need to have done:

  • Creation of new DC’s – You’ll want to ensure that all your new DC’s are at (or above) the level you want to raise your domain/forest to. Example – if you’re wanting to promote from 2003 to 2008 R2, make sure your new DC’s are at least Server 2008 R2.
  • Removal of DC’s – Flipside of the above: if you have any old DC’s you’ll need to get rid of them (and it just so happens there’s a handy guide above!). This means getting rid of those Server 2003 DC’s!
  • Testing – Functional levels are relatively simple and are predominantly about adding features, but there have been some removals and this means you should be testing to make sure it will work for you.

You should be ready, so you’ll need to do the following:

  • Open ‘Active Directory Domains and Trusts’
  • Right-click on your domain and select ‘Raise Domain Functional Level…’
  • Select the domain functional level you’d like and click ‘OK’

No, really. That’s it. It seems so simple – and you’ll probably spend at least 20-30x more time planning it than you do actually implementing it.

Once your domain/s is raised, you can then look at raising the forest…and it’s just as simple:

  • Open ‘Active Directory Domains and Trusts’
  • Right-click on your ‘Active Directory Domains and Trusts’ and select ‘Raise Forest Functional Level…’
  • Select the forest functional level you’d like and click ‘OK’

And you’re done. No fan fare. No streamers. No flashing lights. Trust me, it’s a bit anticlimactic – but you’ve done a good job!

Tips and Tricks

  • Do not promote all DC’s at once! – Trap for young players. Learnt this myself the hard way. Joined 6 new DC’s to my production domain in one hit and promoted them all…at the same time. Poor domain was absolutely flooded. Just glad I did it early in the morning when few people were around and wouldn’t have noticed the delays in auth requests while the new DC’s were replicating…
  • Do not remove all DC’s at once! – This one is one I didn’t get stung by, but still feel the need to pass on as it can catch people out. Especially if you have a number of DC’s you’re removing, I highly recommend decommissioning them in some kind of order. I personally suggest turning them off and leaving each one turned off for a minimum of one week. For my non-production environments, I did one week batches of two at a time (with all removed in 2 weeks). For my production systems, I was a bit more cautious and went with one at a time and left it off for two weeks before proceeding (6 weeks). I also suggest starting with non-FSMO role holders first (as these are least likely to have been used in configurations).
  • Do not run other roles on your DC’s – I thought this was a no-brainer as well, but I received so many comments regarding “You didn’t mention what to do if you run a CA/File Server/WSUS server from the same server?” A DC shouldn’t house any other roles but it’s own and some networking (e.g DNS or DHCP). CA’s should be separate and file servers should never be on the same server as a DC. I don’t care *how* broke your SMB is. If you’re not going to do it properly, don’t bother to do it at all.
  • Conditional Forwarders – We have a few of these in our environment, and they list the AD DNS servers specifically. So, if you add or remove a DC, you need to do the same within your conditional forwarders. If you don’t, DNS timeouts can start to occur from other systems.
  • Check your trusts – Do a “Validate” to be sure that the trust is still working! Sometimes trusts can get finicky when DC’s are added/removed and require that “Validate” just to kick them back into gear.
  • Non-Windows DNS – In large enterprise environments, Windows DNS is not the be-all-and-end-all. We have a non-Windows DNS system that’s in charge of DNS for our entire enterprise – this also has conditional forwarders. However, these kind of systems need to know what the explicit list of DC’s is in order for to function correctly, so be sure to amend these (or get whoever is responsible for it to do so!)
  • Let people know what’s going on – you would be amazed by the number of systems/services/people that rely on the domain. Especially in the case of decommissioning domain controllers, you really do need to let people know what’s going on because they may have dependencies that will need to be fixed up prior to the decommission taking place. This is why change processes are so important!
  • Put in a generic “Domain DC” DNS alias – This is one that has a few people on the fence, because most people are of the belief that everything should be using the domain name (e.g. contoso.com) for anything that requires it to speak to the domain. Unfortunately this isn’t always the case with a number of systems. The way around systems/services (and people!) using specific DC names is to create an alias e.g. DomainDC.domain.com that points to one specific domain controller. When it’s time for that DC to be decommissioned, the alias can just be changed to another DC and no one is any the wiser. This just makes life easier for yourself and your other operations teams.

I’ll be doing a follow-up post on raising domain functional level as an addition to this, as I feel it does deserve its own post!

So that’s it, from start to finish – provision and decommissioning a domain controller and all the stuff that goes with it. Hopefully I’ve covered everything but, as I said, if you feel there’s something missing I should include – more than happy to include it! Let me know in the comments what you think or if you’d do things differently!

14 thoughts on “Adding & Removing Domain Controllers – the good, the bad and everything in between!

  1. James A

    Great guide. Just one small trap we hit when removing a DC – don’t leave it off for too long, otherwise it’ll stop replicating and you’ll have to remove it manually.

    Reply
      1. Joe

        I refuse to shut down domain controllers. If proper change management is in place this shouldn’t be necessary. If people don’t pay attention and have systems hard coded to auth off one DC then they are responsible for that mistake and said system being down.

        I demote and decomm and move on. I am tired of the coddling of users. I have done this a long long time. It’s 2016, not 2003. Time for people to take personal responsibility at work.

  2. Michael Czepiel

    Love the post. Curious how you dealt with the DSRM password in the powershell script? I’ve avoided the non-GUI approach in the past as I didn’t have a good means of protecting the password. I have some new ideas when we move to Server 2016 later this year as we found a way to automatically change the DSRM password post promotion.

    Reply
    1. Axl

      A feature introduced in Windows Server 2008 R2 enables you to copy the password of a domain account to the DSRM password on a domain controller using ntdsutil. This makes maintenance of the DSRM password across an entire domain much easier, since you can schedule a task on each DC to copy the password of the domain account on a regular basis.

      To take advantage of this feature, first create a disabled user account in AD in which you will maintain the DSRM password. Next, run the following ntdsutil commands. Be sure to substitute the username of the account you are using:

      ntdsutil: set dsrm password
      Reset DSRM Administrator Password: sync from domain account dsrm-sync-user
      Password has been synchronized successfully.
      Reset DSRM Administrator Password: quit

      Note If you have multiple domains in your forest, you will need to create and maintain a user account to synchronize the DSRM password with in each domain.

      One of the easiest ways to regularly rotate the DSRM password is to schedule a task using Group Policy Preferences. To do this, first create and link to the Domain Controllers OU a group policy called DSRM Password Sync Policy. Edit the group policy and navigate to Computer Configuration\ Preferences\ Scheduled Tasks.
      • Name: DSRM Password Sync
      • Run: %windir%\ System32\ ntdsutil.exe
      • Arguments: “set dsrm password” “sync from domain account dsrm-sync-user” q q
      • Comments: Synchronizes the DSRM Password with the dsrm-sync-user domain account.

      Schedule the task to run periodically (for example, once a day at 9 A.M.) on the Schedule tab. Once you have created the policy and allowed it time to apply, verify that it was created successfully by checking the DC’s Task Scheduler

      Reply
  3. ahmadsikanderikander

    Can we access data of multiple domain controllers of same domain with one administrator user?.

    Reply
  4. Yia Xiong

    Why even decommission or demote the old DC? Why can’t you simply power it off and wipe it clean if you’re no longer using it.Trying to find this info but no one seems to talk about it or explain why and what takes place during the decommission process. Once you promote and replicate a new DC, doesn’t the new DC simply take over? The migration process requires you to rejoin the clients to the domain anyways so I just can’t see why this process is needed.

    Reply
    1. Yia Xiong

      Actually I see that demoting removes it from the list of available DCs which I can see might potentially cause complications.

      Reply
    2. girlgerms Post author

      Because of the metadata that is stored – it will still be listed within DNS as a domain controller, a global catalogue, LDAP server etc. The computer objects will remain in the Domain Controllers OU, receiving domaon cotroller group policies.

      They need to be demoted. They need to be removed. It’s not optional, unless you want to clear out all that metadata yourself manually.

      Reply
  5. Fred

    Hi,
    Nice post. Really like the general details of the process. One suggestion would be before turning off the DC. enable the firewall without blocking and have it log connections. This will at least allow you to see anonymous LDAP connections and any other app specifically going to that DC. it will give you some backup that you did do your due diligence prior to shutting it down that may cause an outage.

    Reply
  6. Kyle

    I use Wireshark to capture inbound traffic to the old_DC (with a display filter to eliminate traffic from other DCs) for 1-2 weeks before shutting it down. It helped me identify some outliers including remote office copiers that were using that old_DC for DNS.

    Reply

Leave a Reply