Manually Connecting Mailboxes by MailboxGUID or Hey! Where did my mailboxes go?!

On Monday I learned the hard lesson of always making sure that your Active Directory is replicating properly, especially in the middle of a migration. At least this is the best explanation that I can come up with for what happened. We were wrapping up my first Small Business Server 2003 to SBS 2008 migration, with an already less than stellar performance due to the Exchange System Manager refusing to show the new server, and to a few mailboxes that refused to move over. We had to export those monstrosities to pst before we could move them over. Which turned into a mixed blessing later on. Things had come to the point of being ready to remove Exchange 2003 from the SBS 2003 server, which went along fairly smoothly. We rebooted the server ready to breathe a sigh of relief that this portion of the migration was done. Not so lucky, the true nightmare began then. The calls from the users began coming in that they could no longer access their email, even after confirming that they really were pointing to the new server. I fired up the Exchange Management console to find out what was wrong to discover that no mail enabled users showed up, aside from the three we had to import from pst. I took a look for disconnected mailboxes and to my horror we did not have any. Restoring from backup wasn’t an option as we had not been able to take one yet. This began a mostly fruitless 6 hour call with Microsoft. It is very sad when your bluetooth dies and you also are able to recharge it and start using it again during the same call. Fortunately I did not stop doing research into the problem while on the call and eventually cobbled together the solution that I am going to present to you now. My warning is – do NOT do it this way unless you absolutely have to.

Before I had made the call I ran a Get-MailboxStatistics. Interestingly this reported everyone’s mailbox as existing and containing data. This meant that the users had lost their Exchange attributes. To verify this required digging into Active Directory with ADSI edit. Open up ADSI edit and connect to the default view. Drill down to the OU of your user and pull up their properties. You will see a list of attributes set there. Some specific attributes are required to mail enable a user. legacyExchangeDN, homeMTA, mailNickname,  msExchHomeServerName, and finally as well as perhaps the most import is msExchMailboxGuid. homeMDB is required as well but fortunately this was still populated. On the users that were missing mailboxes none of the msExch attributes were set. Fortunately there did remain some users, so looking at those I was able to glean a few of the attributes.

legacyExchangeDN –This attribute needs to be configured to point to the login for your mail enabled user. This drills down through the organization name and the default administrative group name ending down in your user’s login name. Example:  /o=first organization/ou=exchange administrative group (fydibohf23spdlt)/cn=recipients/cn=jland

You can pull this information from your mailboxes actually. Run Get-MailboxStatistics | ft Displayname, LegacyDN

homeMTA – This may not be required under Exchange 2007, but we decided to set it none-the-less. This drills down through Active Directory to where your Exchange server’s MTA resides. Example: CN=Microsoft MTA,CN=SERVER,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=linthicum,DC=local

mailNickname – This one is easy to miss yet is very important. If you user still doesn’t show up as mail enabled go back and make sure you entered in the mailNickname. This generally would be your user’s login name, though you should consult your organization’s naming scheme to be sure.

msExchHomeServerName – This one is fairly self-descriptive. This points to where your server is located in Active Directory, as based off the organization name. An example is: /o=First Organization/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=SERVER

msExchMailboxGuid – This one is the kicker. Exchange won’t know what mailbox to connect your user to without this info. But it isn’t exactly easy to get ahold of either. First run Get-MailboxStatistics | ft Displayname, MailboxGUID. You’ll see everyone’s msExchMailboxGuid listed right there. Easy? No. Now you have to be able to get that into Active Directory. Which is a royal pain. Go down through the properties of your user and open up msExchMailboxGuid to put in some new information. See how you only have the options of decimal, hex, octal and binary? You need to convert this GUID into something usable.

Go to joeware’s great site and download the adfind tool.  Open up a command line and go to where you extracted the tool and run adfind -gc -b “” -binenc -f ” msExchMailboxGUID={{GUID:98ee00d7-df19-4282-bedf-3a1340b8b7c0}}” –dn where of course you replace the GUID with the one you are searching for.  This will return you some interesting output which still isn’t quite usable, though it may look that way at first glance. Your response is mostly hex, but not fully. You need to translate it. Pull up the table at Ascii Table and use this for your translation. Start going through the characters and when you find one that doesn’t match like for instance a lower case j or the number 4 unpaired, look through the red characters in the table for your character and you’ll see the conversion to hex in the separate column. Go through the whole string this way and you’ll eventually get a fully hex string. Go back into your msExchMailboxGUID and put that in and after you click ok you’ll see that the attribute has been populated with the string that you began with. Look very closely at it to make sure it matches. If there’s some deviation go back and check your look-up tables again. This string should match completely, otherwise your user will end up with an empty mailbox created. Here’s an example of how to convert the returned string:

\D70\EE\98\19\DF\82B\BE\DF\3A\13\40\B8\B7\C0

D7 00 EE 98 19 DF 82 42 BE DF 3A 13 40 B8 B7 C0

And another one:

T\BA\A04l\B8\EEM\9F\D6\40m\258\CE\A0

54 BA A0 34 6C B8 EE 4D 9F D6 40 6D 25 08 CE A0

Your user has now been mail enabled as a refresh of your Exchange Management Console will show, but is still missing a number of Exchange attributes. Run Set-Mailbox “My User” –ApplyMandatoryProperties and all the rest will be filled out for you.

The last bit is to clean up Outlook for your users. Even the ones not using Cached Mode still showed as disconnected until we re-put the server back into their profile. For the ones using Cached Mode it was also a good idea to delete their .oab files from their directory to force them to download a newly rebuilt OAB. This may or may not be necessary in your case as that was most likely related to the timing in the migration. If you didn’t delete them it would populate the DN to the user they’re emailing rather than the email address, thusly bouncing back to them when they emailed.

Some extra reading:

Understanding Mailbox GUIDs

How to Re-Home Exchange Mailbox Accounts

Using ADFind Utility

Setting Up Server 2003 as a RADIUS with DD-WRT

A co-worker of mine was having some difficulties in setting up a RADIUS for his wireless network which is what prompted this particular article. For setting up your wireless infrastructure there are times when you need a more centrally controlled solution for the authentication problem. This is where RADIUS, and more to the point Microsoft’s IAS, steps in. For your trivia needs RADIUS stands for Remote Authentication Dial-In User Service, while IAS stands for Internet Authentication Service. Normally I would be setting this up under Server 2008 but our needs were calling for Server 2003. I may follow-up with how to do this under Server 2008 as well and even delve into putting together an IAS farm. The WAP being used is a Buffalo WHR-125 with a fairly current build of DD-wrt v24 SP2 (09/24/09) on it.

First off before installing IAS we will be in need of a certificate for it to use. There are several ways of achieving this. The first method, and easiest/cheapest, is creating a self-signed certificate using the IIS 6 Resource Kit from Microsoft. A particular program need from this is SelfSSL so run through a custom installation and install SelfSSL. Open up a command prompt and navigate to where SelfSSL installed at and here is how we will be constructing a certificate:

C:\Program Files\IIS Resources\SelfSSL>selfssl  /N:CN=server.domain.local /K:1024 /V:1825

This will get you your self-signed certificate. Of course you can use 3^rd party certificates as well. Another method is to issue one from an internal CA. Don’t forget to implement CA best practices when using one. I personally would opt for a self-signed certificate unless you already have a CA available.

Next up is getting our IAS installed. You will find this from Add/Remove Programs Add/Remove Windows Components. In there look for Networking Services and go into Details. Internet Authentication Service will be displayed just a few down. Once installed open up the mmc for IAS and let’s get into configuration. Though we should set up our users first. I went with creating a security group named Wireless Authentication and added my users in there. Note that you will need to allow these users for remote access as well. One way is to go into the user’s properties and on the Dial-In tab select Allow access. This isn’t my preferred method though as it creates more work. The other method I shall detail a bit later.

Bring up your IAS controls and you’ll see categories available. We need to get ourselves configured for our access point. To do this we will create a RADIUS Client. Right click on RADIUS Clients and select New RADIUS Client. Give the policy a name and point it to the address of the access point. Next menu is selecting our vendor which we will want to keep as RADIUS Standard for our configuration, as well as most configurations.  Put in a key for this client and note it down as we will need to configure it in the WAP later on. No need for the Message Authenticator attribute as it is used by default with EAP, which is what we will be configuring. For more information about it read here.

We have our client configured on the server but we are also in need of a Remote Access Policy. Right click the Remote Access Policies and select New Remote Access Policy. We will go with the first option for setting up our policy, though creating a custom policy is easy enough as well. On the next screen Access Method we will select Wireless. On the next screen we can put our group to use. Add in your Wireless Authentication group, unless you prefer to control things at the user level. I prefer security groups so that is what we will use. Select PEAP for the authentication method. Check the configuration of it to ensure that EAP-MSCHAP V2 is selected and that the proper certificate is selected as well. If you get an error when selecting Configure complaining about certificates then you need to go back and verify that you have a properly issued certificate. This is where most problems stem from. In the configuration you may also wish to enable Fast Reconnect. I have read about some clients having issues with this but have not had any problems in my configuration. Your mileage may vary. Disable it if you are having problems authenticating clients routinely. Finish this wizard and you’ll have your policy. We’re not quite done with it yet though.

Bring up the properties on your newly created policy. On the encryption tab you will want only Strongest encryption checked. If there are authentication issues though, you will want to enable the others for diagnostics until you figure out what is properly supported by your WAP. This is also where we can enable the alternate method for allowing our users. Go to the Advanced tab and add Ignore-User-Dialin-Properties set to True. This will ignore the setting on your user’s Dial-in tab and truly allow you to control access via groups. Otherwise user settings will trump group settings, which can make for a headache in troubleshooting. Last thing to do is right click the root folder, Internet Authentication Service, and select Register server in Active Directory. What this does is add your server to the RAS and IAS Servers security group, which enables it to read accounts from your AD. Once we are done here we can finally go configure our access point.

This is specific to DD-WRT, so be sure to verify how to configure your own access point. Connect to your access point and go to the Wireless tab, then Wireless Security. Set it to WPA2 Enterprise and make sure you are using AES, unless you have a reason not to. Put in the address for your IAS server and now would be a great time to make sure that it is a static address. Leave the port as 1812 as IAS listens on that out of the box. Finally put in the preshared key that you configured from earlier. Save then apply and your access point is in business. All that is left is configuring your clients.

This is best done through Windows’ wireless configuration. Manually create a new connection configured with your WAP’s SSID and go into the Security settings on it. Set it to use PEAP and if you are using a non-domain joined machine, that also does not have the certificate that you configured the server with, then tell it not to validate the certificate and also not to use your domain logon and password. Connect wirelessly to your access point and see if you’re successful. If you are not then check your server’s System event log for errors. If you are getting bad username/password errors, and you know your username and password are correct, then start looking at your encryption and configured authentication protocols to make sure they all match. If you are seeing errors about no matching policy then make sure you have your user in the right group or matching the criteria of your policy. That covers the majority of problems you will run into when configuring IAS. Even if you don’t have a use for IAS as a RADIUS it is a good idea to set it up a few times for learning purposes when pursuing an MCSE.

Windows Media Player and Other Libraries

I have been greatly enjoying Windows 7 recently. Microsoft has done a lot right with it. I enjoyed it so much that I actually migrated my primary workhorse from OpenSUSE to Windows 7. Being able to pin programs to the taskbar as well as programs like remote desktop having quick access to recently used links through the start menu are nice little touches. One of the great things about 7 though is Homegroups. They’re a much needed breath of fresh air for your average workgroup. I can see this benefiting small businesses a lot as you get some immediate access to easy file AND printer sharing. The printer sharing part was what I liked best. Just add yourself to the homegroup and ta da it is there. But what I am going to touch on today is something that gave me some grief for a few hours last night and this morning.

I decided to give this streaming media thing a go. The music I was wishing to listen to, which by the way is the fantastic Piano Concert #2 by Sergei Rachmaninov, was on another system. So I thought that this would be a great time to test out the streaming capabilities. I quickly switched it on in the homegroup for that system and started up Windows Media Player on my desktop to give it a go. Sorry! Access denied. WMP was complaining that it cannot access the file. This would happen for everything in the music library on that system. Interestingly enough though, was that videos and pictures would work just fine. I found it odd to be a permissions issue since that was working, and also if I browsed through via Explorer I could play the same music files with whatever media player I chose, including WMP. It didn’t quite seem like a permissions issue, especially since WMP’s error message was so generic that it could be anything, but I wasn’t quite ready to discount it yet. So I gave it a good night’s sleep and returned to the problem in the morning.

After a cup of mocha things became a little bit clearer. I decided to test playing music on the problem system from my desktop, thereby reversing the stream. This worked just dandy. So I gave the music folders a permissions inspection and found the problem. On the system that could stream music from the library there was an extra user with permissions, namely the WMPNetworkSvc who had Read permissions on the folder. The problem system did not have this permission. Unfortunately it wasn’t so simple a fix as just adding the user as the system would report that the user did not exist. Inspecting other folders such as the pictures and video folders did report the proper permissions. Thusly I fixed things with a bit of Powershell magic, and if you use this don’t forget to substitute your account name for MyAccount:

Get-Acl C:\Users\MyAccount\Videos | Set-Acl  C:\Users\MyAccount\Music

This turned a trick! I could stream music to my heart’s desire. Now as for speculation as to why this particular permission is missing, the only guess I have is due to this system being a system that was upgraded from XP, to Vista, to Windows 7. I could quite easily see something getting fouled up along the way, especially since this is over a number of years. It is a good test system though. The other system I have has gone from Vista to 7 and it did not exhibit the same issue.

Windows Server Backup and Exchange 2007 with iSCSI

Since service pack 2 has recently been released for Exchange 2007 this has enabled the long awaited integration of Exchange aware backups with Server 2008’s new Windows Server Backup. WSB is Microsoft’s replacement of the old ntbackup that we all know and love. This new backup is simpler to use than ntbackup and has a number of interesting new features, but it also lacks some of the more useful features of ntbackup as well. One of these missing features that is rather vexatious is that you have to backup whole volumes. You can’t just backup the mailbox stores, or even specific files and folders as well. Another missing feature is that you can’t just back up to a specific folder nor mapped drive. This can cause problems, especially at small businesses that don’t feel like shelling out for a more robust backup program. You will have to dedicate a whole volume to WSB, so this requires a bit more planning ahead. This is a problem that we had to get creative about solving last night on the spot though.

The client has an Exchange 2007 server running on Server 2008. No backup software had been acquired as they had been waiting for SP2 to enable Exchange aware backups. SP2 installed just fine, which was great considering all the other migration issues we had earlier, but then the hang-up we ran into was discovering that WSB wants its own volume for backing up, and doesn’t want to backup to a mapped drive on one of the other servers. This was a source of consternation for a bit as we did not have a spare volume available nor could we just grab an external drive for this either. Fortunately StarWind Software has this great, and free, iSCSI target software. Using StarWind we were able to turn a chunk of storage on the server into a virtual drive and set it up as an iSCSI target. All of this without having to reboot too, which is a huge plus. We connected this to the Exchange server using iSCSI and that meant we were finally able to backup the server and flush those transaction logs that had been building up. This made for a pretty quick and easy fix as StarWind is simple enough to set up.  If you are in need of a quick fix for your backups this is one way to do it.

RSAT for Windows 7

Having recently converted over to Windows 7 one thing I found missing was the Remote Server Administration Tools. Well they are missing no longer. Go and get your RSAT goodies here! Don’t forget after installing you have to go into Programs & Features and add in the tools that you use.

On another note I have to admit that I am really liking Windows 7. I haven’t been using it very long and never made the time to really explore it during beta. But now that I have put it into full time use I have really come to like a number of the UI features. The quick documents/tabs off a program in the start menu or the pinned icons down in the task bar are great. It especially works out for having remote desktop pinned to the start menu and then a quick start list of my favourite servers just off it. I am also really appreciating the ease of use for the network connections icon in the tray. Very simple to cycle through various VPN connections now.

So overall? I like it a lot. No compatibility issues and has been a very painless conversion process.

Exchange 2007 Single Server Migrations for Profit or Headache

I was originally writing up a guide for migrating, actually transitioning, Exchange 2003 to Exchange 2007. There are lots of guides out there that would have better screenshots and perhaps even better written steps. Basically I would not really be meeting a need as there are already plenty out there doing so. So instead I am scrapping all of my original work and concentrating on issues that I believe are not talked about as much out there. Mostly these issues affect those that are doing single server migrations, which is basically you have one Exchange 2007 server holding all of your roles. They have caused me a great deal of headache and drama which I am sure is true for others doing such migrations as well. I would imagine that this is mostly the SMB sector, which is where the majority of my work in this is being done. Let’s talk about the biggest issue now, client access.

The CAS role plays a big part in your Exchange organization as it is the broker for all requests to your mailboxes. You will have MAPI requests as well as HTTPS, POP3 and others coming into this server. By the way as a security side note the recommended set up is to have your CAS role on your internal network with a reverse proxy in your DMZ for proxying requests through to your CAS. The CAS when it receives a request for a mailbox that resides on a 2003 servers proxies that request through to your 2003 server. No issues at all there. The problem that comes up, though, with having a CAS on the same server as your MB is that web requests no longer get proxied to your 2003 servers, they get redirected. This is due to davex.dll handling the requests on a mailbox server, and it will grab the requests first. Exprox.dll is what handles proxying. This redirection is not configurable either. So that causes a problem when it is redirecting an external request to an internal FQDN. That doesn’t work out too well and you get lots of angry OWA users wondering why their logins take them to an invalid address. For a more in depth explanation take a look here. Let’s take a look at a few ideas for mitigating this issue.

First off an easy fix would be to make sure your Exchange 2003 FQDN has a matching public address. This is not a recommended set up though at all. It is against best practices to have your internal domain match your external domain. Not to mention you can get a number of funny DNS issues going on if this is the case unless you’ve planned things out well. Read this article for some more DNS information, and especially look at the split-brain section. All of this can turn your easy fix into a much more complicated fix. If the stars do just happen to be right on your migration though, then go for this. Set up a public record matching your internal Exchange 2003 name and you’ll be set. This will be transparent to your users.

Next up would be to use a reverse proxy such as ISA 2006. This would be great as it keeps the strict definitions of your DMZ as it keeps your Exchange servers from having to blur the lines. This doesn’t seem to be something that most SMBs care about in my experience though. They don’t seem to see the need for security and how having a properly defined DMZ fits into this. But that goes into an entirely separate article and could sound a bit ranty.

Other methods will require a bit more cooperation from your users. Remember, in Exchange 2007 the OWA access by default is /owa. So you will need to communicate this to your users as you migrate their mailboxes over. Then, remove the /exchange virtual directory through the Exchange Shell and recreate it in IIS. Finally, set up /exchange with a custom 403 redirect to a different port on your external address. Mind you that you’ll need to make sure that port does point to your legacy server. This either requires your firewall to be able to do port translation or changing the ports on your 2003 server.

Finally, and the most recommended method, is to set up a temporary virtual machine that will purely host a CAS role. Then everything will be proxied as it is supposed to be. The down side to this is that it would require a separate license in which case you might as well plan for as separate CAS to begin with.

Fortunately as long as everything is configured properly Outlook Anywhere and ActiveSync seem to work just fine. Some dangers with those is if you are having some DNS issues internally or improper communication with a global catalog. This can add to your headache so you will want to cozy up to rcpping which you can grab from Microsoft and get more info about how to work it from here. Another great site I have recently found out about is the Remote Connectivity Analyzer. This site will enable you to test Outlook Anywhere, ActiveSync, SMTP and Autodiscover with detailed error messages about where these break down. It will become your best friend very swiftly.

I guess the moral of all these suggestions is to make sure you have your migration well planned out. Run it through a test lab first if you are able. Definitely make sure you test it out, and definitely don’t spring it on your users unawares. You could be in for quite a “fun” surprise.

DFS On Core — You’re Doing It Replicated

Anyone taken a look at Windows Server 2008 R2 yet? Things I’m excited about in it are PowerShell on Core, AD cmdlets, and the AD Recycle Bin. PS on Core is the most exciting addition though. Maybe later on I will start delving into R2 and talk about working with that on Core. This time, though, we are going to deal with setting up a basic DFS using Windows Server 2008 Core machines.

Core makes for a low resource file server that you can deploy to do its job without letting layers of the OS get in the way. Using it for a DFS will be a step in the right direction towards high availability of your data as well. Further more it can be used as a way to put some controls on your bandwidth utilization through having replicas of your data in locations that are local to your users. Failover is provided by pointing the users at the namespace which will then direct the users to the nearest server. Let’s run through putting together a setup on Core.

Grab our first server and let’s install the DFS NameSpace role.

C:\> start /w oclist DFSN-Server

Once this is complete we can start breaking out our trusty dfsutil.exe tool. We will start out with making a domain based namespace. Set up a share to use for this.

C:\> mkdir TurksNS
C:\> net share TurksNS=C:\TurksNS /GRANT:”Authenticated Users”,FULL

Don’t forget to customize the share and NTFS permissions to your specific needs.

C:\> dfsutil root adddom \\renocore\TurksNS “2008 Namespace”

You can also add V1 or V2 as a parameter. The default is V2. V1 is a Windows Server 2000 mode namespace while V2 is a 2008 mode namespace. Note that a requirement for a V2 namespace is a Windows Server 2008 domain functional level. If you receive any the RPC server is unavailable errors make sure the DFS Namespace service is running. Easiest way is to reboot but you can also run the sc command to start up the service.

C:\> sc start dfs

After that if you are still getting RPC errors then check your firewall and start going down the usualy RPC troubleshooting path. Let’s verify that we have created our domain based namespace.

C:\> dfsutil domain shinra.inc

You will see your newly created namespace there. Of course it isn’t doing much for us right now so let’s create some targets for it. Create another share on this server (or really any server) and add a link.

C:\> dfsutil link add \\shinra.inc\data \\renocore\data

If browse to \\shinra.inc\data via UNC or just map a drive you’ll now see the data available in there. This get us a running DFS, but it really isn’t anything more than a fancy way to share data right now. There are not multiple targets so no replication is occurring. If this server goes down there goes the access to the data. Let’s get some targets in there to fulfill the D in DFS. Jump onto another server, install the DFSN-Server role, and make yourself a share to add to the pool. Don’t forget to make sure it has the same share and NTFS permissions as your first share, otherwise things could get difficult for troubleshooting problems later on. Once you have it ready we can add the target.

C:\> dfsutil target add \\shinra.inc\TurksNS\Data \\RudeCore\Data

We have our links now. But we still have no replication. To get this setup we need yet another role added.

C:\> start /w ocsetup DFSR-Infrastructure-ServerEdition

We will then set up a replication group for our folder here.

C:\> dfsradmin RG New /RgName:TurksData
C:\> dfsradmin Mem New /RgName:TurksData /MemName:RudeCore
C:\> dfsradmin Mem New /RgName:TurksData /MemName:RenoCore

This gives us a replication group with our two servers added in as members. Next we will bring in our data for replication.

C:\> dfsradmin RF New /RgName:TurksData /RfName:TurksData /RfDfsPath:\\shinra.inc\TurksNS\Data /force

We have a folder set for replication, but now we need replication links so that the data may flow. Note that force is required because we set up our namespace target first.

C:\> dfsradmin Conn New /RgName:TurksData /SendMem:RudeCore /RecvMem:RenoCore /ConnEnabled:True /ConnRdcEnabled:True
C:\> dfsradmin Conn New /RgName:TurksData /SendMem:RenoCore /RecvMem:RudeCore /ConnEnabled:True /ConnRdcEnabled:True

Close to the end but we still need to bring in memberships to this replication group.

C:\> dfsradmin Membership Set /RgName:TurksData /RfName:TurksData /MemName:RenoCore /MembershipEnabled:True /LocalPath:C:\Data /IsPrimary:True /force
C:\> dfsradmin Membership Set /RgName:TurksData /RfName:TurksData /MemName:RudeCore /MembershipEnabled:True /LocalPath:C:\Data /IsPrimary:False /force

Replication should start flowing smoothly now shortly. If you don’t have any data in there or if you have prepopulated the shares then you won’t know for sure if replication is working properly. You can run a test from this command line utility.

C:\> dfsradmin PropTest New /RgName:TurksData /RfName:TurksData /MemName:RenoCore

This will start the test from RenoCore and the data will flow to Rudecore. Generate the results with dfsradmin.

C:\> dfsradmin PropRep New /RgName:TurksData /RfName:TurksData /MemName:RenoCore

You’ll find an html and xml file generated to pull up in your web browser. Of course you may just find it easier to do things on your own with creating a new different file on both shares and verifying if it is replicated to the other. But the good thing about the report is that it is detailed and will help you in tracking down any issues you may be having. You can also use dfsradmin to automatically create the folders for you when you use dfsradmin RF. Just add them into the namespace later on. So let’s touch on one last topic here, replication of large amounts of data.

It is ok to run through this with a small amount of data that the DFS may need to replicate initially, but if you get into large amounts, which I generally consider to be amounts over 400 or 500 GB, you will definitely want to prepopulate things. Otherwise your DFS may choke on a few files initially and cause you all sorts of headaches. Not to mention it just plain gives you more control over everything. This all does depend upon the bandwidth available to you, of course. The method I normally use is robocopy. You would want to use /E /SEC /ZB. Instead of /SEC you could use /DATSOU to include the auditing information.

Extra reading:

DFS Step-by-Step

DFS FAQ

Dfsutil Breakdown

DFS Best Practices

Troubleshooting 554 Relay Access Denied NDRs

This week I was called up by a former employer to come in and check out their Exchange 2003 server. They had been having problems with a few domains rejecting their e-mails for the past few weeks and what made it a big problem was that one of the domains was their largest customer. Not a good scene there. They were receiving several NDRs of which the most prominent was something like <mail.domain.com #5.7.1 smtp;554 5.7.1 foo@otherdomain.com: Relay access denied> as well as some from another domain referencing authentication. I don’t have it in front of me so unfortunately I can not quote it. Doing some research I found that there was not much information on troubleshooting this and related problems even though there were lots of people asking. So I am putting together a quick little guide based off how I came to a resolution. Your results may vary.

E-mail problems generally break down to two types. Either you are having DNS problems, or you have a misconfigured server. First thing to do of course is to narrow down whether the problem is on your end or if it is on their end. Generally one would suspect the problem to be on their end when it is only a couple of domains giving you a problem, but that isn’t always the case. Easiest way to narrow it down is to break out the command line utilities. First verify that you are using the same DNS as your Exchange server. This is very important otherwise it may skew the results of your test. Next off you need to find your problem domain’s mail server. You have several options for this. The lazy way is to go to MX Lookup Tool and put in the domain. It should return the mail servers for that domain. You’d be best off going with the first one. The site also has some interesting diagnostic utilities that you could run against your mail server as well as their own. The non-lazy method, and one you should learn anyhow, is using nslookup. Pop open your command line and run nslookup. Here’s the steps to go through to get the MX records for their domain.

> set type=mx
> problemdomain.com
Server: 192.168.1.1
Address: 192.168.1.1#53

Non-authoritative answer:
problemdomain.com mail exchanger = 10 mail.problemdomain.com.

Authoritative answers can be found from:

There’s the mail server that you are working with. Sometimes there are more than one so start from the top and work your way down. Next up is we’re going to send an e-mail the archaic way, through telnet. Note that not all SMTP servers respond with the exact same banners and responses, though the codes should generally line-up. Open up a telnet prompt.

telnet>  open mail.problemdomain.com 25

Trying 64.24.111.68…
Connected to mail.problemdomain.com.
Escape character is ‘^]’.
220 mail.yourdomain.com ESMTP Sendmail 8.9.3/8.9.3; Tue, 27 Aug 2002 16:20:32 -0500

At this point you need to identify yourself as your mail server.

ehlo mail.mydomain.com

250-mail.problemdomain.com mail.mydomain.com [203.32.9.6] pleased to meet you
250-8BITMIME
250-SIZE 10000000
250-DSN
250-ONEX
250-ETRN
250-XUSR
250 HELP

Now it is time to identify who you are.

mail from:yourself@mydomain.com

250 Ok

Next is who you are e-mailing.

rcpt to:testaccount@problemdomain.com notify=success,failure

250 Ok

If you get a 250 Ok at this point then telnet will most likely complete successfully. Also if you get a  510 Bad user that is ok as well, since you are not getting a relay access denied which is what your Exchange server is getting when it communicates with this server. This means their server has passed the telnet test so you need to start digging into your server to fix the problem. If you get a 550 Relaying denied error though that is different. Time to modify the test. Change your DNS server to a known good external DNS server that resolves both forward AND reverse look-ups correctly. If it does not resolve reverse look-ups it is no good to you. Switch your machine over to that DNS server and run through the test against. If it is successful then you’ve tracked the problem down. Time to look at what DNS servers your Exchange is using. Or it may also be time to check on your own internal DNS servers, as they may not be resolving addresses correctly. If, though, you receive a failure from your telnet test with a known good set of DNS servers then things may be a bit more complex. Do a look up on whether your domain is in any Realtime Blackhole Lists (RBL). If it is then definitely check to see if you are running an open relay and get yourself off those lists fast. If not then you had best get on the phone with the admin over at problemdomain.com to find out if your domain has been blacklisted. Finally another thing to verify is your SPF record on your external facing DNS. Here is a site to help craft an SPF record.

For completeness here are the rest of the directions to finish sending your e-mail through telnet:

data
354 End data with <CR><LF>.<CR><LF>
Test e-mail
.
250 Ok
quit
221 Bye
Connection closed by foreign host.
telnet>

In case you were wondering at the client’s site it turned out the Exchange server was set up to use an external DNS server that was not resolving reverse look-ups. This was causing a few sites to return the 554 relay access denied NDRs.

Some other tools that may be helpful in diagnosing your problem:

SMTPDiag.exe

Exchange Best Practices Analyzer

Building Your Fortress with RODCs on Core

Now for the topic that you all have been waiting for. Building an RODC! Read only domain controllers are another one of those awesome additions to Server 2008. An RODC holds read only copies of parts of your AD. They’re ideal for branch offices or even your DMZ where you need heightened security but also still need access to your AD as well. RODCs don’t contain a copy of your credentials but only caches those that you set as per policy. In cases of customized AD-integrated applications you can also mark certain attributes in your AD as filtered. Filtered attributes do not replicate to an RODC, so if the RODC is ever compromised the attacker will not gain this critical information. Furthermore any changes made on an RODC do not replicate back out. If for instance someone makes some changes to the SYSVOL folder, those changes will not replicate out to all the other DCs in the forest. It will make that SYSVOL out of sync with the rest of the forest though and could cause some Group Policy idiosyncrasies. If you are using DFS replication for SYSVOL though this problem is fixed automatically. Later I may talk about how to enable DFS-R.

As a side note, anyone running VirtualBox under Linux and has switched to the newly released 2.6.29 kernel may be having a bit of trouble with their VB installation. If you are receiving an error message like this when starting a VM:

Failed to load VMMR0.r0 (VERR_SYMBOL_NOT_FOUND).
Unknown error creating VM (VERR_SYMBOL_NOT_FOUND).

Then you are in need of editing the vboxdrv Makefile. You should find this in /usr/src/vboxdrv-2.1.4/Makefile. You might need to tweak the version number depending upon your installed version. Uncomment the line # VBOX_USE_INSERT_PAGE = 1. Re-run your /etc/init.d/vboxdrv setup command under your root account (or just use sudo) and you should be good to go. More information about this is available here.

Let’s get a new VM created that we will be purposing for our RODC. Get it installed and joined to the domain, but we’ll be building a different answer file for the dcpromo. One important thing to remember by the way, for practical as well as testing purposes, is that to install an RODC requires a minimum forest functional level of Server 2003. Also for testing and practical purposes remember you only need one DC in your domain running Server 2008. No need to migrate over all your DCs yet. You also have to prep your forest. Login as an Enterprise Admin on your schema master, mount your Server 2008 DVD, and run:

C:\>mkdir C:\adprep
C:\>D:
D:\>xcopy /E D:\sources\adprep C:\adprep
D:\>C:
C:\>cd adprep
C:\adprep>adprep /rodcprep

This copies over adprep files and then preps the forest DNS partitions for replication to an RODC. Now to set up your answer file:

[DCInstall]
InstallDNS=Yes
ConfirmGc=Yes
CriticalReplicationOnly=No
PasswordReplicationAllowed=lablogins
Password=*
RebootOnCompletion=No
ReplicaDomainDNSName=shinra.inc
ReplicaOrNewDomain=ReadOnlyReplica
SafeModeAdminPassword=Pass1word
SiteName=Headquarters
UserName=Administrator

You will also want to specify ReplicationSourceDC= if you have Server 2003 DCs and need to point to your Server 2008 DC. You can also specify PasswordReplicationDenied to deny any additional users/groups replication to this RODC. Once you have your file created run the dcpromo as normal.

C:\>dcpromo /unattend:install.txt

Upon success restart your RODC. If you have your site set up properly they should now be able to log into their systems with authentication through the RODC. Now to do some delving into management of your RODC specifically dealing with the Password Replication Policy (PRP). This is what defines what credentials will be cached and what will never be cached. What happens in the case of a denied password caching the RODC forwards the request on up the WAN to a writable DC for authentication. To view what is currently set for your RODC run:

C:\>repadmin /prp view JenovaCoreRODC allow
C:\>repadmin /prp view JenovaCoreRODC deny

This will show you what is currently allowed and denied for the RODC you have specified.

C:\>repadmin /prp view JenovaCoreRODC auth2

From this you will view all accounts that have been authenticated by this RODC. Finally to know what credentials have been cached by the RODC run:

C:\>repadmin /prp view JenovaCoreRODC reveal

It is important to know what credentials have been cached in case of the RODC being compromised. Now if you are wanting to update the list of what accounts you wish to allow caching for then run:

C:\>repadmin /prp add JenovaCoreRODC allow “CN=Lab Guests,OU=Lab Users,DC=shinra,DC=inc”

This uses the LDAP DN to the account or group that you wish to allow caching for. Something to remember is that an account won’t actually be cached until they have logged in authenticating to that RODC. You can pre-populate credentials via this command:

C:\>repadmin /rodcpwdrepl JenovaCoreRODC CloudCore “CN=Jeffery Land,OU=Lab Users,DC=shinra,DC=inc” “CN=Jeffery Land2,OU=Lab Users,DC=shinra,DC=inc”

You can specify as many users as you would like separated by a space. You will have to specify user accounts and not groups though. Most likely you would want to script this if you’re pre-populating an RODC for a site with limited/sporadic WAN connectivity. Remember that you not only want to allow caching for user accounts but also for any computer and service accounts that require authentication. Otherwise the RODC will attempt to forward the authentication on up and if the WAN is down it will fail due to not having a cached account. You are best off first working with an RODC in a lab environment prior to deployment so that you have worked through all such issues that could arise. Also if an account is both in the allowed and denied lists the account will be denied caching as deny takes precedence.

This should get you up to speed on RODC installation and management. Here is some reading for you to more thoroughly understand RODC implementation and management.

Read Only Domain Controllers
RODC Planning and Deployment
RODC FAS, Credential Caching, and Authentication
RODC Administration

Configuring DNS Zones in Core

Now that you grok more completely the concepts of DNS and how it works we will be going over some of the actual implementation details, on Server 2008 Core of course. We’ll jump on the primary DNS server for our lab and set up a subdomain and put together some records for it. Then we’ll set up another DNS server and do a zone transfer to it and make it authoritative for the zone. We’ll be adding a reverse look-up zone for our ip range as well. That should get you started on managing zones and records from the command line. Let’s begin with a reverse look-up zone for our shinra.inc domain.As mentioned earlier for creating a reverse look-up zone you read from right to left for the ip address. I need to clean-up a bit first, though.

C:\>dnscmd /zonedelete 0.60.10.in-addr.arpa /dsdel

I had a zone left over from some previous work so we are going to remove it and start over. Note the addition of /dsdel to the command. This is require to remove the zone from AD if it is AD integrated. Otherwise you will receive an error such as DNS_ERROR_INVALID_ZONE_TYPE 9611. If you are working with a non-AD integrated zone then it is fine without /dsdel. Now let’s recreate our reverse look-up zone.

C:\>dnscmd /zoneadd 0.60.10.in-addr.arpa /dsprimary

This gets us an AD integrated zone. Pretty much you’ll want to always great AD integrated zones, unless you have requirements such as needing to replicate to a DNS server that is not a DC such as a BIND server set up on your Linux box. AD integrated zones enable you to configure secure dynamic updates. This allows an ACL to secure who can read and update particular records. We’ll set up some PTR records now for our machines.

C:\>dnscmd /recordadd 0.60.10.in-addr.arpa 2 PTR cloudcore.shinra.inc

Now if you execute an nslookup of 10.60.0.2 you’ll find a response of cloudcore.shinra.inc. Here’s the anatomy of how this works. After the /recordadd you specify your zone name which is 0.60.10.in-addr.arpa, then next comes your node which is your ip address relative to the zone name. Since our server is 10.60.0.2 in 0.60.10.in-addr.arpa this would be 2. If the zone was only the first two octets it would be 60.10.in-addr.arpa which would mean our node would be 2.0 for this zone. Then we specify that it is a PTR RR and give the FQDN. We’ll add in a few more records to flesh out the zone.

C:\>dnscmd /recordadd 0.60.10.in-addr.arpa 10 PTR renocore.shinra.inc
C:\>dnscmd /recordadd 0.60.10.in-addr.arpa 12 PTR rudecore.shinra.inc

Note that DHCP clients can add their own PTR records in addition to A records. To verify this list of records we’ve added we’ll do a recordsenum.

C:\>dnscmd /enumrecords 0.60.10.in-addr.arpa @
Returned records:
@ 3600 NS cloudcore.shinra.inc.
3600 SOA cloudcore.shinra.inc. hostmaster.shinra.inc. 13 900 600 86400 3600
2 3600 PTR cloudcore.shinra.inc.
10 3600 PTR renocore.shinra.inc.
12 3600 PTR rudecore.shinra.inc.

This should show that your reverse look-up zone is properly created and populated. We will now move on to our next exercise of creating a subdomain. Since we will also be using this zone for non-AD integrated zone transfers will be creating it is a regular zone which requires it to be stored as a file and not in a directory partition.

C:\>dnscmd /zoneadd lab.shinra.inc /primary /file lab.shinra.inc.dns

We’ll add a few A records for a few non-existent machines to populate the zone. We’ll use a quick batch script to aid in this. Here’s the contents of the script.

for /L %%C in (%1, 1, %2) do dnscmd /recordadd lab.shinra.inc experiment%%C /createptr A 10.60.0.2%%C

Then run from the command line with adddns.bat 10 30. This will populate your zone with a good number of A records. You can verify with dnscmd /enumrecords lab.shinra.inc @. You can also verify that the corresponding PTR record was created with dnscmd /enumrecords 0.60.10.in-addr.arpa @. Now we’ll set up a second server to transfer this zone over to. Configure your server as normal and join it to the AD or not as it doesn’t really matter in this case. If you do join it to the AD you can transfer over AD integrated zones as well though. Let’s get our DNS role installed on it. Here’s a good way to find out the name of a service for installation without having to scroll through a long list.

C:\>oclist | find /I “dns”

Now to install it.

C:\>start /w ocsetup DNS-Server-Core-Role

Once that finishes the role will be installed. We need to configure our original DNS server to allow zone transfers to this new server.

C:\>dnscmd /zoneresetsecondaries lab.shinra.inc /securelist 10.60.0.25

Then we jump back onto our new server to get the zone set up and transferred.

C:\>dnscmd /zoneadd lab.shinra.inc /secondary 10.60.0.2
C:\>dnscmd /zonerefresh lab.shinra.inc

Once this has finished transferring, which with these sizes and being in the same network should be instantaneous, you’ll have a complete read only copy of the zone on this server. Now to make it a master we decommission the old server and make the new one the primary. On the old server we delete the zone.

C:\>dnscmd /zonedelete lab.shinra.inc

Then on the new server we switch it to being a primary zone.

C:\>dnscmd /zoneresettype lab.shinra.inc /primary /file lab.shinra.inc.dns

Then we’ll verify that we were successful from the zone RRs themselves.

C:\>dnscmd /zoneprint lab.shinra.inc

Check your SOA record if you see that your new server is listed then the transfer of the master server was successful. Most likely your NS records will not have been updated properly, so we will go through an recreate those ourselves.

C:\>dnscmd /recordadd lab.shinra.inc @ NS redxiiicore.shinra.inc
C:\>dnscmd /recorddelete lab.shinra.inc @ NS cloudcore.shinra.inc

Now we could even take it a step further and create a stub zone on our previous server for our lab.shinra.inc zone. Hop on your old server and let’s get this created.

C:\>dnscmd /zoneadd lab.shinra.inc /stub 10.60.0.25

Check the zone info and you should be seeing the SOA and NS records in there for the zone, but none of the horde of A records that we had created.

You should be feeling up to speed on managing DNS from the command line on your Core installations. Don’t forget that you can also use these on your full Server 2008 (or even older versions) installations as well. The GUI can be easier but don’t let it be your only tool in your arsenal. Remember that one of the places the CLI can shine is in scripting, as demonstrated earlier. For some reference reading this post will be useful for you as it has a list of commands for dnscmd and a quick example.

DNSCMD Reference