Problems Installing Exchange 2010 Service Pack 2 on SBS 2011

Now these problems that occur are very likely originating from an already rather screwed up installation of SBS 2011. I was not involved in the original setup of this particular server but I do know that there had been a large number of problems originally encountered. In this instance the task was to get Exchange 2010 SP2 installed. There are several hoops that you may have to jump through to get this installed, here I will recount what I was required to do.

Firstly you need to make sure that you have closed any instance of the SBS Console. Otherwise you’ll get a failure in the prerequisites. Also initially you’ll need to stop the Windows SBS Manager service though if you can get the install to progress to the point of working on the installed roles rather than the organization that will no longer be a requirement. Once you’re past those prerequisites in theory your installation should go smoothly. But if that is not the case then read on.
The next problem you may encounter is any error in the Hub Transport Role. From the event logs you’ll find this error:

 Event ID 1002 MSExchangeSetup
 Exchange Server component Hub Transport Role failed.
 Error: Error:
 The following error was generated when "$error.Clear();
 if (get-service MSExchangeServiceHost* | where {$_.name -eq "MSExchangeServiceHost"})
 {
 restart-service MSExchangeServiceHost
 }
 " was run: "Service 'Microsoft Exchange Service Host (MSExchangeServiceHost)' cannot be started due to the following error: Cannot start service MSExchangeServiceHost on computer '.'.".
Service 'Microsoft Exchange Service Host (MSExchangeServiceHost)' cannot be started due to the following error: Cannot start service MSExchangeServiceHost on computer '.'.
Cannot start service MSExchangeServiceHost on computer '.'.
The service cannot be started, either because it is disabled or because it has no enabled devices associated with it

Checking your services you’ll also find all of the Exchange services disabled. Service packs and update rollups usually disable the services to prevent them from starting up unexpectedly while the update is being installed, but in this case for some reason SP2 is jinxing itself by not allowing itself to start a couple of necessary services for it to be able to continue. The easiest way to get around this, though not necessarily the safest, is to make sure that at this point all the Exchange services are set to Manual or Automatic. When you see setup get down to the point of setting up the Hub Transport Role then watch your services and wait for them all to be set to disabled. Once they are pop open a Powershell prompt and run:

Get-Service | {where $_.DisplayName –match “Microsoft Exchange”} | Set-Service –StartupType Manual

Now setup will be able to continue with starting the services that it requires for continuing setup. Which may lead to your next problem, it will fail on generating a new self-signed certificate for the Exchange Transport service. You’ll find this error in the event logs:

Event ID 1002 MSExchangeSetup
 Exchange Server component Hub Transport Role failed.
 Error: Error:
 The following error was generated when "$error.Clear();
 Write-ExchangeSetupLog -Info "Creating SBS certificate";
$thumbprint = [Microsoft.Win32.Registry]::GetValue("HKEY_LOCAL_MACHINE\Software\Microsoft\SmallBusinessServer\Networking", "LeafCertThumbPrint", $null);
if (![System.String]::IsNullOrEmpty($thumbprint))
 {
 Write-ExchangeSetupLog -Info "Enabling certificate with thumbprint: $thumbprint for SMTP service";
 Enable-ExchangeCertificate -Thumbprint $thumbprint -Services SMTP;
Write-ExchangeSetupLog -Info "Removing default Exchange Certificate";
 Get-ExchangeCertificate | where {$_.FriendlyName.ToString() -eq "Microsoft Exchange"} | Remove-ExchangeCertificate;
Write-ExchangeSetupLog -Info "Checking if default Exchange Certificate is removed";
 $certs = Get-ExchangeCertificate | where {$_.FriendlyName.ToString() -eq "Microsoft Exchange"};
 if ($certs)
 {
 Write-ExchangeSetupLog -Error "Failed to remove existing exchange certificate"
 }
 }
 else
 {
 Write-ExchangeSetupLog -Warning "Cannot find the SBS certificate";
 }
 " was run: "The internal transport certificate cannot be removed because that would cause the Microsoft Exchange Transport service to stop. To replace the internal transport certificate, create a new certificate. The new certificate will automatically become the internal transport certificate. You can then remove the existing certificate.".
The internal transport certificate cannot be removed because that would cause the Microsoft Exchange Transport service to stop. To replace the internal transport certificate, create a new certificate. The new certificate will automatically become the internal transport certificate. You can then remove the existing certificate.
Error:
 The following error was generated when "$error.Clear();
 Write-ExchangeSetupLog -Info "Creating SBS certificate";
$thumbprint = [Microsoft.Win32.Registry]::GetValue("HKEY_LOCAL_MACHINE\Software\Microsoft\SmallBusinessServer\Networking", "LeafCertThumbPrint", $null);
if (![System.String]::IsNullOrEmpty($thumbprint))
 {
 Write-ExchangeSetupLog -Info "Enabling certificate with thumbprint: $thumbprint for SMTP service";
 Enable-ExchangeCertificate -Thumbprint $thumbprint -Services SMTP;
Write-ExchangeSetupLog -Info "Removing default Exchange Certificate";
 Get-ExchangeCertificate | where {$_.FriendlyName.ToString() -eq "Microsoft Exchange"} | Remove-ExchangeCertificate;
Write-ExchangeSetupLog -Info "Checking if default Exchange Certificate is removed";
 $certs = Get-ExchangeCertificate | where {$_.FriendlyName.ToString() -eq "Microsoft Exchange"};
 if ($certs)
 {
 Write-ExchangeSetupLog -Error "Failed to remove existing exchange certificate"
 }
 }
 else
 {
 Write-ExchangeSetupLog -Warning "Cannot find the SBS certificate";
 }
 " was run: "Failed to remove existing exchange certificate".
Failed to remove existing exchange certificate

This is a very verbose yet also very helpful error. Chances are you’ll most likely encounter this if you are not using the default self-signed certificates but have installed a third party certificate. Though I didn’t check in this case, reviewing the commands being run it may be choking on a third party certificate that has a friendly name of Microsoft Exchange. To fix this one first make sure you have a copy of your third party certificate available and if you don’t then export a copy as you’ll be in need of it later. Once you have that available then run through the SBS Set up your Internet address wizard. This will generate you another self-signed certificate and replace the third party certificate you have in place. It will also remove the third party certificates from your certificate store, which is why you need to make sure you have a copy of the certificate available. Once you have done this re-run setup and you’ll be able to finish your installation of SP2. Don’t forget to put the third party certificate back in place and also it would be a good idea to run ExBPA to make sure you are still in compliance. You’ll also want to make sure that all of your Exchange services are set back to their appropriate startup values as you may be left with all the services set to disabled.

Tutorial on Configuring and Migrating Redirected Folders

In recent migrations I’ve seen that there is some confusion in how to work with redirected folders. Let’s first go over a few reasons for the existence and usage of redirected folders. The most important reason is that it is absolutely critical in an RDS farm if you want any sort of user data persistence between servers, not to mention it will help cut down on the amount of local disk space used by each server. Your users will be able to be load balanced from server to server without worrying about which is their “home” server or having to configure their account on each server. They can still keep their habit of saving critical data to their Documents folder as well. Another reason is that you’ll get all of your users’ profiles stored in a central location. Which means their Documents and Desktop folders will be stored centrally. Which means you’ll be able to back those up. Now when you have this implemented as just a roaming profile all of that data is copied down to the server and then synched back to the central location. This slows things down for everyone since you have network bandwidth being taken up unnecessarily at logon and also longer logon times for the local user. Here is where folder redirection jumps in to help. With your Desktop and Documents and Pictures and so forth being redirected then everything is pulled off a share rather than being copied down to the server. That frees up a lot of bandwidth and speeds up login times so everyone is a lot happier. You’ll want to nip those PSTs right away though, otherwise you could end up with a lot of performance problems.

Anyhow let’s go on to the implementation. We’ll begin with configuring the redirected folders. Create a share, we’ll name it Folders, and configure the share permissions with Everyone:Full. Generally whenever you create a share you want to configure the share permissions as Everyone:Full unless you have a very good reason not to. Normally all permissions you would want to control through NTFS. This simplifies management and troubleshooting. Now your NTFS permissions you’ll first want to disable including inheritable permissions. The permissions you want on this folder are Full Control for SYSTEM, CREATOR OWNER and Administrators, and for Authenticated Users you’ll need to set advanced permissions. You’ll want Create Folders/Append Data, Read Permissions, Read Attributes and Read Extended Attributes. This will create a folder where the data is secure from prying eyes yet administrators will still be able to access it without breaking redirection.

Next up is creating the group policy for configuring folder redirection. Create a new policy and name it Folder Redirection. The section we’ll be working in is User Configuration/Policies/Windows Settings/Folder Redirection. You’ll want to plan out your folder redirection strategy before you start implementing. What folders are important to you, how are you getting the data there, and perhaps even most importantly how are you going to back this policy out when you’re done. Once you’re done planning then start editing your policy. For this tutorial we’re erring on the side of simplicity.

The first setting gives you two options, Basic and Advanced. Most times you will want to use Basic but it depends upon what you are trying to achieve. With Basic you point the folder to the share that you want, as a UNC of course i.e. \\storageserver\Folders\. Once selected you normally will want the option of Create a folder for each user under the root path. It will even show you what the path will look like at the bottom. With these options everyone affected by the policy will be redirected to the same location. With the Advanced option you get more flexibility in how you configure user’s redirected folders since now you can use group membership to configure share selection for the storage of the redirected folders. The next tab over we have Settings. By default users are granted exclusive rights to the folder. Also by default the contents of the folder will be moved to the new location. This simplifies the job of moving content, but the down side is that it prevents you from pre-staging the move instead of having it happen at logon. But you will have planned this out already, right? The last unchecked option is to apply to 2000/XP/2003 operating systems. You’ll want to check this depending upon where these folders will be used. This will disable some redirection options in Vista/7 though.

Now the final option is Policy Removal which you will have also planned out ahead of time. If you select leave the folder in the new location then when the policy is removed their profile still redirects to \\storageserver\Folders\ and the data still remains there. If you select redirect the folder back to the local user profile then what happens depends upon what you checked for Move the contents to the new location. If you have it checked then the folder redirects to their local profile and the data is copied, not moved, to the local profile. You’ll still need to clean up the old location. If you have the option unchecked then the folder will redirect to their local profile but all the data will still stay on the share. Your users will end up with empty local folders. This is why you’ll want to plan your exit strategy because at some point some or all your users data will end up being stored somewhere else. Since we’re preparing a migration scenario most likely everything will be setup with the defaults so that is what we are going to do here, setup the folders with the defaults. We’ll configure redirection for the Desktop, Documents, Pictures, Music, Videos, Favorites, Downloads. Not all of these will be available depending upon what versions of windows you are working with. Also note that there is an option for Pictures, Music and Videos to follow the Documents folder which is what you’ll want to select unless you have a reason to split them amongst multiple shares. Don’t forget to allow time for the policy to replicate to any other DCs or force replication, and that you may need to run gpupdate on the client to force immediate pick-up of the change.

Now that we have configured our folder redirection go ahead and populate a few profiles with data. If you check the Folders share that you created you’ll see that it is getting populated with account names and the redirected folders. Test logging into a few different servers as well to make sure that the folders are following your accounts. You can also pull up the properties on them to verify the path pointing to the share. If that is all working fine then let’s look at migrating the redirected folders.

We’ve got several options for migrating the folders. The simplest method and definitely the one you’ll want to use when dealing with small amounts of data is to let the policy take care of it for you. Let’s test it out. Create a share somewhere else named NewFolders and configure it with the same share and NTFS permissions as listed earlier. Edit your folder redirection policy and change the path to point to your new server. Also make sure you’ve checked Move the contents to the new location. That’s the part that is doing the work for us. Once you’re done with the changes give it a test. You’ll probably see a longer logon the first time as data is copying across. There’s also a chance the it won’t be picked up until the next logon due to asynchronous policy processing. Note that the data was actually moved, not copied. This is great for when there isn’t much data to move, and you can also do it in phases moving one folder at a time. Something else you could do if you want to migrate accounts in phases is to create policies for redirection and link them to migration OUs that you create lower than where the original redirection policy is linked.

When you’re working with larger amounts of data though you may want to pre-stage the data rather than have it be moved at first logon. This requires a bit of work. Since the folders get locked down by default if you have Grant the user exclusive rights checked, the administrator account does not have access to the folders. If you take ownership of the folders, that will break redirection since the policy checks for ownership of the folder. What you’ll need to do is go into the policy and uncheck the exclusive rights option everywhere. At the same time you’ll also want to uncheck Move the contents to the new location. This is best done as earlier as possible in the migration just to make sure all clients have picked up the updated settings to cut down on the amount of weirdness you may encounter. Now once this is done make sure that mentioned NTFS permissions are configured on the top level folder for the share. Now go in and if the Administrators group doesn’t have ownership of the folder take ownership of it, then check the box to replace owner on subcontainers and objects. Ok out of everything then open up the advanced NTFS permissions. Check the box for Replace all child object permissions with inheritable permissions from this object. Now use whatever method you prefer to copy the data from there to the new redirected folders share. Robocopy is my preference.

You’ve now pre-staged all the data and policies are configured so that permissions do not break anything it is time to update the policies to point to the new shared folder. Same as last time just update the UNC to the new location, once again making sure that Move the contents to the new location is unchecked. You will probably want to take the old share offline just to be safe. This will flush out any systems that are not processing group policy properly.

Now what happens if you go ahead and delete the group policy rather than reconfigure it for anything. Reference back to the section on Policy Removal which is on paragraph 5. Assuming the policy you deleted was left at the defaults for policy removal all clients will be left pointing at the old share until told differently. To fix this is simple, create a policy with the new redirection settings and once it is picked up the user will be pointed to the new location. What if you are just trying to remove folder redirection altogether? Hopefully you set the policy removal to redirect back to the local user profile. But if you have not; create a policy and set each redirected folder’s target location to Redirect to the local user profile location. Once this policy has been applied everywhere at that point it is safe to delete altogether.

References:

http://technet.microsoft.com/en-us/library/cc732275.aspx

http://support.microsoft.com/kb/288991

USB Drive Disappears from Removable Storage on XenServer after a Reboot

Quick fix for an annoying problem I ran across where the removable storage no longer shows the attached usb drives after a reboot under XenServer 5.6. Pop open a console window on your XenServer host:

modprobe -r usb_storage … this removes the usb_storage kernel driver

modprobe usb_storage … this reinstalls the usb_storage kernel drives

That should get you your drives back and if you don’t see them then just do a rescan.

xe sr-list | grep -i removable -B 1 … use this to find the UUID of your removable storage SR

xe sr-scan uuid=<uuid of removable storage> … your usb drives should be showing up now ready to be attached to your VM

Quick review of flushdns, registerdns, and DNS queries

There seems to be a bit of a misconception on how DNS cache flushing works. I’ve heard techs talking about running ipconfig /flushdns and ipconfig /registerdns to flush the DNS cache. It looks like there needs to be a bit of clarification on how these commands work:

ipconfig /flushdns: “Flushes and resets the contents of the DNS client resolver cache. During DNS troubleshooting, you can use this procedure to discard negative cache entries from the cache, as well as any other entries that have been added dynamically”

ipconfig /registerdns: “Initiates manual dynamic registration for the DNS names and IP addresses that are configured at a computer. You can use this parameter to troubleshoot a failed DNS name registration or resolve a dynamic update problem between a client and the DNS server without rebooting the client computer. The DNS settings in the advanced properties of the TCP/IP protocol determine which names are registered in DNS.”

Now as you can see from the above documentation that the parameters operate independently. You would only issue a /registerdns parameter in cases where the client system’s name is not being resolved. There is no requirement to run it with the /flushdns parameter.

Something that you may find of interest is that there is also a parameter to show the contents of the DNS cache. ipconfig /displaydns will print out in the terminal window the entire contents of the DNS cache. You can verify from there whether it truly has the correct address for whatever you’re having issues resolving or not.

A quick refresher on how name resolution works. First the name is submitted for DNS resolution. The system checks to see if the name is a FQDN, single label or multi label. This is determined by the dots within the name i.e. www.microsoft.com. is an FQDN while www.microsoft.com is a multi label and just www is a single label. Note the terminating period on the FQDN and the lack of a terminating period on the multi label name. Let’s first check how resolution works for an FQDN:

1.       Checks DNS cache (this is built from previous DNS queries and the hosts file, hosts file always win)

2.       Queries primary DNS server

3.       If no response in two seconds it queries all remaining DNS servers

4.       Resends queries to all servers at the four and eight second marks

5.       Returns time outs for all queries after thirty seconds

6.       Query is evaluated on whether it is 15 bytes or less

7.       If less then query is submitted for NetBIOS resolution

8.       Query finally fails if no resolution has been achieved

Now if a multi label name was submitted such as www.microsoft.com (note the lack of a terminating period) then the resolver terminates it with a period to make it an FQDN and submits it to the same resolution list as above, with a slight difference:

1.       Checks DNS cache (this is built from previous DNS queries and the hosts file, hosts file always win)

2.       Queries primary DNS server

3.       If no response in two seconds it queries all remaining DNS servers

4.       Resends queries to all servers at the four and eight second marks

5.       Returns time outs for all queries after thirty seconds

6.       Queries are re-issued with the connection specific DNS appended to the query

7.       Queries are then reissued devolving the parent DNS until only two labels are left

8.       Query is evaluated on whether it is 15 bytes or less

9.       If less then query is submitted for NetBIOS resolution

10.   Query finally fails if no resolution has been achieved

For a single label name the connection specific DNS is appended immediately and then it is submitted to the same resolution order as the FQDN.

For more information and flow charts look at the documentation links below.

Documentation taken from here:

http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/ipconfig.mspx?mfr=true

http://technet.microsoft.com/en-us/library/cc961411.aspx

The Case of the Mysterious Crashing Application

A recent client had migrated off their terminal server and onto a virtualized 2008 R2 RDS server. Actually a farm of them but for this case it did not matter. Their previous setup had been all contained on one 2003 server which also ran their AD, print server, and whatever else was crammed into the kitchen sink. This new setup had some proper separation and centralized storage all on 2008 servers. Now for all of their data and programs they would reach into a file share on the SAN. This was working great except for one program they had would keep on crashing unless the data files were local to the server. Event IDs were as such with one immediately following the other:

Event ID 1000

Application Error

Description:

Faulting application name: OMNIS7.exe, version: 8.0.0.0, time stamp: 0x3bb82293

Faulting module name: OMNIS7.exe, version: 8.0.0.0, time stamp: 0x3bb82293

Exception code: 0xc0000006

Event ID 1005

Application Error

Description:

Windows cannot access the file  for one of the following reasons: there is a problem with the network connection, the disk that the file is stored on, or the storage drivers installed on this computer; or the disk is missing. Windows closed the program Omnis 7 core executable because of this error.

Program: Omnis 7 core executable

File:

The error value is listed in the Additional Data section.

User Action

1. Open the file again. This situation might be a temporary problem that corrects itself when the program runs again.

2. If the file still cannot be accessed and

- It is on the network, your network administrator should verify that there is not a problem with the network and that the server can be contacted.

- It is on a removable disk, for example, a floppy disk or CD-ROM, verify that the disk is fully inserted into the computer.

3. Check and repair the file system by running CHKDSK. To run CHKDSK, click Start, click Run, type CMD, and then click OK. At the command prompt, type CHKDSK /F, and then press ENTER.

4. If the problem persists, restore the file from a backup copy.

5. Determine whether other files on the same disk can be opened. If not, the disk might be damaged. If it is a hard disk, contact your administrator or computer hardware vendor for further assistance.

Additional Data

Error value: C00000C4

Disk type: 0

It was very swiftly recognize that the situation was not a temporary problem though unfortunately a sporadic one. Patterns noted were that crashed the most often in the morning when everyone would sign on and the afternoon when everyone was closing out. Now 0xC00000C4 is STATUS_UNEXPECTED_NETWORK_ERROR but that doesn’t provide much to go on. Grabbing some performance logs also showed that there shouldn’t be a network performance problem either bandwidth-wise. The first thing that was tried was disabling rss and offloading but that did not help matters. Doing more research I was lead to believe that the problem was being caused by oplocks.

Oplocks, short for Opportunistic Locking, is a process in the SMB protocol that was designed to allow multiple processes to lock a file while providing client side caching. The purpose of this is to improve performance for the local clients on the network. For more reading on this consult this document and this document. So all the crashing basically came down to cache integrity since the database used by the client was a flat file instead of transactional database.

To disable oplocks on the server you go into this key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters

And set EnableOplocks to 0. If it is not there create it as a REG_DWORD. Reboot to take effect.

Unfortunately Server 2008 introduces a new problem. Server 2008 will communicate via SMB2 to any client using Vista or newer. SMB2 also does not allow oplocks to be disabled. The work around for this is that if SMB2 is disabled on either the client or the server then communication will fall back to using SMB. Easiest way to fix this then is to disable on the server.

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\LanmanServer\Parameters

Create a REG_DWORD named SMB2 and set it to 0. Reboot to take effect.

You may notice that the server takes substantially longer to start up after making this change. They were severe enough that I decided to test an alternative method for disabling SMB2. Since communication will default to SMB if either the client or the server did not support SMB2 then SMB2 could be disabled on the client side. Disabling on the client side is a bit different since you’re actually disabling a service.

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\LanmanWorkstation

You may want to backup this key for easy restoration. Then edit DependOnService and remove MRxSmb20.

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\mrxsmb20

You may want to backup this key as well. In this key set Start to 4. Reboot your client and SMB2 will now be disabled.

Ever since implementing these changes the client’s applications have been running solid as a rock. For oplocks reading from Microsoft check here.

Active Directory Internal Naming and DNS Strategy

This post touches on something that is rather simple, yet I’ve seen it done improperly at many of the SMB clients that I work with by a previous provider. This has resulted in some unnecessary complexity and even migrations to a new forest to meet requirements such as for Exchange 2007 when it did not support single label domains. When you are first creating your Active Directory forest you want to put some thought into what you are naming it. You need to first think about the company’s internet facing domain names and what sort of traffic is being generated through them. This changes your DNS strategy depending upon, for instance, if the company’s website is being hosted by the company or if it is hosted by a third party. You will also need to think about where your public DNS is being served from. Another thing to throw into the mix is security, which ties into the previous issue. Let’s take a look at a few things here.

Microsoft has some best practices guidelines here. My personal preference is to go for a non-registered TLD such as .internal or .local so as to provide no confusion with TLDs such as .com or .net. Microsoft would prefer for you to go with a subdomain of your external domain i.e. corporate.contoso.com for your AD forest while using contoso.com on the internet facing side. Either way of doing it a benefit you reap is that name resolution for contoso.com is done externally. The reason for this is that while your internal DNS is authoritative for contoso.local or corporate.contoso.com it is not authoritative for contoso.com itself so it will find a server that is. This will return the internet facing IP address for whatever is in contoso.com. The reason you would want this is because, especially for the majority of SMBs that I work with, most often their website is hosted at a 3rd party provider. If your AD forest was contoso.com that would add complexity as you would have to manage internet addresses both internally and externally as you would no longer be able to forward requests to your public DNS provider. For example for the record of www.contoso.com if you switched 3rd party hosting providers you would need to update that A record on your public DNS. You would also need to update that record internally, otherwise the next day your client will be calling in to let you know that their website is “down.”

Now what if you were hosting your own DNS? For security you would want to put your public DNS into your DMZ serving different zones than your private DNS servers. The reason for this is to restrict public access to your internal DNS hierarchy. Access to that would give hackers a huge amount of information on your internal network such as naming conventions, internal ip addressing and even names of your DCs. Your private DNS would then forward requests for contoso.com to your public DNS and management is simplified since internal changes would not affect external changes and vice versa.

Next obstacle to face is what if you were hosting some addresses internally but others are hosted at a 3rd party, such as www .contoso.com goes to your company’s website but mail.contoso.com goes to your OWA. Creation of a zone internally for that specific address would allow internal requests to be managed by your internal DNS while still forwarding requests for the company site to the public DNS side. This simplifies DNS management as well. You would have your mail.contoso.com zone and you could be migrating from one Exchange server to another and all you would have to manage internally is the mail.contoso.com zone. Your public IP address has not changed at all so your public A record for mail.contoso.com has no need to be updated. All those remote users hitting mail.contoso.com would not notice a difference, unless of course you have forgotten to change your NAT and firewall rules but that is an entirely different subject. Also if the reverse is true and you are changing your public IP address you would still only be changing your public DNS records. Private DNS would not be impacted whatsoever.

So what if you were to go with contoso.com for your AD forest as well as your public DNS? DNS changes would be more complex. You would need to manage addresses both externally and internally. An example, you have your mail.contoso.com address created externally and your remote users are using OWA. If they come into the office suddenly all their OWA requests are failing since an A record internally is not created. You create your A record pointed to your Exchange server internally and everything works properly again. Then there is the scenario of the company website which is hosted by a third party. Users are able to access www.contoso.com outside the company but inside the company the requests fail. You create an A record pointed to the 3rd party site and everything works again, until you switch your hosting provider. People will be unable to access the site again until you also update the internal DNS record.

There is also the single label domain name to think about. Microsoft recommends to avoid this and I would also recommend avoiding it since it requires even more initial management to get things working properly. It can also cause problems with cross forest trusts.

Keep your DNS simple and you will have less late nights trying to figure out why mail.contoso.com does not work on the company network.

Addressing P2V 0x7b Issues

The other night I was P2Ving several systems and on one I ran into the issue of it blue screening on boot. It is unfortunate but not too uncommon as usually you need to enable IDE drivers on the system prior to the P2V. Microsoft’s article here works for all versions of XP and Server 2003, though I found I needed to expand the mentioned drivers directly from the cd for the SBS system I was working. That unfortunately did not resolve my 0x7b blue screen the other night. This article turned out to be the key to what I needed. Now the part that neither of these mentions is how to fix the problem if you can’t even boot that VM, so as to avoid having to do another P2V of the system. With Server 2008 this is possible to avoid and it can save you a lot of time, especially if the systems are large.

Server 2008 contained a great feature of being able to mount VHDs which is what we’ll be doing. For the first method you’ll want to mount the VHD to a drive letter and then expand the drivers to the \windows\system32\drivers folder in the VHD. Pull up regedit and select the HKLM key. Go to File->Load Hive and open the system registery from the \windows\system32\config\ and give it an easily identifiable name. You’ll find the registery loaded in HKLM under the name that you gave. Now loading the registery this way you won’t find a CurrentControlSet under the SYSTEM key. CurrentControlSet is just a pointer to ControlSetxxx. To find out which ControlSet number the system is set to boot with look ing SYSTEM\Select. The Current dword contains the number that it is using which in most cases will be 1, so go into that particular ControlSet i.e. for 1 it will be ControlSet001. In there you can manually implement the keys from the first article or the second article. In the case of the problem I ran into I had to set the Group Value of wdf01000 to WdfLoadGroup as it was part of the base group. If you want to learn more about service orders take a look at this article and this article.

Once done with those changes unload the hive and close out of regedit. Dismount the VHD and your virtual machine should be good to go.

Outlook pulling the wrong user’s mailbox?

Recent problem I ran into at a client’s site was that when they were attempting to setup an Outlook profile for a specific user it would keep on pulling a different user’s mailbox. Also same thing happened when they pulled the e-mail from the global address list, though oddly enough if done from a Blackberry it was just fine. Checking the two mailboxes involved in Exchange 2007 yielded no wrong information. All the names and aliases were correct. So to dig deeper I broke out adsiedit.msc. Pulling up the user’s properties and checking the mail property showed that both accounts had the exact same e-mail address. A swift change of that to the proper smtp address corrected the problem immediately on all accounts.

Manually Connecting Mailboxes by MailboxGUID or Hey! Where did my mailboxes go?!

On Monday I learned the hard lesson of always making sure that your Active Directory is replicating properly, especially in the middle of a migration. At least this is the best explanation that I can come up with for what happened. We were wrapping up my first Small Business Server 2003 to SBS 2008 migration, with an already less than stellar performance due to the Exchange System Manager refusing to show the new server, and to a few mailboxes that refused to move over. We had to export those monstrosities to pst before we could move them over. Which turned into a mixed blessing later on. Things had come to the point of being ready to remove Exchange 2003 from the SBS 2003 server, which went along fairly smoothly. We rebooted the server ready to breathe a sigh of relief that this portion of the migration was done. Not so lucky, the true nightmare began then. The calls from the users began coming in that they could no longer access their email, even after confirming that they really were pointing to the new server. I fired up the Exchange Management console to find out what was wrong to discover that no mail enabled users showed up, aside from the three we had to import from pst. I took a look for disconnected mailboxes and to my horror we did not have any. Restoring from backup wasn’t an option as we had not been able to take one yet. This began a mostly fruitless 6 hour call with Microsoft. It is very sad when your bluetooth dies and you also are able to recharge it and start using it again during the same call. Fortunately I did not stop doing research into the problem while on the call and eventually cobbled together the solution that I am going to present to you now. My warning is – do NOT do it this way unless you absolutely have to.

Before I had made the call I ran a Get-MailboxStatistics. Interestingly this reported everyone’s mailbox as existing and containing data. This meant that the users had lost their Exchange attributes. To verify this required digging into Active Directory with ADSI edit. Open up ADSI edit and connect to the default view. Drill down to the OU of your user and pull up their properties. You will see a list of attributes set there. Some specific attributes are required to mail enable a user. legacyExchangeDN, homeMTA, mailNickname,  msExchHomeServerName, and finally as well as perhaps the most import is msExchMailboxGuid. homeMDB is required as well but fortunately this was still populated. On the users that were missing mailboxes none of the msExch attributes were set. Fortunately there did remain some users, so looking at those I was able to glean a few of the attributes.

legacyExchangeDN –This attribute needs to be configured to point to the login for your mail enabled user. This drills down through the organization name and the default administrative group name ending down in your user’s login name. Example:  /o=first organization/ou=exchange administrative group (fydibohf23spdlt)/cn=recipients/cn=jland

You can pull this information from your mailboxes actually. Run Get-MailboxStatistics | ft Displayname, LegacyDN

homeMTA – This may not be required under Exchange 2007, but we decided to set it none-the-less. This drills down through Active Directory to where your Exchange server’s MTA resides. Example: CN=Microsoft MTA,CN=SERVER,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=linthicum,DC=local

mailNickname – This one is easy to miss yet is very important. If you user still doesn’t show up as mail enabled go back and make sure you entered in the mailNickname. This generally would be your user’s login name, though you should consult your organization’s naming scheme to be sure.

msExchHomeServerName – This one is fairly self-descriptive. This points to where your server is located in Active Directory, as based off the organization name. An example is: /o=First Organization/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=SERVER

msExchMailboxGuid – This one is the kicker. Exchange won’t know what mailbox to connect your user to without this info. But it isn’t exactly easy to get ahold of either. First run Get-MailboxStatistics | ft Displayname, MailboxGUID. You’ll see everyone’s msExchMailboxGuid listed right there. Easy? No. Now you have to be able to get that into Active Directory. Which is a royal pain. Go down through the properties of your user and open up msExchMailboxGuid to put in some new information. See how you only have the options of decimal, hex, octal and binary? You need to convert this GUID into something usable.

Go to joeware’s great site and download the adfind tool.  Open up a command line and go to where you extracted the tool and run adfind -gc -b “” -binenc -f ” msExchMailboxGUID={{GUID:98ee00d7-df19-4282-bedf-3a1340b8b7c0}}” –dn where of course you replace the GUID with the one you are searching for.  This will return you some interesting output which still isn’t quite usable, though it may look that way at first glance. Your response is mostly hex, but not fully. You need to translate it. I’ve included a utility at the bottom of this post that you can use to convert this output into full hex. Pull up the table at Ascii Table and use this for your translation. Start going through the characters and when you find one that doesn’t match like for instance a lower case j or the number 4 unpaired, look through the red characters in the table for your character and you’ll see the conversion to hex in the separate column. Go through the whole string this way and you’ll eventually get a fully hex string. Go back into your msExchMailboxGUID and put that in and after you click ok you’ll see that the attribute has been populated with the string that you began with. Look very closely at it to make sure it matches. If there’s some deviation go back and check your look-up tables again. This string should match completely, otherwise your user will end up with an empty mailbox created. Here’s an example of how to convert the returned string:

\D70\EE\98\19\DF\82B\BE\DF\3A\13\40\B8\B7\C0

D7 00 EE 98 19 DF 82 42 BE DF 3A 13 40 B8 B7 C0

And another one:

T\BA\A04l\B8\EEM\9F\D6\40m\258\CE\A0

54 BA A0 34 6C B8 EE 4D 9F D6 40 6D 25 08 CE A0

Your user has now been mail enabled as a refresh of your Exchange Management Console will show, but is still missing a number of Exchange attributes. Run Set-Mailbox “My User” –ApplyMandatoryProperties and all the rest will be filled out for you.

The last bit is to clean up Outlook for your users. Even the ones not using Cached Mode still showed as disconnected until we re-put the server back into their profile. For the ones using Cached Mode it was also a good idea to delete their .oab files from their directory to force them to download a newly rebuilt OAB. This may or may not be necessary in your case as that was most likely related to the timing in the migration. If you didn’t delete them it would populate the DN to the user they’re emailing rather than the email address, thusly bouncing back to them when they emailed.

Some extra reading:

Understanding Mailbox GUIDs

How to Re-Home Exchange Mailbox Accounts

Using ADFind Utility

Update:

I’ve finally gotten around to writing a simple little utility for converting the resulting GUID output from adfind into full hex for pasting back into adsiedit. Usage is guidconvert.exe <adfind guid output>.  Here’s the utility. C# source has been included as well.

Setting Up Server 2003 as a RADIUS with DD-WRT

A co-worker of mine was having some difficulties in setting up a RADIUS for his wireless network which is what prompted this particular article. For setting up your wireless infrastructure there are times when you need a more centrally controlled solution for the authentication problem. This is where RADIUS, and more to the point Microsoft’s IAS, steps in. For your trivia needs RADIUS stands for Remote Authentication Dial-In User Service, while IAS stands for Internet Authentication Service. Normally I would be setting this up under Server 2008 but our needs were calling for Server 2003. I may follow-up with how to do this under Server 2008 as well and even delve into putting together an IAS farm. The WAP being used is a Buffalo WHR-125 with a fairly current build of DD-wrt v24 SP2 (09/24/09) on it.

First off before installing IAS we will be in need of a certificate for it to use. There are several ways of achieving this. The first method, and easiest/cheapest, is creating a self-signed certificate using the IIS 6 Resource Kit from Microsoft. A particular program need from this is SelfSSL so run through a custom installation and install SelfSSL. Open up a command prompt and navigate to where SelfSSL installed at and here is how we will be constructing a certificate:

C:\Program Files\IIS Resources\SelfSSL>selfssl  /N:CN=server.domain.local /K:1024 /V:1825

This will get you your self-signed certificate. Of course you can use 3^rd party certificates as well. Another method is to issue one from an internal CA. Don’t forget to implement CA best practices when using one. I personally would opt for a self-signed certificate unless you already have a CA available.

Next up is getting our IAS installed. You will find this from Add/Remove Programs Add/Remove Windows Components. In there look for Networking Services and go into Details. Internet Authentication Service will be displayed just a few down. Once installed open up the mmc for IAS and let’s get into configuration. Though we should set up our users first. I went with creating a security group named Wireless Authentication and added my users in there. Note that you will need to allow these users for remote access as well. One way is to go into the user’s properties and on the Dial-In tab select Allow access. This isn’t my preferred method though as it creates more work. The other method I shall detail a bit later.

Bring up your IAS controls and you’ll see categories available. We need to get ourselves configured for our access point. To do this we will create a RADIUS Client. Right click on RADIUS Clients and select New RADIUS Client. Give the policy a name and point it to the address of the access point. Next menu is selecting our vendor which we will want to keep as RADIUS Standard for our configuration, as well as most configurations.  Put in a key for this client and note it down as we will need to configure it in the WAP later on. No need for the Message Authenticator attribute as it is used by default with EAP, which is what we will be configuring. For more information about it read here.

We have our client configured on the server but we are also in need of a Remote Access Policy. Right click the Remote Access Policies and select New Remote Access Policy. We will go with the first option for setting up our policy, though creating a custom policy is easy enough as well. On the next screen Access Method we will select Wireless. On the next screen we can put our group to use. Add in your Wireless Authentication group, unless you prefer to control things at the user level. I prefer security groups so that is what we will use. Select PEAP for the authentication method. Check the configuration of it to ensure that EAP-MSCHAP V2 is selected and that the proper certificate is selected as well. If you get an error when selecting Configure complaining about certificates then you need to go back and verify that you have a properly issued certificate. This is where most problems stem from. In the configuration you may also wish to enable Fast Reconnect. I have read about some clients having issues with this but have not had any problems in my configuration. Your mileage may vary. Disable it if you are having problems authenticating clients routinely. Finish this wizard and you’ll have your policy. We’re not quite done with it yet though.

Bring up the properties on your newly created policy. On the encryption tab you will want only Strongest encryption checked. If there are authentication issues though, you will want to enable the others for diagnostics until you figure out what is properly supported by your WAP. This is also where we can enable the alternate method for allowing our users. Go to the Advanced tab and add Ignore-User-Dialin-Properties set to True. This will ignore the setting on your user’s Dial-in tab and truly allow you to control access via groups. Otherwise user settings will trump group settings, which can make for a headache in troubleshooting. Last thing to do is right click the root folder, Internet Authentication Service, and select Register server in Active Directory. What this does is add your server to the RAS and IAS Servers security group, which enables it to read accounts from your AD. Once we are done here we can finally go configure our access point.

This is specific to DD-WRT, so be sure to verify how to configure your own access point. Connect to your access point and go to the Wireless tab, then Wireless Security. Set it to WPA2 Enterprise and make sure you are using AES, unless you have a reason not to. Put in the address for your IAS server and now would be a great time to make sure that it is a static address. Leave the port as 1812 as IAS listens on that out of the box. Finally put in the preshared key that you configured from earlier. Save then apply and your access point is in business. All that is left is configuring your clients.

This is best done through Windows’ wireless configuration. Manually create a new connection configured with your WAP’s SSID and go into the Security settings on it. Set it to use PEAP and if you are using a non-domain joined machine, that also does not have the certificate that you configured the server with, then tell it not to validate the certificate and also not to use your domain logon and password. Connect wirelessly to your access point and see if you’re successful. If you are not then check your server’s System event log for errors. If you are getting bad username/password errors, and you know your username and password are correct, then start looking at your encryption and configured authentication protocols to make sure they all match. If you are seeing errors about no matching policy then make sure you have your user in the right group or matching the criteria of your policy. That covers the majority of problems you will run into when configuring IAS. Even if you don’t have a use for IAS as a RADIUS it is a good idea to set it up a few times for learning purposes when pursuing an MCSE.