HelpMaster v10.x.x and above.
After upgrading to v10 to v12 from v8 to v9.1, Active Directory authentication on the Web Module elicits a Challenge authentication response and no matter how many times you enter your Windows Domain credentials the challenge dialogue remains. This is due to Microsoft's Dot.Net Framework v3.5 SP1 security enhancements as outlined in detail in the Microsoft MSDN article below. In short this is because the default domain DNS resolved machine name or FQDN is not being used in the URL, e.g. Http://[MachineName].[DomainName].com/[VirtualDirectory]/Winlogin.aspx, but you are using the machine IP address, machine name only, a registered domain name, or an alias rather than the registered and resolved name returned by DNS on your domain so that the Windows Authentication is fails as a result.
After upgrading to v10 to v12 from v8 to v9.1, Active Directory authentication on the Web Module elicits a Challenge authentication response and no matter how many times your Windows Domain credentials are entered the challenge dialogue remains as follows;
NOTE: If your credentials are accepted the first time you enter them at the Windows Security challenge dialogue, then your configuration is not suffering from this issue but your problem is that Windows Authentication is not enabled on your browser or the host name hasn't been entered as a Trusted Site. In this case see the Knowledge Base article below under "Related Articles".
To cater for the security enhancement of the Dot.Net Framework v3.5 SP1, your IP address, registered domain name, machine name, FQDN, and/or alias needs to be added to a registry key on the IIS server hosting the HelpMaster Web Module as follows;
1. Click Start, click Run, type 'regedit', and then click OK.
2. In Registry Editor, locate and then click the following registry key:
3. Right-click MSV1_0, point to New, and then click Multi-String Value.
4. Type 'BackConnectionHostNames', and then press ENTER.
5. Right-click BackConnectionHostNames, and then click Modify.
6. In the Value data box, type the host name or the host names for the sites (the host name used in the request URL) that are on the local computer, each on a new line and then click OK.
7. Quit Registry Editor, and then restart the workstation or server. The problem will not be resolved until the machine has been restarted.
PRD Software Knowledge Base articles;
Internet Explorer settings - http://www.helpmasterpro.com/Community/Discussion-Board/aft/126.aspx#126
Microsoft MSDN article - http://msdn.microsoft.com/en-us/library/cc982052%28v=vs.90%29.aspx
Microsoft Knowledge Base article - http://support.microsoft.com/kb/896861