Hmmm...
What is the client operating system?
Had a quick goole around and found this.
Troubleshooting: "Error 789 processing error"
You may get the following error:
"Error 789: The L2TP connection attempt failed because the security layer encountered a processing error during initial negotiations with the remote computer."
This error may occur in the following cases:
The certificate (plus private key) has not been installed correctly.
Windows 2000/XP's built-in IPsec stack has been disabled, e.g. when a third-party IPsec client (e.g. SSH Sentinel) was installed and then removed.
You are connected to a Windows 2000 Server through a Terminal Server (RDP) session. On that Windows 2000 Server you have configured a VPN connection to an L2TP/IPsec server. You attempt to bring up this connection but you get the Error 789 (see also MS KB Q326751).
In the first case, try to import the certificate again following the instructions. Verify in MMC that certificates actually have been installed for both the CA and for the user, including the private key. When you view the details of your certificate, you should see the message "This certificate has a corresponding private key". When MMC asks where you want to store the certificate, be sure to select "Local Computer", and not "My user account". Verify that the certificates are valid (check the start and end dates) and issued by the same CA as used on your FreeS/WAN box. Check the internal clock of your computer: if it is set to a strange date (say, 1970 or so), your computer will think that the certificate is not (yet) valid.
In the second case, you can re-enable the Windows IPsec service as follows. Click Start -> Programs -> Administrative Tools -> Services. Select "IPSec Policy Agent" from the list and check if the Startup type is set to "Automatic". If it is not, this is the problem. Set Startup type to "Automatic", click Apply and then Start.
Microsoft's error message is misleading. The user may easily get the impression that the Linux server is at fault. However, the error pops up immediately after clicking "Dial". I also sniffed the network communication between the client and the server: there was none. If there are no packets exchanged between Windows 2000/XP and the Linux server then it is impossible that the server is to blame. It seems to me that other error messages would have been more appropriate, for instance the error 781 mentioned above.
Removing SSH Sentinel is described here (unfortunately these instructions have not been updated for version 1.4 but you'll get the idea).
The third case is a bit atypical and I am only mentioning it for completeness. Q326751 says this is a known problem in Windows 2000 Server. According to Oleksander Darchuk the problem does not occur if you use VNC or any other remote administration tool, as long as it is not RDP. Alternatively, you could make a VPN connection from the client itself, bypasssing Windows 2000 Server. Or perhaps the problem has been fixed in Windows 2003. Either way, the L2TP/IPsec VPN server is not to blame.
Time for me to spend some time with my family so see you tomorow.
Shane
16 Replies
2885 Views