Here is a question that came into my forum and I thought it needed wider coverage.

Q: Can you explain the Yahoo! Messenger challenge response algorithm?

The Yahoo! Messenger challenge response sequence is quite complex and unique to Yahoo! The challenge comes from the server and is then run through an algorithm on the client. When looking at the challenge and response in ASCII view it almost looks like a mathematical equation but it is not.

This complex algorithm came from several years ago when the username and password was sent in plain text over the network and was eventually exploited. Basic encryption such as MD5 was then added. This is when things got interesting and politics stepped in. In 2004 Yahoo! was having a battle with several third-party applications such as Trillian as to if they could make their own client and join the Yahoo! Messenger network (http://www.theinquirer.net/en/inquirer/news/2004/06/24/yahoo-blocks-trillian-from-its-networks). Messenger has an ad driven revenue model so Yahoo! did not appreciate having an unofficial client not displaying ads. The solution from Yahoo! was to implement an outrageous and very complex authentication algorithm that the other companies could not reverse engineer. As anyone with a third person view could have predicted, the new monster algorithm did hold off Trillian for a while but was eventually cracked and the code was leaked to the Net.

Several years later the authentication code is not a huge secret but Yahoo! sill uses this beast to authenticate their users. I have never been able to find the original leaked code but it does live on in Pidgin which is an open source multi-platform client. To get a look at the code go to the Pidgin website (pidgin.im) and download the source for the latest build (http://sourceforge.net/project/showfiles.php?group_id=235&package_id=230234). Pidgin is written in c/c++ so it can be hard to read for someone not familiar with c. The code is also very integrated with the Pidgin base so it is next to impossible separate it out without having to rewrite the entire code base.

I have looked at the code and studied it for many hours and have come to the conclusion that it is overly complex and a nightmare to decipher. If Yahoo! wanted to make an algorithm that is hard to reverse engineer then this is a successful effort on their part. The downside is that a person would have to spend an insane amount of time to write their own representation of the code. The algorithm is a custom hash that has no direct relation to any common hash or encryption function. Parts of the code resemble MD5 while other parts look like DES. The majority of the code is based in lookup tables which is a common encryption technique.

A few years ago I wanted to make my own implementation in .NET because the DLL that YCC Trainer uses has been marked as a “virus” by most of the antivirus companies. (The DLL is not a virus but it appears that is has been marked that way because it is commonly distributed with booters. Most booters are also not viruses but in the infinite wisdom of the antivirus companies, we should be protected from ourselves but this is a different article all together.) After spending about a week trying to get the basics to work I realized that I hadn’t even scratched the surface and gave up. I still use the shady booter DLL that I found many years ago.

In the end I don’t want to discourage you from looking at the code for yourself but this is one fight that I decided not to take. It is ugly, nasty, and complex to the point of being a coding nightmare. If you do decide to look at the code I would love for you to post your findings, especially if you make another implementation. For now I am happy with my shady virus laden DLL that I once found in the far corners or the Internet.