Token-based authentication (authenticating at the ArcGIS Server tier) is explained comprehensively here but customers have asked us what is happening behind the scenes when authentication is done at the web tier.
How does ArcGIS Server Web-tier authentication work?
When web-tier authentication is used, authentication is handled between IIS on the web server and the web browser. If the user specifies "Windows Authentication", then an authentication approach called Negotiate/SPNEGO is used which then leads to either NTLM or Kerberos authentication schemes being used. These are protocols that Microsoft has defined for http authentication.
Both NTLM and Kerberos are mechanisms that don't involve sending user passwords directly but instead send a hash based on the password. IIS then directly authenticates the user's hash against Active Directory. During this process Esri software is not involved at all however when the user authentication is successful, the web adaptor gets called for the first time.
The web adaptor then asks IIS for the username of the authenticated user. The web adaptor cannot get the password because IIS itself never got the password, just a hash based on the password. The web adaptor then makes a .Net call to ask which groups the user is in using the standard .Net mechanism. The user and the groups the user belongs to is then sent to the rest of ArcGIS Server through an encrypted mechanism that prevents tampering with the information using an established trust mechanism that prevents spoofing attacks. That's how authentication works for ArcGIS Server when using integrated windows authentication when accessing ArcGIS Server services in 10.1.x and 10.2.x
For administrative requests at 10.1, ArcGIS Server issues tokens after directly authenticating the user against the Active Directory using a simple bind over SSL/TLS. This is also a fully supported and secure means of authenticating against Active Directory.
ArcGIS Server securely handles credentials in all cases. If you wish to use Kerberos exclusively then it's recommended that you use 10.2.x with web-tier authentication and do all administration through the web adaptor.