HTTP Basic, Digest, NTLM and AWS S3 Authentication

Sample Number

110

Level

Advanced

Description

This sample demonstrates HTTP authentication support to secure services, as well as connect to already secured services

Use Case

The HTTP protocol supports transport level request authentication, and the most common schemes are: Basic Authentication and the more secure Digest Authentication scheme and NTLM authentication. With large scale use of the Amazon S3 infrastructure, the AWS S3 authentication over REST is also being used in many systems connecting with the Amazon S3 infrastructure. This article explains these authentication schemes, and describes how Proxy services deployed on the UltraESB could use Basic or Digest authentication to secure backend services, and also shows how Basic, Digest, NTLM or AWS S3 secured services can be accessed through the UltraESB - by allowing the ESB to perform transparent authentication on behalf of the clients. Each of these schemes maybe used over HTTPS as well.

The HTTP Authentication Schemes

Pre-emptive and Challenge-response authentication

HTTP Authentication mechanisms can be briefly divided into two popular categories. Those that allow the authentication information to be passed along upfront with the request; and those that requires a dynamic response performed over a challenge presented by the server. Pre-emptive authentication can be considered as performing slightly better than challenge-response authentication that requires a re-issuing of the request with a response to the challenge presented. However a challenge-response authentication scheme would offer a higher level of security. The Basic and AWS S3 schemes typically supports pre-emptive authentication, while the Digest and NTLM schemes support challenge-response based authentication.

Basic Authentication

HTTP Basic Authentication is the most simple authentication scheme that’s well described by the linked Wikipedia article. When pre-emptively used, it sends a HTTP header called 'Authorization' with the Base64 encoding of the "<username>:<password>" string as follows:

Basic Authentication Secured HTTP Request

GET /private/index.html HTTP/1.0
Host: localhost
Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==

If the authentication information is validated successfully, the normal / expected response is returned. If authentication information was not passed when making a request to a secured resource (i.e. if the request above was issued without the 'Authorization' header) or if the credentials fail, the server would respond back with a HTTP status code of 401 as follows, requesting authentication - to which the request would then be re-issued by the client with the 'Authorization' header.

HTTP Response requesting Basic Authentication

HTTP/1.0 401 Authorization Required
Server: HTTPd/1.0
Date: Sat, 27 Nov 2004 10:18:15 GMT
WWW-Authenticate: Basic realm="Secure Area"

Note: The Basic authentication scheme sends the password in an easily decrypt-able manner, and thus is not safe unless the request is made over an SSL secured connection.

Digest Authentication

HTTP Digest Authentication is again well described by the linked article from Wikipedia. A request that does not provide valid authentication credentials fails with a response from the server with a challenge as follows.

HTTP Response requesting Digest Authentication

HTTP/1.0 401 Unauthorized
Server: HTTPd/0.9
Date: Sun, 10 Apr 2005 20:26:47 GMT
WWW-Authenticate: Digest realm="testrealm@host.com",qop="auth,auth-int",nonce="dcd98b7102dd2f0e8b11d0f600bfb0c093",opaque="5ccc069c403ebaf9f0171e9517f40e41"

A client is expected to re-submit the request as a response to the challenge presented by the server (nonce, opaque, etc - See Wikipedia article for more details)

Digest Authentication secured HTTP Request

GET /dir/index.html HTTP/1.0
Host: localhost
Authorization: Digest username="Mufasa", realm="testrealm@host.com", nonce="dcd98b7102dd2f0e8b11d0f600bfb0c093", uri="/dir/index.html",
 qop=auth, nc=00000001, cnonce="0a4f113b", response="6629fae49393a05397450978507c4ef1", opaque="5ccc069c403ebaf9f0171e9517f40e41"
NTLM Authentication

NTLM authentication is described by its Wikipedia definition and some of its references, especially the one by Eric Glass. This could be a complex protocol to understand to someone unfamiliar with it, and involves multiple steps which are best explained from the previous links.

Amazon S3 Authentication

The Amazon S3 authentication scheme is defined by this article, and uses the AWSAccessKeyId and the AWSSecretAccessKey of the client to authenticate. A typical request could be presented as follows where the authentication header is computed as "Authorization: AWS AWSAccessKeyId:Signature" with the <Signature> component calculated with respect to the HTTP verb, request URI, date and other headers signed with the AWSSecretAccessKey.

AWS S3 Authentication secured HTTP Request

GET /photos/puppy.jpg HTTP/1.1
Host: johnsmith.s3.amazonaws.com
Date: Tue, 27 Mar 2007 19:36:42 +0000
Authorization: AWS 0PN5J17HBGZHT7JJ3X82:xXjDGYUmKxnwqr5KXNPGldn5LbA=

Securing Proxy Services with HTTP Basic and Digest Authentication

The UltraESB sample # 110 includes a complete - fully functional and annotated configuration used by each example presented in this article. It could be started and used to try out all of the following scenarios and more. The graphical SOA Toolbox will help to test some of the scenarios or a Web Browser may be used as the client. To start this example, invoke Sample # 110 configuration as follows from the /bin directory of your installation

Running the sample

On Linux   : $./ultraesb.sh -sample 110
On Windows : >ultraesb.bat -sample 110
authentication

The scenarios first define a service "rest-mock" that’s exposed over HTTP port 8281 and secured with Basic authentication, as well as over HTTP port 8282 and secured with Digest authentication. This service is hosted as a service implementation on the UltraESB, and will return request information. Its available at the following URLs:

Accessing the above URLs after starting the sample #110 configuration will request for user credentials using HTTP Basic (port 8281) or Digest (port 8282) authentication. If the correct credentials (username: asankha / password: adroitlogic) is presented, the user is shown a response as shown below.

Sample response from the "rest-mock" service

<response>
  <user>asankha</user>
  <roles>[ROLE_MANAGER, ROLE_USER]</roles>
  <method>GET</method>
  <uri>/service/rest-mock</uri>
  <query>{}</query>
</response>
Configuring Basic Authentication

Basic authentication is configured on the HTTP/S transport listener by defining the BasicAuthenticationFilter as a request filter. This causes each request received over this transport to be first passed through this filter, and the authentication credentials validated before being allowed to proceed. The realm name is configured as a property of the filter.

 1<!--Enforce Basic authentication on HTTP over port 8281-->
 2<bean id="http-8281" class="org.adroitlogic.ultraesb.transport.http.HttpNIOListener">
 3 <constructor-arg ref="fileCache"/>
 4 <property name="port" value="8281"/>
 5 <property name="requestFilters">
 6 <list>
 7     <bean class="org.adroitlogic.ultraesb.transport.http.auth.BasicAuthenticationFilter">
 8         <property name="realmName" value="adroitlogic"/>
 9     </bean>
10 </list>
11 </property>
12</bean>
13<!-- Standard Spring security authenticaion provider definitions are used to enforce basic and digest authentication
14Usernames/Password used is asankha/adroitlogic -->
15<s:authentication-provider>
16 <!--<s:password-encoder hash="md5"/>
17 <s:user-service>
18     <s:user name="asankha" password="abac6d7582d9ab52c629f7490fd3eb2f" authorities="ROLE_ADMIN, ROLE_USER"/>
19 </s:user-service>-->
20 <s:user-service>
21     <s:user name="asankha" password="adroitlogic" authorities="ROLE_USER, ROLE_MANAGER"/>
22 </s:user-service>
23</s:authentication-provider>

The authentication provider is configured using the standard Spring mechanisms, and maybe based on a simple in-memory list, a database, LDAP or many other mechanisms as supported by Spring Security. In the above example, a simple in-memory implementation is used with the usernames, passwords and roles described inline. The password maybe provided as an encrypted hash (See commented section), or encrypted using techniques described in the article Securing Configurations

Configuring Digest Authentication

Digest authentication is configured similarly to Basic authentication as follows using the DigestProcessingFilter. See above for more details.

 1<!--Enforce Digest authentication on HTTP over port 8282-->
 2<bean id="http-8282" class="org.adroitlogic.ultraesb.transport.http.HttpNIOListener">
 3 <constructor-arg ref="fileCache"/>
 4 <property name="port" value="8282"/>
 5 <property name="requestFilters">
 6 <list>
 7     <bean class="org.adroitlogic.ultraesb.transport.http.auth.DigestProcessingFilter">
 8         <property name="realmName" value="adroitlogic"/>
 9     </bean>
10 </list>
11 </property>
12</bean>
The Proxy service secured with Basic and/or Digest authentication - [Path A and Path B]

This is the "rest-mock" proxy service that responds with the authenticated username, roles and the HTTP method, URI and query parameters of the request. By specifying one or more transport ID’s a proxy maybe exposed over one or more transports. In this example, the same service is exposed over transports defined as "http-8281" and "http-8282" which was configured in the previous sections to use HTTP Basic and Digest authentication respectively.

To test this service, point your browser at URL [http://localhost:8281/service/rest-mock] - Path A, or [http://localhost:8282/service/rest-mock] - Path B. Your browser will request for the credentials, and you will be presented with the expected response after successful authentication if the correct credentials asankha/adroitlogic is presented.

 1<u:proxy id="rest-mock">
 2 <u:transport id="http-8281"/>
 3 <u:transport id="http-8282"/>
 4 <u:target>
 5     <u:inSequence>
 6     <u:java import="org.adroitlogic.ultraesb.api.transport.http.HttpConstants;"><![CDATA[
 7         System.out.println("User is : " + msg.getMessageProperty(HttpConstants.USERNAME));
 8         System.out.println("Roles are : " + msg.getMessageProperty(HttpConstants.USERROLES));
 9         Message res = msg.createDefaultResponseMessage();
10         mediation.setPayloadFromString(res,
11         "<response>" +
12         "<user>" + msg.getMessageProperty(HttpConstants.USERNAME) + "</user>" +
13         "<roles>" + msg.getMessageProperty(HttpConstants.USERROLES) + "</roles>" +
14         "<method>" + msg.getMessageProperty(HttpConstants.METHOD") + "</method>" +
15         "<uri>" + msg.getDestinationURL() + "</uri>" +
16         "<query>" + msg.getMessageProperty(HttpConstants.QUERY_PARAM_MAP) + "</query>" +
17         "</response>");
18         mediation.sendResponse(res, 200);
19     ]]></u:java>
20     </u:inSequence>
21 </u:target>
22</u:proxy>

Proxy Services performing authentication on behalf of users

Basic Authentication - [Path C and Path F]
 1<u:proxy id="basic-auth-proxy">
 2 <u:transport id="http-8280"/>
 3 <u:target>
 4     <u:inSequence>
 5     <u:java import="org.adroitlogic.ultraesb.api.transport.http.HttpConstants;"><![CDATA[
 6         msg.addMessageProperty(HttpConstants.AUTH_USERNAME, "asankha");
 7         msg.addMessageProperty(HttpConstants.AUTH_PASSWORD, "adroitlogic");
 8     ]]></u:java>
 9     </u:inSequence>
10     <u:inDestination>
11         <u:address>http://localhost:8281/service/rest-mock</u:address>
12     </u:inDestination>
13     <u:outDestination>
14         <u:address type="response"/>
15     </u:outDestination>
16</u:target>
17</u:proxy>

This configuration defines the credentials to perform a non-preemptive basic authentication against the backend service, only if requested by the service. When using basic authentication, using pre-emptive authentication (described next) will usually allow better performance where the authentication requirements are already known.

Pre-emptive Basic authentication - [Path C and Path E]
 1<u:proxy id="preemptive-basic-auth-proxy-1">
 2 <u:transport id="http-8280"/>
 3 <u:target>
 4     <u:inSequence>
 5     <u:java><![CDATA[
 6         mediation.addPreemptiveBasicAuthentication(msg, "asankha", "adroitlogic");
 7     ]]></u:java>
 8     </u:inSequence>
 9     <u:inDestination>
10         <u:address>http://localhost:8281/service/rest-mock</u:address>
11     </u:inDestination>
12     <u:outDestination>
13         <u:address type="response"/>
14     </u:outDestination>
15 </u:target>
16</u:proxy>

Basic authentication maybe pre-emptively performed by the UltraESB on behalf of a client request in either of two ways. The example above depicts how pre-emptive basic authentication maybe added to a message within the mediation code. To test this service, point your browser at URL [http://localhost:8280/service/preemptive-basic-auth-proxy-1]. The request will be pre-emptively authenticated by the UltraESB, and thus you will be presented with the expected response after successful authentication.

 1<u:proxy id="preemptive-basic-auth-proxy-2">
 2 <u:transport id="http-8280"/>
 3 <u:target>
 4     <u:inDestination>
 5         <u:address>http://localhost:8281/service/rest-mock</u:address>
 6             <u:property name="ultra.http.auth_username" value="asankha"/>
 7             <u:property name="ultra.http.auth_password" value="adroitlogic"/>
 8             <u:property name="ultra.http.auth_scheme" value="basic"/>
 9         </u:inDestination>
10     <u:outDestination>
11         <u:address type="response"/>
12     </u:outDestination>
13 </u:target>
14</u:proxy>

The above example depicts how pre-emptive basic authentication maybe performed by specifying credentials specified as properties of a destination endpoint. This method could be used to specify basic, digest or Amazon S3 authentication credentials at the destination endpoint level. To test this service, point your browser at URL [http://localhost:8280/service/preemptive-basic-auth-proxy-2]. The request will be pre-emptively authenticated by the UltraESB, and thus you will be presented with the expected response after successful authentication.

Digest authentication - [Path C and Path G]
 1<u:proxy id="digest-auth-proxy">
 2 <u:transport id="http-8280"/>
 3 <u:target>
 4     <!--<u:inSequence>
 5     <u:java import="org.adroitlogic.ultraesb.api.transport.http.HttpConstants;"><![CDATA[
 6         msg.addMessageProperty(HttpConstants.AUTH_USERNAME, "asankha");
 7         msg.addMessageProperty(HttpConstants.AUTH_PASSWORD, "adroitlogic");
 8     ]]></u:java>
 9     </u:inSequence>-->
10     <u:inDestination>
11         <u:address>http://localhost:8282/service/rest-mock</u:address>
12             <u:property name="ultra.http.auth_username" value="asankha"/>
13             <u:property name="ultra.http.auth_password" value="adroitlogic"/>
14         </u:inDestination>
15     <u:outDestination>
16         <u:address type="response"/>
17     </u:outDestination>
18 </u:target>
19</u:proxy>

Digest authentication credentials could be supplied in the same manner as credentials supplied for Basic authentication in the last section. The Commented code line numbers 6-7 shows how they could be specified programmatically, and the lines 12-13 depicts how they could be provided at the destination endpoint level.

Amazon S3 authentication - [Path C and Path D]
 1<u:proxy id="s3-auth-proxy">
 2 <u:transport id="http-8280">
 3     <u:property name="ultra.transport.url" value="s3-auth-proxy*"/>
 4 </u:transport>
 5 <u:target>
 6     <u:inSequence>
 7     <u:java import="org.adroitlogic.ultraesb.api.transport.http.HttpConstants;"><![CDATA[
 8         // read AWS S3 AccessKey and SecretKey from an environment variables
 9         msg.addMessageProperty(HttpConstants.AUTH_USERNAME, System.getenv().get("AWS_S3_USERNAME"));
10         msg.addMessageProperty(HttpConstants.AUTH_PASSWORD, System.getenv().get("AWS_S3_PASSWORD"));
11         System.out.println("Using AWS Key : " + msg.getMessageProperty(HttpConstants.AUTH_USERNAME));
12         System.out.println("Using AWS Secret Key : " + msg.getMessageProperty(HttpConstants.AUTH_PASSWORD));
13     ]]></u:java>
14     </u:inSequence>
15     <u:inDestination>
16         <u:address type="prefix">http://downloads.adroitlogic.com.s3.amazonaws.com</u:address>
17         <u:property name="ultra.http.auth_scheme" value="aws.s3"/>
18     </u:inDestination>
19     <u:outDestination>
20         <u:address type="response"/>
21     </u:outDestination>
22 </u:target>
23</u:proxy>

The example above has intentionally been made to read the Amazon S3 credentials of the user from the system environment variables "AWS_S3_USERNAME" and "AWS_S3_PASSWORD" for security reasons. However these maybe easily specified in-line as per previous examples, and optionally encrypted in the configuration.

To upload a file to your Amazon S3 account using the above proxy service, change the destination endpoint URL to your own S3 account, add message properties "transport.auth.username" and "transport.auth.password" as your Amazon S3 AWS Key and Secret Key. You may now upload a text file using the SOA Toolbox as follows.

Set the URL to "http://localhost:8280/service/s3-auth-proxy/test.txt", HTTP method to"PUT", Context type to"text/plain" and the payload body to say "Hello Amazon S3 from UltraESB". Pressing the"Send" button will now yield a successful response as shown below.

s3 upload

To read the file just stored again from Amazon S3, request for the URL "http://localhost:8280/service/s3-auth-proxy/test.txt" from your browser or the ToolBox, and you could see the content that was just stored in the previous step. Using the ToolBox you could use the REST methods GET, DELETE and PUT etc, to retrieve, delete or update this content. A GET request from the SOA Toolbox will receive a response as follows:

Sample response for the AWS S3 secured request

HTTP/1.0 200 OK
x-amz-request-id: 7116031469BAF93F
ETag: "6ef184923dc9c5c5e75c6739b0a02f2e"
Date: Sat, 29 May 2010 16:18:50 GMT
Last-Modified: Sat, 29 May 2010 16:10:41 GMT
x-amz-id-2: hxJ+FHh012dwgjRm4TCL8RlYlQFFeIo/hqWU0h1nl1NJGhN6MYjp9rQv8O0+qDHj
Content-Type: text/plain; charset=UTF-8
Server: UltraESB/2.6.1
Content-Length: 29
Connection: close

Hello Amazon S3 from UltraESB
NTLM authentication - [Path C and Path H]
 1<u:proxy id="ntlm-auth-proxy">
 2 <u:transport id="http-8280"/>
 3 <u:target>
 4     <u:inSequence>
 5     <u:java import="org.adroitlogic.ultraesb.api.transport.http.HttpConstants;"><![CDATA[
 6         msg.addMessageProperty(HttpConstants.AUTH_USERNAME, "administrator");
 7         msg.addMessageProperty(HttpConstants.AUTH_PASSWORD, "RhU2Xfwcmew");
 8         msg.addMessageProperty(HttpConstants.NTLM_DOMAIN, "WORKGROUP");
 9     ]]></u:java>
10     </u:inSequence>
11     <u:inDestination>
12         <u:address>http://ec2-184-73-124-76.compute-1.amazonaws.com/test.asmx</u:address>
13     </u:inDestination>
14     <u:outDestination>
15         <u:address type="response"/>
16     </u:outDestination>
17 </u:target>
18</u:proxy>

The above example is for illustration only, and shows the authentication against a page secured with NTLM authentication

Related Topics and Extension

The UltraESB could easily be configured to accept or send messages over any of its supported transports such as HTTP/S, FTP/S, SFTP, File, Email (POP3/IMAP), MLLP/S or JMS etc. Thus one could easily write a proxy service that may fetch a file from an SFTP location on a given CRON schedule, and post it to Amazon S3 using the S3 authentication, or to accept a REST request with Digest authentication and send an email message of the request using SMTP etc. In addition, you may configure authentication against any Spring Security supported backend service such as an in-memory list, Database, LDAP or custom implementation etc.

WS-Security Username Token Authentication

WS-Security based Username Token authentication is not discussed in this article, but is supported by the UltraESB with its WS-Security support. See relevant sections for more details.

XACML Authorization

The UltraESB supports XACML authorization to allow fine grained access control. See documentation on the XACML support for more information.

In this topic
In this topic
Contact Us