How to hide credentials used in mediation configuration using Secure Vault in WSO2 ESB

Eventhough we use secure vault to encrypt password, it is not possible to use secure vault directly in the mediation configuration. As an example imagine you need to hide a password given in a proxy.

All you have to do is using Secure Vault Password Management screen in WSO2 ESB.


1. Run sh ciphertool.sh -Dconfigure and enable secure vault
2. Start the WSO2 ESB with
3. Go to  Manage -> Secure Vault Tool and then click Manage Passwords
4. You will see the below screen.

5. Click on Add New Password to encrypt and store and specify values.

Vault Key - The alias for the password. E.g., sfpassword
Password -  The actual password.
Re-enter password - Re enter the password.

6. Once you save you will be able to save the newly created vault key.

7. Now you can use it in your proxy or any other mediation configuration as below.

E.g.,

   <inSequence>  
      <property name="password" expression="wso2:vault-lookup('sfpassword')"/>  
      <log level="custom">  
       <property name="InMsg" value="Invoking the proxy"/>  
       <property name="pickPassword" expression="get-property('password')"/>  
      </log>  
      <property name="OUT_ONLY" value="true"/>  
      <send/>  
    </inSequence>  

8. If you want to hide the salesforce password,

Salesforce Configuration :

  <salesforce.getUserInfo configKey="MySFConfig"/>   
    <salesforce.query>   
    <batchSize>1</batchSize>   
    <queryString>{$ctx:CompleteLeadQuery}</queryString>   
  </salesforce.query>   

MySFConfig with salesforce connection details :

 <salesforce.init>  
   <username>xxxx@abc.com</username>  
   <password>{wso2:vault-lookup('sfpassword')}</password>  
   <loginUrl>https://login.salesforce.com/services/Soap/u/27.0</loginUrl>  
   <blocking>false</blocking>  
 </salesforce.init>  

Simple WSO2 ESB API which Queries salesforce and build a json array using payloadFactory Mediator

1. Download WSO2 ESB and Salesforce connector.
2. Add the salesforce connector and enable it.

For more information please follow : https://docs.wso2.com/display/ESBCONNECTORS/Working+with+Salesforce+Connector+Operations


Then the below API can be used to query User object using profile id and then build a json object using payload factory mediator.

• MySFConfig should have the required login information.
• 00e90000001aVwiAAE is the profile id of a user.

 <api xmlns="http://ws.apache.org/ns/synapse" name="leads1" context="/leads1">  
   <resource methods="GET">  
    <inSequence>  
      <property name="LeadQuery" value="Select u.Username, u.ProfileId, u.Name, u.LastName, u.Email From User u where ProfileId='" scope="default" type="STRING"/>  
      <property name="Apostrophe" value="'" scope="default" type="STRING"/>  
      <property name="ProfileId" value="00e90000001aVwiAAE" scope="default" type="STRING"/>  
      <property name="CompleteLeadQuery" expression="fn:concat($ctx:LeadQuery, $ctx:ProfileId, $ctx:Apostrophe)" scope="default" type="STRING"/>  
      <salesforce.getUserInfo configKey="MySFConfig"/>  
      <salesforce.query>  
       <batchSize>1</batchSize>  
       <queryString>{$ctx:CompleteLeadQuery}</queryString>  
      </salesforce.query>  
      <property xmlns:ns="urn:partner.soap.sforce.com" xmlns:sf="urn:sobject.partner.soap.sforce.com" name="Name" expression="//ns:queryResponse/ns:result/ns:records/sf:Name/text()" scope="default" type="STRING"/>  
      <property xmlns:ns="urn:partner.soap.sforce.com" xmlns:sf="urn:sobject.partner.soap.sforce.com" name="Username" expression="//ns:queryResponse/ns:result/ns:records/sf:Username/text()" scope="default" type="STRING"/>  
      <property xmlns:ns="urn:partner.soap.sforce.com" xmlns:sf="urn:sobject.partner.soap.sforce.com" name="LastName" expression="//ns:queryResponse/ns:result/ns:records/sf:LastName/text()" scope="default" type="STRING"/>  
      <property xmlns:ns="urn:partner.soap.sforce.com" xmlns:sf="urn:sobject.partner.soap.sforce.com" name="Email" expression="//ns:queryResponse/ns:result/ns:records/sf:Email/text()" scope="default" type="STRING"/>  
      <log level="full" separator=","/>  
      <payloadFactory media-type="json">  
       <format>{ "ProfileId": { "source": "SALESFORCE", "id": "$1" },  "Name": "$2", "Username": "$3", "LastName":  "$4", "Email": "$5"  }</format>  
       <args>  
         <arg evaluator="xml" expression="$ctx:ProfileId"/>  
         <arg evaluator="xml" expression="$ctx:Name"/>  
         <arg evaluator="xml" expression="$ctx:Username"/>  
         <arg evaluator="xml" expression="$ctx:LastName"/>  
         <arg evaluator="xml" expression="$ctx:Email"/>  
       </args>  
      </payloadFactory>  
      <property name="messageType" value="application/json" scope="axis2" type="STRING"/>  
      <respond/>  
    </inSequence>  
   </resource>  
 </api>  

Enable email login in WSO2 carbon products

To enable email address the below steps can be followed in any carbon product.

1. EnableEmailUserName in carbon.xml

<EnableEmailUserName>true</EnableEmailUserName>

2. Then provide the correct regex to allow email address in user store configuration in user-mgt.xml for JDBC user store

E.g.,

    <Property name="UsernameJavaRegEx">[a-

zA-Z0-9@._-|//]{3,30}$</Property>

3. Create admin user with email address in user in user-mgt.xml.

   <AdminUser>
                     <UserName>admin@wso2.com</UserName>

                     <Password>admin</Password>
  </AdminUser>

By the above configurations, it will enable email address.

If you want to give the both support, email address and username, you can include the below property in user store configuration.

4.  <Property name="

UsernameWithEmailJavaScriptRegEx">[a-zA-Z0-9@._-|//]{3,30}$</Property>

      

To know how to do this for a LDAP, refer this well explained blog post [1] done for Identity server which is applicable for other carbon products as well. This document also explains the properties [2]

[2] http://xacmlinfo.org/2014/10/07/email-username-with-identity-server/
[3] https://docs.wso2.com/display/IS500/Working+with+Properties+of+Primary+User+Stores

A reason for getting com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Sometimes you will end up in getting an exception like below when starting a server pointed to MySQL DB.

ERROR - DatabaseUtil Database Error - Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server


One possible reason can be since you have mapped, bind-address to an IP in /etc/mysql/my.cnf

The default is 0.0.0.0 which is all interfaces. This setting does not restrict which IPs can access the server, unless you specified 127.0.0.1 for localhost only or some other IP.

bind-address            = 0.0.0.0

Once this is done, restart mysql server.

 sudo service mysql restart

Simple explaination for URL configurations done during IS as Key Manager in WSO2 API Manager

When you configure as IS as Key Manager, this document can be refereed. The below explanations for some of the URLs given in the configuration.

In WSO2 IS Side :

Make the following changes in the api-manager.xml file you just copied.

• Change the <RevokeAPIURL> so that it points to the API Manager server. Note that if API Manager is running in distributed mode (has a separate node for the Gateway), you need to point this URL to the Gateway node. This is done so that when the token is revoked, the Gateway cache is updated as well. The port value you enter here must be the NIO port. See Default Ports of WSO2 Products for more information.

 <RevokeAPIURL>https://${GATEWAY_SERVER_HOST}:{nio/passthrough port}/revoke</RevokeAPIURL>

Why do we point to API Manager ? This is because this will call the Gateway node in API manager which points to the Key Manager

Why we point to the NIO/Passthru port ? This is because, here we will call the _RevokeAPI_.xml which is deployed in synapse folder of the Gateway. (In a distributed scenario, we point to the _RevokeAPI_.xml that resides in the Gateway worker node)

Note : So in a distributed scenario (APIM distributed scenario), In store if you call Revoke or Regenerate, store will call the key Manager/Validator and key Manager/Validator will call the gateway.

So in Store we have to configure KeyValidator/Key manager server url and in Key Manager/Validator we have to configure gateway server url (Passthru). Inside Gateway the apis will be called (NIO)

• Change the <ServerURL> occurring under the <APIGateway> (of the Key Manager/Key Validator node) section so that it points to the API Manager server. If you are using distributed mode, this needs to point to the Gateway node as well. This is done so that when the token is regenerated, the Gateway cache is updated as well. The port value you enter here must be the management transport port.

<ServerURL>https://${GATEWAY_SERVER_HOST}:{port}/services/</ServerURL>

Why we point to API Manager ? This  is because this will call the Gateway node in API manager. This is to identify the gateway node.

Why we point to servlet port ? This is because it calls the admin services.

Note : This is same like as above explained. Since IS is the Key Validator/Manager it calls the gateway.

In WSO2 APIM Side :

Open the api-manager.xml file found in the <APIM_HOME>/repository/conf directory and change the following. 

• Change the ServerURL of the AuthManager to point to IS.

  <ServerURL>https://${IS_SERVER_HOST}:{port}/services/</ServerURL>

Why IS : This is because authentication will be done via Key manager/validator

Why we point to servlet port ? This is because it calls the admin services.

• Change the ServerURL of the APIKeyManager to point to IS.

    <ServerURL>https://${IS_SERVER_HOST}:{port}/services/</ServerURL>

Why IS : This is because authentication will be done via Key manager/validator
Why we point to servlet ports : This is because it calls admin services

Usage of ports in a gateway cluster when fronted by Nginx:

GW manager : Use to publish APIs - So only the servlet ports will be used
GW worker : Use when invoking - So only the passthrough/NIO ports will be used

Note : When you configure API endpoints in gateway (Which is in synapse folder  (E.g.,     _AuthorizeAPI_.xml, _RevokeAPI_.xml, _TokenAPI_.xml), you should edit them only in gateway manager since it will depsync to workers. Otherwise it can cause issues.

How to Enable and test Custom SSL Profiles in WSO2 ESB used for SSL communicating

For this I have used WSO2 ESB 4.8.1 and WSO2  Application Server 5.2.1.
WSO2 ESB uses the truststore for SSL communicating and keystore-truststore pair for Mutual SSL communicating.

In here I have used a trust store for SSL communicating.

Configure App Server as backend

Configure backend :

1. Use app server as backend

2. Create a new keystore in App server in <Appserver_Home>/repository/resources/security

keytool -genkey -alias appserver -keyalg RSA -keysize 1024 -keypass password -keystore appserver.jks -storepass password

3. Export in to a pem file by following command

keytool -export -alias appserver -keystore appserver.jks -storepass password -file appserver.pem

4. Edit the carbon.xml in appserver as below :

 <KeyStore>  
    <!-- Keystore file location-->  
      <Location>${carbon.home}/repository/resources/security/appserver.jks</Location>  
 <!-- Keystore type (JKS/PKCS12 etc.)-->  
     <Type>JKS</Type>  
 <!-- Keystore password-->  
     <Password>password</Password>  
 <!-- Private Key alias-->  
     <KeyAlias>appserver</KeyAlias>  
 <!-- Private Key password-->  
     <KeyPassword>password</KeyPassword>  
 </KeyStore>  

Configure ESB :

1. Created a new keystore.

keytool -genkey -alias esb -keyalg RSA -keysize 1024 -keypass password -keystore esb.jks -storepass password

2. Copy and paste appserver.pem in to the <ESB_HOME>repository/resources/security folder Import appserver.pem in to esb.jks by following command.

keytool -import -alias appservernewesb -file appserver.pem -keystore esb.jks -storepass password

3. Configure esb for custom profile in axis2.xml as below.

 <parameter name="customSSLProfiles">  
  <profile>  
    <servers>10.100.0.31:9443</servers>  
    <TrustStore>  
     <Location>repository/resources/security/esb.jks</Location>  
     <Type>JKS</Type>  
     <Password>password</Password>   
    </TrustStore>  
  </profile>  
 </parameter>  

Invoke and Test :

1. Restart Appserver (offset=0) and ESB (offset = 10) by the command :

 "sh wso2server.sh" or  "sh wso2server.sh -Djavax.net.debug=ssl:handshake " to view the detailed logs.

Following logs should be printed during restart.

[2015-04-27 18:33:19,397]  INFO - ClientConnFactoryBuilder HTTPS Loading Identity Keystore from : repository/resources/security/wso2carbon.jks
[2015-04-27 18:33:19,400]  INFO - ClientConnFactoryBuilder HTTPS Loading Trust Keystore from : repository/resources/security/client-truststore.jks
[2015-04-27 18:33:19,408]  INFO - ClientConnFactoryBuilder HTTPS Loading custom SSL profiles for the HTTPS sender
[2015-04-27 18:33:19,408]  INFO - ClientConnFactoryBuilder HTTPS Loading Trust Keystore from : repository/resources/security/esb.jks
[2015-04-27 18:33:19,409]  INFO - ClientConnFactoryBuilder HTTPS Custom SSL profiles initialized for 1 servers



2. Create the below proxy in ESB.

 <?xml version="1.0" encoding="UTF-8"?>  
 <proxy xmlns="http://ws.apache.org/ns/synapse"  
     name="SecureHello"  
     transports="https,http"  
     statistics="disable"  
     trace="disable"  
     startOnLoad="true">  
   <target>  
    <outSequence>  
      <send/>  
    </outSequence>  
    <endpoint>  
      <address uri="https://localhost:9443/services/HelloService/"/>  
    </endpoint>  
   </target>  
   <publishWSDL uri="http://localhost:9763/services/HelloService?wsdl"/>  
   <description/>  
 </proxy>  

3. Invoke the Proxy.

 <body>  
   <p:greet xmlns:p="http://www.wso2.org/types">  
    <!--0 to 1 occurrence-->  
    <name>ushani</name>  
   </p:greet>  
 </body>  

Following response will be received.

 <ns:greetResponse xmlns:ns="http://www.wso2.org/types">  
   <return>Hello World, ushani !!!</return>  
 </ns:greetResponse>  

How to monitor the created tcp connections for a particular proxy service in WSO2 ESB

As an example if we want to monitor the number of tcp connections created during a proxy invocation, the following steps can be performed.

1. Assume you need to monitor the number of tcp connections created for the following proxy service :

 <?xml version="1.0" encoding="UTF-8"?>  
 <proxy xmlns="http://ws.apache.org/ns/synapse"  
     name="Proxy1"  
     transports="https,http"  
     statistics="disable"  
     trace="disable"  
     startOnLoad="true">  
   <target>  
    <inSequence>  
      <property name="NO_KEEPALIVE" value="true" scope="axis2"/>  
      <clone>  
       <target>  
         <sequence>  
          <send>  
            <endpoint>  
             <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>  
            </endpoint>  
          </send>  
         </sequence>  
       </target>  
        <target>  
         <sequence>  
          <send>  
            <endpoint>  
             <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>  
            </endpoint>  
          </send>  
         </sequence>  
       </target>  
       <target>  
         <sequence>  
          <send>  
            <endpoint>  
             <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>  
            </endpoint>  
          </send>  
         </sequence>  
       </target>  
      </clone>  
    </inSequence>  
    <outSequence>  
      <aggregate>  
       <completeCondition>  
         <messageCount/>  
       </completeCondition>  
       <onComplete xmlns:m0="http://services.samples" expression="//m0:getQuoteResponse">  
         <send/>  
       </onComplete>  
      </aggregate>  
    </outSequence>  
   </target>  
   <description/>  
 </proxy>  

You will see there are 5 clones. Therefore it should create only 5 tcp connections.

2. We have used SimpleStockQuoteService service as the backend.

3. Since you need to monitor the connections created, we should delay the response coming from the backend. Therefore we need to change the code slightly in the SimpleStockQuoteService.

We have include a Thread.sleep() for 10 seconds until we monitor the number of connections.

Therefore go to <ESB_HOME>/samples/axis2Server/src/SimpleStockQuoteService/src/samples/services/SimpleStockQuoteService.java

and add the Thread.sleep(10000); as below to hold the response for sometime.

 public GetQuoteResponse getQuote(GetQuote request) throws Exception {  
 Thread.sleep(10000);  
 if ("ERR".equals(request.getSymbol())) {  
 throw new Exception("Invalid stock symbol : ERR");  
 }  
 System.out.println(new Date() + " " + this.getClass().getName() +  
 " :: Generating quote for : " + request.getSymbol());  
 return new GetQuoteResponse(request.getSymbol());  
 }   

4. Build the SimpleStockQuoteService once again by “ant” in here, <ESB_HOME>/samples/axis2Server/src/SimpleStockQuoteService

5. Now start the axis2server

6. Now you have to open a terminal and provide the following netstat command to get the process id.

sudo netstat --tcp --listening --programs

You will see the relevant SimpleStockQuoteService which is up in port 9000 like below.

tcp6 0 0 [::]:9000 [::]:* LISTEN 20664/java

So your process ID will be 20664.

7. Then open a terminal and provide the below command to view the tcp connections for the particular process id.

watch -n1 -d "netstat -n -tap | grep 20664"

8. Now open your soapui and send the following request to Proxy1

 <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ser="http://services.samples" xmlns:xsd="http://services.samples/xsd">  
   <soapenv:Header/>  
   <soapenv:Body>  
    <ser:getQuote>  
      <!--Optional:-->  
      <ser:request>  
       <!--Optional:-->  
       <xsd:symbol>IBM</xsd:symbol>  
      </ser:request>  
    </ser:getQuote>  
   </soapenv:Body>  
 </soapenv:Envelope>  

9. View the tcp connections created in the terminal which you have been monitoring as soon as you send the request. You should be able to view only 3 connections since we have configured like that in proxy using clone mediator.

tcp6 0 0 127.0.0.1:9000 127.0.0.1:44218 ESTABLISHED 20664/java

tcp6 0 0 127.0.0.1:9000 127.0.0.1:44219 ESTABLISHED 20664/java

tcp6 0 0 127.0.0.1:9000 127.0.0.1:44217 ESTABLISHED 20664/java