Automation within Information Services Applications Team Delivers Efficiency Gains

Recent work within Information Services Applications has realised benefits by significantly reducing the amount of time required to release APIs (code used by developers to innovate and build further tools).

By utilising tools and technologies which partly automate the work involved in rolling out APIs the Development Services teams have managed to reduce the lead time prior to release from around five days to a matter of minutes.  The tools mean that the release process is significantly more straightforward and requires much less setting up and configuration to the point where releasing an API is almost a case of pushing a button.  The automation also ensures that the process is more reliable and less prone to errors.

Furthermore, multiple APIs can be released in parallel meaning that efficiency gains are further increased as more APIs are developed by the University.

The team demonstrated this new capability in a recent deployment of a microservices tool where several existing APIs were redeployed as part of the release.

Further Information on APIs Programme:

There is a need to transform the online experience of students, applicants, alumni, staff and other members of the University community to provide tailored information via personalised digital services.

To deliver these personalised digital services requires a way to plug into information from central systems and use it in new and innovative ways , like new websites, apps for mobile devices and what’s called ‘portlets’ for MyEd that operate like web pages within web pages.

Plugging into the information can be achieved by using Application Programming Interfaces (APIs). Here at the University we use open source tools for APIs – the type of code you can get from online communities where developers collaborate to make even better products and software.

While API technology has been around for a long time, its use beyond the integration of large, complex business systems has grown rapidly in recent years and with the proliferation of devices and mobile technology, API use in getting data from one place to another – from connecting business systems, to mobile devices and apps – has expanded exponentially.  That’s because APIs allow data to be accessed securely and consistently across multiple devices, a tremendously valuable resource for the University.

The University’s Enterprise APIs Programme (part of Digital Transformation) has been set up to deliver APIs to support projects like the User Centred Portal Pilot, the Service Excellence Programme and other digitalisation and transformation Programmes across the University.  In addition to enhancing central services such as MyEd, APIs will provide a consistent way for software developers across the University to build flexible systems at a lower cost, securely, and consistently across multiple systems.

Further Links:

Digital Transformation Website

API Technology Roadmap

 

From 0 to OAuth2 with Spring Boot

Following on from the previous post about documenting MicroServices with Swagger, we also wanted to have a uniform authorisation/authentication model for access to our services.

Our basic requirements were as follows:

  • They must support client-side authorisation (e.g. via Javascript calls in browsers)
  • They should have a single authorisation point
  • Session authorisation timeout must be controllable
  • They should be multi-domain ready (e.g. authentication from <user>@ed.ac.uk or <another-user>@another.ac.uk

After reviewing our options, OAuth2 was the obvious contender. We already have a web single sign-on solution called EASE which uses Cosign, so we need to use that for any web-based user authentication.

The remainder of this article shows how we went about setting up an OAuth2 service using Spring Boot.

Continue reading “From 0 to OAuth2 with Spring Boot”

Adding embedded Tomcat AJP support to a Spring Boot application

We currently use Apache with mod_jk in front of our Tomcat application servers. I was exploring how to use an embedded Tomcat while enabling an AJP connector. I wanted all the configuration to be property driven, allow the specification of HTTP/AJP ports, and allow the switching off of AJP for running the app locally.

Here’s how I went about it.

Application class

Firstly in the Spring Boot Application class you can tell the application on startup to use custom settings for the embedded Tomcat. Out of the box if you specify server.port as a property it will change the port of the standard http connector. I specified that property and some other values specific to AJP.

server.port=8082
tomcat.ajp.port=9090
tomcat.ajp.remoteauthentication=false
tomcat.ajp.enabled=true

They were then wired into the Application class using @Value annotations. The server.port is already handled by Spring Boot so I don’t have to do anything.

@Value("${tomcat.ajp.port}")
int ajpPort;

@Value("${tomcat.ajp.remoteauthentication}")
String remoteAuthentication;

@Value("${tomcat.ajp.enabled}")
boolean tomcatAjpEnabled;

Then I added in a specific Bean which defines the Tomcat settings, and whether or not to switch on AJP based on a property being set.

@Bean
public EmbeddedServletContainerFactory servletContainer() {

    TomcatEmbeddedServletContainerFactory tomcat = new TomcatEmbeddedServletContainerFactory();
    if (tomcatAjpEnabled)
    {
        Connector ajpConnector = new Connector("AJP/1.3");
        ajpConnector.setProtocol("AJP/1.3");
        ajpConnector.setPort(ajpPort);
        ajpConnector.setSecure(false);
        ajpConnector.setAllowTrace(false);
        ajpConnector.setScheme("http");
        tomcat.addAdditionalTomcatConnectors(ajpConnector);
    }

    return tomcat;
}

Then when I start up the application, I end up with an HTTP connector running on a specific port, and also optionally an AJP connector running on a specific port.

2015-06-24 08:40:09.514 INFO 93685 --- [ main] org.apache.coyote.ajp.AjpNioProtocol : Initializing ProtocolHandler ["ajp-nio-9090"]
2015-06-24 08:40:09.516 INFO 93685 --- [ main] org.apache.coyote.ajp.AjpNioProtocol : Starting ProtocolHandler ["ajp-nio-9090"]
2015-06-24 08:40:09.521 INFO 93685 --- [ main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8082 (http) 9090 (http)
2015-06-24 08:40:09.523 INFO 93685 --- [ main] uk.ac.ed.ca.centralauthms.Application : Started Application in 4.178 seconds (JVM running for 4.6)

 

Documenting Spring Boot Microservices with Swagger

It’s a fairly well known fact that many developers don’t like to write documentation, often muttering things like “the code is the documentation” in a half hearted manner suggesting even they don’t believe themselves. Recently I was looking to write a Microservice, so I wanted to also look at ways in which we could make nice easy to use documentation in a consistent manner.

Swagger

This led me to look at Swagger. It’s a way to produce elegant and powerful interactive documentation on your REST API without having to write pages of documentation. And when used with annotations and Spring Boot, truly the code *is* the documentation.

Spring Boot Example

I’m completely taken with Spring Boot already. It takes a lot of the complexity out of getting an initial application up and running, and allows you to add in features easily. In my application, I produced a REST API for getting identities out of LDAP, to support both a lookup on the logged in persons identity, and also other identities.

The POM file

We still use Maven (although it’s getting more and more tempting to switch to Gradle). My pom file contains the following dependencies:

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>    
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-rest</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency> 
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>com.jayway.jsonpath</groupId>
        <artifactId>json-path</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.ldap</groupId>
        <artifactId>spring-ldap</artifactId>
        <version>1.2.1</version>
        <type>jar</type>
    </dependency>
    <dependency>
        <groupId>com.mangofactory</groupId>
        <artifactId>swagger-springmvc</artifactId>
        <version>1.0.2</version>
        <type>jar</type>
    </dependency>
</dependencies>

Most of the dependencies are covering the Spring Boot side of things, REST support, web application and LDAP/JSON support. Note though the bottom dependency, which is including support for Swagger Spring MVC.

The Controller

The controller is fairly simple, it maps two URLs. The key thing of note is the @ApiOperation annotation, where we describe what the method does.

@RestController
public class UserLookupController {
    
    @Autowired
    LdapService ldapService;
    
    @ApiOperation(value="Get the currently logged in users details",notes="Uses the remote user logged in")
    @RequestMapping(value="/my",method=RequestMethod.GET)
    public @ResponseBody Person getMyDetails(HttpServletRequest request) throws ServletException
    {
        if (request.getRemoteUser()==null)
        {
            throw new ServletException("Remote user is null.");
        }
        return ldapService.getPerson(request.getRemoteUser());
    }

    @ApiOperation(value="Get a specific users details",notes="Requires uid of user to look up")
    @RequestMapping(value="/id/{uid}",method=RequestMethod.GET)
    public @ResponseBody Person getUserDetails(@PathVariable("uid") String uid)
    {
        return ldapService.getPerson(uid);
    }
    
}

Swagger Config

We also need a config class to tell Swagger what to do, and also provide high level documentation about what the Service is providing.

@Configuration
@EnableSwagger
@EnableAutoConfiguration
public class SwaggerConfig {
    
    private SpringSwaggerConfig springSwaggerConfig;
 
    @Autowired
    public void setSpringSwaggerConfig(SpringSwaggerConfig springSwaggerConfig) {
        this.springSwaggerConfig = springSwaggerConfig;
    }
    
    @Bean
    public SwaggerSpringMvcPlugin customImplementation() {
        return new SwaggerSpringMvcPlugin(this.springSwaggerConfig)
                //Root level documentation
                .apiInfo(new ApiInfo(
                        "Central Authorisation Service JSON API",
                        "This service provides a JSON representation of the LDAP identity data held in the Central Authorisation Service",
                        null,
                        null,
                        null,
                        null
                ))
                .useDefaultResponseMessages(false)
                //Map the specific URL patterns into Swagger
                .includePatterns("/id/.*","/my");
    }
    
}

The Application Class

The Application class is simple, and just hooks into our LDAP config:

@Configuration
@ComponentScan("uk.ac.ed.ca")
@EnableAutoConfiguration
public class Application {
    
    public static void main(String[] args)
    {
        SpringApplication.run(Application.class, args);
    }
    
    @Bean
    @ConfigurationProperties(prefix="ldap.contextSource")
    public LdapContextSource contextSource() {
        LdapContextSource contextSource = new LdapContextSource();
        return contextSource;
    }

    @Bean
    public LdapTemplate ldapTemplate(ContextSource contextSource) {
        return new LdapTemplate(contextSource);
    }
    
}


HTML UI

Finally, we add in a static set of HTML/CSS/JS to cover the API. You can get the static pages from https://github.com/swagger-api/swagger-ui. Put them in your project under (src/main/)resources/static and they’ll automatically get mapped into the application.

The end result

Swagger UI screenshot

The end result is a service which also provides the following documentation. As the documentation is interactive you can also try to call the id service and see what kind of response it gives. Very neat, very powerful, and a very easy way for us to provide API documentation!

Creating a ‘Bootiful’ new Visitor Registration System

The time has come to replace the venerable Visitor Registration System that has served the University now for quite some time. In June the team established for the COM011 project, destined to fulfil this task, ran user workshops and collected some 480 ‘user stories’ from interested parties around the University who use the incumbent system. User Stories are the cornerstone of the Agile methodology which has been chosen for the project.

Using Agile will allow the team to adapt to changing requirements and produce an end product that reflects the will of the users. What better way to complement this than by also introducing a new light weight, flexible, development tool that encourages rapid development and prototyping into IS apps technology stack? Spring Boot, (or just ‘Boot’), which was released in April, is the culmination of an effort by the huge Java/Spring community to prove the speed and ease with which Java applications can be created. This technology was showcased, to massive excitement, when it was shown that Boot could deliver an entire running web application in a tweet.

Boot has been used as the basis of the new Visitor Registration project; we now have a framework in our code repository that can be reused by anyone who wants to quickly setup a fully functional web application, with responsive front end, security enabled for various user roles, Rest endpoints, Soap endpoints and backend Oracle integration. And all of this functionality is fully unit and integration tested – in keeping with the goal of Agile that software quality should always be paramount. The new Visitor Registration System, using cutting edge technologies, will hopefully stand the test of time as well as its predecessor.

Oracle SOA vs Spring – SOAP Web Service throughput testing

We are soon going to embark on a major project to introduce enterprise notification handling at the University. Part of that will be the ability to handle a large number of messages in an efficient and robust manner. We already use Oracle SOA Suite here at the University, but wanted to test its throughput versus a lighter approach, that of Java and the Spring framework.

The scenarios

We chose four scenarios to test:

  • Basic assign, parameter passed in is passed out as response
  • DB Write , parameter passed in is written to Oracle Database
  • DB Read, parameter passed in is used to read value from Oracle Database
  • DB Read/Write, parameter passed in is written to Oracle Database, then read back out again

Testing constraints

We then applied the same constraints to both Oracle SOA and Java:

  • A connection pool must be used with the same settings (min 1, max 10 connections)
  • The same table structure/setup must be used with both technologies
  • We use the same back-end Oracle database
  • Testing would be done using a SOAP UI load test

For Oracle SOA, we set up a simple composite which tested the various features.

For Java Spring, we used Spring Boot, Spring Web Services, and Spring JPA.

The results

The results were as follows (total timings are rounded up to the nearest second):

Oracle SOA

500 calls 2000 calls 5000 calls
Assign 2 sec | 293 ms avg 6 sec | 504 ms avg 16 sec | 593 ms avg
Write 3 sec | 1284 ms avg 10 sec | 861 ms avg 29 sec | 1094 ms avg
Read 2 sec | 389 ms avg 9 sec | 838 ms avg 21 sec | 803 ms avg
Write Read 3 sec | 1038 ms avg 18 sec | 1644 ms avg 36 sec | 1403 ms avg

Java (Spring framework)

500 calls 2000 calls 5000 calls
Assign 1 sec | 101 ms avg 1 sec | 82 ms avg 2 sec | 72 ms avg
Write 1 sec | 112 ms avg 2 sec | 232 ms avg 5 sec | 203 ms avg
Read 1 sec | 73 ms avg 1 sec | 116 ms avg 3 sec | 116 ms avg
Write Read 1 sec | 271 ms avg 3 sec | 256 ms avg 6 sec | 234 ms avg

Conclusions

It is clear that the Java Spring solution is giving better throughput times,, and that is especially evident when we increase the load. However it would be unfair to use throughput times alone in looking at what Oracle SOA provides. It gives for example an “out of the box” message resilience and  support for automated message retry that would have to be coded in when using Java even with the benefit of Spring frameworks. However, Spring can provide a very useful high throughput entry point into Oracle SOA.

We want to benefit from the strengths of each of the technologies, so we are going to use the following:

  • Java Spring Web Services will be used as the initial entry point for creating/editing/deleting notification messages
  • The Java Spring WS will put a message in a queue for Oracle SOA
  • Oracle SOA will poll the queue for messages, then will apply the necessary business processing and rule logic for pushing notifications out
  • Oracle SOA will handle message retry in the event of processing failures
  • Java Spring Web Services will be used for pulling user notifications out for subscriber systems

As with most of the modern web, building a solution is about choosing the right set of technologies and not choosing a single technology approach. We’re confident now that we can introduce the necessary scale to handle a modern enterprise notifications system.