It’s been under wraps for a while now but the team have been working really hard and at long last, our new Python development platform is nearing its July launch. In this post, I provide a preview of some of the features we will be delivering for Developers in the first release of our new Python web development platform.
Wasn’t programming so much simpler in the old days? When I started here at Edinburgh we were an Oracle shop: our databases were all Oracle and therefore our applications were also all written in Oracle (anyone remember Oracle Forms?) with reporting being provided by Oracle Reports. Modern software development has certainly moved on a pace since then! With the shift to web based applications, we chose ColdFusion as our development platform of choice as it made it much easier to connect simple html pages to a database. However, today’s customer demands are far more complex requiring developers to manage a large supporting cast of databases, languages and frameworks. It’s time to move on from ColdFusion.
A web languages review (APP004-25) identified Python as our preferred web development language to replace ColdFusion; and Django as the preferred web application framework to use with Python. For the last few months a small team have been working hard on a project (INF115) to develop an initial production-ready development platform for developers within Apps by the end of this financial year.
We were given a fixed budget to develop our platform, so the team has taken an Agile approach to ensure that the must-have features are available from day one. It is however recognised that this is just the beginning and that the platform will evolve over time as we gain more experience in what we need (there is a follow-up project already being planned for next year).
Our vision is to create an enterprise, production-quality, Python web development platform that is:
- Consistent, flexible, responsive, dynamic and disposable.
- In step with our Division’s automation strategy
- Suitable for use as a replacement for our existing ColdFusion web development platform
To achieve this vision, the team have developed a number of infrastructure and development features that will be available in Release #1 which I have provided more details below.
Release #1 Infrastructure Features:
1. Have a standard Python Infrastructure service available for applications:
Falling in line with our current practices, our Python infrastructure will be provisioned as follows:
- Three new load balanced environments (DEV TEST & LIVE)
- Six VM’s (two per environment – KB and AT)
2. Have a standard configuration of key server technologies:
The following lists the key elements within the server configuration:
- Operating System: CentOS 7
- Web Server: Apache 2.4
- Memory caching system: Memcached 1.4.15
- Automation Client: Puppet
- Language: Python 3.4 (for R#1 it will be installed and held at server level but Python services will be isolated from each other)
- Web Framework: Django 1.9 (per application)
- Web Server Gateway Interface: Gunicorn 19.4.5 (per application)
- Database Support: MySQL, Oracle and SQL Server
3. Have environment creation fully automated:
Automation will be fulfilled by a combination of two tools: Puppet and Bamboo.
Puppet will:
- Install & Config apache web server
- Install Python
- Install cosign, ssl certificates, oracle clients, tnsnames, memcached
- Create standard directory structure
Bamboo will:
- Create the application’s python virtual environment
- Install the application’s python dependencies. This will include Django and Gunicorn.
4. Have a mechanism to maintain configuration scripts in source control:
With infrastructure configuration as software, there is a need to standardise and define on configuration code management:
- Automation configuration scripts will be held in the ITI Git repository.
- The model for source code management to be utilised is based on recommendations from recent Puppet consultancy.
5. Have an understood and agreed DR, resilience, patching and EOL strategy:
We need to have an agreed understanding of how and when the various components of the platform infrastructure will be tested as well as upgraded. To achieve this, we will have in place an agreed and published:
- Patching and upgrade strategy
- Understanding of End of Life Characteristics
- Description of the disaster recovery & resilience characteristics of the platform
- Ongoing licensing and server costs
6. Have staff able to support the service:
Critical to the success of the project:
- Staff in both DevTec and TecMan will be trained in the skills required to support the Python Service including the use of Puppet and Bamboo where appropriate. As we grow the service, so too will we grow the number of staff in these teams who have the ability to support it.
Release #1 Development Capability:
Based on having this modern, consistent and flexible infrastructure as foundation, our web development capability will:
1. Have a standard local development environment:
All staff developing in Python will be able to use a standard common virtual environment with the following features:
- Representative of the corporate DEV/TEST/LIVE environments
- VM built with Vagrant to facilitate rapid, consistent, disposable working environments.
- Uses Virtual Box for the VM
- Uses Puppet to provision the software and configures the OS
- Supports Basic Auth as a Cosign alternative
- Supports application build and automated testing
- Comes with a default starter application built in.
2. Have Python Coding Guidelines:
Coding rules will be based largely on PEP8 (public domain) and the Google Python Style Guide (licensed under CC-By 3.0). Additionally topics covered will include:
- At a glance guidance on the python stack.
- Guidance on the deployment workflow
- Guidance on coding rules
- Common solutions guidance
- Guidance in how to connect to MySQL, SQL Server and Oracle using Django
- A training programme to introduce staff new to Python and Django
3. Have a Default Django Starter Application:
Building on the success of the ColdFusion Starter Application that has evolved over the last few years, it is our intention to do something similar with Python. The idea is that it will start as something quite simple but will be enhanced with each successive application we build. For the first release, our Starter Application will:
- Support automated testing (Unit testing, UI testing, Integration testing and code analysis/lint testing)
- Support for database connectivity (all 3 of our standard database platforms)
- Support cosign (basic auth)
- Support memcached
- Support automatic schema updates based on models defined by Django
- Support externalised config
- Include standard packages including Healthcheck
- Support scheduled tasks
- Be hosted in GitLab
4. Have a standard repository for code management:
In line with our version control strategy, we will ONLY be using the GIT version control system for source code control and GitLab as our web-based repository manager. The following will be provided:
- Instructions in how to set up GIT locally
- Instructions in how to access and use GitLab
5. Have a new model for source control:
In line with our version control strategy, we will standardise on the Gitflow model originally described by Vincent Driessen. In practice this means
- A new clearly defined and documented version control process that supports both project development and production working practices
- Fully integrated with automated deployment processes
- Easily identifiable production code
- Developers work on feature branches and merge into a development branch
6. Have a standard method of deployment using Bamboo:
Features include:
- Standard build plan template based on the Starter Application
- Externalised configuration
- Runs tests on DEV server
- Triggered automatically from git push
- Automatic feature branch plan creation
- Standard deployment plan template based on the Starter Application
- Builds application in python virtual environment
- Carries out any databse migrations (schema updates)
- Integrates with our Gitflow branching model
- Have instructions on best practice for build and deploy plans
- All application-level configuration to be stored separate to the code (i.e. not directly replaced into the code during deployment). This will allow us to adopt good deployment practice regarding configuration; our deployable artefacts are agnostic to the environment to which they are being deployed and configuration changes do not require version-changing software builds.
And what of the future?
It will take us some time to develop as rich a development capability as we currently have with ColdFusion and Java. After all, we have been using these for almost 20 years and Python is a completely new platform for us. I hope from the above that you can see that the Python web application development platform we have in mind for R#1 will provide a sound foundation on which to build. Each application we build will add to our knowledge and help us develop the platform further. Many of you may be thinking “so what happened to my feature request?”. Regrettably not all features requested by our Developer community have made it into R#1, there are a couple of notable omissions: Containerisation would be a massive step forward in realising our vision but in order to meet our go-live deadline this has not been possible to put in place. Likewise, baking EdGel into the platform (and in particular into the Starter App) was high on our list but again couldn’t be done within our deadlines. Happily, I can advise that both of those are candidates for next year’s project and I’d like to think will be part of the base platform before too long. And what of the other features? I hear you ask. Well these are still on our backlog and will be prioritised before being included in the next release of the platform. I also expect that up-coming projects will help to add to the basic platform feature set as we extend our capability
May your programming be awesome with our new python development platform 4 U.