This week, a few members of IS have decamped to Amsterdam to attend DrupalCon 2014, which brings together people involved with all aspects of Drupal for a week of talks, labs, Birds-of-a-Feather sessions and many hours of coding on Drupal 8. The focus of Tuesday’s opening Prenote, a DrupalCon fixture beautifully compered by JAM and Robert Douglass, was life-changing Drupal experiences. Whether or not DrupalCon changes your life, the breadth and depth of sessions and associated discussions to be found here this week is undeniably absorbing. For those of us who are currently working on the University’s new central Drupal CMS, DrupalCon provides a unique opportunity to both validate the approach we are taking with our development processes, coding standards and infrastructure, and to discover new modules, best practices and techniques which will benefit our new Drupal CMS.
After Dries’ Keynote, the conference kicked off in earnest. We have crowdsourced some of the highlights of our first day of sessions below. Many thanks to Aileen, Arthur, Riky, Tim, Andrew, Adrian and Stratos, for contributing their thoughts on sessions they attended; any errors, omissions or misinterpretations in their edited summaries are entirely mine. The overriding impression from discussing the sessions we have all attended is that Drupal is often at the bleeding edge of development tools and technologies by virtue of the commercial and community pressures in the Open Source environment. Drupal’s presence as a tool in our development portfolio both challenges our own best practices and introduces new, innovative means of developing quality applications which anticipate the needs of an increasingly diversified technological world.
Most of the sessions mentioned below, along with many more interesting talks, are recorded and available on the DrupalCon website, so if the summaries pique your interest, visit the DrupalCon site for more information!
Development Processes, Deployment and Infrastructure
Whilst the focus of this session was Drupal DevOps, Kris Buytaert’s talk applies more generally, covering the reasons why DevOps is not just a team or C.I. or Puppet. It is a cultural attitude that requires long-term thinking and a degree of co-operation from all teams. It’s not just about the lifetime of a particular project, but the lifetime of an application or service. The impact of putting off changes to deployment strategy is an increase in the “technical debt”; it only defers the issues, which then become support problems.
The proposed approach is mainly to expand best practices developed for writing code, such as version control and testing, down into the infrastructure and up into the monitoring tools. For example, the importance of repeatability is heavily emphasised; everything needs to versioned, and this includes infrastructure as well as artifacts. Using such DevOps techniques, we can better map and evaluate the impact of changes. The payoff should be safer, quicker sites or applications that do what people want, and developers get more feedback about why things went wrong (“It works on my machine.” is never an excuse).
This session covered the Drush Deploy plugin, which allows you to create drush configuration files to describe each of your environments so you can deploy consistently to all servers with one command, reducing human error.
It was interesting to contrast the approach of this plugin vs Capistrano or services like Bamboo or Jenkins. We use drush heavily in the automated deployment process for the University’s new central Drupal CMS, but Bamboo still handles the code deployment to ensure consitency across environments. We use Ant scripts to describe the pre- and post- code deployment tasks for Bamboo to carry out, whereas these are specified in a drush configuration file for the Deploy plugin and are limited to drush functions. It was interesting to compare their approach to handling code rollbacks with our plans for this, even though they do not explicitly include resinstating the database as part of that process. However, we are in a different position as normally we would roll back immediately upon failed deployment rather than some hours/days later when content could have changed significantly.
The importance of adopting an appropriate Git workflow to support deployment where there is a branch for live deployment which is always deployable was also discussed. Having separate live and dev branches is very important, and making use of separate branches for hotfixes and features is recommended: http://nvie.com/posts/a-successful-git-branching-model/.
WF Tools is a Continuous Delivery framework for Drupal sites, which was also shown off at DrupalCamp Scotland earlier this year. It is used to deploy code and configuration from git into a freshly spun-up Development virtual host environment. These code changes are all separate “jobs”, working from git branches; WF Tools allows you to tag these jobs with JIRA issues and trigger runs from a build tool such as Jenkins or Bamboo. After a successful run, the Development environment can be assigned to another user for peer review, and their GUI view shows a change log and git diffs for comments and approval or rejection. Any approved jobs continue along the Dev/Test/Staging/Live deployment pipeline.
WF Tools is an interesting solution which could wrap around our existing processes quite well, and the Pfizer implementation of the GUI looks good. However, as we’re already using a lot of Bamboo functionality, and we’re only developing one Drupal site centrally at the moment, it might not be perfect for our current requirements.
This talk by Josh Waihi covered ways in which a system could be created to fulfil a client’s needs using vertical and horizontal optimisation techniques, supplemented by profiling tools to help find and fix bottlenecks. ‘The building blocks of performance’ were broken down into different important categories, understand, build and optimise:
- understanding the resource requirements before an application is built;
- building the infrastructure in a suitable manner which balances complexity with performance;
- using logging and load testing to optimise the performance of the Drupal system.
Vertical optimisation involves hosting the components in the system – load balancer, http cache, web server with PHP and Drupal, other files, database and database cache – on different servers. In addition, assigning more resources to a site makes for easier location of bottlenecks. Once the vertical optimisation is done then horizontal optimisation can begin. In general the web server is duplicated many times, all referring to the same database and shared files. The main limiting factor here is cost. And finally, load testing and profiling tools help to ensure your system is using the correct amount of resources.
Unlike other sessions where new, cutting edge techniques and technologies were discussed which could be used in the future for Drupal, this session was beneficial because it vindicated the techniques that we are already using:
- scaling our infrastructure vertically before we scale it horizontally;
- the importance of using business rather than technical metrics for our performance and capacity testing;
- the different caching tiers we use to boost performance were aspects of our current practice validated by the expert from Acquia.
However, using PHP-FPM instead of mod_php and interesting diagnostic tools such as XHProf are interesting new ideas I’ll bring back to DevTech.
Front End Concerns
Yes, it did open with a pic of Dorothy et al! This was a very well attended session covering when and why you should/could use the different layout options available for Drupal. With clever use of a ‘Garfield’ rating system it was clear that all have pros and cons depending on use and complexity of the site.
Here’s the “Janet and John” bit…
- Context provides flexibility by extending core blocks to provide reusable blocks and regions, but Blocks are still hard to maintain and there is only one set of regions for layouts.
- Panels is powerful with a high level of granularity and a default variant which provides a failover structure. However, the codebase for Panels is heavy and the functionality provided may be overkill for easy layouts.
- Display Suite has a simpler UI and similar flexible layouts to Panels, but only Entity layouts are supported and because there is no structure across different layouts, things can easily become complicated.
The consensus seems to be that Display Suite ticks most boxes but each method has its merits. You may want to do some research to find the best for your particular project.
Although this session was titled for Drupal 8, both modules covered, picture and breakpoints, have been backported to Drupal 7 and we have recently implemented them to support the theme for the University’s new central Drupal CMS. At this point we only have four variants of the group banner image being served, one per breakpoint that has been defined in the theme.
The session also covered the module’s use of the sizes attribute to serve optimised versions of images according to the viewport width in steps between the breakpoints where required. This is not something we are currently implementing, but we will be in the coming weeks as we approach the initial distribution release of our new Drupal theme.
The Front End is moving forward faster than anything else in drupal. Display targets used to be 640 X 480 or 1024 X 760 for IE and Netscape using tables; easy! Now HTML5, CSS, JS, responsive design, etc. add significant complexity to Front End development, and these are not Drupal skills!
Frameworks come and go (e.g. 960 grid, blueprint, bootstrap) and we may catch up or even get ahead, but not for long when things change in the Front End so fast, and techniques fall in and out of fashion. However, Front End is A Thing, and is pushing Drupal forward in the post-responsive world. There are multiple frameworks for everything and too much scope to play design for design’s sake; the goal is truly device-independent design. It’s necessary to accept that you may be ahead, but only ever for a short time; however, there are many tools out there to support the drive to keep up with the rapidly changing world of Front End development.
Where you have to generate feeds from Drupal for high volume of requests, caching is sometimes not an option because requests from downstream clients include timestamps per second (to retrieve things that have changed since last request), or there are user filtered requests which aren’t likely to be repeated.
The approach taken here was to use an indexer module to generate de-normalised tables for the Drupal data on a MongoDB database, optimised for delivery, and put a fast Node.JS REST API in front. In their case-study, Drupal could be as slow as around 1 request per second where many records were being returned in one request, whereas NodeJS could handle 800-3000 requests per second. Response times dropped from up to a minute down to 80-150ms.
To support developing for many parallel/asynchronous requests in NodeJS, there are npm’s such as promises to help.
In order for a site to respond quickly enough for a user not to get bored waiting and give up to go elsewhere, it’s generally accepted that pages should be served in under a second. This can be challenging enough in complex Drupal pages, but there are added constraints to consider with mobile devices on slow networks. The DNS lookup, TCP connection, TLS handshake and HTTP request over 3G can come to 800-1000ms alone before you get to the time Drupal takes to serve the content and then the time taken by the client to paint the page. Given that many countries primarily use mobile devices now to access the internet, this is becoming more important.
When painting the page there are some blockers that delay rendering of the content. In particular, moving JS to the footer whenever possible and using async and defer was recommended. The magic module can help achieve this, and any critical JS can be made inline.
CSS can also be split out into what’s required for mobile devices and then LoadCSS can be used to load the remainder without blocking the initial rendering of the page. However, it is not always practical to achieve this.
With TCP, most of the time taken is not due to limitations in bandwidth, but latency on each round trip with initiating each request (handshakes, etc). CDNs can help by putting content closer to the client and allowing more parallel connections (because they are split across different domains), but this is expensive and will likely be blocked in China. Aggregating files and spriting so there are fewer files to load, even inlining very small assets; all of these things can help. Remove things you don’t need. The target is to get first response within 10 packets, 14.6k (RFC 6928), although it’s extremely difficult and very few sites are able to achieve this.
Preparing for SPDY/HTTP2.0 can conflict with some of the HTTP1.1 optimizations above, such as domain sharding and concatenation. Some work has started on supporting SPDY features, such as server push, in Drupal.
Project Management and Engagement
In this experimental session, the Vesa Palmu CEO of Wunderroot shared some of his lessons learned over the past 10 years using Agile on IT projects.
A core difficulty in getting agreement to use Agile can be the lack of trust between parties. The reasons for this mistrust can be a lack of understanding of what Agile actually means. How does using Agile translate for the business, in terms how this changes their activity during projects? Customers are unsure about the cost versus what will actually be delivered or are unwilling to engage by providing a product owner.
One of the key messages is that in order to sell Agile successfully, one needs to focus on selling the benefits of Agile. Some of the benefits:
- Collaborative development approach
- Testing development as it progresses
- Creating value faster the multiple deliveries
- Delivering better quality
- Making better decisions along the way
Mixing Agile and Waterfall should be avoided, the benefits of Agile cannot be realised when project teams are in two mind sets.
Equally not all projects sizes are suitable for Agile, as illustrated by the following slide:
In this session, speaker Jason Coghlan examined whether the self-organising team is a reality, especially in a commercial or public sector Drupal services environment, focusing on tips for research rather than specific examples of how to coach and build self-organising teams.
The session covered the need to differentiate between control and accountability: the former must be relinquished; the latter, especially individual accountability, is extremely important. In the context of self-organising teams, “Leaders are required, managers are optional”. George will take care of it! The conclusion is that whilst self-organising teams are not suitable for all projects, they are an ideal approach to technology-driven projects where a clear product or solution is delivered, focusing on value and return.
This was a really interesting session from Design Researcher, Dani Nordin, highlighting the challenges of integrating User Testing, UX & Design guidelines in an already established community such as the Drupal community. It’s very obvious that it is difficult to engage developers and designers to cooperate in such an open environment, so one thing I took from this session is that there is a strong need for integrating UX into the process of module contribution. Even though it might sound restrictive (especially for people contributing in their own time) it will pave the way for a more user friendly and intuitive Drupal UX. Drupal 8 might be a good opportunity to explore this as well.
Looking to the Future
Having attended DrupalCon last year as a relative Drupal newbie, and with most of our current internal development focus on Drupal 7, I approached Larry Garfield’s technical Drupal 8 overview session with mild trepidation. I needn’t have worried. This well structured introduction to Drupal 8 and its use of Symfony2 was very accessible. The code samples were clear and progressively illustrated each concept to give a pretty good high level overview of what to expect from Drupal 8. Coming from a Java/C++/OO background, I can say that it truly seems “Drupal8 is finally not weird”. Lots of familiar code even in the context of an unfamiliar framework!
The current Drupal 7 theming system involves a mixture of markup generated by modules and the theme primarily through theming hooks (functions). This means the front-end developer does not have full control of the markup (and CSS) that’s output.
By converting all the hook functions to Twig templates in the new Classy theme for Drupal 8, the themers can be in full control of all the output. The hope is that this will make frontend developers engage with Drupal in the future and make it easier for web design companies to engage with Drupal.
Changing menus and pagers was demonstrated, two of the most complex components to theme since they are currently buried in module functions, but in Classy these elements are themed in single twig template files which can easily be re-written to change CSS dependencies and markup without needing to know the inner workings of the core pager and menu functions.