Posted on

IS at DrupalCon – Sessions Day 1

DrupalCon Barcelona 2015On Tuesday I posted some comments on the start of DrupalCon 2015 in Barcelona, where myself and a few colleagues are spending this week. The major strands this year are Docker, performance and scalability issues, the Symfony framework in a Drupal 8 context, and using headless Drupal with an alternative toolkit to provide the front-end.  So far it’s been really interesting to see how sessions on Symfony and Drupal 8 this year have progressed from last year’s DrupalCon in terms of the complexity of what is being covered. It’s also interesting to see how many attendees are using tools like Vagrant, Docker and Jenkins to take the pain out of manual configuration and deployment.  We have been using automated deployment tools for application code for some time now in IS Apps; configuration management and automatic creation of server environments is something that could further streamline our deployment workflow.

Our experiences of some of Tuesday’s DrupalCon sessions are outlined below. Thanks to Riky, Tim, Andrew, Adrian and Chris for contributing their thoughts on sessions they attended; any errors, omissions or misinterpretations in their edited notes are entirely mine. Most of the sessions mentioned below, along with many more interesting talks, are recorded and available on the DrupalCon YouTube channel.

Symfony for Dupal Developers

Drupal 8 makes use of many Symfony components, and this session covered the differences between the two frameworks to help decide which to use for projects. Drupal uses about a third of the Symfony components and you don’t need to know Symfony to develop Drupal.

Visualisation of Symfony components in Drupal 8
Visualisation of Symfony components in Drupal 8

Some differences are more obvious than others, such as while the application entry point in Drupal 8 is index.php, in Symfony it’s web/app.php and web/app_dev.php. These two entry points arise from the fact that Symfony enforces a programmatic toggle between Development and Production modes: you push generated code to production heads; you do not compile on production boxes.

Drupal uses the kernel a little differently and imposes stricter coding standards. For instance Drupal always uses the View event whereas this is discouraged in Symfony. Drupal 8 coding standards are quite strict in prescribing when to use YAML or Annotations for configuration. With Symfony configuration you are free to use PHP, XML, YAML or Annotations, although best practice is to pick one and stick to it.

Coming from Drupal 7, one of the fundamental differences in Symfony is that there are no functions; it’s all methods, with only a few static functions available. All meaningful logic in Symfony is in services, which are stateless objects.

Paths and routing are handled differently in Symfony and Drupal 8. In Drupal 8, you can only use module routing.yml files or events, not annotations, XML or PHP. Also, you don’t have path nesting or slugs in Drupal 8.

While both frameworks now use twig in the theming layer, when working in Drupal 8 you work with multiple twig files for each element, mirroring templating system in Drupal 7. Symfony templating uses only one file which extend twig files and override blocks defined in the parent twig file(s). Also Drupal always requires a render array; you don’t return rendered output directly in controllers.

Drupal has multiple APIs for storing content data, Symfony doesn’t have anything: you use Doctrine (or something else). Doctrine can only store primitive data and being a stand-alone PHP project has different event listeners from Symfony.

This talk highlights the fact that whilst Drupal 8 is using Symfony components, these are very much used with a Drupal flavour.  Comprehending Drupal 8 and how it uses Symfony requires an understanding of what has gone before in previous versions of Drupal as well as knowledge of Symfony concepts and techniques.

Estimation: from waterfall to agile

In this session, Danish Digital Agency Adapt covered the background to their move from a Waterfall project methodology to an Agile approach, describing their experience of that transition and specifically targeting how they tackled estimation. Starting from a position where they were losing money on 50% of their projects due to inaccurate estimations and the need for their customers to prioritise scope, they adopted an Agile “light” methodology, only to quickly realise that with Agile it needs to be all or nothing.

The presentation continued with an outline of the process used to define user stories, use of planning poker, and the need for clearly defined roles and just-in-time management. The key to relative estimation, i.e. measuring feature size in story points not hours, is to involve all of the project team members in the process. Time boxed planning poker sessions, with limited discussion, allowed the knowledge level across the project team to be increased leading to more accurate forecasting. The iteration or sprint velocity was then calculated by breaking down the tasks (from development to testing) into the hours expected to take to complete. If a story took longer to complete than expected, the user story points associated with that user story did not change but this information was then used to forecast how much could be delivered within the project; this allowed the customer to prioritise what remained in the backlog.

The two main benefits derived from this change in approach to delivering projects was that, firstly, they were now in a position to keep to fixed budgets and secondly, knowledge was gained during the estimation process. However there were also negatives, primarily concerned with small projects where this approach has proved difficult to implement and there is often not sufficient time for people to become accustomed to this approach.

My main takeaways from this session are “Customise processes along the way” and “Know what you don’t know and accept that!”. Projects and customers (business units) are different and there is a need to refine and adapt the Agile process where experience shows that refinement is required. The more experience the project team have using Agile, the easier this process becomes. At the outset of a project, especially with larger scale projects, there are inevitably unknowns; it’s crucial to identify and accept these unknowns. As a project progresses, unknowns should become knowns and these can be requirements, risks or opportunities.

Configuration Deployment Best Practices in Drupal 8

This was an engaging (if somewhat caffine fueled!) session. Drupal currently has a problem with the separation of configuration from content. Code is developed in DEV then pushed out to TEST, STAGING, then LIVE. Content, however, is created on LIVE and the other environments are refreshed from it. “Content” can be thought of as the database, which has tables for both the created content and the configuration, but as we want Drupal configuration to be developed in DEV and tested through the environments, what is really needed is for configuration to be treated like code.

Drupal 7 does not have a good way to deal with this, although the Features contrib module can be used, as we are doing in our own Drupal CMS, to make our configuration fully deployable through our automated deployment process.  In Drupal 8, configuration management on a managed workflow seems to offer many benefits over the older version.  Configuration is totally separated out into YAML files. These files can then be committed into Version Control System like code, providing accountability and the ability to audit configuration changes.  All of this makes Continuous Integration much easier, and may make it configuration rollback possible. The YAML configuration files are imported into the database for added performance, allowing a more robust method of configuration over hook_update_n, or using the current feature module in ways for which it wasn’t strictly designed; drush has also been extended to work with this functionality.

This was an interesting talk which made some very useful suggestions for how configuration should be managed between environments when using Drupal 8, as well as how exceptions can be handled.  This area should be explored further when planning for the Drupal 8 upgrade; we should look into replacing our reliance on the features module for exported configuration with the equivalent in YAML configuration.

Altering, Extending and Enhancing Drupal

One of the many sessions on what is new in Drupal 8 versus Drupal 7, this was an interesting talk giving a high level summary of the mechanisms for customising and extending Drupal 8.  Topics covered were plugins, services, events and hooks.  In Drupal 8, the principle for plugins is “Learn once, apply everywhere”, moving away from the inconsistencies between modules and how they are used by having plugin classes implement an interface, so there is a common approach.  Services in Drupal 8 are very well decoupled and can easily be swapped out, for example for testing purposes.  Event handling allows modules to react to Drupal application actions and/or conditions in a standard manner that is common in OOP rather than using hooks to react when something happens. Hooks still exist in Drupal 8, but are primarily for modifying metadata which has been gathered by other means, or to alter forms.

One interesting question that was raised at this session was how to determine whether what is needed to implement a feature is a plugin or a service.  A useful way to decide this is to think about a service as something that you would usually only have one of for any Drupal instance, for example, a caching service.

This succinct outline of mechanisms for extending Drupal 8 highlighted the fact that these mechanisms are less specific to Drupal than in previous versions.  Hooks remain, but whereas previously they would have been used for everything, Drupal 8 leverages Symfony to add new ways of doing things that help improve code structure and re-usability.  The patterns and techniques are familiar from other contexts where OO is used, which brings a consistency and helps developers to better avoid the unnecessary, and at times frustrating and confusing, proliferation of different ways of doing things.  These approaches also help with documentation – creating common patterns for implementing new modules means that Drupal developers are not so much at the mercy of how good the documentation for a particular module is.  All of these factors should improve the development experience in Drupal 8.

Fundamentals of Front-End Ops

As more application logic is being handled client-side, Front-end Ops is a response to the proliferation of front-end tools and frameworks. This session was not about the Drupal framework, but instead looked at tools for automating front-end development tasks, managing dependencies and generating scaffolding.

For scaffolding tasks, Yeoman was demonstrated. Yeoman recommends that your workflow involves Bower for dependency management and Grunt or Gulp for task automation. The session covered installing and using Yeoman, Bower, Grunt and Gulp as well as comparing the merits of Grunt and Gulp.

The talk covered the BBC’s Wraith, which leverages PhantomJS and SlimerJS to provide visual diffs of screenshots between two environments, as well as other visual regression tools, namely Huxley and PhantomCSS. There was also a discussion of the test rendering engines available: PhantomJS, SlimerJS, CasperJS and GhostLab.

Finally some of the front-end debugging tools available were covered, including Chrome DevTools Remote Debugging which allows you to connect a mobile or tablet to your desktop machine and use the development tools on the desktop browser to inspect the DOM, etc, on the mobile device.

Docker powered team and deployment

This session gave a basic overview of the features and benefits of Docker and of infrastructure as code. It focused on Bowline as an easy way to get started using Drupal on Docker, offering great flexibility and minimum requirements via a suite of BASH scripts that can be included in your Drupal code repositories.  The scripts add a method for container installation, and can also be used to “hoist” (start up) other containers at the same time for local development, such as a Behat container for testing. The aim is to simplify and facilitate the configuration and linking of containers for Drupal setups. This definitely looks like the way forward for the provisioning and support, at the very least, of our Dev environments. Taking the paradigm further than just sandboxes for development, the ability to move docker containers through an environment pipeline from Dev to Test and into Production was very interesting and worth further investigation.

While much of the session was used to introduce Docker, it was interesting to see the various methods, like Bowline, that are being used to enable developers to work with Docker containers as the next step on from Vagrant. It was also interesting that almost everyone in the room was using Vagrant and had Jenkins as their CI server, and about half were developing on Linux – none of which are currently true for Development Services in general.

Using Docker (or something similar) could be beneficial for us, mainly because it would give us readily available, standard and up-to-date environments for development or support use.
We do however already achieve something quite similar with virtual machine deployment environments on local machines.

Docker in the DrupalCI test infrastructure

As seems to be the current standard, this Docker session also started with the obligatory shipping metaphor-laden introduction to containerisation which I won’t repeat (see https://www.docker.com/whatisdocker for the official version). DrupalCI is a project to make it easier for developers to do local testing, and to enable testing of different combinations of PHP versions and database backends. It works by having a base Docker image, from which a PHP base image and a database base image are created. With only small adjustments containers with variations of these, like PHP 5.5 and PHP 5.6 environments, can be produced.

The talk also raised important security points which arise from using containerisation: one needs to think about running a private image registry (an ‘IS Apps Docker Hub’); SELinux should be used to reduce the possibility for malicious containers to break out into the host; and the responsibility for updating the images, and the running containers, is also vital.

While it is frustrating to keep seeing the same Docker introductions, it is interesting that so many sessions this year have been dedicated to the technology showing it is being used by many in the community.

Solving Drupal Performance and Scalability Issues

In this session, Tine Sørensen drew on her years of experience in optimising performance for Drupal sites and troubleshooting scalability issues to highlight some of the techniques that can be used to diagnose issues, pick off the ‘low hanging fruit’ and achieve great improvements without great expense.  There is no real value in spending 6 months rewriting some aspect of a module that is not performant if there is only a very small improvement at the end of that time.  The recommended starting point when a performance issue is found is to collect data from the site, analyse it, choose where to apply effort – prioritising where ratio of effort to gains is the greatest, then repeat the process until performance is satisfactory.

The core message of this talk was the importance of collecting the data to demonstrate what is actually happening, and the huge gains in efficiency when pinpointing performance issues that can be achieved simply by using monitoring tools.  Tine focused on New Relic as that is where her experience lies.  As we found when we had external consultancy for our Drupal CMS project, New Relic is the tool of choice for monitoring Drupal and diagnosing performance issues; when using the Pro version, the tool has even more functionality such as granting XHProf-like profiling. Like ourselves in Apps, 50% of the audience were using New Relic for Drupal monitoring. Monitoring tools like New Relic can give an extremely useful picture of performance bottlenecks.  For example, using the Pro version, it’s possible to see a list of PHP functions being called listed in order of execution time; this shows up any particular function that might be causing problems.  A developer can then go straight to the function in question and analyse the code to identify any issues. We have used this technique ourselves whilst developing our Drupal CMS and it certainly can save hours, if not days, of time spent drilling down through XHProf reports.

Examples of quick wins were also provided, such as switching from GD to ImageMagick, disabling Views UI, tuning caching and tuning queries that are performing poorly.  Of these, the GD/ImageMagick switch is the only one that was unfamiliar.  It was particularly interesting to see that the server settings and tuning recommendations correspond to those we already use internally; it is useful to have our current approach validated by the speaker who is an experienced consultant.

For me, the main takeaways from this talk were that it is absolutely essential to understand what is happening on the servers when performance is poor, and that the less time you have to spend on collecting that data, the more quickly and efficiently you can resolve the problem.  It’s also important not to simply throw hardware at a performance issue; that can resolve things in the short term, but ultimately it only masks the problem, particularly if that problem is not fully understood, with the risk that it could resurface in a more damaging way in future.

Drupal 8 Plugin Deep Dive

This session covered the new Plugin system in Drupal 8 which replaces the hook_info() and hook_info_alter() pattern. This is one of the areas where Drupal differs from other CMS’s and frameworks; Symfony bundles are hard-coded whereas Drupal plugins are configurable and discoverable.

Plugins do away with many of the Drupal 7 hooks, in favour of the new PluginManagerInterface model and examples of these in core were covered.

Plugin autoloading, dependency injection, service containers and annotations were all covered before going on to demonstrations of building your own plugins.

The session was very technical, covering low-level code samples in detail.  It was a good companion piece to the earlier session on altering and extending Drupal 8.

Drupal 8 The Backend of Frontend

The Drupal 8 theming layer has been re-written and now uses Twig as its template engine. Theme functions are pretty much done away with now and replaced with Twig templates. The theme process hooks are also done away with now that Twig is used. You still have the two levels of template_preprocess and hook_preprocess hooks, but now everything they return in the variables array needs to be a render array.

The session also covered writing Twig templates and how to extend Twig with your own custom functions/filters in Drupal.

Drupal enables Twig’s auto-escaping; the security issues around this, and how to mark strings as safe so they are not escaped, were also covered.

Symfony2: The journey from the request to the response

This session was presented by the Head of Sensio Labs (creators of Symfony), Sarah Khalil. It covered the components involved in processing a HTTP request starting with the front controller (app.php in Symfony, index.php in Drupal) passing the Symfony HttpFoundation request through HttpKernel.

The Routing component YAML files are named and located differently in Drupal, but in essence the process is unchanged for the router passing the request through the controller and on to returning an HttpFoundation response.

Symfony’s Event Dispatcher component was covered in some detail with examples of event listeners and subscribers, the differences between them, and examples of how Drupal core implements these.

After listing the seven kernel events that you should know together with the Symfony components used that Drupal uses, the Dependency Injection component’s three key concepts of service container, services and parameters were detailed.

Finally a quick look at the concepts in Twig, with the caveat that Drupal does not use all the features available in Twig.

Caching at the Edge: CDNs for everyone

This session, although fairly technical, was well presented by clearly very knowledgeable speakers and touched on upcoming technology such as service-worker (client-side caching), ESI and Big Pipe.

Content Delivery Networks are external, multi-sited hosts which enable content to be delivered with lower latency from caches local to the user. They can be used just for static assets (JS, CSS, images), and also for dynamic content, although the latter is far more complicated. As proven by our infrastructure with Varnish, delivering anonymous content from a cache is fairly simple as it rarely changes; this advanced session focused mainly on caching authenticated content with CDNs.

As we have seen in earlier DrupalCon sessions, Drupal 7 is limited in what it can offer in this conext; it can only use “max-age” caching, or scripted/manual purging of stale content from CDNs. Drupal 8 looks to have taken a big step forward in terms of the effects of caching on website performance, providing three main cache invalidation techniques:

  • cache tags, which show where data dependencies for caching exist;
  • cache contexts, which give the context of dependencies for requests;
  • cache max-age, as found in Drupal 7, which give the time dependencies of what is cached.

Together these enable placeholders and auto-placeholdering, whereby Drupal 8 “knows” what content makes up the page and so knows which can be retrieved from a CDN and which needs to be dynamically requested from Drupal.

Using this mechanism, it’s possible to perform Edge Side Includes with authentication being cached per session. This could be used with Varnish rather than a CDN, enabling us to cache a lot of our HTTPS traffic, which is where we are currently experiencing our worst performance. The demonstration of the response time improvements from caching user specific content was very impressive. While complicated, I strongly recommend that using Varnish for authenticated users is investigated when upgrading to Drupal 8. This would perhaps be one of the main reasons for considering upgrading when it is finally released.

BigPipe ( https://github.com/bigpipe/bigpipe ), a node.js framework which can be used with Drupal 8, was also demonstrated. It breaks up pages into smaller chunks so they can load asynchronously, decreasing the time for the first elements of page content to load.

Cut the crap. Practical tips and real world examples for removing waste from your development process.

This session dealt with a rather different approach to managing projects, albeit there is a caveat that this covers smaller pieces of work. Basically anything at all that can be deemed as not adding tangible benefit should be removed from the project. We all know that getting decision makers to make decisions quickly can be challenging. So the first thing is to identify your decision maker. In the example given the presenter, Jason Mark, relayed a project that was delivered within 4 weeks. The first week was used for requirements, design and templates and starting the build. The remaining three weeks involved working with the client/partner to test and refine what was delivered.

If this fast track approach is to work, it needs several things to fit, the main four being:

  1. Decision maker;
  2. No hidden stakeholders;
  3. The right people – low ego and ability to be flexible are key characteristics;
  4. Technology needs must fit and the requirement needs must fit the technology.

If there are blockers, especially people, find ways to turn them into champions by using positive creative language.

The top takeaways from this session are twofold. Firstly, take a step back and look at the project in terms of the 4 points above. Can these points be answered positively? If not, what can be done to turn this around and make the process fit? Secondly, because this is a fast approach to turning a piece of work around, the planning will not be complete before the work starts. This makes change inevitable, which needs to be embraced at the outset and not seen as something negative. To repeat a quote attributed to Buddha,

“Change is never painful, only the resistance to change is painful”

Above all, only focus on things that add value! As Project Manager you can ask this everyday!

Headful Drupal

This session was about headless Drupal. Perhaps the only advantage for us with this is the security advantages of removing some front-end admin components. Alternative means of achieving what we require do however seem quite time consuming and careful consideration  of the particular development context would be needed before going down the headless Drupal route.

Visualizing Logfiles with ELK Stack

This was an interesting session that would have benefited from demonstrating a concrete example. The material presented was somewhat abstract and the potential for an escalation in complexity and the associated infrastructure requirements of such a system seemed a little daunting without something to tie it to a real world example. However, a centralised logging system would be a great asset to IS Apps even beyond the context of Drupal and EdWeb. An ELK stack, in some incarnation, should be a serious consideration.

Migrating a running service (Mollom) to AWS without service interruptions and reduce costs

A disappointing session that was difficult to generalise from, and only seemed relevant in terms of the specific service that the speakers were trying to move into the Amazon Cloud. The Mollom spam protection system seemed far removed from anything managed by IS Apps. It was claimed that the switch to AWS reduced the number of alerts that were received by the Ops team, but no other metrics were presented in terms of savings to the business as a whole. The takeaway from this session seemed simply to be that you need to think differently about services in AWS due to the ephemeral nature of the server instances, these being  discarded and new ones spun up any time configuration changed or applications crashed.

Lightning talks

At the Lightning Talks session, there were 3 commercial companies pitching ideas and providing insight into their products and services.

PhpStorm for Drupal Development

This is a good looking and powerful tool which could be useful for us when working with PHP in Drupal. It has Drupal-specific functionality such as being able to track hooks across your codebase. It also incorporates Git functionality and provides a graphical means for diffing files as well as back tracking changes. In general it seems quite neatly put together and well thought out.

Interoute Virtual Data Centre – Proven to be the fastest cloud

One of the main advantages as part of their pitch was hosting in remote locations to assist with latency caused by long distance connections. I cannot see that this is applicable for us.

How to setup Nginx/Varnish Full Page Caching for Drupal

The main point taken from this session is that reverse proxy (Varnish) full page caching is not possible in Drupal 7, but will be in Drupal 8.

 

Posted on

IS at DrupalCon – Hola Barcelona!

Driesnote keynote talk at DrupalCon Barcelona 2015It’s 7.30 in the morning in our hotel and I have already overloaded my breakfast plate with so much that I need help to finish what I can’t eat.  It must be DrupalCon!

This week myself and some colleagues in IS have escaped from autumnal Edinburgh to balmy Barcelona for this year’s European Drupal conference, DrupalCon 2015.  For three days we are in the midst of a whirlwind of sessions on all aspects of working with Drupal, many of which touch on issues and experiences that are common to all web developers and site owners.

Our journey to create a new Drupal CMS for the University’s main website began before DrupalCon in Prague in 2013. Since last year’s DrupalCon in Amsterdam our new Drupal CMS has moved from its embryonic state following the initial MVP release in 2014 to a production system with a name, EdWeb, and upwards of 140 sites have so far been migrated across to the new system.  You can read more about our Drupal journey at the University Website Programme site in EdWeb.

Soon we embark on the exciting process of planning the next set of features to add to our shiny new responsive website and I’m sure that as before we will find much to inspire us at DrupalCon.  It’s also a fantastic opportunity to explore how other organisations are managing scalability and performance when running an Enterprise level CMS.  Judging by the number of cloud hosting companies in the exhibition hall this year, the answer to that problem for many organisations is to let someone else feel at least some of the pain for you!

As we did last year, over the next few days we will be gathering together short summaries of some of the sessions we attend here at DrupalCon, with our own reflections on what we see and hear. This year we are fortunate enough to have brought a group of colleagues who have played a range of roles in the creation of EdWeb, from development through to project management and production staff and there is something at DrupalCon for everyone!

But before we share any session notes, the Prenote and Keynote sessions from the first day of DrupalCon have already given food for thought.

At the centre of the yesterday’s 8am Prenote, which is always an entertaining way to start DrupalCon, was the notion of dreaming the impossible dream, expressed charmingly in song by Adam Juran, in this case the seemingly impossible dream of getting Drupal 8 released.  Having been involved with the huge undertaking of building EdWeb from the outset, that sentiment was very familiar!  Throughout the ups and downs of such a large scale and complex Agile project, it’s been important to keep our end goal in sight, and to believe that what we are trying to achieve with EdWeb is both possible and necessary.  Now we have our production CMS, and it was announced at DrupalCon today that Release Candidate 1 for Drupal 8 will ship on 7th October 2015.  Those impossible dreams can be realised!

The morning continued with the opening Keynote by Drupal founder Dries Buytaert, and once again we see parallels between the process of getting Drupal 8 to a shippable release and our own experience of building a large scale CMS.  This year, Dries’ theme was “We need to talk about that”, covering some of the uncomfortable questions in the Drupal community.  Two aspects of his talk in particular struck a chord as they relate to problems that we have also had to solve.

In talking about the extended timescale that’s been necessary to get Drupal 8 ready, Dries proposed an alternative model for Drupal’s code management, a branching strategy rather than the current approach of having all development on the trunk.  This would embrace the difficult reality of getting all features ready for a release; instead, what goes into the release is only what is ready.  During the course of our own CMS development, we have had to solve exactly that problem, releasing feature bundles into the production system at the end of each development iteration without releasing code that is not ready to ship.  Our initial workflow was to do development on the trunk, but we quickly realised that this creates problems, particularly as we run our migration project in parallel with ongoing CMS development work.  We also had to allow for the release of a security patch for Drupal itself, or for a contrib module we are using, which would need to take precedence over any feature development and go into production sooner, without including features that are not ready.  To solve those problems in a way that would support our automated deployment process, we moved to a workflow that turned out to be a variation on Gitflow Workflow and this has served us extremely well, allowing us to manage parallel development work and release only code that is production-ready into our live system.  It was very interesting to hear how Dries has come to the same conclusion as us – that for large-scale development work involving multiple developers and many features with a complex life cycle, feature branches are the way to go.  The detail of our own approach is a topic for a future post!

Another theme of Dries’ talk was usability in Drupal and how features that improve the experience for editors can be sacrificed in favour of features that add new functionality.  In creating EdWeb, we have involved users from the outset, whether via quick paper prototyping sessions to determine the best approach for a particular interface detail, or by running sessions where all members of our team, including developers, were able to watch editors actually use EdWeb so we could pinpoint usability problems.  That process has contributed hugely to the usability of EdWeb, but it’s undoubtedly true that when the pressure is on to develop new features, it’s very difficult to hold to the discipline of prioritising usability.  That problem is not unique to Drupal development; it’s a perennial problem that is particularly troublesome for Agile projects.

So there it is – before we even got to 10am on the first morning of DrupalCon, there was already a lot to think about!  Watch this space for daily posts with session notes from our team.  And if you want to see what we’re so excited about, check out the DrupalCon YouTube channel for session recordings!

Posted on

IS at DrupalCon Amsterdam – Day 2

Yesterday I posted some session summaries from the first full day of DrupalCon 2014 in Amsterdam, where a few members of IS are spending this week. DrupalCon Day 2 began on Wednesday with a Keynote from Cory Doctorow, a thought-provoking talk on freedom and the internet, a subject about which some of us had previously heard him give at the IT Futures Conference in 2013, and one which has significance well beyond the context of Drupal for anyone who uses the web. The broader relevance of Cory’s speech is reflected in many of the sessions here at DrupalCon; topics such as automated testing or developments in HTML and CSS are of interest to any web developer, not just those of us who work with Drupal.  In particular, the very strong DevOps strand at this conference contains much that we can learn from and apply to all areas of our work, not just Drupal, whether we are developing new tools or managing services.

Our experiences of some of Wednesday’s DrupalCon sessions are outlined below.  Once again, thanks to Aileen, Arthur, Riky, Tim, Andrew, Adrian and Stratos for contributing their thoughts on sessions they attended; any errors, omissions or misinterpretations in their edited summaries are entirely mine. Most of the sessions mentioned below, along with many more interesting talks, are recorded and available on the DrupalCon website, so if the summaries pique your interest, visit the DrupalCon site for more information!

Development Processes, Deployment and Infrastructure

How Cultivating a DevOps Culture will Raise your Team to the Next Level

The main idea explored in this session was how to create a single DevOps team rather than have separate teams. DevOps is a Movement, a better way to work and collaborate. Rather than make a new team, the current teams should work together with fewer walls between them. The responsibility for adding new features and keeping the site up can then be shared, but this does mean that information needs to be shared between the teams to enable meaningful discussion.

The session was very dense and covered many aspects of implementing a DevOps culture, including:

  • common access to monitoring tools and logging in all environments;
  • the importance of consistency between environments and how automation can help with this;
  • the need for version control of anything that matters – if something is worth changing, it is worth versioning;
  • communication of process and results throughout the project life-cycle;
  • infrastructure as code, which is a big change but opens up many opportunities to improve the repeatability of tasks and general stability;
  • automated testing, including synchronisation of data between environments.

The framework changes discussed here are an extension of the road we are already on in IS Apps, but the session raised many suggestions and ideas that could usefully influence the direction we take.

Using Open Source Logging and Monitoring Tools

Our current Drupal infrastructure is configured for logging in the same way as the rest of our infrastructure – in a very simple, default manner. Apache access and error logs and MySQL slow query logs in the default locations, but not much else. Varnish currently doesn’t log to disk at all as its output is too vast to search. If we are having an issue with Apache on an environment, this could mean manually searching through log files on four different servers.

Monitoring isn’t setup by default by DevTech for our Linux hosts – we would use Dell Spotlight to diagnose issues, but it isn’t something which runs all the time. IS Apps is often unaware that there is an issue until it is reported.

We are able to solve these issue by using some form of logging host. This could be running a suite of tools, such as the ‘ELK stack’, which comprises Elasticsearch, Logstash and Kibana.

By using log shipping, we can copy syslog files and other log files from our servers to our log host. Logstash can then filter these logs from their various formats to a standard type, which Elasticsearch, a Java tool based on the Lucene search engine, can then search through. This resulting aggregated data can then be displayed using the Kibana dashboard.

We can also use these log “monitors” to create metrics. Logstash can write out to Graphite which can act as a counter of this data. Grafana acts as a dashboard for Graphite. As well as data from the logs, collectd can also populate Graphite with system data, such as CPU and memory usage. A combination of these three tools could potentially replace Spotlight for some tasks.

We need this. Now. I strongly believe that our current logging and monitoring is insufficient, and while all of this is applicable to any service that we run, our vast new Drupal infrastructure particularly shows the weaknesses in our current practices. One of the 5 core DevOps “CLAMS” values is Measurement, and I think that an enhanced logging and monitoring system will greatly improve the support and diagnosis of services for both Development and Production Services.

Drupal in the HipHop Virtual Machine

When it comes to improving Drupal performance, there are three different areas to focus on. The front end is important as render times will always affect how quickly content is displayed. Data and IO at the back end is also a fundamental part; poor SQL queries for example are a major cause of non-linear performance degradation.

While caching will greatly increase page load times, for dynamic content which can’t be cached, the runtime is a part of the system which can be tuned. The original version of HipHop compiled PHP sites in their entirety to a C binary. The performance was very good, but it took about an hour to compile a Drupal 7 site and it resulted in a 1 Gb binary file. To rectify this, Java Virtual Machine-like Just In Time compilation techniques were introduced for HipHop Virtual Machine (HHVM) which runs as a FastCGI module.

Performance testing has shown that PHP 5.5 with OPcache is about 15% faster than PHP 5.3 with APC, which is what we are currently using, and HHVM 3.1 has about the same performance improvement again over PHP 5.5. However, despite the faster page load times, HHVM might not be perfect for our use. It compiles pages with Hack, which uses strong typing, rather than PHP, and it doesn’t support all elements of the PHP language. It is still very new and documentation isn’t great, but this session demonstrates that it is worth thinking about alternatives to the default PHP that is packaged for our Linux distribution. There are also other PHP execution engines, PHPng (which PHP 7 will be based off of), HippyVM and Recki-CT.

In IS Apps, we may want to start thinking about using the Red Hat Software Collections repository to get access to a supported, but newer, and therefore potentially more performant, version of PHP.

Content Staging in Drupal 8

This technical session provided a very nice overview of content staging models and how these can be implemented in Drupal 8. There was a presentation of core and contrib modules used, as well as example code. The process runs by comparing revisions and their changes using hashcodes and then choosing whether to push to the target websites.

What I would take from this session is that it will be feasible to build content staging in Drupal 8 using several workflows, from simple Staging to Production, up to multiple editorial sandboxes to production or a central editorial hub to multiple production sites. One understandable caveat is that the source and target nodes must share the same fields otherwise only the source fields will be updated, but this can be addressed with proper content strategy management.

Whilst this session focused on Drupal 8, the concepts and approach discussed are of interest to us as we explore how to replicate content in different environments, for example between Live and Training, in the University’s new central Drupal CMS.

Testing

Automated Frontend Testing

This session explored three aspects of automated testing: functional testing, performance testing and CSS regression testing.

From the perspective of developing the University’s new central Drupal CMS, there were a number of things to take away from this session.

In the area of functional testing, we are using Selenium WebDriver test suites written in Java to carry out integration tests via Bamboo as part of the automated deployment process.  Whilst Selenium tests have served us well to a point, we have encountered some issues when dealing with Javascript heavy functionality.  CasperJS, which uses the PhantomJS headless WebKit and allows scripted actions to be tested using an accessible syntax very similar to jQuery, could be a good alternative tool for us.  In addition to providing very similar test suite functionality to what is available to us with Selenium, there are two features of CasperJS that are not available to us with our current Selenium WebDriver approach:

  • the ability to specify browser widths when testing in order to test responsive design elements, which was demonstrated using picturefill.js, and which could prove invaluable when testing our Drupal theme;
  • the ability to easily capture page status to detect, for example, 404 errors, without writing custom code as with Selenium.

For these reasons, we should explore CasperJS when writing the automated tests for our Drupal theme, and ultimately we may be able to refactor some of our existing tests in CasperJS to simplify the tests and reduce the time spent on resolving intermittent Selenium WebDriver issues.

On the performance testing front, we do not currently use any automated testing tools to compare such aspects of performance as page load time before and after making code changes.  This is certainly something we should explore, and the tools used during the demo, PageSpeed and Phantomas, seem like good candidates for investigation. A tool such as PageSpeed can provide both performance metrics and recommendations for how to resolve bottlenecks. Phantomas could be even more useful as it provides an extremely granular variation on the kind of metrics available using PageSpeed and even allows assertions to be made to check for specific expected results in the metrics retrieved. On performance, see also the blog post from DrupalCon day 1 for the session summary on optimising page delivery to mobile devices.

Finally, CSS regression testing with Wraith, an open source tool developed by the BBC, was demonstrated.  This tool produces a visual diff of output from two different environments to detect unexpected variation in the visual layout following CSS or code changes.  Again, we do not do any CSS regression testing as part of our deployment process for the University’s new central Drupal CMS, but the demo during this talk showed how easy it could be to set up this type of testing. The primary benefit gained is the ability to quickly verify for multiple device sizes that you have not made an unexpected change to the visual layout of a page. CSS regression testing could be particularly useful in the context of ensuring consistency in Drupal theme output following deployment.

I can highly recommend watching the session recording for this session.  It’s my favourite talk from this year’s DrupalCon and worth a look for any web developer.  The excellent session content is in no way specific to Drupal.  Also, the code samples used in the session are freely available and there are links to additional resources, so you can explore further after watching the recording.

Doing Behaviour-Driven Development with Behat

Having attended a similar, but much simpler and more technically focused, presentation at DrupalCamp Scotland 2014, my expectation from this session was to better understand Behaviour Driven Development (BDD) and how Behat can be used to automate testing using purpose written scripts. It was showcased how BDD can be integrated easily in Agile projects because its main driver of information is discussions regarding business objectives. In addition to user stories, examples were provided to better explain the business benefit.

I strongly believe that this testing process is something to look deeper into as it would enable quicker, more comprehensive and better documented user acceptance testing to take place following functionality updates, saving time in writing long documents and hours of manual work. Another clear benefit is that the examples being tested reflect real business needs and requests, ensuring that deliverables actually follow discussed user stories and satisfy their conditions. Finally, this highlights the importance of good planning and how it can help later project stages, like testing, to run more smoothly and quickly.

UX Concerns

Building a Tasty Backend

This session was held in one of the smaller venues and was hugely popular; there was standing room only by the start, or even “sitting on the floor room” only. Obvious health and safety issues there!

The focus of this session was to explore Drupal modules that can help improve the UX for CMS users who may be intimidated by or frankly terrified of using Drupal, demonstrating how it is possible to simplify content creation, content management and getting around the Admin interface without re-inventing the wheel.

The general recommended principle is “If it’s on the page and it really doesn’t need to be, get rid of it!”.  Specific topics covered included:

  • using the Field Group module to arrange related content fields into vertical tabs, simplifying the user experience by showing only what the user needs to see;
  • disabling options that are not really required or don’t work as expected (e.g. the Preview button when editing a node) to remove clutter from the interface;
  • using Views Bulk Operations to tailor and simplify how users interact with lists of content;
  • customising and controlling how each CMS user interacts with the Admin menu system using modules such as Contextual Administration, Admin Menu Source and Admin Menu Per Menu.

The most interesting thing about this talk in light of our experience developing the University’s new central Drupal CMS is how closely many of the recommendations outlined in this session match our own module selection and the way in which we are handling the CMS user experience.  It is reassuring to see our approach reflected in suggested best practices, which we have come to through our knowledge and experience of the Drupal modules concerned, combined with prototyping and user testing sessions that have at times both validated our assumptions and exposed flaws in our understanding of the user experience.  As was noted in this session, “Drupal isn’t a CMS, it’s a toolkit for building a CMS”; it’s important that we use that toolkit to build not only a robust, responsive website but also a clear, usable and consistent CMS user experience.

Project Management and Engagement

Getting the Technical Win: How to Position Drupal to a Sceptical Audience

This Presentation started with the bold statement that no one cares about the technology be it Drupal, Adobe, Sitecore or WordPress. Business’s care about solutions and Drupal can offer the solution. Convincing people is hard, removing identified blockers is the easier bit.

In order to understand the drivers for change we must ask the correct questions. These can include:

  1. What are the pain points
  2. What is the competition doing, and most importantly
  3. Take a step back and don’t dive into a solution immediately.

Asking these kinds of questions will help build a trusted relationship. To this end it is sometimes a necessity in certain situations to be realistic and sometimes there is the need to say no. Understanding what success will look like and what happens if change is not implemented are two further key factors.

The presentation then moved on to technical themes. It is important to acknowledge that some people have favoured technologies. While Drupal is not the strongest technology, it has the biggest community and with that huge technical resources, ensuring longevity and support. Another common misconception is around scalability. However, Drupal’s scalability has been proven.

In the last part of the presentation attention turned to the sales process, focussing on the stages and technicalities involved towards closing a deal. The presentation ended with a promising motto “Don’t just sell, promise solutions instead.”

Although this was a sales presentation it offered valuable arguments to call upon when encouraging new areas to come aboard the Drupal train.

SellingDrupal

Looking to the Future

Future-Proof your Drupal 7 Site

This session primarily explored how best to future-proof a Drupal site by selecting modules chosen from the subset that have either been moved into Drupal core in version 8 or have been back ported into Drupal 7.  We are already using most of the long list of modules discussed here for the University’s new Drupal CMS.  For example, we recently implemented the picture and breakpoints modules to meet responsive design requirements, both of which have been back ported to Drupal 7.  This gives us a degree of confirmation that our module selection process will be effective in ensuring that we future-proof the University’s new central Drupal CMS.

In addition to the recommended modules, migrate was mentioned as the new upgrade path from Drupal 7 to Drupal 8, so we should be able to use the knowledge gained in migrating content from our existing central CMS to Drupal when we eventually upgrade from Drupal 7 to Drupal 8.

Symfony2 Best Practices from the Trenches

The framework underpinning Drupal 8 is Symfony2, and whilst we are not yet using Drupal 8, we are exploring web development languages and frameworks in other areas, one of which is Symfony2. As Symfony2 uses OO, it’s also useful to see how design patterns such as Dependency Injection are applied outside the more familiar Java context.

This best practices covered in this session seem to have been discovered through the bitter experience of the engaging presenter, and many of them are applicable to other development frameworks.  Topics covered included:

  • proper use of dependency injection in Symfony2 and how this can allow better automated testing using mock DB classes;
  • the importance of separation of concerns and emphasis on good use of the service layer, keeping Controllers ‘thin’;
  • appropriate use of bundles to manage code;
  • selection of a standard method of configuration to ensure clarity, readability and maintainability (XML, YAML and annotations can all be used to configure Symfony2);
  • the importance of naming conventions;
  • recommended use of Composer for development using any PHP framework, not just Symfony2.

I have attended two or three sessions which talk about Symfony2 at this conference as well as a talk on using Ember.js with headless Drupal.  It’s interesting to note that whilst there are an increasing number of web development languages and tools to choose from, there are many conceptual aspects and best practices which converge across those languages and frameworks.  In particular, the frequent reference to the MVC architecture pattern, especially in the context of frameworks using OO, demonstrates the universality of this particular approach across current web development languages and frameworks. What is also clear from this session is that standardisation of approach and separation of concerns are important in all web development, regardless of your flavour of framework.

The Future of HTML and CSS

This tech-heavy session looked at the past, present and future of the relationship between HTML and CSS, exploring where we are now and how we got here, and how things might change or develop in future. Beginning with a short history lesson in how CSS developed out of the need to separate structure from presentation to resolve cross-browser compatibility issues, the session continued with an exploration of advancements in CSS such as CSS selectors, pseudo classes, CSS Flexbox, etc. and finally moved on to briefly talk about whether the apparent move in a more programmatic direction means that CSS may soon no longer be truly and purely a presentational language.

There was way too much technical detail in this presentation to absorb in the allotted time, but it was an interesting overview of what is now possible with CSS and what may be possible in future.  In terms of the philosophical discussion around whether programmatic elements in CSS are appropriate, I’m not sure I agree that this is necessarily a bad thing.  It seems to me that as long as the ‘logic’ aspects of CSS are directed at presentation concerns and not business logic, there is no philosophical problem.  The difficulty may then lie in identifying the line between presentation concerns and business concerns.  At any rate, this is perhaps of less concern than the potential page load overhead produced by increasingly complex CSS!

Posted on

IS at DrupalCon Amsterdam – Day 1

This week, a few members of IS have decamped to Amsterdam to attend DrupalCon 2014, which brings together people involved with all aspects of Drupal for a week of talks, labs, Birds-of-a-Feather sessions and many hours of coding on Drupal 8. The focus of Tuesday’s opening Prenote, a DrupalCon fixture beautifully compered by JAM and Robert Douglass, was life-changing Drupal experiences. Whether or not DrupalCon changes your life, the breadth and depth of sessions and associated discussions to be found here this week is undeniably absorbing. For those of us who are currently working on the University’s new central Drupal CMS, DrupalCon provides a unique opportunity to both validate the approach we are taking with our development processes, coding standards and infrastructure, and to discover new modules, best practices and techniques which will benefit our new Drupal CMS.

After Dries’ Keynote, the conference kicked off in earnest. We have crowdsourced some of the highlights of our first day of sessions below. Many thanks to Aileen, Arthur, Riky, Tim, Andrew, Adrian and Stratos, for contributing their thoughts on sessions they attended; any errors, omissions or misinterpretations in their edited summaries are entirely mine. The overriding impression from discussing the sessions we have all attended is that Drupal is often at the bleeding edge of development tools and technologies by virtue of the commercial and community pressures in the Open Source environment. Drupal’s presence as a tool in our development portfolio both challenges our own best practices and introduces new, innovative means of developing quality applications which anticipate the needs of an increasingly diversified technological world.

Most of the sessions mentioned below, along with many more interesting talks, are recorded and available on the DrupalCon website, so if the summaries pique your interest, visit the DrupalCon site for more information!

Development Processes, Deployment and Infrastructure

State of Drupal DevOps

Whilst the focus of this session was Drupal DevOps, Kris Buytaert’s talk applies more generally, covering the reasons why DevOps is not just a team or C.I. or Puppet. It is a cultural attitude that requires long-term thinking and a degree of co-operation from all teams. It’s not just about the lifetime of a particular project, but the lifetime of an application or service. The impact of putting off changes to deployment strategy is an increase in the “technical debt”; it only defers the issues, which then become support problems.

The proposed approach is mainly to expand best practices developed for writing code, such as version control and testing, down into the infrastructure and up into the monitoring tools. For example, the importance of repeatability is heavily emphasised; everything needs to versioned, and this includes infrastructure as well as artifacts. Using such DevOps techniques, we can better map and evaluate the impact of changes. The payoff should be safer, quicker sites or applications that do what people want, and developers get more feedback about why things went wrong (“It works on my machine.” is never an excuse).

Deploying your Sites with Drush

This session covered the Drush Deploy plugin, which allows you to create drush configuration files to describe each of your environments so you can deploy consistently to all servers with one command, reducing human error.

It was interesting to contrast the approach of this plugin vs Capistrano or services like Bamboo or Jenkins. We use drush heavily in the automated deployment process for the University’s new central Drupal CMS, but Bamboo still handles the code deployment to ensure consitency across environments. We use Ant scripts to describe the pre- and post- code deployment tasks for Bamboo to carry out, whereas these are specified in a drush configuration file for the Deploy plugin and are limited to drush functions. It was interesting to compare their approach to handling code rollbacks with our plans for this, even though they do not explicitly include resinstating the database as part of that process. However, we are in a different position as normally we would roll back immediately upon failed deployment rather than some hours/days later when content could have changed significantly.

The importance of adopting an appropriate Git workflow to support deployment where there is a branch for live deployment which is always deployable was also discussed. Having separate live and dev branches is very important, and making use of separate branches for hotfixes and features is recommended: http://nvie.com/posts/a-successful-git-branching-model/.

WF Tools – Continuous Delivery Framework for Drupal

WF Tools is a Continuous Delivery framework for Drupal sites, which was also shown off at DrupalCamp Scotland earlier this year. It is used to deploy code and configuration from git into a freshly spun-up Development virtual host environment. These code changes are all separate “jobs”, working from git branches; WF Tools allows you to tag these jobs with JIRA issues and trigger runs from a build tool such as Jenkins or Bamboo. After a successful run, the Development environment can be assigned to another user for peer review, and their GUI view shows a change log and git diffs for comments and approval or rejection. Any approved jobs continue along the Dev/Test/Staging/Live deployment pipeline.

WF Tools is an interesting solution which could wrap around our existing processes quite well, and the Pfizer implementation of the GUI looks good. However, as we’re already using a lot of Bamboo functionality, and we’re only developing one Drupal site centrally at the moment, it might not be perfect for our current requirements.

Understanding the Building Blocks of Performance

This talk by Josh Waihi covered ways in which a system could be created to fulfil a client’s needs using vertical and horizontal optimisation techniques, supplemented by profiling tools to help find and fix bottlenecks. ‘The building blocks of performance’ were broken down into different important categories, understand, build and optimise:

  • understanding the resource requirements before an application is built;
  • building the infrastructure in a suitable manner which balances complexity with performance;
  • using logging and load testing to optimise the performance of the Drupal system.

Vertical optimisation involves hosting the components in the system – load balancer, http cache, web server with PHP and Drupal, other files, database and database cache – on different servers. In addition, assigning more resources to a site makes for easier location of bottlenecks. Once the vertical optimisation is done then horizontal optimisation can begin. In general the web server is duplicated many times, all referring to the same database and shared files. The main limiting factor here is cost. And finally, load testing and profiling tools help to ensure your system is using the correct amount of resources.

Unlike other sessions where new, cutting edge techniques and technologies were discussed which could be used in the future for Drupal, this session was beneficial because it vindicated the techniques that we are already using:

  • scaling our infrastructure vertically before we scale it horizontally;
  • the importance of using business rather than technical metrics for our performance and capacity testing;
  • the different caching tiers we use to boost performance were aspects of our current practice validated by the expert from Acquia.

However, using PHP-FPM instead of mod_php and interesting diagnostic tools such as XHProf are interesting new ideas I’ll bring back to DevTech.

Front End Concerns

Panels, Display Suite, and Context – oh my! What to use when, why and how

Yes, it did open with a pic of Dorothy et al! This was a very well attended session covering when and why you should/could use the different layout options available for Drupal. With clever use of a ‘Garfield’ rating system it was clear that all have pros and cons depending on use and complexity of the site.

Here’s the “Janet and John” bit…

  • Context provides flexibility by extending core blocks to provide reusable blocks and regions, but Blocks are still hard to maintain and there is only one set of regions for layouts.
  • Panels is powerful with a high level of granularity and a default variant which provides a failover structure. However, the codebase for Panels is heavy and the functionality provided may be overkill for easy layouts.
  • Display Suite has a simpler UI and similar flexible layouts to Panels, but only Entity layouts are supported and because there is no structure across different layouts, things can easily become complicated.

The consensus seems to be that Display Suite ticks most boxes but each method has its merits. You may want to do some research to find the best for your particular project.

Drupal 8 breakpoints and responsive images

Although this session was titled for Drupal 8, both modules covered, picture and breakpoints, have been backported to Drupal 7 and we have recently implemented them to support the theme for the University’s new central Drupal CMS. At this point we only have four variants of the group banner image being served, one per breakpoint that has been defined in the theme.

The session also covered the module’s use of the sizes attribute to serve optimised versions of images according to the viewport width in steps between the breakpoints where required. This is not something we are currently implementing, but we will be in the coming weeks as we approach the initial distribution release of our new Drupal theme.

The State of the Front End

The Front End is moving forward faster than anything else in drupal. Display targets used to be 640 X 480 or 1024 X 760 for IE and Netscape using tables; easy! Now HTML5, CSS, JS, responsive design, etc. add significant complexity to Front End development, and these are not Drupal skills!

Frameworks come and go (e.g. 960 grid, blueprint, bootstrap) and we may catch up or even get ahead, but not for long when things change in the Front End so fast, and techniques fall in and out of fashion. However, Front End is A Thing, and is pushing Drupal forward in the post-responsive world. There are multiple frameworks for everything and too much scope to play design for design’s sake; the goal is truly device-independent design. It’s necessary to accept that you may be ahead, but only ever for a short time; however, there are many tools out there to support the drive to keep up with the rapidly changing world of Front End development.

Performance

Turbocharging Drupal syndication with Node.JS

Where you have to generate feeds from Drupal for high volume of requests, caching is sometimes not an option because requests from downstream clients include timestamps per second (to retrieve things that have changed since last request), or there are user filtered requests which aren’t likely to be repeated.

The approach taken here was to use an indexer module to generate de-normalised tables for the Drupal data on a MongoDB database, optimised for delivery, and put a fast Node.JS REST API in front. In their case-study, Drupal could be as slow as around 1 request per second where many records were being returned in one request, whereas NodeJS could handle 800-3000 requests per second. Response times dropped from up to a minute down to 80-150ms.

To support developing for many parallel/asynchronous requests in NodeJS, there are npm’s such as promises to help.

Getting content to a phone in less than 1000ms

In order for a site to respond quickly enough for a user not to get bored waiting and give up to go elsewhere, it’s generally accepted that pages should be served in under a second. This can be challenging enough in complex Drupal pages, but there are added constraints to consider with mobile devices on slow networks. The DNS lookup, TCP connection, TLS handshake and HTTP request over 3G can come to 800-1000ms alone before you get to the time Drupal takes to serve the content and then the time taken by the client to paint the page. Given that many countries primarily use mobile devices now to access the internet, this is becoming more important.

When painting the page there are some blockers that delay rendering of the content. In particular, moving JS to the footer whenever possible and using async and defer was recommended. The magic module can help achieve this, and any critical JS can be made inline.

CSS can also be split out into what’s required for mobile devices and then LoadCSS can be used to load the remainder without blocking the initial rendering of the page. However, it is not always practical to achieve this.

With TCP, most of the time taken is not due to limitations in bandwidth, but latency on each round trip with initiating each request (handshakes, etc). CDNs can help by putting content closer to the client and allowing more parallel connections (because they are split across different domains), but this is expensive and will likely be blocked in China. Aggregating files and spriting so there are fewer files to load, even inlining very small assets; all of these things can help. Remove things you don’t need. The target is to get first response within 10 packets, 14.6k (RFC 6928), although it’s extremely difficult and very few sites are able to achieve this.

Preparing for SPDY/HTTP2.0 can conflict with some of the HTTP1.1 optimizations above, such as domain sharding and concatenation. Some work has started on supporting SPDY features, such as server push, in Drupal.

Project Management and Engagement

Selling Agile

In this experimental session, the Vesa Palmu CEO of Wunderroot shared some of his lessons learned over the past 10 years using Agile on IT projects.

A core difficulty in getting agreement to use Agile can be the lack of trust between parties. The reasons for this mistrust can be a lack of understanding of what Agile actually means. How does using Agile translate for the business, in terms how this changes their activity during projects? Customers are unsure about the cost versus what will actually be delivered or are unwilling to engage by providing a product owner.

One of the key messages is that in order to sell Agile successfully, one needs to focus on selling the benefits of Agile. Some of the benefits:

  1. Collaborative development approach
  2. Testing development as it progresses
  3. Creating value faster the multiple deliveries
  4. Delivering better quality
  5. Making better decisions along the way

Mixing Agile and Waterfall should be avoided, the benefits of Agile cannot be realised when project teams are in two mind sets.

Equally not all projects sizes are suitable for Agile, as illustrated by the following slide:

AgileProjectSizes

The Myth of the Meerkat: Organising Self-Organising Teams

In this session, speaker Jason Coghlan examined whether the self-organising team is a reality, especially in a commercial or public sector Drupal services environment, focusing on tips for research rather than specific examples of how to coach and build self-organising teams.

The session covered the need to differentiate between control and accountability: the former must be relinquished; the latter, especially individual accountability, is extremely important. In the context of self-organising teams, “Leaders are required, managers are optional”. George will take care of it! The conclusion is that whilst self-organising teams are not suitable for all projects, they are an ideal approach to technology-driven projects where a clear product or solution is delivered, focusing on value and return.

Engaging UX and Design Contributors

This was a really interesting session from Design Researcher, Dani Nordin, highlighting the challenges of integrating User Testing, UX & Design guidelines in an already established community such as the Drupal community. It’s very obvious that it is difficult to engage developers and designers to cooperate in such an open environment, so one thing I took from this session is that there is a strong need for integrating UX into the process of module contribution. Even though it might sound restrictive (especially for people contributing in their own time) it will pave the way for a more user friendly and intuitive Drupal UX. Drupal 8 might be a good opportunity to explore this as well.

Looking to the Future

Drupal 8: The Crash Course

Having attended DrupalCon last year as a relative Drupal newbie, and with most of our current internal development focus on Drupal 7, I approached Larry Garfield’s technical Drupal 8 overview session with mild trepidation. I needn’t have worried. This well structured introduction to Drupal 8 and its use of Symfony2 was very accessible. The code samples were clear and progressively illustrated each concept to give a pretty good high level overview of what to expect from Drupal 8. Coming from a Java/C++/OO background, I can say that it truly seems “Drupal8 is finally not weird”. Lots of familiar code even in the context of an unfamiliar framework!

Twig and the new Drupal 8 Theme system

The current Drupal 7 theming system involves a mixture of markup generated by modules and the theme primarily through theming hooks (functions). This means the front-end developer does not have full control of the markup (and CSS) that’s output.

By converting all the hook functions to Twig templates in the new Classy theme for Drupal 8, the themers can be in full control of all the output. The hope is that this will make frontend developers engage with Drupal in the future and make it easier for web design companies to engage with Drupal.

Changing menus and pagers was demonstrated, two of the most complex components to theme since they are currently buried in module functions, but in Classy these elements are themed in single twig template files which can easily be re-written to change CSS dependencies and markup without needing to know the inner workings of the core pager and menu functions.

Posted on

uPortal release 4.10

uPortal  version 4.10 has been recently released, which has lots of great new features, most notably the move over to a Bootstrap 3 responsive design.

http://www.apereo.org/uportal/download/uportal-4-1-0

We’re also pleased to note that Ross Nicoll in Development Services has contributed to this release. That’s great to see, and hopefully something we can do more of in subsequent releases.

Thanks Ross :)!

Posted on

Drupalise me – confessions of a born-again Drupalist

In September 2013, in the early stages of the ongoing project to replace the University’s central CMS with Drupal, some of the project team attended DrupalCon in Prague. It proved to be an educational and fun introduction to the Drupal world. There were many fascinating sessions, focusing not just on Drupal but on web and Open Source topics in general, and we met tonnes of interesting people who are passionate about what they do. Drupalists are a vibrant, welcoming and supportive bunch; when they say ‘Come for the software, stay for the community.’, they’re not kidding.

On the 9th and 10th of May, the University was one of the sponsors for DrupalCamp Scotland, giving us an ideal opportunity for a more local dose of the Drupal community spirit, and some socialising afterwards! The keynotes on both days were insightful, and there were talks from other people doing interesting things with Drupal – from DevOps, to testing with Behat and Selenium, as well as working with particular Drupal modules. We met other Drupal developers from around Scotland and the UK who share our enthusiasms and frustrations, including Drupalists from the University of Dundee who are very active in the Scottish Drupal community. This time, however, we weren’t just there to observe and absorb, we were taking part and sharing our own experiences with Drupal.

On Friday the focus was the Business perspective. Stratos Filalithis from the University Website Programme team gave a great presentation on the vision for the University’s central Drupal CMS. On Saturday, things got more technical!  Myself and another developer on our Drupal team, Adrian Richardson, presented a session on how we’re combining our automated deployment toolkit with Drush and Features to support the deployment of the new Drupal CMS. For me personally it was a daunting prospect, especially given the technical credentials and Drupal kudos of the other DrupalCamp speakers! However,  the audience were friendly and interested, and asked really good questions at the end of our talk. Later, people were keen to discuss their own thoughts on the topics we covered. We got some great feedback on our approach to deployment and testing, giving us confidence that we’re going in the right direction. It was a rewarding experience and, unexpectedly, it was fun!

What has struck me about both DrupalCon and DrupalCamp is the genuine sense of community evident at Drupal events.  The obvious drive of those present to share their experiences and to be involved with the community is infectious.  I will be at the next DrupalDrinks in Edinburgh to talk more to the people I met on Saturday, and whilst I haven’t yet contributed to code on Drupal.org, I’m now keen to attend a code sprint and get on the Drupal Ladder!  There are many developers all around the University who are using Drupal to build websites and by virtue of using Drupal, we are already part of this amazing community.  On Saturday the keynote speaker, Robert Douglass, talked about how easy it is to get caught up in the trials and tribulations of our own complex Drupal projects and to lose sight of the bigger picture. Being at DrupalCamp was a reminder that by placing Drupal at the core of the University’s website we are placing ourselves firmly in the midst of a dedicated, talented, enthusiastic and supportive Open Source community.  We can contribute to that community and we can learn from them, and we can also be inspired by their ethos when it comes to our own internal Drupal community and the future of Drupal at the University.

Here is a PDF of the slides from our presentation at DrupalCamp, including notes, along with a link to Adrian’s prezi covering the technical aspects of our talk:

Posted on

Open Source updates

Open Source projects and code are an important aspect of what we do, so it’s good for us to be able to contribute and share with the wider community. Here’s an update of some of the recent work we’ve been doing.

uPortal

On the uPortal front we’ve contributed a bug fix to the core uPortal framework which should be incorporated into the uPortal 4 patch releases.

And the Blackboard  Virtual Classroom portlet which we originally contributed to the Apereo foundation (then JASIG) has been updated by the University of Wisconsin to version 2.

Blackboard Learn

We have submitted a building block we wrote a while ago to do dynamic web forms to project oscelot, which is an open source community site based around eLearning. We chose to release the bb-webform-tool (as we call it) code on Github.

We have also submitted the a Building Block which allows creation and integration with Confluence Wiki spaces to Project Oscelot, and have put the bb-confluence-tool code on Github in preparation.

The portlet and building block code above have been released under the MIT license.