21 February 2012

Story header graphic logos associated with the Drupal 8 initiatives

I was already planning to provide an overview of what’s been going on in the various Drupal 8 initiatives even before last week, when Dries announced the timeline for Drupal 8, which includes a “feature freeze” for Drupal 8 in only a little more than nine months from now, and planned release at the DrupalCon Europe, in late August 2013.

Drupal 7’s Plateau of Productivity?

I think we reached the Plateau of Productivity for Drupal 6 sometime in late 2009, about 18 months after its release. At that point there was no advantage to developing in Drupal 5, and Drupal 7 was still a long way off. --Dries Buytaert, June 8, 2011 (about 5 months after Drupal 7’s release)While most of the top Drupal 6 modules are now available, in some state or another, for Drupal 7, and I would certainly choose Drupal 7 for a large Drupal-based project that is not expected to be deployed for some time, from the outcry of protests I think there are a lot of people who would not agree that Drupal 7 is yet at its Plateau of Productivity. I would choose development in Drupal 7 for most projects, but there is still plenty of reason for site builders to work with Drupal 6, especially if they need particular features (e.g Nodewords / Metatag functioning properly, among others) and if they need to deploy the site now, with those features ready for use. Dries indicated that he thought Drupal 6 reached its Plateau of Productivity in late 2009, about 18 months after its initial release. At that point, there were fewer than 20,000 sites using Drupal 5 and more than 200,000 sites using Drupal 6. While this order-of-magnitude-greater-usage is not likely to ever be seen comparing Drupal 7 vs Drupal 6 usage (at least not before Drupal 8 is released), I do think that it’s significant that Drupal 7 usage has finally overtaken Drupal 6. That said, I don’t think we are truly at Drupal 7’s Plateau of Productivity, the point where building a new site on Drupal 6 would be “pointless”. Both in terms of time-after-release and usage statistics, it is arguably premature to say we are quite to that point yet.

Quibbling aside, I also don’t think it matters whether we believe we have reached the Plateau of Productivity for Drupal 7, or not — and it certainly doesn’t matter whether we are all in agreement about that. I do think Drupal 7 is very “ready for use”, though and I do think we are ready to see core development for Drupal 8 get kicked into high gear and I don’t think it will significantly delay the development of certain lagging contrib modules or resolving core issues in Drupal 7 which are the final barrier, in my view, to truly reaching its Plateau of Productivity. Additionally, many of the fixes and features going into Drupal 8 are regularly being back-ported to Drupal 7, and there is increased discussion of relaxing the criteria for what can be back-ported to Drupal 7, so I see the increased attention to Drupal 8 core development as exciting: a win-win for the whole Drupal community. We now have a release date for Drupal 8, which is important for business decisions, and a better timeline to facilitate a roadmap for the final stages of determining feature inclusion.

Drupal 8 Core Initiatives

Currently there are 6 official Drupal 8 Core Initiatives which are working on various aspects of desired improvements to core. There are others likely to be added to the list as soon as a bit more progress has been made on the current list and/or as qualified individuals step up to take on some of the other “top 10” desired improvements we had on our collective community wishlist. Some of the improvements require fixes to issues plaguing Drupal 7 and 6 and have been backported. Most of the others involve dozens, if not hundreds, of related issues. Following is a brief summary of each of the current core initiatives and what their priority goals are for Drupal 8. In the interest of brevity, the explanations leave out a lot of juicy details, but for those who haven’t been paying close attention and who might like to get involved, I hope this summary is useful:

Web Services and Context Core Initiative

The Web Services and Context Core Initiative (WSCCI, pronounced “Whiskey”), formerly referred to as the “Butler” project, is a core initiative led by Larry Garfield of Palantir.net, aka “Crell” on Drupal.org. While the traditionally typical HTTP request has been for HTML pages, the modern Web has brought with it the need for HTTP services which deliver information which is not necessarily in the form of HTML. This is especially true for mobile applications, but also applies to feeds and other communications via HTTP. The goal is to “take Drupal from being a first-class Web CMS to being a first-class REST server which includes a first-class Web CMS”. Really, this initiative spans a huge range of related issues and without writing an article many times the length of this one, I could not possibly cover everything, but…

03 February 2012

While this article focuses primarily on the state of Drupal “contrib” (modules and themes which are not part of the “core” Drupal download), it also takes a look at the greater “State of Drupal” in terms of sites known to be running on some version of Drupal, comparisons of the rate of uptake after Drupal 6 and Drupal 7 release, and a small case study involving attempting to perform a “major upgrade” from Drupal 6 to Drupal 7 on a site using a significant number of contributed modules.

The recent history of Drupal core usage

dries-quote_drupal-growth.pngAs a starting point, I think it is helpful to look at the recent history of Drupal core usage and compare the uptake of Drupal 6, after its release, with the uptake of Drupal 7. On June 22nd, 2008, when Drupal 6.0 was released, there were already significantly more sites using Drupal 6 than Drupal 5 (almost 32,000 on Drupal 6 vs almost 17,000 on Drupal 5). Both core versions of Drupal steadily gained users for a time, with Drupal 5 reaching a peak of about 24,000 sites about 7 months later, but by that time Drupal 6 was running on more than 100,000 sites. By late July 2009 (a similar point to now in terms of months after the major version release), Drupal 5 usage had dropped to about 20,000 sites and Drupal 6 was running on more than 160,000 sites; more than eight times as many installations. Since then, Drupal 5 usage has tapered to about 7,000 sites; a bit more than 1% of total Drupal usage (please note: it’s likely that many of the existing Drupal 5 sites do not report usage back to Drupal.org).

Now let’s look at the usage of Drupal 6 and Drupal 7 since the time of Drupal 7’s release. Drupal 6 peaked with about 355,000 sites, shortly after Drupal 7’s release in January 2011. At that time Drupal 7 was running on about 24,000 sites, a fraction of Drupal 6 usage at that time. Since then, while sites running on Drupal 7 have steadily increased to their present values, about 280,000 sites, Drupal 6 has hovered around the same value, drifting between about 320,000 and 350,000 sites, but not yet significantly dropping. Almost 13 months after Drupal 7’s official release, we still have more sites running Drupal 6 than Drupal 7 (and I suspect that a significant percentage of the Drupal 7 sites are in development rather than production). But what does this really say? Let’s look a bit closer at the numbers and trends:

Note: I banged this graph out in Excel since the Google chart of Drupal usage, normally displayed on project pages, seems to fail as “too large to process” for “core” usage statistics.

drupal_usage_june-2008_to_jan-2011-sm.png

Drupal usage has grown by leaps and bounds since Drupal 6’s release. In June 2008, there were fewer than 50,000 sites using Drupal 5 and 6 combined. Now, a bit more than three-and-a-half years later, there are more than 615,000 sites running on some version of Drupal — more than a 12-fold increase in that time period! A year ago, this figure was less than 400,000, so Drupal 7 sites make up a large proportion of the more-than-200,000 Drupal sites added since then. The growth was steeper after Drupal 6’s release, but we still did not have 200,000 sites, total, by July 2009. In any case, it’s safe to say that for most use cases, we have the modules necessary to build a good site based on Drupal 7, so if you are hesitant to use it, don’t be. There are many great advantages to Drupal 7 and with the continual improvement of the contributed modules, we should probably build new sites on Drupal 6, only if modules critical to the use case are lacking for Drupal 7 (or if the “new” site is another site in an existing Drupal 6 “multi-sites” installation). Even if a “critical” module exists for Drupal 6 and not yet for Drupal 7, it may still be worth building the site on Drupal 7 if you have the coding experience to port the Drupal 6 module to Drupal 7, which would help alleviate the current issue that many significant modules are not yet available for Drupal 7.

State of Drupal 7 contrib (modules)

Good news: Almost all “Top 100” Drupal 6 Modules are ported to Drupal 7

state_of_top_100_modules-sm.pngThe good news, especially for site builders creating a new Drupal 7 site, is that most of the top 100 modules are ready for use on Drupal 7. Nine of them are redundant (now included in “core”), 43 have “stable” releases, 23 have beta or RC, 11 have an alpha release, and 9 are in “dev” status, while a couple others recommend using another module which performs similarly.

25 January 2012
Apache Solr logoTomcat logo

Installing Tomcat6 and Apache Solr

Installation procedure

While there are several ways to install Tomcat 6 and Apache Solr, we will use the repository version to gain the benefit of automatic updates.

What is needed:

  1. Tomcat6 as Servlet container

    sudo apt-get install tomcat6
    sudo apt-get install tomcat6-admin

  2. Apache Solr Search Server

    sudo apt-get install solr-tomcat

Once everything has been correctly installed, you should see the message, “It works!” at http://localhost:8080 and “Welcome to Solr!” at http://localhost:8080/solr/

Configuring Tomcat 6

In the default Tomcat installation, no privileges are created for the Tomcat Manager, so in order to make use of the Tomcat Manager GUI, we still have to create the proper role and a corresponding user.

16 December 2011

[…] there are both advantages and disadvantages to backward compatibility. […] If you start dragging baggage along, your product will, eventually, be replaced by something that offers the same functionality but without the baggage. — Dries Buytaert, May 17 2006 (just after Drupal 4.7 releaseThe topic of Drupal’s backward compatibility issues has come up in various ways over the years and has been an issue of debate in most cases. When we responded to Dries’s “State of Drupal 2011” survey, only about 8.2% of the community members who responded indicated that improving backward compatibility was one of Drupal’s “biggest challenges”. But this is actually a pretty large number of respondents, considering that there were 19 other options for the question, and we could only select three, and also considering that the majority of those who took the survey might not represent the majority of Drupal stakeholders who would most benefit from improved backward compatibility: people who might not work with Drupal full-time, and might not personally maintain any code, and who want to upgrade their sites to the next major versions, but cannot do so easily because of missing modules and the fact that, regardless of whether a module might be very close to compliant with the next major-version API, modules are only for one major Drupal version.

If this lack of API backward compatibility were common with Windows or Mac applications, or Firefox plugins, or just about any other software written for a platform, far fewer people would make these upgrades or they would wait a very long time to do so (and might end up changing operating systems or browsers to avoid the headache). The opposite is actually true in the case of Mac apps; Apple has historically supported legacy code and legacy processor types with compatibility bridges which, although not particularly great for performance, at least allowed needed software to run. Firefox and Safari, my two most-used browsers, regularly release major updates which don't break every plugin (at least with Firefox, some add-ons are identified as incompatible and disabled until a compatible version is released). And PHP deprecates functions, but usually doesn’t make it impossible to use code that worked in the previous major PHP release. But Drupal introduces major changes that result in developers having to learn a whole new API just to get their modules or themes ready to run again. This made sense in the past for a variety of reasons, at least back when Drupal was still in diapers, but there might be sufficient reason, now, to revisit the issue and explore whether this approach still makes sense in the foreseeable future.

Lest it sound like I think this approach was always a bad idea, I should say that I do understand the logic behind breaking compatibility, at least historically. Too much attention to backward compatibility and it limits innovation, more bugs creep in, and the code starts to bloat. There are even former Drupal developers who claim they now work on slimmer platforms in order to avoid what they consider too much bloat in Drupal, already. It’s doubtful that Drupal would be as popular as it is today if the core team had focused on maintaining backward compatibility in the API just to keep from breaking legacy code. But what worked in the early days of Drupal may not work so well when there are thousands of modules which are not considered high enough priority to port to the next major Drupal release and most larger Drupal sites seem to have at least one of those that they consider a necessity.

Bar chart from Dries’s summary of the State of Drupal 2011 survey results:
State of Drupal 2011 Survey results — Biggest Challenges For the purposes of this post, I’ve marked up the image to discuss a few points. Yes, there are “only” 269 respondents who chose backward compatibility as one of Drupal’s biggest challenges. But it is also arguable that this challenge is directly related to many of the other challenges which ranked higher.

09 December 2011

Carsten Müller presents the Profiler library.This Thursday, 9 December, Cocomore hosted the Frankfurt/Rhein Main region’s third Drupal Meetup, which again was a success with a decent turnout.

01 December 2011

drupal8_release_date_confusion.pngOkay, so the title of this post is a bit tongue-in-cheek (it’s a reference to a particular Drupal 8 usability issue); I don’t really think this misunderstanding is “universal” but there does seem to be a bit of confusion about when to expect Drupal 8. The confusion seems common among the greater Drupal user-base, and it seems to affect more than just new Drupal users and those evaluating or considering using it, but also Drupal stakeholders who have several years’ of experience. According to Dries, we should not expect to see Drupal 8 released for at least another 18 months. Yes, I’m aware that he was talking about 18 months after Drupal 7 hits its “productivity plateau”, and since he also indicated that he would announce when he thought we were at that plateau (and to my knowledge, he has not yet) it’s likely to be more than 18 months before we see Drupal 8’s release.

And yet, for the past few months, we have been seeing increasing questions about Drupal 8 (in the greater developer and discussion community); questions which make the assumption that Drupal 8 is very close to release; a couple examples of such questions:

One user wrote:
... I have several years working with Drupal 6 and I am thinking to start building sites and develop over Drupal 7, but it seems that Drupal 8 will be launched very fast. Is it best to wait some months and learn about Drupal 8?
It’s easy to just think “Well, that’s just silly!” when we see one question like this, but when we see many such questions popping up, we have to know this points to some murkiness in terms of the Drupal release cycle.

13 November 2011

An exciting change has finally been committed to Drupal; the directory structure has been improved to move basically all of the files which are replaced during Drupal updates into a '/core' subdirectory. This should make it much simpler to manage updates and should also mean that there is less chance of new users unwittingly placing their downloaded ‘contributed’ modules and themes into the top-level directories. While it might seem like a very simple change, there is a reason that the issue which proposed the reorganization, first created over six-and-a-half years ago and originally intended as a change for the not-yet-released Drupal 4.7, took so long to be committed — and that it had over 300 comments on it by the time it was closed and marked as “fixed”.

28 October 2011

Some time back, I promised another short article in the WYSIWYG set-up series for Drupal 7, one which covers BUEditor. First, we should note that the BUEditor is not actually “WYSIWYG”, but it offers some nice features which might make it a bit better than the WYSIWYG options, depending on your use case. It also does not integrate with the Wysiwyg module. You add it separately (and instead of Wysiwyg), but it does have some great supporting modules and code libraries. This article covers some of the basics about use and installation of the BUEditor on a Drupal 7 site (most of the information applies equally to Drupal 6, where the BUEditor module is also available). I’ve also got some good tips for some ways to extend the default button-set. (And you can download my modified button code here to easily import the buttons into a new editor profile.)

20 October 2011

As part of the Knowledge Lab at Cocomore on 2 September, 2011, I outlined how to use Jenkins to implement Continuous Integration for a Drupal-based project.

14 October 2011

I only realized a few hours before yesterday's Drupal Meetup that I hadn't actually signed up to attend it, and Cocomore was playing host, so of course I wanted to be there. As luck would have it, one of our managers had canceled since he was unexpectedly called out of town, so there was a slot available, but I was surprised to see that we'd filled the “capacity” for the event in online signups and even more surprised that not only did everyone show up, but there were a few extras in attendance.