One of the things we do in Gradwell is write an internal blog. We include three things:

1. We documents all our system changes, so if something is planned, or has changed, we have any easy reference for it.
2. We write up internal case studies of our customers, and feedback from our birthday lunches
3. Weekly, on Friday afternoons, we do a round up of what all the teams (tech, sales, support, billing) got up to that week.

Unfortunately we can’t make the whole internal blog public, but it is interesting to use a new tool I found, Wordle to produce a “word cloud” for that internal blog. A word cloud is an image made from your blog text which gives greater prominence to words that appear more frequently in the text:

We can also compare that to the word cloud that you get from the public Gradwell blog:

It’s good to see the themes are consistent:

• Lots of focus on happy customers, through tickets and Birthday Lunches.
• Lots of continued work on infrastructure and scaling, particularly email and storage, as well as migrating from legacy servers to our newer platforms.
• A huge amount of work being completed and delivered, with many small incremental improvements.

We’d be interested in your feedback as to how we’re doing, via our online survey. Thanks!

Last week, we became aware of a number of customers websites being altered, with the addition of some extra hidden code to customer’s home pages.

No personal data has been compromised (apart from an encrypted version of FTP passwords) and all sites have been restored as far as possible.

We have been investigating this and whilst we first thought that this attack was possible through the exploitation of weak customer passwords, we have now proved that the hacker was manipulating our ftp server username and password database, and adding additional usernames into the system in order to gain access to customer’s websites.

We have been able to successfully monitor this attack in action, which has allowed us to exactly understand what went on, how it happened and most importantly, the full extent of the attack.

The Attack – What happened?
A customer’s website was compromised, probably through some insecure php code, and the attackers uploaded some code which allowed them to browse a mysql database.

Our web servers have access to our management database, so that they can pull down the configuration for customer websites. It seems likely that at some point we inadvertently miss-configured the permissions on the configuration file storing the database access credentials.

From this, the attacker was able to scan the server configuration files and determine a mysql username on our web hosting database.

This user was able to edit the ftp server database table, and they created additional ftp usernames that mirrored customer’s usernames (but with different passwords), which they then used to access the customer’s website accounts. The extra usernames were then deleted, to hide their trace.

How do we know?
From our FTP server logs we have been able to identify which files were edited, and it is clear that the only change was to alter the index/home page on a website.

We were also able to add monitoring to our database servers and determine which username and password was used, and what changes they made to our database.

From our logging of the network traffic, we can see that no other data was misappropriated, specifically no personal customer data.

What have we done to rectify & prevent future occurrences?

• We have restored affected customer files as far as possible.
• We have improved the firewalling in our network, so that customer web servers do not connect to our management database to get their configuration data.
• We have undertaken a further precautionary password changing exercise.
• We have planned additional work, to implement automated scanning for future attacks on our ftp servers and web servers.
• We implementing further partitions to our database security, so that web server configuration and ftp access are secured using different users.
• We have fed the lessons learned from this exercise into our web cluster redesign work, to improve the protection we give to customer websites.

Conclusion
We would like to apologise to customers for any inconvenience this episode may have caused.

As you may know from following our blog, we are engaged in a program of evolving our email and web hosting infrastructure and problems like this highlight areas for improvement in our initial systems designs from a several years ago.

Finally, we would also recommend customers reset their FTP passwords, because, whilst they are encrypted in our database, there is the possibility that a 3rd party has a copy of that list, and they could be decrypted and used again.

This can be done online at: https://hosting.gradwell.net/login/ftpmanage?menu_req=75

We wanted to update customers on the progress with our email systems migration.
Customers will be aware that we are currently engaged in the process of building a completely new email infrastructure and migrating customers mailboxes from a number of different systems onto a single, consistent platform.

The challenge has been to build a cost effective email platform which would allow us to offer a high level of uptime and protection against physical server and disk failure, for our low cost email hosting products. We have built a system using multiple storage arrays, on top of a VMWare Server cluster and have nearly completed the migration process.

Our email system consists of front end servers, which handle the connections from customers, using imap, pop3 and the sending of email, using SMTP. We also then have the backend servers, which store customers email files.

To recap:

• We announced the plan in December 2007, where we outlined the proposed developments for the first half of the year.
• In February 2008, we completed the deployment of 15 new servers, handling pop3, imap and smtp.
• One of the hurdles for deploying the new IMAP platform was that customer mailboxes needed to be restructured, so we completed a restructuring project of the 43,000 mailboxes we host, which was completed by 9th May 2008. Customers were largely unaffected by this process, although unfortunately a small percentage of customers had to reconfigure their email programs.

This completed the majority of the “front end” work, although we still have a handful of customers whom are logging into the old mail platform and we are going through the process of contacting those manually, to correct their settings.

The second phase of the project has been to migrate the backend of the system – the customers mailbox storage – to a more reliable storage platform.

• Previously, we put customers mailboxes on individual servers, however, this meant that it would take a long time to recover from a server crash and we were reliant on a single disk array.
• Therefore, we adopted a high availability file system from Redhat (GFS), which provided excellent resilience, but at the expense of performance, and it is still reliant on a single disk array.
• We have therefore migrated all customers mailboxes to a set of 9 file servers instances on top of our VMWare platform, which allows us to have quick recovery (sub 5 min) in the event of the operating system crashing, and, whilst the files are still stored on top of a single disk array, we can move them from one disk array to another with no down time.

During this process, the migration of customers mailboxes has been exacerbated by:

• Slow performance and unreliability on old servers.
• Capacity issues on our new VMware cluster, due to our accelerating the migration process, and licensing delays at VMWare, causing some of the new file servers to crash, particularly early in the morning, when the load has been highest.
• The need to migrate over a terabyte of storage in fairly short time windows.

During this migration period in May, customers have experienced short periods of not being able to access their email whilst we adjusted settings and moved mailboxes for which we apologise.

As a third phase of the project, we have designed a redundant backend server, which allows us to store customer mail files on two independent disk arrays (potentially in two locations) with sub-1 minute level fail over, and we plan to test this in late May, before retrofitting it to our new storage setup in early June if successful.

At the end of May had the following short term work to complete, and as at 2nd June, this work is done:

• We need to add servers and RAM to our VMWare cluster, to increase available computing power on the new mail cluster. Done.
• Migrate the remaining 200 mailboxes from our Redhat GFS Cluster. Done.
• Ensure all customers are using the new platform, and correct any old customer configurations. Done.
• Fixed an issue in the Linux Kernel, related to disk i/o timeouts under heavy load, which causes the file servers to crash. Done.

Finally, to take advantage of the new system customers should ensure they check their email from pop3.gradwell.net and use relay.gradwell.net as their outbound smart host.

Outstanding Issues

Out VMWare based solution is performing well, giving good speed and has resolved many of the issues customers were experiencing. However, a number of elements continue to need to be addressed Mid-Term.

• Implement the high availability solution for our storage servers, which we hope to complete by early July. We have a proof of concept in operation and are beta testing it (at 14th June).
• Implement a multi-site VMware cluster, giving us multi-site redundancy, which will be complete for autumn 2008. We have ordered the redundant networking equipment required to do this (at 14th June) and expect it to be implemented by mid-August 2008.