You can read the full interview here.

Register at twittex.com today!

If you can, then you should think about working for Gradwell.

Your responsibilities will be processing customer orders, predominantly configuring and dispatching VoIP Telephone hardware, maintaining our stock, processing deliveries and assisting the support and operations teams.

Gradwell dot com Limited is a rapidly growing Internet Services Provider who focuses on VoIP (www.gradwell.com/voip/), email and web service solutions. We are a pioneer in the exciting new world of Internet telephony and this section of our business is growing rapidly.

As a new and fast moving technology, VoIP is a demanding product to provide and our customer support team is regularly faced with evaluating new and different types of network routers, phones, broadband services etc. to make sure it all fits together correctly so our clients can make phone calls.

We have won a number of awards in 2004 and 2005 in various DTI schemes including “Best use of Broadband” for our VoIP product and “Best Teleworking Company” and “Best Online Customer Service” for both of which we were highly commended. 2006 and 2007 have been equally exciting and we are continuing to double every year in terms of customer base, financial turn over and staff. We are a team of 20 with annualised turn over of £2 million.

Prior experience of the Internet, IT and Telecoms Sectors will be highly beneficial as our core business is selling outsourced telecoms, email and web hosting services to IT consultants, designers, media agencies and other small and medium sized technology based businesses.

About the job

Your responsibilities will be processing customer orders, predominantly configuring and dispatching VoIP Telephone hardware, maintaining our stock, processing deliveries and assisting the support and operations teams.

You will also be involved in configuring a wide range of linux servers and PCs in a busy TCP/IP network environment.

You will need to have a good familiarity with web, email and computer networks, an understanding of broadband, experience with MS Windows operating systems or linux based systems . Experience with VoIP services will be a plus.

You will be well organised, enthusiastic about new technology, eager to learn, good at asking questions to solve problems and have good written and spoken English with a clear telephone manner.

This position will appeal to those with a keen interest in the latest internet and telecoms technology and who are looking to gain exposure to the world of work and who wish to gain experience prior to a job in customer/it technical support.

Salary is £13,000 per year. We also offer a range of benefits (including free parking in Bath).

This is a full time position (37 hours a week), based at our Bath offices in Cheltenham St, Bath, reporting to our Operations Manager. You will receive 24 days holiday per year.

Application Details

For more information, please view our website at http://www.gradwell.com or contact Barrie Millett (barrie.millett@gradwell.net, 01225 800 888).

Please apply via email to Barrie Millett, or post to: gradwell dot com Limited, 26 Cheltenham St, Bath, BA2 3EX.

Our offices are located on the Lower Bristol Road in Bath, a 10 min walk from the train station, next to the Green Park Tavern.

We are continually expanding our provisioning team, so if you are interested in working with us, please get in contact.

Up until 14th August 2008, Twitter provided a free SMS alerts service for its users in the UK. From inside Twitter, you could register your mobile phone, and decide which of your friends you wanted to follow via SMS. The cost of providing this service for free was one that Twitter couldn’t continue to carry, and the service was withdrawn as a result.

At Gradwell.com, we’d recently launched a new service news website so that our customers could be kept informed of alerts and maintenance of our major services (broadband, email, hosting and VoIP). The new service published updates to Twitter, so that our customers could use Twitter’s SMS alerts for when they were away from their computers. With Twitter no longer sending out SMS alerts, we needed an alternative. So we built twittex!

To get started, create an account at twittex.com, add your mobile phones and your twitter accounts, and some credit (£1, £5, £10 or £20), and you’re all set.

1. We documents all our system changes, so if something is planned, or has changed, we have any easy reference for it.
2. We write up internal case studies of our customers, and feedback from our birthday lunches
3. Weekly, on Friday afternoons, we do a round up of what all the teams (tech, sales, support, billing) got up to that week.

Unfortunately we can’t make the whole internal blog public, but it is interesting to use a new tool I found, Wordle to produce a “word cloud” for that internal blog. A word cloud is an image made from your blog text which gives greater prominence to words that appear more frequently in the text:

We can also compare that to the word cloud that you get from the public Gradwell blog:

It’s good to see the themes are consistent:

• Lots of focus on happy customers, through tickets and Birthday Lunches.
• Lots of continued work on infrastructure and scaling, particularly email and storage, as well as migrating from legacy servers to our newer platforms.
• A huge amount of work being completed and delivered, with many small incremental improvements.

We’d be interested in your feedback as to how we’re doing, via our online survey. Thanks!

Gradwell dot com has been aware of this issue with DNS servers since early July, thanks to a story posted on the popular Linux.com website.  We immediately checked our DNS servers, and determined that our nameservers are not vulnerable to this attack, using the testing tools published by the security researcher who discovered this issue.

If you run your own nameserver at all, the current advice from security experts is to ensure that you have upgraded it to the very latest version available.  I personally use Linux Weekly News‘ excellent weekly security round-up page to keep track of all the reported security updates for the different Linux distributions; your Linux distribution of choice will also publish regular security advisories.

I hope this clarifies the situation for our customers.  If you do have any concerns that I have not addressed, or at any time believe you’ve spotted a security problem that we haven’t addressed, you can raise an incident with our Customer Services team in the usual way, or contact me directly as firstname.lastname@gradwell.net.

Best regards,
Stu
–
Stuart Herbert, Technical Manager, Gradwell dot com Ltd

Please visit our porting section or contact our porting team for more information about porting your Virgin Media number to Gradwell!

Technically:

• We’ve been working on the stability of the email file servers, and had a couple of weeks with no nagios alerts for them (touch wood). We also managed to speed up our backups and shorten the time window.
• We added a further 7 servers to our VMWare cluster, we have 100 Ghz and 168 Gig of RAM deployed under vmware now, and quite a lot more servers to migrate over.
• We’ve installed £48,000 worth of Cisco Switching gear in our London data centres, as part of a plan to upgrade and improve the redundancy of our core ethernet network, and agreed the purchase of further switch upgrades to our Telecoms network (worth EUR 120,000), to be implemented in July and August.
• We’ve implemented a new security policy, and restructured all our internal web applications to further partition and secure them.
• 4 Industrial placement students joined our development team (3 will leave this summer) and have completed their inductions.
• Stuart Herbert, our new Technical Manager, has been with us now for three months and is getting to grips with the wide scope of work we do.
• We’ve done huge amount of other minor fixes and improvements - with over 200 entries logged in our change management system, mainly on email, voip and billing systems.

Products:

• Our broadband product has launched and been well received, with every customer who has switched to resolve a VPN or VoIP quality issue having been statisfied! Premium Broadband does make a difference.
• We’ve worked on the implementation and testing of the new Tiscali Annex M service, an enhancement to our broadband range which will give customers a 2.5 Megabit upstream link - due 10th July.
• We’ve completed 999 test calls on our VoIP Platform
• We are looking at how we can package our expertise in VoIP, networks and products like Trixbox (and IP PBX) into a managed services offering.
• Zimbra testing continues. Zimbra is an email service that provides diary, calendaring and contacts integration, like Microsoft Exchange. Please get in touch if you would like to take part.
• We’ve continued to deploy our newsip VoIP platform for Centrex customers, with good results. This gives more redundancy, scalability, and the ability to view the status of other customers on the phone via Lit Lamps.
• We tested integration with SpinVox, a service which converts voicemails to text, and hope to roll this out in July 08.

Customer Service:

• The support team has been slightly restructured, merging into a broader customer service unit which covers technical support, customer service & billing and product delivery (e.g. number porting).
• As part of this, we are starting to assign support incidents to individual members of staff, so they take a more case orientated approach to resolving customer queries.
• Our new online customer portal now provides a view for support tickets, and will soon track the generic service issues/bugs we know about as well.
• We’ve built the prototype of the replacement for our Gradnoc status service, and hope to get that released in early July.
• The team of 11 continue to handle approximately 600 queries from customers per week, across both technical and billing queries.

Sales & Marketing:

• Sales have continued very strongly in the summer months, breaking a few of our previous records, which is always excellent news. Phone calls are up 79% since January 2008.
• We’ve telephoned nearly all our resellers this month (some 200) and driven 2936 miles in May talkin to them, bringing them upto date, getting positive fantastic and useful feedback - some times it’s just good to talk.
• We held the first Gradwell lunch in Bristol, meeting 15 customers, from one of the first, to one of the most recent. The experience was very enjoyable and again, customers. Come and see us in Portsmouth in July, or Edinburgh in August.
• Emails to promote Unlimted IAX Trunks, Broadband and Customer Lunches were sent out and good a strong response.

Hopefully this gives a brief taste of what has happened behind the scenes at Gradwell, and much more is already underway in July!

No personal data has been compromised (apart from an encrypted version of FTP passwords) and all sites have been restored as far as possible.

We have been investigating this and whilst we first thought that this attack was possible through the exploitation of weak customer passwords, we have now proved that the hacker was manipulating our ftp server username and password database, and adding additional usernames into the system in order to gain access to customer’s websites.

We have been able to successfully monitor this attack in action, which has allowed us to exactly understand what went on, how it happened and most importantly, the full extent of the attack.

The Attack – What happened?
A customer’s website was compromised, probably through some insecure php code, and the attackers uploaded some code which allowed them to browse a mysql database.

Our web servers have access to our management database, so that they can pull down the configuration for customer websites. It seems likely that at some point we inadvertently miss-configured the permissions on the configuration file storing the database access credentials.

From this, the attacker was able to scan the server configuration files and determine a mysql username on our web hosting database.

This user was able to edit the ftp server database table, and they created additional ftp usernames that mirrored customer’s usernames (but with different passwords), which they then used to access the customer’s website accounts. The extra usernames were then deleted, to hide their trace.

How do we know?
From our FTP server logs we have been able to identify which files were edited, and it is clear that the only change was to alter the index/home page on a website.

We were also able to add monitoring to our database servers and determine which username and password was used, and what changes they made to our database.

From our logging of the network traffic, we can see that no other data was misappropriated, specifically no personal customer data.

What have we done to rectify & prevent future occurrences?

• We have restored affected customer files as far as possible.
• We have improved the firewalling in our network, so that customer web servers do not connect to our management database to get their configuration data.
• We have undertaken a further precautionary password changing exercise.
• We have planned additional work, to implement automated scanning for future attacks on our ftp servers and web servers.
• We implementing further partitions to our database security, so that web server configuration and ftp access are secured using different users.
• We have fed the lessons learned from this exercise into our web cluster redesign work, to improve the protection we give to customer websites.

Conclusion
We would like to apologise to customers for any inconvenience this episode may have caused.

As you may know from following our blog, we are engaged in a program of evolving our email and web hosting infrastructure and problems like this highlight areas for improvement in our initial systems designs from a several years ago.

Finally, we would also recommend customers reset their FTP passwords, because, whilst they are encrypted in our database, there is the possibility that a 3rd party has a copy of that list, and they could be decrypted and used again.

This can be done online at: https://hosting.gradwell.net/login/ftpmanage?menu_req=75

The challenge has been to build a cost effective email platform which would allow us to offer a high level of uptime and protection against physical server and disk failure, for our low cost email hosting products. We have built a system using multiple storage arrays, on top of a VMWare Server cluster and have nearly completed the migration process.

Our email system consists of front end servers, which handle the connections from customers, using imap, pop3 and the sending of email, using SMTP. We also then have the backend servers, which store customers email files.

To recap:

• We announced the plan in December 2007, where we outlined the proposed developments for the first half of the year.
• In February 2008, we completed the deployment of 15 new servers, handling pop3, imap and smtp.
• One of the hurdles for deploying the new IMAP platform was that customer mailboxes needed to be restructured, so we completed a restructuring project of the 43,000 mailboxes we host, which was completed by 9th May 2008. Customers were largely unaffected by this process, although unfortunately a small percentage of customers had to reconfigure their email programs.

This completed the majority of the “front end” work, although we still have a handful of customers whom are logging into the old mail platform and we are going through the process of contacting those manually, to correct their settings.

The second phase of the project has been to migrate the backend of the system – the customers mailbox storage – to a more reliable storage platform.

• Previously, we put customers mailboxes on individual servers, however, this meant that it would take a long time to recover from a server crash and we were reliant on a single disk array.
• Therefore, we adopted a high availability file system from Redhat (GFS), which provided excellent resilience, but at the expense of performance, and it is still reliant on a single disk array.
• We have therefore migrated all customers mailboxes to a set of 9 file servers instances on top of our VMWare platform, which allows us to have quick recovery (sub 5 min) in the event of the operating system crashing, and, whilst the files are still stored on top of a single disk array, we can move them from one disk array to another with no down time.

During this process, the migration of customers mailboxes has been exacerbated by:

• Slow performance and unreliability on old servers.
• Capacity issues on our new VMware cluster, due to our accelerating the migration process, and licensing delays at VMWare, causing some of the new file servers to crash, particularly early in the morning, when the load has been highest.
• The need to migrate over a terabyte of storage in fairly short time windows.

During this migration period in May, customers have experienced short periods of not being able to access their email whilst we adjusted settings and moved mailboxes for which we apologise.

As a third phase of the project, we have designed a redundant backend server, which allows us to store customer mail files on two independent disk arrays (potentially in two locations) with sub-1 minute level fail over, and we plan to test this in late May, before retrofitting it to our new storage setup in early June if successful.

At the end of May had the following short term work to complete, and as at 2nd June, this work is done:

• We need to add servers and RAM to our VMWare cluster, to increase available computing power on the new mail cluster. Done.
• Migrate the remaining 200 mailboxes from our Redhat GFS Cluster. Done.
• Ensure all customers are using the new platform, and correct any old customer configurations. Done.
• Fixed an issue in the Linux Kernel, related to disk i/o timeouts under heavy load, which causes the file servers to crash. Done.

Finally, to take advantage of the new system customers should ensure they check their email from pop3.gradwell.net and use relay.gradwell.net as their outbound smart host.

Outstanding Issues

Out VMWare based solution is performing well, giving good speed and has resolved many of the issues customers were experiencing. However, a number of elements continue to need to be addressed Mid-Term.

• Implement the high availability solution for our storage servers, which we hope to complete by early July. We have a proof of concept in operation and are beta testing it (at 14th June).
• Implement a multi-site VMware cluster, giving us multi-site redundancy, which will be complete for autumn 2008. We have ordered the redundant networking equipment required to do this (at 14th June) and expect it to be implemented by mid-August 2008.