SRV1 Downtime Post-Mortem More →
May 9th, 2012
As many of our customers will already be aware, we had an extended outage affecting one of our servers this weekend caused by a hardware problem. Now that everything is mostly back to normal, we wanted to explain in detail what happened.
In the second half of last week, we had a couple of outages on SRV1 where the server slowed down and stopped responding. The same thing happened again first thing Saturday morning, with the exception that this time the server’s logs were filled with disk IO errors. We immediately alerted the on-site technicians at the datacenter, and the server was taken down while the hard drives were scanned. This scan found errors on one of the server’s hard drives, and the decision was made to immediately replace that drive and rebuild the RAID array. Ordinarily this can be done with no downtime while the server is running, since the drives are hot-swappable, so we were anticipating a nice straightforward hardware replacement. As a precaution, a full backup was started to make sure that our backups were as up-to-date as possible before any work was carried out on the drives.
After the backup process had been running for several hours, but before it could be completed, the server stopped responding again. This time, when the server was brought back online it was clear that the filesystem was now in a terrible state and that the server was barely functional. It now looked like the problem was not limited to just one hard drive, but two of the server’s drives (out of four total) needed to be replaced. This meant that simply swapping in new hardware and allowing the RAID array to rebuild itself was no longer an option and we needed to re-install the server’s operating system and restore from our backups.
The hard drives were no longer working well enough for us to finish the backup which had been started that morning, but we were able to merge the partial backup with an older backup to restore as fresh and complete copy of the data as possible. Unfortunately the older backup was several days old; the most recent scheduled backup had been interrupted by one of the earlier outages, which meant the newest full backup we had was from the first half of the week. Merging with the partial backup helped, but we were not able to avoid the loss of some of the newest files on the system which had been uploaded in the couple of days beforehand.
Re-installing the operating system took a few hours, and we then started the process of restoring the backups. This process took over 36 hours to complete, during which time most web sites (and email) were unavailable (individual sites became operational as they were restored, so some customers experienced less downtime than others). In total, the time from the start of the outage on Saturday to when the backup restore finished (on Monday) was about two and a half days.
This incident has highlighted a few issues which we will be addressing:
- While RAID 10 is resilient when it comes to hard drive failures, it is not indestructible. We will be immediately setting up more detailed proactive monitoring to see if we can get earlier warnings of hard drive problems in the future.
- Our scheduled backups were not as up-to-date as we would have liked. The system is supposed to provide daily backups, but the backups take so long that this target can sometimes be missed (and if a backup is interrupted it can take too long to catch up). The backup system needs to be improved so that backups finish more quickly, and so that we can provide better than daily backups.
- The process of restoring the backups took far too long. This is the first time we’ve had to perform a full restore using our current backup system, and we were unhappy to see how long the whole process took. In an emergency, we need to be able to restore everything in a maximum of one or two hours.
To address the second two points, we will be investing in a more sophisticated backup system which should be operational by the end of this week and will not only allow us to perform multiple backups per day, but will reduce the total time to recover from this kind of incident to a fraction of what it took this time.
We sincerely apologise for both the duration and the severity of this outage, and we want to thank our customers for being patient while we worked to restore service.
November Update More →
November 14th, 2011
It’s been a little while since we made a blog post however things have been anything but quiet here at Valcato towers so I thought I’d update you on what we’ve been up to.
We’ve just brought a new server online, it’s the latest in our new 16 core, 12 GB RAM infrastructure with RAID 10 hard drive redundancy providing plenty of processing power to serve your websites at blistering speeds. To continue our naming convention this new server is designated “SRV2″ and all new accounts will be provisioned on it. We’ve also moved some accounts from our existing SRV1 server across which has alleviated the performance problems some of you have noticed during peak times.
To complete our migration to the new infrastructure customers currently hosted on SH1 will be migrated next weekend. We’ve emailed all customers involved explaining the simple process, but if you have any questions feel free to get in contact.
Next I have some unfortunate news, VeriSign – the administrators of .com and .net domains – have announced an increase in their wholesale price. In the past we’ve absorbed price increases but this time we are forced to pass it on. So on the 13th of January 2012 the price of .com and .net domains only will be increased by $1.
We’ve got some more exciting announcements to come in future, including free website optimisation, local proxies, access to a global Content Distribution Network (CDN) and some great Christmas Promotions. So stay tuned!
Updated VPS Options More →
September 7th, 2011
I’m pleased to announce that we’ve just updated our range of virtual private servers. We’ve vastly simplified the range of options available, and we’ve moved to a new platform which provides even better reliability and performance than before.
Some highlights from this update:
- Cloud storage is now optional – you can now choose local (RAID 10) storage if you want high performance.
- IPv6 ready – all virtual private servers come with the option of a free block of IPv6 addresses.
- Expanded US locations – we’ve had to remove our European locations (for now), but we now offer a choice of four US locations covering a larger geographical area across North America.
Click here to read more about our VPS options.
.Eu on sale for $7 More →
August 3rd, 2011
Target the European region of over 500 million people with your own .eu domain. Now with this exclusive promotion you have the opportunity to register your very own .EU domain at a highly competitive rate.
This promotion runs until the end of September, new registrations are slashed from $12.00 to just $7.00 per year!
Shared Cloud is Now Valcato Shared Hosting More →
July 20th, 2011
A few months ago we launched our updated range of shared hosting plans, with increased storage quotas, unmetered data transfer, and improved data backups. We also added a layer of cloud-based virtualisation, allowing for sophisticated hardware redundancy with automated recovery in the unlikely event of a hardware failure. This redundancy comes at the expense of disk performance, but during testing we found that the system performed well within acceptable limits so we decided that it was worth the performance hit to gain the extra redundancy.
As is often the case, what happens in testing does not always reflect what will happen when the system goes live. Although we managed to resolve our initial teething problems by moving to a different cloud platform, and everything has been operating smoothly since, we still haven’t seen the level of performance that we would like to see from the system; performance has been acceptable, but not excellent.
As a company we’re never happy with simply being “acceptable”, and we’re always looking for areas in which we can improve. We’ve considered various options over the past few weeks, and we’ve come to the conclusion that we can vastly improve performance while still maintaining an improved level of hardware redundancy (as compared to our services before the launch of Shared Cloud) by using a RAID storage system rather than cloud-based storage. As a result, we’ve decided to remove the cloud virtualisation and storage layer and move to a more traditional shared hosting system backed by RAID storage.
We’re renaming the service to reflect this change. As of today, Shared Cloud is now known simply as Valcato Shared Hosting. All customers who sign up from now on will be set up on the new RAID system, and we’ll be moving our existing customers to the new system over time, though let us know if you wish to be moved early.
.Nz Prices Slashed More →
July 15th, 2011
We are pleased to announce that the price of all .Nz domains have been slashed by over 60%! You can now register .co.nz, .net.nz or .org.nz for just $25/year which makes Valcato one of the cheapest places to register your New Zealand domain. To put it in perspective, GoDaddy charge nearly $60!
So head over to the Domain page and register today, or if you’re paying too much for an existing domain now is the perfect opportunity to transfer it to Valcato.
.Name Wholesale Increases More →
June 24th, 2011
Following in the recently trodden footsteps of other registries, the authority overseeing the .name TLD will be increasing the wholesale cost of registering domains. Unfortunately the increase is of such proportions that we are being forced to increase our price in response.
So from the 30th June 2011 our price for new registrations, renewals and transfers of .name domains will increase by 1 dollar to $12/year
Other TLDs are unaffected.
Announcing Shared Cloud More →
April 26th, 2011
Please see here for an update regarding Shared Cloud.
Today we’re pleased to announce the availability of Shared Cloud, a new service which replaces our previous shared hosting service. Shared Cloud is designed to have the ease and flexibility of shared hosting, with the power and reliability of cloud computing.
Key feature highlights:
- No unnecessary limits – we don’t place artificial restrictions on your account.
- As simple as legacy shared hosting – your account functions in exactly the same way as any other cPanel-based shared hosting account.
- Automated hardware failover – hardware problems are detected automatically, and services are immediately moved to new hardware to avoid disruption.
- Redundant daily backups – your data is backed up daily through two separate backup systems.
All existing shared hosting customers will be automatically upgraded to Shared Cloud in the coming weeks. We’ll be in touch with everyone individually with more details.
Visit the Shared Cloud page to learn more.
Testers Wanted More →
April 6th, 2011
We’re working on some new things (to be announced later this month) and we’re looking for a few volunteers to help us test them. Ideally we’d like to pick a handful of people with a range of different web sites and content management systems. Volunteers must be existing customers of ours, and if chosen will receive the following in return for helping us out:
- Free web hosting for the duration of the testing — likely to be a week or two at most (i.e. we’ll extend your renewal date by that much).
- A $5 account credit once testing is completed, which can be used against future purchases and renewal invoices.
If you’re interested, just open a support ticket letting us know and we’ll let you know more of the details.
Also, on an unrelated topic, some of you may have noticed a few problems with our web site over the past few days. We’ve had some teething problems with the new server we moved our site to recently, but it should all be fixed now.
2011 Survey Results More →
March 27th, 2011
100% of our customers would recommend us to a friend, and 86% of those surveyed have done so already!
Over the course of the past two months we have been grilling our customers in our annual customer survey to gather their honest opinions on our service, support and plans for the future. The results are in and we are pleased to announce that for the second time our customers have been so overwhelmingly positive about our service. We’ll be emailing our customers directly with responses to some of the most frequently raised points.
You can find some of the comments received on our customer reviews page and a selection of the results below:
The speed your website loads
1 meaning slowly, 5 meaning extremely quickly
The reliability of your website
1 meaning unreliable, 5 meaning extremely reliable
The flexibility of your account
1 meaning not at-all flexible, 5 meaning extremely flexible
Valcato staff worked on my issue until it was resolved
1 meaning strongly disagree, 5 meaning strongly agree
Valcato staff answered my question in a timely manner
1 meaning strongly disagree, 5 meaning strongly agree
Have you ever recommended us to a friend?
These results can be compared against our 2008 Customer Survey and 2009 Survey Results, as you can see there is a dominance of turquoise blue which means the vast majority of our customers rated us 5/5 in these key areas. We welcome feedback at any time of the year and have a special address set up for your comments: feedback@valcatohosting.com.