Wasn’t expecting to do one of these but I need to talk about the site outages that have been happening around the time the spoiler is posted to varying degrees the last few weeks.
A lot of you have been seeing database errors during the hours of 10pm-11pm on the night the spoiler is posted. You aren’t the only ones, I’ve had them too which has meant it has taken me several attempts to actually post the spoiler during those weeks.
For those of you who don’t know, much of the site’s dynamic content such as the posts, comments and polls is stored in a database which this site connects to and queries in order to generate the pages you see. Many other websites on the internet, especially those with dynamic changing content such as blogs like this work in exactly the same way. If the site can’t connect to the server hosting the database, it can’t display the content which is why the errors come up.
Those of you who were around last year will know how much I don’t like having site outages. It lets you guys down, isn’t professional and of course if it happens repeatedly you may go somewhere else for your spoiler. It also meant the social features of this site like the polls and comments were not working so the discussions weren’t happening as much whilst the site wasn’t working. I’ve taken pride in how much discussion is taking place on this site now about the shows, the spoiler and the results including during the final weekend when a lot of you came to the site to discuss the final even though there wasn’t a spoiler. The discussion and poll features however add to the amount of times the database gets queried on each page view which will not have helped these issues.
The reason these outages have occurred are more or less similar to the last outages that caused me to move to my current host, just on a larger scale and are simply down to the volume of traffic especially when a lot of you are refreshing the site to see if the spoiler has come in. The first time outages occurred last year my host at the time was struggling to cope with 10,000 visits over a Strictly weekend. Which is frankly not a lot of hits and they really should have been able to cope. My new/current host is starting to fall down now but the number of hits is getting much higher. During the semi final weekend there was over 323,000 views (including views at dave-thorp.me.uk where I temporarily mirrored the spoiler due to the issues but not including any views where you got a database error). My current host coped fine at all times apart from the peak time between 10pm and 11pm when a lot of you are refreshing the site to try and see if I’ve posted the spoiler. The lateness of that week’s spoiler probably also compounded the issues as more people were checking and refreshing the longer it took for the spoiler to come in.
I’ve been discussing the issues with my current host these last few weeks and their solution would be to move the site to a dedicated server with its own resources rather than the current shared servers on which resources are shared with websites other than mine. Whilst I accept that the site may need this level of resources at the peak time when the spoiler is posted and a lot of you are refreshing the site I don’t agree that it needs it the rest of the time when the current solution is coping just fine. The following graphs are taken from my logs both on the site’s back end and through google analytics.
This graph shows the views on a day by day basis. It doesn’t take too much to realise that the peaks are the days of a Strictly weekend. Mondays to Fridays are significantly quieter as you can see. You can also see how much traffic has increased week on week (excluding Blackpool which is always an anomaly as the spoiler is usually about 90 minutes later)
This graph from google analytics shows the page views over the whole of the semi final weekend each hour. As you can see it starts to peak whilst strictly is on, building up after the live show has finished with a massive peak between 10pm and 11pm when you are all trying to view the spoiler. It tapers off overnight with a slight peak in the morning when people wake up, steady traffic throughout the day (although the graph doesn’t reflect this properly due to the size of the main peak) and another slight peak during the not live results show (not sure why I get this – maybe people are checking if I was right or possibly someone’s favourite has been in the dance off and they’ve wanted to check they survive it).
As I mentioned before my current host has suggested (and I am inclined to agree) that in order to cope with this peak for one hour a week the site needs the resources of its own dedicated server. This costs £249+VAT a month with my current host (although in researching I have found cheaper alternatives). This is an awful lot of money especially for something that I only really need at the moment for 1 hour a week for 3 months of a year. I currently pay £7.19 a month to host this (and all my other) site(s) so there is a significant difference in costs.
So what am I going to do? Well in researching my options I think I have come up with a solution that will give me the best of both worlds. I’ve found flexible servers which will allow me to adjust the resources available to them on the fly and pay per minute for the resources I use. So I can have the resources set pretty low during the quieter periods but increase them as needed during the traffic spikes so I get the power I need when I need it but only pay for it for the few hours a week I actually need it. Operating this way would mean I could run a server every bit as powerful (if not more powerful) than the server my current host is proposing but only pay for it when I need that power. This would bring my hosting costs to something more in the region of £40-£50 a month whilst Strictly is on and a broadly similar level to what they are now when Strictly isn’t on which is obviously a lot more manageable than the £250 a month currently being proposed. £250 a month would be a huge struggle for me even with the ad revenue I currently receive and even if I only used a dedicated server during the Strictly months. There would also be £100 setup fees with my existing host that I would have to pay each year (unless I kept the server all year round which would be completely unaffordable) adding further to the costs.
I’d like to also add that I’m not criticising my current host in the slightest. I was very critical of my previous host and their inability to cope with what at the time were very minor traffic spikes of a few thousand views. My current host has coped fantastically with spikes of tens of thousands and occasionally a hundred thousand. It’s only now that we start getting into several hundred thousand views during a spike that they’ve started to struggle and that’s not really their fault. I am getting an awful lot of traffic now to this site and it’s just clearly outgrown cheaper shared hosting solutions. I will likely keep my other sites hosted with them as it provides me a separate backup site just in case issues hit like I’ve made use of this year and it also means that I can dedicate the resources of the new server to this site which is the one that needs those resources. I can still recommend my current host which is Vidahost for anyone with moderately high traffic sites looking to keep their hosting costs low. Sadly this site, at least during the main period where traffic spikes, has outgrown them.
Either way there won’t be any move in hosting until at least August/September next year. Traffic to this site will drop significantly now the series is over and won’t start to peak again until the 2016 series of Strictly so there is little need to switch hosting until nearer the start of the next series. Not to mention there’s also the chance that the BBC might do something radical like a live results show, especially now I have their attention, and there would be no point in moving hosting if the main thing that draws people to this site suddenly becomes redundant although as final weekend showed, quite a few of you will still come to the site even without a spoiler as there were almost 40,000 views over the course of the final weekend and significant discussion taking place during and after the final which is testament to the community that I’ve built!
In the meantime thank you for all your support especially when the site has had difficulties. The site continues to grow at a rate that is both surprising and occasionally alarming to me and the site’s unusual traffic pattern makes it difficult to find a solution to the site’s growing needs that wouldn’t be a waste of money during the quieter periods. However I think I have found the ideal solution and hopefully next year the site will be able to cope with pretty much anything you guys throw at it!