The BBC have just told me we are all doooomed in 6 years, and it turns out that the council isn’t in a much better state (http://www.theregister.co.uk/content/63/36161.html) well it’s 29.9 minutes longer than us.
On a more serious note, if something like this makes news in the IT industry, we are a huge story waiting to happen?
For a while now we have been running with no UPS, because the thing is more flakey than the real supply. We have lost power at least once because of this, but we lost power to our machine room once because of this, but we have lost power more times when the ups and backup generator was in, because of dodgy wires and underspecified kit.
Currently we have not offsite solution for most of the data in the machine room, we have just got a budget to spend a small fortune on two EVA’s to split our data across sites, but it is seriously looking like we won’t be able to do that for the main systems like the student database because it’s just to darn expensive and complex to get running.
Our problem stems from budgets, as a department we have wanted desperately to fix this problem, but power falls under the remit of estates, Who have other priorities like roofs not falling down, and buildings not catching fire. So we don’t get this fixed. In the end the solution for us, has been to get funding for a new machine room, so we can put in the price a new UPS solution. This does mean moving all our expensive kit from the former bullion vault two floors underground to a less secure above ground floor of the same building, but hey it avoids awkward internal politics.
To be fair to us and the council they probably do have offsite backups, but like us not offsite kit. So it would only take the time to get replacement kit restore the systems rewire their network and do full integrity checks on the systems before they would be able to be back up and running.
And all this is far less far fetched than that stupid If program on the BBC, as it doesn’t require long term 6 year weather forecasts and predications on the world economy (I am a bit upset with the BBC dumbing down news!)