Lightning Strike KOs Amazon, Microsoft EuroClouds 189
1sockchuck writes "A lightning strike has caused power outages at the major cloud computing data hubs for Amazon and Microsoft in Dublin, Ireland. The incident has caused downtime for many sites using Amazon's EC2 cloud computing platform and Microsoft's BPOS (Business Productivity Online Suite)."
So Cloud v Cloud.... (Score:5, Funny)
...nature wins?
Re:So Cloud v Cloud.... (Score:4, Funny)
Re: (Score:3)
And with all eggs in one basket you can be sure to crack them all in one punch.
Re: (Score:2)
Hmm, 20% of the eggs in each basket I'd say.
Re: (Score:2)
Sephiroth is rumored to have been stuck in the middle.
Re: (Score:3, Funny)
There is bad fanfiction depicting this event. Lots of it.
Re:So Cloud v Cloud.... (Score:4, Informative)
I live in Dublin, and that was some seriously targeted lightning. No sign of storms here, that I saw...
Re: (Score:3)
Re: (Score:2)
Re: (Score:3)
Yeah I just asked around the office here and no one heard any either. I'm beginning to wonder if someone was just playing with a set of balloons and had an unfortunate accident involving static...
Re: (Score:3)
Re: (Score:2)
"History shows again and again how nature points out the folly of men....GODZILLA!" - Blue Oyster Cult
Nature always "wins" (Score:2)
Nature bats last.
Re: (Score:3)
US is not another availability zone -- it's a different region. There are multiple AZs per region and -- if Amazon is doing their job -- a lightning strike should not take down more than one AZ in a region.
Re:So Cloud v Cloud.... (Score:4, Informative)
That might be true if Amazon didn't have multiple AZs in single datacenters.
The fault isn't necessarily Amazon's for stuff like this. The whole point of cloud infrastructure is that you use many cheaper instances to scale load and provide high availability caused by the failure of any one (or group of) node. Take Netflix, for example. While they do have their share of outages, they were completely unaffected by Amazon's big EC2 failure a few months ago, despite the fact that a significant portion of Netflix' infrastructure was hosted out of the affected region. Why? Because they built failure into their system, to the extent where they have a process that goes around killing random instances to keep them on their toes. They've planned for and built their system around the possibility that large chunks of the system might just up and vanish without warning.
If you're building a large-scale cloud system, *geographic* diversity should obviously be a part of any high availability plan. I'd also say that having provider diversity isn't a bad idea, but it seems like a lot of big cloud customers just stick with one provider.
Re: (Score:3)
I dislike "the cloud" term as much as the next /.er but right now its working as intended.
That seems to be true for Amazon; the outage is exactly what they documented in the case of the loss of a data centre. I'll give them (tentative) points for a job acceptably done*. You can't, however, say the same for Microsoft. They had a user visible application level outage for the loss of a single location. That's a screw up and clearly shows why you shouldn't trust anything "business critical" to just one cloud.
* we still don't have clarity about the physical separation between their generator an
Re: (Score:2)
You can't, however, say the same for Microsoft. They had a user visible application level outage for the loss of a single location. That's a screw up and clearly shows why you shouldn't trust anything "business critical" to just one cloud.
Don't you mean 'you shouldn't trust anything "business critical" to Microsoft'?
Re: (Score:2)
Re: (Score:2)
That would be an Illudium Q-36 explosive space modulator, not a lightning strike.
Re: (Score:2)
"Epic Zapp Battles of History"
FF7/Star Fox crossover (Score:2)
Re: (Score:2)
The biting humor is you also spelled it wrong. Considering it was a reference to The Highlander, the correct spelling would be with only one "L", not two. MacLeod.
Re: (Score:2)
yeah... obvious typo... it was too early to be typing accurately... ... but I should have known all along because...
"There can be only one!!!... L in MacLeod"
Just imitiating Verizon. (Score:5, Funny)
My Sympathies (Score:4, Insightful)
Considering that my radio stations have been getting hammered for weeks now by this horrible weather in the Southern United States, my sympathies are with them.
I don't care how much protection you put on your system (and when you have giant lightning rods that are hundreds of feet tall, like we do, you DO try to protect things), an occasional strike is going to slip through. When it does it can get ... messy. :)
Re: (Score:3)
"When it does it can get ... messy. :)"
I'm often amazed at all the weird failures even when lightning doesn't hit directly and just induces currents.
Used to see long RS232 runs that woudn't fail instantly, but would act flakey soon after a near strike, and then take a day or two to fail completely.
Re: (Score:3)
And that, my friend, is why the good lord gave us RS-422 to use for long runs instead RS-232.
Balanced, differential signalling and sensible grounding FTW.
Re:My Sympathies (Score:5, Interesting)
Even with ball gaps, chokes, and all the other effort, ultimately the transmitter has to be connected to the tower. 50 ohms is not that much different than "the shortest path to ground" when you put a few thousand KV against it.
It took several years after my career change to enjoy the spectacle of a lightning storm
Re: (Score:2)
And those 50 ohms aren't resistive ohms either, so the lightning just ignores them.
A lightning can be very powerful - a 2" steel tube can get disintegrated completely by it.
Re: (Score:2)
The component failure mode was "disappearance".
The Cloud (Score:4, Funny)
Sounds about like
http://xkcd.com/908/ [xkcd.com]
Don't say I didn't warn you! (Score:2)
Re: (Score:2)
Re: (Score:3)
Re: (Score:2)
Weren't those the same zones that turned out to be hosted in the same facility? Or am I misremembering that.
Re: (Score:2)
Looks like the old "Good, fast, cheap: pick two" adage might need a little rewording. How about "Fast, reliable, cheap: pick two"?
Re: (Score:3)
Looks like the old "Good, fast, cheap: pick two" adage might need a little rewording. How about "Fast, reliable, cheap: pick two"?
Since when was "reliable" anything other than one of the classic metrics for "good"? The old adage needs no changes at all.
Re: (Score:3)
Re: (Score:2)
Rubbish. We stripe across two EC2 zones. We would always want at least two servers anyway in case of any server failure. We still had 100% uptime over last night because failover worked correctly.
We are now seeing servers come back up gracefully and so far have not had to take any remedial action - a watching brief.
And it's cheap.
Re: (Score:2)
In my capacity as a Certified Solution Architect(tm), I often warned that The Cloud was suitable only for dynamic workloads. But did you listen? Oh, no, you just went and let your static workloads build up in the Cloud, increasing TCO and, now, bringing down Disaster on your heads!
Let me see if I understand you. Static buildup in the clouds caused a sudden discharge redistributing the static to a different cloud with its own static buildup, which then failed to discharge its duties after being charged by the other cloud's discharge.
Re: (Score:2)
It just isn't the same without a blond in there somewhere.
Cloud fail (Score:5, Insightful)
My understanding of the point of cloud computing was that it would be distributed. I.e. the failure of any one data or computing center would mean the data was still available. Hence, the term "cloud": nebulous, non-localized. Apparently, someone forgot to tell Microsoft and Amazon what the buzzwords they were using actually mean. I more or less expected that of M$, but the fact that Amazon failed too, well, thats pretty a little surprising. I guess it's kinda the norm for all large corporations.
Glancing at the article, it looks like this outage effected only a certain area, but still, cloud should mean other data centers would take over. I particularly love the quote "Dublin has become a key cloud computing gateway." If one city serves as a "gateway", its not a cloud system. I understand using it as one data center, but others should take over automatically for that area in case of a failure. If you don't have a failover system, you don't have a real cloud computing platform. You have a wannabe cloud computing platform. Or maybe they are just taking a buzzword and redefining it to suit their purposes. That's... exactly what we should expect, I suppose.
Or am I completely misunderstanding the meaning of this latest buzzword? It's quite possible, I never quite got down what "Web 2.0" was supposed to mean either. Beyond lots and lots of Flash.
Re:Cloud fail (Score:5, Informative)
For EC2, it's only distributed if you pay to have your "service" running in more than one availability zone.
Re: (Score:2)
As near as I could tell, Web 2.0 boiled down to one thing: The HTTP request object in javascript.
Where I work, people who are normally major control freaks are seemingly eager to let their processing and data storage go to The Cloud. It will be interesting to see how long The Cloud is down from this lightning strike. The control freaks don't like it when a company web app isn't accessible for even a few minutes, let alone an hour.
Re: (Score:2)
As near as I could tell, Web 2.0 boiled down to one thing: The HTTP request object in javascript.
Well that was the main thing but greater maturity of DOM, HTML and CSS played their part. Even before XMLHttpRequest turned up it was still possible to do something analogous by doing a form submitting within an invisible iframe and parsing the result. XMLHttpRequest made things a lot easier although same origin policy was a huge pain. Most modern browsers allow cross origin requests these days but it's still a pain to set up.
Re: (Score:3)
Or am I completely misunderstanding the meaning of this latest buzzword?
The main feature of cloud computing is the ability to almost instantly increase your capabilites and capacities.
At its most basic level, you're talking about a pool of computing resources that can be doled out without regards to the underlying hardware.
There's no promise that this cloud (the aforementioned pool of hardware) is geographically distributed.
Like any other hosting, you have to pay extra if you want your data replicated at another hosting center.
Re:Cloud fail (Score:5, Insightful)
At the moment, my company is aggressively encouraging our customers to avoid the Cloud at all costs. Let me explain why.
Whilst the technology exists for the cloud to deliver fault tolerant distributed storage, when you choose to put data in the cloud, you are choosing to relinquish control of the data. You are placing it in the hands of someone else. Quite probably an organisation that you do not know intimately. Quite probably an organisation that is based in a different legislative region - probably another country.
You have little to no capacity to audit their system. You have little to no capacity to test their fault tolerance. And here's the sucker punch - you have little to no legal comeback for the consequences if something bad happens.
If your data contains any personal information about another person, you are placing the privacy of that person in the hands of an organisation you do not control, and upon whom you cannot enforce any legislative restrictions.
So, unless you are seriously geared up to investigate and audit your prospective cloud provider - and they are willing for you to do so, the only data you can safely put in the cloud is data that would be basically irrelevant to your core business anyway. Until the fundamental issues of privacy, security and accountability are resolved - or dramatically improved - placing core business data in the Cloud is a massive corporate risk.
They should not have called it the 'Cloud'. They should have called it the 'Arse' - because if your management are planning to stick their heads in one, they may as well stick their heads up the other. I don't imagine that 'Arse Computing' would be as popular though.
Re: (Score:3)
I quit paying attention to your explanation/rant after one particular choice of words:
avoid the Cloud at all costs.
I immediately envision a scenario where the cost of setting up management, infrastructure, and equipment is significantly larger than the cost of losing a portion (or perhaps even all) of a company's data or processing capacity. Rejecting cloud services as a viable option regardless of the actual cost is just as asinine as rejecting the option for turning on hot water in the bathroom sink, because it just might be too warm
Re:Cloud fail (Score:4, Interesting)
As for auditing, uptime, and legal consequences, you've apparently never dealt with a service contract. If the contract mandates five nines of uptime, and includes a clause making them liable for all damages and loss, that's a pretty hefty legal comeback.
I agree with you entirely, it's an absolutely beautiful piece of legal comeback. But every service contract I've ever seen is so full of ifs, buts and other assorted get-outs that it's very rare to actually be able to hold someone to it.
The one time I have seen an SLA that was actually quite good, the company in question didn't refuse to honour it. Oh no. They went one better - they hadn't even told their staff that it existed, so if you asked about it you'd get a response along the lines of "What's an SLA, then?" The only way you'll get an SLA honoured in those circumstances is to take your provider to court, and you can bet that if you do they'll drop you like a hot potato. So you probably wouldn't bother in any but the most egregious of circumstances.
Re: (Score:3)
(1) How would you not be backing up to a local resource?
(2) Why would you not be using encryption at your user site before sending data to the remote server?
If you're small, choose a small service like SpiderOak.
If you're large, build a custom front end.
I don't have particularly sensitive data, so I don't encrypt at my end, though in theory I could (but it would be a pain) and SO had issues with their distributed synchronization back then (note: if I were big enough, I could have hired an IT person to manag
Re: (Score:2)
Whilst the technology exists for the cloud to deliver fault tolerant distributed storage, when you choose to put data in the cloud, you are choosing to relinquish control of the data. You are placing it in the hands of someone else. Quite probably an organisation that you do not know intimately. Quite probably an organisation that is based in a different legislative region - probably another country.
Which is the real issue: No way for a European company to use a US cloud provider - Amazon, Azure, Google. The Patriot Act is prohibitive here. [google.com]
Re: (Score:2)
"At the moment, my company is aggressively encouraging our customers to avoid the Cloud at all costs. Let me explain why."
Because it is eating your profits.
The same lightning strike we are talking about could have just as easily taken out a in-house data center. I would also argue with that being the case the situation would have likely even been worse for the customer.
distributed != replicated (Score:2)
Re: (Score:3)
Perhaps it would be a good idea to start by defining what exactly "cloud computing" means?
Because looking at the Wikipedia article [wikipedia.org] I see only a brief mention of reliability through redundancy: Reliability is improved if multiple redundant sites are used, which makes well-designed cloud computing suitable for business continuity and disaster recovery.
As CTO of a data hosting "cloud services" provider myself, I'm proud of our track record for reliability and redundancy. All our systems are backed up offsite t
Re:Cloud fail (Score:4, Informative)
Cloud computing is a buzzword meaning "don't run your own hardware, run your business on someone elses". Which might mean anything from a virtual server that you manage at one end of the sophistication scale to a SaaS product at the other.
All sorts of aspects of this are optional. Including:
1. Whether or not you manage the underlying operating system - including things like security patches and hardening. You can choose a cloud computing provider that has sysadmins deal with that for you and just run the application yourself; they are a LOT more expensive than Amazon.
2. How much effort your provider puts into making their systems geographically redundant. Few will talk openly about this; I'm prepared to bet hard cash this is because the vast majority that offer you a virtualised server are just using a web interface to expose a fairly vanilla Xen-with-a-SAN infrastructure to the world with everything sat in one place. Providers that will run the OS for you and can honestly say their infrastructure accounts for complete data centre loss are like hen's teeth.
3. If you've gone for a SaaS provider - how much effort their developers went to to ensure their application can stand up to everything up to and including total loss of a data centre. And whether or not they test for such an occurrence.
Re: (Score:2)
Well, according to the trade press, "the cloud" is both a floor wax and a dessert topping (plus a bag of chips).
In reality, it's fine grained virtual server rental with an API. Nothing more and nothing less.
While that is quite useful if you need it, it doesn't magically fix any problems other than highly variable server loads. As we see here, it isn't a magic wand for uptime/availability.
Re: (Score:2)
In reality, it's fine grained virtual server rental with an API. Nothing more and nothing less.
That's exactly what IaaS is. Higher levels (notably SaaS) are a bit more; in those cases you're buying the service — wherever that's running $mdash; and not the server. You get less control but have to do less of the legwork yourself. It's the classic value-add model, and it makes a bunch of sense for many people. (Not everyone, but that's OK. Nobody can be all things to all people.)
Re: (Score:2)
You were lied to.
Re: (Score:2)
Oh silly underinformed person. There is a datacentre in Dublin. It is one of six Amazon datacentres. The others were unaffected, as were (our) public facing services, because only some (of our) servers are placed in Dublin.
It looks like a cloud from the outside. Those of on the inside know where the servers are because we want to choose where we place them for latency / redundancy reasons.
Re: (Score:2)
The selling point... (Score:2)
... And if we aren't 1000%, absolutely, positively reliable may God Strike Us... BLAM!!!!!!
Re: (Score:2)
... And if we aren't 1000%, absolutely, positively reliable may God Strike Us... BLAM!!!!!!
Sorry sir. The thousands place is the sign bit in our percent calculations.
Re: (Score:2)
(posting to undo wrongful mod, a bug in the system? dropdown mod combo box, use arrows to scroll, press escape to cancel, it then applies the last highlighted mod option :/)
was meant to mod funny btw :)
Re: (Score:2)
(posting to undo wrongful mod, a bug in the system? dropdown mod combo box, use arrows to scroll, press escape to cancel, it then applies the last highlighted mod option :/)
It's not a bug, it's a feature. If you want to cancel your mod, you have to unplug the RJ45 at the back of your computer first. RTFM for God's sake !!!
Microsoft renamed its product (Score:5, Funny)
Office 364
Re: (Score:2)
Or M$ could port it to Xbox (Score:2)
Serves them right (Score:2, Informative)
I'd call a few lightning strikes the least of the punishments those data centers - and the entire infrastructures to which they're attached - really deserve.
Re: (Score:3)
The AWS services out of dublin aren't through an Irish subsidiary. It's just regular AWS.
If you know differently, please document it.
Re: (Score:2)
The AWS services out of dublin aren't through an Irish subsidiary. It's just regular AWS.
I don't think that's what the GP was getting at.
Due to their complex tax [strike]avoidance[/strike] management plans, American based multi-nationals end up with billions of dollars overseas.
They can't bring those billions back to the USA without being taxed, which would completely negate the point of booking them offshore to begin with.
The end result is that they invest it overseas instead of domestically, because it looks better for their bottom line.
Just because AWS LLC isn't an Irish subsidiary doesn't m
Re: (Score:2)
Also, please not, that if the UK had more a progressive, enterprise friendly tax system......
A comparatively small country will always be able to set lower taxes for enterprises than their larger neighbors. This is because it only needs to attract a small percentage of the larger countries taxable income to offset the loss due to the lower tax rate, whereas the big country only lowers its income when it does this. Fiscal competition in an open market is a fraud against the people of larger countries, preventing them to set the tax rates of the companies that do business there.
Re: (Score:2)
Because it is for US citizens/residents, and corporations are people right?
Re: (Score:2)
Power Co-Generation (Score:4, Interesting)
To mitigate this threat, Chevron worked with Mississippi Power to operate a power generation facility at the refinery.
I would think that anywhere there is a substantial "data processing farm" with critical power requirements, business arrangements should be made with the power generation utilities to run a natgas power plant in the immediate area.
The utilities often run these plants as "topping" plants, as they needed anyway to even out short-time load variances on the line.
But, in the event of a serious loss of grid power, it can be awful handy to have a few megawatts of power coming from down the street.
Re: (Score:3)
"Normally, upon dropping the utility power provided by the transformer, electrical load would be seamlessly picked up by backup generators,â Amazon said in an update on its status dashboard. âoeThe transient electric deviation caused by the explosion was large enough that it propagated to a portion of the phase control system that synchronizes the back
Re: (Score:2)
When I was an undergrad I worked a crappy job at a Florida amusement park. Lightning capital of the world is in Pasco County Florida which was about 30 minutes away.
They generated their own power and the power lines and even the roller coasters were designed to be struck by lightning. Let me tell you they were struck every 3 or 4 days during the raining season in summer from the monsoons from the Carribean and Gulf. During a bad storm the power lines and rides could be struck 2 to 3 times each in a 30 minut
Re: (Score:2)
Yeah but how many times did the amusement park get taken offline before they came up with a solution. The old saying "it is not a matter of if, it is a matter of when" often applies. Unless compelled by some law, upper management often listens to the bean counters until something goes wrong ... then they start asking questions and opening the checkbook.
Re: (Score:2)
Some people may have problems with that kind of availability and prefer the risk of the datacenter going down 1 day over a period of years instead of going down every lightning storm. But they are just being overly picky I am sure.
Obviously (Score:2)
Come to the cloud... (Score:2)
How shocking! (Score:5, Informative)
2 things tto remember (Score:2)
Murphy was an Irishman
Every Silver Cloud has a leather lining.
THAT's NOT a CLOUD (Score:3)
If ALL your computing power is in ONE SINGLE DATACENTRE then what you have is a DAMP SPOT not a CLOUD.
Good! Now maybe people will know! (Score:2)
All of the cloud computing hype has business everywhere, once again, buying what they don't understand just as they did during the dotCOM bubble. That particular bandwagon caused all sorts of damage to the industry including a flood of people unsuited to the line of work and suppression of wages that don't seem to have ever returned. Now business continues to crave cheap, Walmart-ized IT services and are seeking to get it any way they can; outsourcing to 3rd world nations and most recently entrusting clou
Sounds strange (Score:2)
(see below) Why does synchronization have to be manual? Or is this a safety feature that the automatic synchronization decides to go offline if things get weird?
From the Article:
âoeNormally, upon dropping the utility power provided by the transformer, electrical load would be seamlessly picked up by backup generators,â Amazon said in an update on its status dashboard. âoeThe transient electric deviation caused by the explosion was large enough that it propagated to a portion of the phase cont
marketing bullshit (Score:5, Insightful)
And there is the marketing bullshit revealed. All the promises of the cloud - down by one lightning strike.
Because, let's face it, the whole "cloud" thing as they sell it is just advanced virtual hosting with a different name. The only real cloud capabilities are those the big companies build for themselves, and they did things like that 10 years ago already, when nobody had ever heard the term "cloud" used in computing contexts.
In the end, it's about selling something to people who already have the older version and convincing them to buy the new one. So you give it a different name because a "new" product sells easier than the upgraded version of an "old" product.
Anyone remember when "Web 2.0" was all the hype? It really wasn't a 2.0 as we all know. There was nothing new in it, all components had been around for a long time. It was a conceptual bundle, but not a new version like the name suggested.But "we're doing more Javascript" now doesn't sell nearly as good as "we're moving to Web 2.0 now".
Re: (Score:3)
But they weren't down. I have servers in Dublin. I also have striped redundancy across other EC2 datacentres. We had 100% uptime last night and I'm now watching the Dublin based machines recover gracefully.
Yes, I agree it's "advanced virtual hosting with a different name". But it didn't break its promises.
Re: (Score:2)
Yes, I agree it's "advanced virtual hosting with a different name". But it didn't break its promises.
That depends on what promises you mean.
"The Cloud" has been hyped as this mystical thing that means you just move your servers "into the cloud" (whatever that means) and mystically physical location ceases to matter.
And that promise was broken. You need to worry about things like redundancy, multiple data centers, etc. etc. - just like you did before. I've designed HA systems myself many, many years ago when "cloud" was something in the sky. We set up our own redundant upstreams and taught our routers BGP.
Re: (Score:2)
But it didn't break its promises.
I can read countless stories of people who use things like EC2 and express disdain toward users afflicted by downtime because they didn't understand how to work it. When all is said and done, the users using EC2 'right' seem to end up having to worry all the frustrating details of HA they worried about before, 'Cloud' doesn't change that. The 'promise' is that you naively slap your workload into a 'cloud' provider and you don't have to think so hard about all of that. Even if Amazon doesn't explicitly ma
Microsoft's BPOS (Score:4, Funny)
Business Productivity Online Suite?
I always thought it stood for "Big Piece of Sh..... never mind
Zeus Anonymous (Score:2)
at that level the safety's tipped foreing a manual (Score:2)
at that level the safety's tipped foreing a manual cut over to backup. Also Surge Protectors can't really take a direct lighting strike.
But any ways for the cloud to work you will need data centers all over the place for good lag and for back up. Now Dublin is good for not needing lot's of cooling but still you don't want to put all the severs in one area so other stuff like over seas data line cuts can't take down systems far away. If on side severs at least you can get some work done with no or a slow bac
Re:at that level the safety's tipped foreing a man (Score:5, Informative)
Also Surge Protectors can't really take a direct lighting strike.
But lightning arrestors can. [iceradioproducts.com] A serious lightning arrestor is a spark gap (sometimes open air, sometimes in an inert gas) to ground, with a very heavy cable or busbar to multiple ground rods, and no sharp turns in the path to ground. This is followed up by an inductor which is a few turns of busbar. This gear is usually placed where power lines or antenna feeds enter a building. MOV-type protection is further downstream.
Antenna towers are struck by lightning frequently, and the associated radio gear routinely continues to operate. This isn't rocket science. It's big hunks of copper.
The Hartford Steam Boiler Inspection and Insurance Company, in their publication "The Locomotive" (they've been at this since 1867) has a good article on lightning protection. [google.com] Hartford Steam Boiler insures not only against boiler explosions, but things like downtime due to lightning strikes. But only after their inspectors (they have 1200) have visited the plant and are satisfied with the equipment.
A question to ask your "cloud" provider - who handles your business interruption insurance, and do they inspect your faclities?
Re: (Score:3)
Have you ever seen a surge protector after a direct strike? The MOVs don't help much once they vaporize.
A surge protector is mostly useful against the more common near misses.
Re: (Score:2)
Have you ever seen a surge protector after a direct strike? The MOVs don't help much once they vaporize.
It's amazing how many people who call themselves engineers don't get that surge suppressors are non-resettable.
"what do you mean the power supplies got destroyed. We've had surge suppressors in there for years..."
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Is it somehow Amazon's fault that the developer didn't RTFM and understand EC2 is effectively the same as remoting into a system booted off a LiveCD with no hard drive attached? Great for computing rainbow tables or, indeed, farming BitCoins; rather poor choice for running a database.
Re: (Score:2)
This could result in another series of fuckups like this [ycombinator.com], where a bitcoin exchange lost its wallet.dat due to a misconfigured EC2 instance.
GEEZ I didn't know that.
I admit this is not very clear from EC2 users (that is unless you read the docs), but Amazon should have a provision where if it shuts down because of them, then no data is lost.
And btw use S3 and s3cmd, very good and very easy (even with encryption)