** UPDATE: Rackspace representative explain that what happened here is not supposed to be their standard process, see my latest comments below. Wish they will be more transparent on these issues in future **
This is a story about a web startup provide simple form / survey creation services, and it get totally shutdown by Rackspace Cloud since they "think" one of our user's form is phishing, and this may happen on your startup too! Some background information first...
Some Background
We run a web startup (PandaForm), which is a simple forms creation services which integrate with a CRM for small businesses to handle their customer's forms such as who signed up to what events, who ordered which products, etc. We were self-funded by doing web consultation aside. We have a few thousands users right now, and we have 10 servers on Rackspace CloudServer (and a Rackspace Cloudsites), Slicehost and Amazon Web services for our consultation clients and the web service.
I always love Rackspace Cloud over others due to it's price and performance, but I can never imagine this happens to me..
Time Line
This is 2010 Sep 1, 1:11am HKT, after 2 hours of down time, exhausted for apologize to my clients.... and this is what happened in brief (in Hong Kong Time):
- Aug 31, 9:57pm - An email sent from Rackspace Cloud, saying a ticket is created so I have to respond immediately. (I didn't pick this email up yet)
- Aug 31, 10:42pm - 2 servers running our service was taken down as reported by pingdom.
- Aug 31, 10:50pm - I read the email sent to me at 9:57pm on my iPhone, and shocked when I clicked on the ticket and read the content (see below) on my Safari. (The ticket didn't highlight my servers were taken down and the ticket system didn't work on my iPhone Safari browser anyway). I added a comment as a respond to the ticket immediately. Tried to explain that I run a web service and if one of my users is phishing I will be investigating, request to take my servers back online immediately.
- Between 10:50pm - 11:15pm:
- Tried to talk with support on live chat, it didn't work and they asked me to call their hotline;
- Talk on the hotline for 15minutes (was put on hold for 10 mins);
- They asked me to fix the server within 3 hours, and I insist that I need 12 hours.
- 12:01am - Server back online finally. I have a chance to look at the subdomain they're complaining (nothing special about it for me), and clicked the "Disable" button on our administrative panel painfully. Apologize and explain to my clients on email and phone.
- 12:54am - Got another email from Rackspace, saying "If we received 1 more complaint we will have to terminate the client account per our AUP policy and per legal policy. " Great! Now they force me to move my servers to other services provider (see the full message below)
Why am I so annoyed with it?
Fellows web startups founders -- imagine one day your users uploaded content got a phishing / copyright complain, and your server was taken down without notice within 1 hour -- all these things happened during mid-night and you got no explaination of what is the complain, who filed the complaint, etc.
I'm definitely willing to be responsible and to investigate when there are complaint to our users, but I think I deserve a better notice with more information, and more time to do the investigation. But not taken down your server in a hour during mid-night.
And why infrastructure provider have such a great power to terminate their client's servers? Isn't that the phishing complaint should be forwarded to the services provider? Is the infrustrature provider the best people to judge what's phishing or violated copyright and what's not?
At the end, thanks to those support people who help me and make me speak to the right people.
Lesson Learnt?
Now I've 12 hours to resolve the "issues", I've disabled the user's account but I don't know what will happen since they've warned with 1 more complaint they will take out my whole rackspace account. I will have to move all of my servers (including my customer's dedicated servers on the same account) to somewhere else, I don't wanna risk losing all of my servers so suggestions are welcomed.
So with Rackspace Cloud taking down my servers, I'm trying to ask all major cloud servers providers how they handle these incident before doing the migration, I will keep updating this table after I heard their feedback.
| When you received Phishing / Copyright complaint... |
Rackspace Cloud |
Linode |
AWS |
How will you notify us?
|
Email |
Email
|
|
Will you take down the server? How much time was given to respond?
|
Yes (in my case 1 hour before take down)
|
Yes, generally 24 hours are given for clients to verify the material were removed or the calm was false
|
|
Now I wish I don't have to wake up my whole team for this incident, but they wake up and trying to backup everything now on the servers... god bless us til tomorrow morning when we start migrating the servers!
How Rackspace notify me in their ticket system....
Hello,
We have been notified that there is a phishing site located on your server.
(http://[xxxxxx].pandaform.com/pub/[xxxxxxx]/new)
A phishing site is set up to steal personal information from unsuspecting consumers to use in instances such as identity theft. I am forwarding this to a member of your support team to disable at this time. Please contact us immediately so that we can resolve this issue.
Sincerely,
Renee Graves
Rackspace AUP
How Rackspace follow up...
Ben,
Thank you for calling in and raising concern about the phishing site on your server. I do understand that you are upset and dissatisfied with our customer support however we are responsible for thousands of other customers on our network and we have legal responsibilities as well.
I conferred with our AUP department and they confirmed that the customer who has a phishing site must take immediate action. If we received 1 more complaint we will have to terminate the client account per our AUP policy and per legal policy.
Again, I do apologize for the short notice however those are the terms and conditions you agreed to when you chose to host with the Rackspace Cloud.
Regards,
Harris Collins
Account Manager