Categories
All Blogging Consulting iHeavy Newsletter

Mailchimp fraudulently charged my credit card for spambot activity… Really!

via GIPHY

Wait seriously, you ask? Isn’t Mailchimp in the business of identifying, and protecting us from spam? Uh, yes indeed they are.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

I’m still in disbelief myself. And while I got the problem cleared up in the end, I really have to share the story.

1. The precipitating event – a charge to my credit card

I love when I pay for a service, and their method of communicating with me is to charge my credit card. Of course I pay attention to when someone is taking my money, and I perk up.

At first I thought they were raising the prices again. They’ve done that recently, so I thought it was odd.

But sure enough I got an email with the following message:

Your account has been adjusted to another billing tier. Old plan $22.49, new plan $31.49.

Wait huh? I add about 2-3 new subscribers per day. How could this be?

Read: What I learned from 10 years of blogging

2. After digging I found spam emails

After looking at my list, I found that I had added 600 new subscribers last week in three days. How is that even possible? I wasn’t mentioned on BBC. That must be spam, I thought.

So I emailed support. They sent me all sorts of links, but didn’t seem to understand the issue. So I emailed back again and they said they were working on it.

Related: 6 Devops interview questions

3. Mailchimp communication – a warning

This *warning* is problematic. For one thing is it buried through various menus and pages. Only because I was looking for spam did I find it.

Plus Mailchimp doesn’t take responsibility.

In fact they kind of imply that I’m a bad actor here. Seriously? Is that how you communicate with your customers?


Warning

We noticed a 0.55% abuse rate on your campaign “Welcome Message”. This is above industry standards, so we strongly recommend you review your collection process, audience management, and sending frequency.

Internet service providers set strict limits on unsubscribe rates, undeliverable mail, and abuse complaints. Mailchimp is required to observe these limits. If your emails continue to generate high rates of unsubscribes, bounces, or abuse complaints, we may need to review or restrict your account. Please take the opportunity to address this now.

Read: High availability what is it and why is it important?

4. Can’t get someone on the phone

I did some google searching because I could not find the phone number. Turns out you *CANNOT* call Mailchimp. A lot of these services internet companies are going this route. Sure it saves them lots of money, but the customer service goes straight to the trash.

So I begrudgingly jump on a chat session. It took

Read: Service Monitoring – what is it and why is it important?

5. The chat transcript in full

Sean Hull
I've been hacked.

THEN MAILCHIMP CHARGED ME!

This is strange.

Does mailchimp protect me?

Mailchimp Support
We apologize for keeping you waiting and appreciate your patience. Our operators are busy at the moment. One of our agents will be with you as soon as possible.
Sean Hull
Thank you ... waiting patiently.
Mailchimp Support
We didn't forget about you. We apologize for keeping you waiting and appreciate your patience. Our agents are busy at the moment. One of our agents will be with you as soon as possible.
Sean Hull
thank you robot person...
how is the progress?
7 more! :)
4 more!
we are almost there!
Neo joined the chat
Sean Hull
hi neo
Neo
Hey there Sean, thanks for reaching out to Mailchimp support. Give me just a moment while I pull up your account.
Sean Hull
ok thank you
Neo
Alright Sean, what is the exact issue you are facing within your account?
Sean Hull
mailchimp charged me for fake subscribers.
if you look at my email list, you'll see it typically grows by 2 or 3 maximum per day
recently a hacker dumped 200+ per day into my list.
Mailchimp didn't monitor things, and then CHARGED ME to my credit card.
Does mailchimp protect me?
hi Neo, are you still there?
Neo
I'm still with you Sean. One of the main ways that Mailchimp prevents spam signups is through the use of ReCAPTCHA. This is a setting you can add to your embedded form from the "Audience name and defaults" page, which you can read more about here: https://mailchimp.com/help/about-fake-signups/#How_we_prevent_it
Sean Hull
ok. that is helpful. for the time being i enabled double opt-in.
but I also see that mailchimp has a WARNING.
about recent activity on my account, and possibly shutting it down. do you see that?
Neo
Are you referring to the "Account issue" that is referenced in the bar at the top of the screen?
Sean Hull
Warning

We noticed a 0.55% abuse rate on your campaign "Welcome Message". This is above industry standards, so we strongly recommend you review your collection process, audience management, and sending frequency.

Internet service providers set strict limits on unsubscribe rates, undeliverable mail, and abuse complaints. Mailchimp is required to observe these limits. If your emails continue to generate high rates of unsubscribes, bounces, or abuse complaints, we may need to review or restrict your account. Please take the opportunity to address this now.
This is what it says...
so to explain more...
first off this is fraudulent activity.
so I'm concerned that mailchimp would just charge my account, without warning of some problem.
and further, it seems that mailchimp *monitors* to INCREASE BILLING and monitors to DISABLE YOUR ACCOUNT, but they don't monitor to protect their customers.
Is that correct? Because if there is some type of monitoring I can enable, that would certainly be very helpful.
Also is it possible to DISABLE AUTO PAYMENT on my credit card?
Neo
For the sake of clarity, let's tackle your questions one at a time. I'm getting some more information for you at the moment and will follow up with you shortly. Thank you for your patience.
Sean Hull
thank you Neo.
you're awesome !
do i need *both* double-opt-in and RECAPTCHA? or is RECAPTCHA enough?
Neo
It certainly couldn't hurt to use both. Using double op-t in will help ensure higher engagement rates overall and less likely to present warnings such as what you've seen. Here is some more information on double opt-in: About Double Opt-In: https://eepurl.com/dyij4v
Sean Hull
i mean the warning is a mistake from mailchimp isn't it?
because these automated systems just sent that because of the hacking.
i feel mailchimp should be protecting me, so I'm confused by that.
I am a paying subscriber of the service. and the price has gone up in recent months. so i think we can agree there should be protection from spambots.
is it possible to disable AUTOPAY on my credit card? BC i don't want to get further fraudulent charges from mailchimp, because of a spam problem.
does that make sense?

Neo
The method of protecting your account would be through tools provided to avoid spam signups, which would be double opt-in and ReCAPTCHA. Additionally, our teams, such as Compliance and Billing, would be happy to look into your account with you to help resolve any issues you may be experiencing.

I would also like to let you know that I understand the situation you are facing is frustrating, so I will be submitting feedback on your behalf internally.
Sean Hull
anyway, could you scan my account for further spam signups? I think around july 19th there was a bump in signups of 80 people. I tried the segment method but couldn't find which day they were from.
Neo
Sure thing, allow me a moment to take a look.
Sean Hull
thank you Neo, i do appreciate that.
i mean at the end of the day I'm not an email spam expert, so that's why I pay for a service like mailchimp. to avoid problems and run a really clean list.
so for example that RECAPTCHA thing should be ON BY DEFAULT. probably that would have avoided all this to begin with.
also mailchimp should SCAN FOR SPAM FIRST. not charge customers first, then realize there is spam and charge back. Because that is a fraudulent charge. which i find super frustrating. I do realize these are all automated systems. But mailchimp should be more sophisticated to protect good customers like me.
Neo
I can certainly see how that is frustrating and would be useful to users such as yourself, so I would highly recommend leaving feedback on the matter via the "Feedback" tab at the right side of your screen.
Additionally, I’ll certainly be routing your concerns and feedback to our internal teams.
Sean Hull
Is that part of "chat comments"?
i left a good review of your help :)
okay thx again Neo.
I'll see if I can find that feedback tab
have a nice night :)

Neo
You can find it within your Mailchimp account on any page on the right side of the screen. Please don't hesitate to reach back out if you have any further issues, and have a great rest of your night.
Sean Hull
thx
laters

The remedy as you can see above was the enable recaptcha and also double-opt-in. That was fairly easy once I knew where to look.

From there I created a "segment" which is a collection of emails. And I selected the date range for the three days where I got spambot hit. And then clicked UNSUBSCRIBE for all those.

Why didn't Mailchimp do this for me? Read more to find out why I think they don't automatically fix this.

Read: How do I migrate my skills to the cloud

6. What Mailchimp did wrong

o They monitored the account to increase a SALE.
o They monitored my account to warn me about shutdown.
o They did not warn me about RECAPTCHA.
o They did not alert me when I got aberrant signups. When you see a 100x increase in signups, it doesn't take a rocket scientist to see that's a hacker of some kind.

o Then Mailchimp fraudulently charged my account!

Read: How to hire a developer that doesn't suck?

7. The sinister side

This charge could be innocent. A left over part of an automation system that hasn't evolved with the spambots. But I wonder.

From the help forums, they are clearly AWARE of the problem.

And the company *could* go and FORCE enable RECAPTHA for all lists. They could then email customers about the change, and for those who this poses and problem, the user could then go and manually DISABLE it.

They haven't done that. And certainly RECAPTCHA was not enabled by default.

Please don't call me paranoid when I say that there is a *huge* revenue stream to be had for users who fail to notice, this, get charged, and don't even know their own negligence. Think of all that revenue. My list has under 1500 subscribers, but others have 5000 or 10,000. Imagine how easily a *higher billing tier* could get overlooked.

Yes folks, it's dark.

And I'm not happy with Mailchimp now.

I'm happy to pay for a service, when it is done well. But I'm not a fan of these dirty tricks.

Read: 5 things toxic to Scalability

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don't work with recruiters

Categories
All CTO/CIO iHeavy Newsletter War Stories

When You Have to Take the Fall

Also find Sean Hull’s ramblings on twitter @hullsean.

One of the biggest jobs in operations is monitoring. There are so many servers, databases, webservers, search servers, backup servers. Each has lots of moving parts, lots that can go wrong. Typically if you have monitoring, and react to that monitoring, you’ll head off bigger problems later.

A problem is brewing

We, myself & the operations team started receiving alerts for one server. Space was filling up. Anyone can relate to this problem. You fill up your dropbox, or the drive on your laptop and all sorts of problems will quickly bubble to the surface.

Also check out – Why generalists are better at scaling the web.

As we investigated over the coming days, a complicated chain of processes and backups were using space on this server. Space that didn’t belong to them.

Dinner boils over

What happened next was inevitable. The weekly batch jobs kicked off and failed for lack of space. Those processes were not being monitored. Business units then discovered missing data in their reports and a firestorm of emails ensued.

Hiring? Get our MySQL DBA Interview Guide for managers, recruiters and candidates alike.

Why weren’t these services being monitored, they wanted to know.

Time to shoot the messenger

Having recently seen a changing of the guard, and a couple of key positions left vacant, it was clear that the root problem was communication.

Looking for talent? Why is it so hard to find a mythical MySQL DBA or devops expert these days?

I followed up the group emails, explaining in polite tone that we do in fact have monitoring in place, but that it seemed a clear chain of command was missing, and this process fell through the cracks.

I quickly received a response from the CTO requesting that I not send “these types of emails” to the team and to direct issues directly to him.

You might also like: A CTO Must Never Do This

A consultants job

As the sands continued to shift, a lead architect did emerge, one who took ownership of the products overall. Acting as a sort of life guard with a higher perch from which to watch, we were able to escalate important issues & he would then prioritize the team accordingly.

Are you a startup grappling with scalability? Keep in mind these 5 things toxic to scalability

Sometimes things have to break a little first.

What’s more a consultants job isn’t necessarily to lead the pack, nor to force management to act. A consultant’s job is to provide the best advice possible & to raise issues to the decision makers. And yes sometimes it means being a bit of a fall guy.

Those are the breaks of the game.

Want more? Grab our Scalable Startups monthly for more tips and special content. Here’s a sample

Categories
All Web Operations Website Basics

Service Monitoring – What is it and why is it important?

Data centers are complex beasts, and no amount of operator monitoring by itself can keep track of everything.  That’s why automated monitoring is so important.

So what should you monitor?  You can divide up your monitoring into a couple of strategic areas.  Just as with metrics collection, there is business & application level monitoring and then there is lower level system monitoring which is also important.

Business & Application Monitoring

  • If a user is getting an error page or cannot connect
  • If an e-commerce  transaction is failing
  • General service outages
  • If a business goal is met – or not
  • Page timeouts or slowness

Systems Level Monitoring

  • Backups completed and success
  • Error logs from database, webserver & other major services like email
  • Database replication is running
  • Webserver timeouts
  • Database timeouts
  • Replication failures – via error logs & checksum checks
  • Memory, CPU, Disk I/O, Server load average
  • Network latency
  • Network security

Tools that can perform this type of monitoring include Nagios,

Quora discussion – Web Operations Monitoring

Categories
All Business iHeavy Newsletter

iHeavy Insights 68 – Transparency

The analogy du jour for cleaning up the financial mess is that sunshine makes the best disinfectant.  The idea is to push for more corporate transparency as a cleaning agent upon our current financial troubles.   Whether this cleaning job will have longstanding impact remains to be seen, however it’s clear that transparency is good for markets and economic stability.
In computing that same sunshine can be put to work as a disinfectant as well.  Transparency is as important for your cloud hosted application or traditional servers alike.  So how does it work?
Your typical internet application consists of a whole fleet of servers working together to do work for you.  Unlike automobiles, bridges, buildings or even most electronics however, the construction is constantly changing.  In effect these are buildings that are always being built, and bridges always being expanded.  Due to their changing nature, their behavior changes as well.  That’s where transparency comes in.
There are a number of great historical data tools specifically designed to capture the myriad of different metrics on your servers and then analyze and graph that information for you offline.  We like offline because that means the monitoring itself won’t affect or impact the performance of your application and servers.  Some of the tools of choice today include Munin, Cacti, and Collectd.  They each have their own strengths and weaknesses in terms of installation, configurability and so forth.  What they all have in common though is the transparency they provide.
Once installed, they will begin happily collecting information and monitoring your servers, all day and all night long even while you are enjoying your sunday brunch.
Are you looking at an outage that you encountered yesterday at 11pm?  Did your customers have trouble ordering your products, or utilizing your service? Fire up your cacti graphs, and drill down to that time window, and then review the various metrics to see what they reveal.
Having the right information at your fingertips is the first step in being able to resolve troubles.  Only with the right information can you fix these problems, and serve your customers what they expect.  So follow the analogy of using sunshine as a disinfectant and shine some light into your complex cloud environments. Let transparency lead you to the root of the problem and clean it up before it touches your customers.

The analogy du jour for cleaning up the financial mess is that sunshine makes the best disinfectant.  The idea is to push for more corporate transparency as a cleaning agent upon our current financial troubles.   Whether this cleaning job will have longstanding impact remains to be seen, however it’s clear that transparency is good for markets and economic stability.

In computing that same sunshine can be put to work as a disinfectant as well.  Transparency is as important for your cloud hosted application or traditional servers alike.  So how does it work?

Your typical internet application consists of a whole fleet of servers working together to do work for you.  Unlike automobiles, bridges, buildings or even most electronics however, the construction is constantly changing.  In effect these are buildings that are always being built, and bridges always being expanded.  Due to their changing nature, their behavior changes as well.  That’s where transparency comes in.

There are a number of great historical data tools specifically designed to capture the myriad of different metrics on your servers and then analyze and graph that information for you offline.  We like offline because that means the monitoring itself won’t affect or impact the performance of your application and servers.  Some of the tools of choice today include Munin, Cacti, and Collectd.  They each have their own strengths and weaknesses in terms of installation, configurability and so forth.  What they all have in common though is the transparency they provide.

Once installed, they will begin happily collecting information and monitoring your servers, all day and all night long even while you are enjoying your sunday brunch.

Are you looking at an outage that you encountered yesterday at 11pm?  Did your customers have trouble ordering your products, or utilizing your service? Fire up your cacti graphs, and drill down to that time window, and then review the various metrics to see what they reveal.

Having the right information at your fingertips is the first step in being able to resolve troubles.  Only with the right information can you fix these problems, and serve your customers what they expect.  So follow the analogy of using sunshine as a disinfectant and shine some light into your complex cloud environments. Let transparency lead you to the root of the problem and clean it up before it touches your customers.

Book Review:  The Ascent of Money – Niall Ferguson

When I think back to the dot-com days, I recall euphoria in people’s eyes.  It was that excitement in the face of making boat loads of money off the stock market that I remember clearly.  It is the excitement of the gambler, the thought of taking the shortcut, of getting something for nothing.  I remember seeing that same look in people’s eyes when they talked about housing just a short few years ago.  Talk of flipping houses and making money without adding anything.

It’s after the bubble bursts that everyone starts to think clearly again.  The tide has receded and we are left wondering how there could be bathers who weren’t wearing bathing suits, while it’s now plain for all to see.

Niall Ferguson’s book chronicles money’s use through history both the good and the bad.  By putting the current financial mess into historical perspective, he offers us new insights into our current predicament, helping us chart the way forward.  For anyone wanting to understand the financial forces around us, this is definitely a book worth reading.