Category: Consulting

Categories
All Consulting CTO/CIO Devops Startups

I’m building a startup. What should I watch out for?

via GIPHY

I’ve worked with close to 200 startups over the years. So I’ve seen a lot of stuff. Some of the get-rich-quick managed to get broke even quicker. And others saw great engineers leave behind a clunky maze of an architecture with little or no documtation.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Here are some thoughts on what to be mindful of…

1. Keep the stack simple

Are you asking yourself questions like these:

o if we build x,y,z it will save us time later
o we need kubernetes because everyone is using it
o let’s deploy on Amazon’s Cloud, everyone is there

If so you may have sidestepped keeping the stack simple. If you are building a proof of concept, mvp, chances are you do not need to build all the bells and whistles.

Relentlessly apply the discipline to keep things simple.

Read: Are pioneers and process people different breeds?

2. Use boring technology

You may be tempted to use some exotic new language. I know this new framework solves all the problems that have come before. No!

Let someone else live on the bleeding edge. I’m not suggesting to use COBOL, though there is still a surprising amount of that still around. What I’m saying is 5 year old tech is *not* legacy.

Not sure? Do a quick search on a job site for the language or technology. How many candidates do you get? If they are rarer you will pay more. And replacing your genius developer will be even harder.

But boring technology is not just about finding talent. It means many of the problems have already been solved. This is huge. There are proven ways to implement with the tried and true.

Related: Why is Martin Weiner special?

3. Beware of sharks in the water

Sharks can come in many forms. Investors that want to take 50%, a google or an Amazon that could rearchitect your solution in a week. These are real and present dangers.

Let’s not forget overspending on co-working spaces, and hidden costs around benefits, and surprising fees on your data or aws bill.

Related: How can 1% of something equal nothing?

4. Outsourcing – trust but verify

You may be able to hire one or two devs locally in new york, but if you need a larger team, you’ll likely look to outsourcing. There are tons of talented and brilliant engineers worldwide. So it makes sense to leverage this.

Beware though of handing all control to a team outside our shores. That’s because if you stumble into a legal trouble, you cannot use legal methods to resolve them. Some examples…

o what if you lost your aws keys and the outsource team holds you hostage?
o what if you are hacked, and the outsource team abandons the project?
o what if the outsource team tries to steal your IP?

I’m not suggesting here that outsource teams are any more likely to do these things than some firm locally. I think these events are rare in business. But if they happen local you can use the levers of the justice system to resolve. Remote may not lend to that.

Related: The art of resistance – when you have to be the bad guy

5. Relentlessly manage time

Your time is going to be short, and you’re trying to do the impossible. Use the tech wisely:

o use monitoring systems to alert you – no need to manually check things
o beware of slack and email – as they can destroy productivity
o automate where you can see real benefits
o keep the architecture simple!
o network for hiring people

Related: Is maintenance a forgotten art?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Cloud Computing Cloud Migrations Consulting CTO/CIO

What can we learn from NASA’s AWS fail?

via GIPHY

I was just reading the Register, which is sort of the UK’s version of Slashdot, and they had a jaw dropping title. NASA moved 247 petabytes into AWS and then later learned about EGRESS costs

OMG! Face palm. Wow.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

To say this is a disaster is an understatement. Could it have been prevented? Not likely by 100% strategic thinking. I believe a certain amount of real-world testing & prototyping is the only way.

Here are my thoughts…

1. Expect hidden costs

Everytime I check there are more AWS services. Just now I did some googling, and the number stands at 170. Not only is it tough to keep up with all of those, but the offerings are constantly evolving, getting new features and so forth. That means the pricing and costs are evolving too.

All this means an infinitely complex web of interconnecting pieces, so it is near impossible to predict prices in advance. The solution? Prototype.

And this would have helped save NASA. Because you would build, feature test and load test too. In all of that would have come a small estimate of cost which would include EGRESS costs.

There are no guarantees in this game, but it is surely getting complicated.

Read: How can 1% of something equal nothing?

2. Vendor lock-in is not dead

With the receding of some of the big old world vendors like Oracle, many have forgotten the shark like tactics they used with startup companies.

The model was something like this. Send in the big guns, nicely dressed to get you on board. Finesse the sale. Offer deep discounts, and get the customer on Oracle. After a year, maybe too, start squeezing. You’d be surprised how much blood comes out of diamonds.

These days we feel more free to port our applications to different cloud vendors. Even if mostly everybody is on Amazon already. But this NASA story really highlights the great organizational cost to migrate to the cloud. You architect your application, you do cost planning, and so on. So once you’re there, it’s hard to unravel.

Related: Is Fred Wilson right about dealing in an honest, direct and transparent way?

3. New possible hacking vector

Since costs are tied to usage in the public cloud, this could have implications for hacking. If a bad actor wants to cause you harm, then can now just use your service more.

Don’t like company A? Write some bots to access them from obscure locations, and ramp up those egress costs. With all the complexity of the cloud, are most firms monitoring for this sort of thing? I don’t see it in my engagements.

Something similar happened to me. I wrote: when mailchimp fraudulently charged my credit card. It really happened. Do I think it was intentional? You’ll have to read the article to get my 360 degree take on it.

Related: What mistakes did you make when starting as a consultant?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Consulting

What happens when a bartender doesn’t get the job, but files a lawsuit?

via GIPHY

I stumbled on this interesting article, and thought I’d share it. Bartender doesn’t get job, but wins suit for consulting fees

Now some of you may have already made some prejudgements. But I ask that you hold your conclusions, and take a listen.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Now let’s for a moment, distinguish big firm consultants, with independent consultants. I’m speaking about the latter, the freelancers of the world.

1. Time spent evaluating a consultant

If you’ve hired a consultant or freelancer before, you know you’ll spend time evaluating. You’ll talk about your business problem, and they’ll share how they can help. They may even start brainstorming with you. This alone can be valuable to a firm, as it can give them new ideas and new perspectives to dig into their problem.

Notice too, that while you are a fulltime employee, all that time on the phone, and at your desk, you are getting paid. While that freelancer, is *not* getting paid.

My point is not to complain here. Just merely to point out that there is a lot of work that happens before you are paying your consultant. Before they are even billing. Whether it is preparation, leveling up on knowledge, networking, business dinners, or prospecting.

So when you consider the cost of consulting, figure that there is 25-30% more time that they are *working* though it doesn’t show up on the invoice.

Related: A CTO must never do this

2. Straightforward or naive?

In the case of the bartender story, he did indeed spend time researching for the propsect. Whatever that may entail. There is a point where the consultant goes out on a limb. Some saavy is required to avoid misunderstandings, but ultimately both parties also need to be fair.

Although we all want to make an effort to be straightforward, despite what Fred Wilson says, parties are not always transparent. I would argue if you were too naive in this regard, you would not succeed in business.

Related: What mistakes did you make when starting as a consultant?

3. Avoid the legal route at all costs

Unlike the bartender, I personally would never choose legal arbitration. Is it ever really worth it? In the case of the bartender, he says $35/hr and 5.5 hours work, so $192.50. But how much time did he spend filing the suit? Paperwork, phone calls, emails, whatever. And then driving to the courthouse, cost of gas etc. No way this could come out cost-effective. To my mind many legal cases come down to ego. One party wanting to SHOW the other party they are wrong. Not worth it!

My feeling is always take the high road. Talk with them, and explain where you’re coming from. Also listen to them, and try to understand their perspective. There is always a middle ground that can be found.

Read: How to avoid legal problems in consulting

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Consulting CTO/CIO Devops Hiring

Should I join this new startup Delicious Data?

via GIPHY

I’ve been asked this before by folks.

Hey, you know technology, what stock picks would you recommend?

Join 35,000 others and follow Sean Hull on twitter @hullsean.

It’s a tough question, with a lot of intangibles. It’s no wonder people ask friends for advice. You have to think about what matters to you? Your free time? Your income? Your time to commute? What about the team you’re working with? Or what your job contributes to the world?

Many of those I can’t quantify for you. What you can quantify money, so it’s worth doing that!

1. What are their prospects for success?

When asked about the chances of a companies success, knowing the industry may be one small part. You also have to know how many competitors they have, and where they are along in the process. And it’s not just developing technology, but team dynamics that are huge. From what I hear VCs hire more for team than for idea.

What factors outside domain expertise come into play? Lots! The weather, financial markets, or the big guys like google or amazon coming into the market. They may not buy you, they may just replicate your idea. Then where are you?

Read: How to hack job search the smart way

2. How can I apply mathematics to money?

My answer is always the same, go for the S&P 500. If the S&P beats 90% of all stocks, then nine out of ten times you will win this way. That’s it, calculation done.

Yeah but how does that pertain to joining a startup?

How indeed. I still say invest in the index, not in one pony. So use that advice as you will.

Gambling on one company is something for gamblers. If you want to become a vc, that’s a different question. In that case you would do a lot of due diligence on team and idea, to be sure you’re putting your money in a smart place.

Can’t I do that as an employee? Yes sure, but the intangibles remain strong.

How can 1% of something equal nothing?.

Related: Is Fred Wilson right about dealing in an honest, direct and transparent way?

3. How does all this help me?

It leaves out the intangibles. Don’t count paper as part of your compensation package. If money is a key factor, divide the number of hours per year by your salary plus real benefits – health insurance and so forth – to come up with a real number. Compare that to other jobs.

The heck with these finance jobs that pay $200k and offer a $50k bonus, but ask you to work 90-100 hours per week. Why not get two $180k/yr jobs at 45 hours per week? You see the logic right?

And what else? Of course if you’re going to be commuting in to an office everyday, and joining the family, you want to have great coworkers. So make sure you like the place where you’re working. I don’t know how much this is worth to you, but I would say it’s quite valuable!

Related: What to do when prospects mislead you?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Consulting CTO/CIO Devops Software Development

Do you fear you are an imposter? Join the club

via GIPHY

I was reading another delicious hacker news thread, this time on a psychology question. How do you work with the fear of your own incompetence?

Join 35,000 others and follow Sean Hull on twitter @hullsean.

It’s a great question. I’ve had this suspicion for years, and it was only after stumbling on psychology books that I even knew it was a thing.

So how *do* you manage this fear?

1. Demonstrate that it is a fear

Fear is a funny thing. It can color reality. You may not even realize it’s happening. When it comes to imposter syndrome, prove yourself wrong. Do the work, and then step back and show yourself the evidence.

You’re a logical rational engineer, so you should be able to weigh the evidence, and see that you made a mistake.

Doing good work is not about perfectionism. It is about knowing you can execute, and delivering quality. That doesn’t not mean there are no imperfections. That means good enough. That means equal to or better than the team you’re working in.

That means you’re improving the bottom line for the firm you’re part of. Help them deliver new features, new code, new product. And help other team members do the same. That’s the name of the game.

Read: How can 1% of something equal nothing?

2. Look at your history

Whenever I have this feeling, I look at my own history. Then it makes me sorta chuckle. I have a list of twenty companies that I worked for recently, and they’ve all been really happy with my work.

How do I know I did good work? They paid me handsomely, paid me on time, and then recommended me to other colleagues.

That’s how I know I’m not an imposter. Am I perfect? Nope. Do I know everything? Nope? But I do good work, and I take ownership, admit when I’m wrong, and play well with others.

If you want to stand out, take a look at these two pieces:

Check out: What do the best engineers do better?

And this: How to think like a senior engineer

Those will help you on your way…

Related: Is Fred Wilson right about dealing in an honest, direct and transparent way?

3. Realize your perfectionism

I think a lot of engineers or bright people have this problem. They want everything to be perfect. They want to produce documents without spelling errors, and code without bugs. They want to deliver everything on time perfectly every time. And they want to feel they know everything.

But it doesn’t play to your benefit. People resent this type of thinking, and it’s unhealthy besides. Take a deep breath, realize we’re all working towards the same goal, and keep your eye on the ball. That means have a sense of humor. You’re probably *way* harder on yourself then others will ever be.

Related: What mistakes did you make when starting as a consultant?

4. Be easier on yourself and easier on others

As you begin to be “easier” on yourself, hopefully you’ll also be a little bit easier on others. Be patient with mistakes. Understand that people have a lot going on in their life. Notice that they are trying.

Sure even after you gain a sense of humor, there will be some people who are not trying, who don’t care or who are really incompetent. But have your default position be patience, and give them and yourself the benefit of the doubt.

Usually if said person is really that bad, others will also complain and the problem will come to management’s attention. It is their job, after all to manage the team as a whole, and keep it productive.

Have fun!

Related: Why did mailchimp fraudulently charge my credit card?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Consulting CTO/CIO Security

How do we secure an existing aws hosted application?

via GIPHY

What if you don’t have the luxury of a greenfield. You are looking at an already built application, and asking yourself, how do I secure this?

Join 35,000 others and follow Sean Hull on twitter @hullsean.

One can think of it as a giant labyrinth, with many turns and many paths. Some of those paths have not had light shining in them for some time. So you’ll need to be cautious, thorough, and vigilant.

Here are some notes on where to start.

1. Scanning – code

One area you’ll need to dig into is the application code itself. If you don’t have the luxury to push new code, you’ll need to verify what version is deployed, and scan the repository for keys or passwords. You can also scan on the server itself. Better to double your efforts.

Read: What do the best engineers do better?

2. Scanning – network

Your VPC is obviously your first layer of defense. Scan the routing table policies, to make sure there aren’t open ports or whitelisted IPs.

Do the same sort of review for security groups, as those are an alternative method for configuring access to servers.

AWS has a service called Flowlogs, which can be enabled. These give you detailed network layer logging, which you can then scan for trouble.

Related: Is Fred Wilson right about dealing in an honest, direct and transparent way?

3. Scanning – IAM, keys & console

Your existing devs probably have keys to some or all of the EC2 boxes. If you don’t want to relaunch all of these boxes with new keys, or don’t have the luxury to do that, you’ll need to lock down the security groups, whitelisted IPs and VPC routing rules.

You’ll also need to carefully review IAM roles & policies. Amazon Inspector may be a useful tool to scan your environment, and find glaring holes and enforce best practices. But you’ll also want to do your own scanning both automated and manually eyeballing the accounts.

You’ll also want to lock down console access, especially the root account, and any others that have adminstrator privileges. Enable password policies and password rotation, as well as multi-factor authentication. There is also a nice toggle for “alert on login”. You certainly want to know about those!

Related: What mistakes did you make when starting as a consultant?

4. Scanning – services

Review all of the AWS services that are deployed. Ask yourself some of these questions:

o which regions & availability zones am I deployed in?
o what elastic IPs do I have configured where?
o what IAM roles & policies do I have created?
o what databases, API gateways & S3 buckets are configured
o etc…

Cloudtrail can be a great help here as it can log all sorts of useful information. You can then scan those logs for problems.

Related: Why did mailchimp fraudulently charge my credit card?

5. Rebuilding

The scanning approach can work, but there is a strong need to be thorough. If you miss one whitelisted IP or existing ssh key, you can leave the whole network open to a crafty intruder.

Another option is to rebuild the whole application. This gives you the time to:

o automate the whole stack with terraform
o test that everything is working
o plan for failover
o ensure that every bucket is secure with lifecycle policies enabled
o ensure that every EBS volume is encrypted
o enabled cloudtrail, cloudwatch etc

o potentially setup in a *brand new* aws account, for even more confidence
o backup all the pieces of the application as you go

Read: Did Disney+ have to fail?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Consulting CTO/CIO Devops Startups

What do the best engineers do better?

via GIPHY

I’m fascinated by this topic.

I recently found another thread on HN about it What do top engineers you know do that others don’t.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

As always for hacker news, there’s a feisty debate about what such character includes.

Here’s my take.

1. Tackle learning quickly

Whether it means getting up to speed on a new service that AWS has launched, building a new api for an application that has never been built before, or getting up to speed with a new platform. Learning is ongoing.

Top engineers can make this a seemless part of their daily routine. Getting going quickly with new concepts and technologies, means wading into the water at first, to gain the general lay of the land. Now you can talk intelligently about the features, limitations and challenges.

From there he or she can dive in quickly to the specific area required for the project, and move forward with that technology comfortably.

Related: How do I migrate my skills to the cloud?

2. Customer & product perspective

When building code, it’s easy to get mired in libraries, sorting algorithms, and API minutiae. And all of that is very important. But what are you building, and why are you building it?

Understanding your customer, what they do day-to-day is not always easy. It means using the product yourself, and also talking with sales teams regularly to hear what they are hearing.

Then pouring all this into your user stories. For top engineers it will inform their decisions, and help them communicate to product & project managers about what issues their encountering. Tradeoffs about features, coding, performance and technical debt can be better evaluated with more information.

Read: Is Fred Wilson honest about transparency?

3. Dig Deeper

Does your code run slowly? Have you tried to figure out why?

Is it related to:

o latency in production that doesn’t appear our your laptop
o untuned production database queries
o untuned connection pooling
o slow API calls
o weird kubernetes or orchestration issues
o web host issue with memory shortage
o web host issue with slow unoptimized code
o issues on the client side

Top engineers have seen applications slow down or fail in a myriad of ways. This allows them to imagine how a new application might be failing, and investigate those.

Related: What mistakes did you make when starting as a consultant?

4. Great communicators

In startups, your engineers need to communicate to many folks who don’t have an engineering background. Product & UI/UX folks probably are quite technical on their own. But what about sales teams who are dealing directly with customers? Or C-suite folks who watch the business bottom lines, but may not have the same low level technical understanding?

Great communicators can find the right metaphor to explain hurdles and holdups, technical debt, or the latest performance challenges. And explaining those in terms that resonate for others is incredibly valuable to the team and business velocity.

Related: Can Mailchimp fraudulently charge your credit card?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Business Consulting CTO/CIO

Is Fred Wilson right about dealing in an honest, direct & transparent way?

via GIPHY

I was just reading Fred Wilson’s excellent blog AVC. While I enjoy & respect his work immensely, I don’t always agree with what he has to say.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

The recent post was titled No Shenanigans.

He writes that these are part of Twiliio’s company values…


Be thoughtful. Always deal in an honest, direct, and transparent way.

While anyone can appreciate the sentiment, and everyone wishes things always worked well, sometimes the world of business can be a bit tougher than these words imply.

To me the words themselves are disingenous, ie they are *not* honest and direct to begin with. They sugar coat things in a Disney sort of way, when we know there are horror stories out there.

Buyer, customer & employee alike beware.

1. Disappearing stock options

In an article How can 1% of something equal nothing I wrote about a guy from Dogpatch lab whose company got bought for 100 million, and who didn’t get a penny.

You might think these stories are anomalies. In my personal experience working in the dot-com era & more recently, I’ve seen quite a few of these unfold. As a consultant I’ve always been on the outside, but I sure do have sympathy for the employees who were dealing in an honest, direct, & transparent way, until they found out four years of hard work got them no dividend.

Ouch!

Read: Why I ask clients for a deposit

2. Not getting paid

I’ve had my own experiences of being slighted. I wrote When clients don’t pay to tell my own story. I’ve learned from scrapes & bruises. And I tend to be more careful now, especially with new prospects.

That’s another way of saying, when I was young & green, I was naive. I was honest, direct and transparent. And because of it I was taken advantage of.

So again folks, buyer beware in this world.

Related: What’s the luckiest thing that’s happened in your career?

3. Aspirational

I think at the end of the day, the words Fred quotes are an aspiration. We would like to shoot for that ideal. But that doesn’t mean we shouldn’t be careful as well.

Protect your own interests, and read between the lines. You never know when things will go south.

Trust is something that is earned, over time. Especially in the business world. And all the company value statements in the world can’t rewrite human nature.

Read: What mistakes did you make when you first started out as a consultant?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Consulting CTO/CIO Startups

What mistakes did you make when starting as a consultant ?

via GIPHY

I was recently reading a Hacker News thread on mistakes made in consulting. While most of the discussion I didn’t agree with folks *at all*, it did get me thinking about my own lessons.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

Since these threads in HN often end up in debate, I decided to blog my answers instead. Here they are!

1. Not getting a deposit

I wrote before about When clients don’t pay. It doesn’t happen often, but it does on occasion.

Most of the time, you have a solid relationship because you’re referred by someone who you have also worked with. And a bad actor will get a bad rep among others in the network. This also by the way keeps consultants honest too, as your reputation is at stake with every new customer.

But there have been a few. And there are other good reasons Why I ask clients for a deposit.

It can be a nominal amount of a few hundred dollars. But it gets you into the finance system, provides a small hurdle that the client must also jump over, and generally provides good will to both parties. It says “we’re serious”.

And that’s important!

Read: What I learned from 10 years of blogging

2. Giving away free advice

In the first few years of consulting, I worked for so many great companies, that I figured they were all great. Then along comes one shady shop, for whom I was called in to help with scaling an application.

For a couple of hours we met face to face to discuss their challenges. What I found was an application that had grown beyond it’s original ambitions. I suggested they evaluate the product, provide new service levels to their customers, at different price points. And then for the higher paying customers, build out new hardware just for them.

It was a great solution, if I do say so myself. The customer thought so too. They went and implemented it themselves, without hiring me! I think they even offered to pay me a couple of hours for the meeting.

Suffice it to say I was pissed. There wasn’t much I could do because we didn’t have any agreement in place at that stage. I had no idea people would do stuff like this.

But I learned the hard way. Sad to say it’s the few bad actors that make us all play more carefully in business. Buyer beware!

Related: 6 Devops interview questions

3. Billing hourly

Billing hourly is how many start out. Some think of it as an industry standard. Lawyers do it, hey why not?

But hourly billing can be very confusing to customers. If you work 10 hours to solve a problem one week and 60 the next, they will find these invoices confusing and frustrating. What’s more on the 10 hour end of the spectrum you’ll be answering questions why the work was so “easy” and on the 60 hour end, what did you do wrong?

Customers often just want a problem solved. They want you and they want your availability. And the value of that may vary quite a lot. If you can figure out a way to bill weekly or monthly such that the client is happy with the value you’re providing, this will simplify your life immeasurably. Customers will be happier, and so will you.

I also recommend keeping daily notes and providing progress reports to your client.

Read: High availability what is it and why is it important?

4. Don’t step on toes

As a consultant, you will often be working with a team of fulltime folks. Not always, but sometimes there can be a tiny bit of resentment. Maybe because you’re outside opinion is threatening, or maybe for a million other reasons.

So I recommend treading carefully. Try to reassure your teamates that you aren’t trying to outshine them. Inevitably you’ll be in meetings with folks smarter than you. Certainly they will know more as they have boots on the ground, but sometimes, they are just plain & simple smarter than you. 🙂

Feel things out, and don’t step on toes. Some need to be right. Let them be that.

Read: High availability what is it and why is it important?

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don’t work with recruiters

Categories
All Blogging Consulting iHeavy Newsletter

Mailchimp fraudulently charged my credit card for spambot activity… Really!

via GIPHY

Wait seriously, you ask? Isn’t Mailchimp in the business of identifying, and protecting us from spam? Uh, yes indeed they are.

Join 35,000 others and follow Sean Hull on twitter @hullsean.

I’m still in disbelief myself. And while I got the problem cleared up in the end, I really have to share the story.

1. The precipitating event – a charge to my credit card

I love when I pay for a service, and their method of communicating with me is to charge my credit card. Of course I pay attention to when someone is taking my money, and I perk up.

At first I thought they were raising the prices again. They’ve done that recently, so I thought it was odd.

But sure enough I got an email with the following message:

Your account has been adjusted to another billing tier. Old plan $22.49, new plan $31.49.

Wait huh? I add about 2-3 new subscribers per day. How could this be?

Read: What I learned from 10 years of blogging

2. After digging I found spam emails

After looking at my list, I found that I had added 600 new subscribers last week in three days. How is that even possible? I wasn’t mentioned on BBC. That must be spam, I thought.

So I emailed support. They sent me all sorts of links, but didn’t seem to understand the issue. So I emailed back again and they said they were working on it.

Related: 6 Devops interview questions

3. Mailchimp communication – a warning

This *warning* is problematic. For one thing is it buried through various menus and pages. Only because I was looking for spam did I find it.

Plus Mailchimp doesn’t take responsibility.

In fact they kind of imply that I’m a bad actor here. Seriously? Is that how you communicate with your customers?


Warning

We noticed a 0.55% abuse rate on your campaign “Welcome Message”. This is above industry standards, so we strongly recommend you review your collection process, audience management, and sending frequency.

Internet service providers set strict limits on unsubscribe rates, undeliverable mail, and abuse complaints. Mailchimp is required to observe these limits. If your emails continue to generate high rates of unsubscribes, bounces, or abuse complaints, we may need to review or restrict your account. Please take the opportunity to address this now.

Read: High availability what is it and why is it important?

4. Can’t get someone on the phone

I did some google searching because I could not find the phone number. Turns out you *CANNOT* call Mailchimp. A lot of these services internet companies are going this route. Sure it saves them lots of money, but the customer service goes straight to the trash.

So I begrudgingly jump on a chat session. It took

Read: Service Monitoring – what is it and why is it important?

5. The chat transcript in full

Sean Hull
I've been hacked.

THEN MAILCHIMP CHARGED ME!

This is strange.

Does mailchimp protect me?

Mailchimp Support
We apologize for keeping you waiting and appreciate your patience. Our operators are busy at the moment. One of our agents will be with you as soon as possible.
Sean Hull
Thank you ... waiting patiently.
Mailchimp Support
We didn't forget about you. We apologize for keeping you waiting and appreciate your patience. Our agents are busy at the moment. One of our agents will be with you as soon as possible.
Sean Hull
thank you robot person...
how is the progress?
7 more! :)
4 more!
we are almost there!
Neo joined the chat
Sean Hull
hi neo
Neo
Hey there Sean, thanks for reaching out to Mailchimp support. Give me just a moment while I pull up your account.
Sean Hull
ok thank you
Neo
Alright Sean, what is the exact issue you are facing within your account?
Sean Hull
mailchimp charged me for fake subscribers.
if you look at my email list, you'll see it typically grows by 2 or 3 maximum per day
recently a hacker dumped 200+ per day into my list.
Mailchimp didn't monitor things, and then CHARGED ME to my credit card.
Does mailchimp protect me?
hi Neo, are you still there?
Neo
I'm still with you Sean. One of the main ways that Mailchimp prevents spam signups is through the use of ReCAPTCHA. This is a setting you can add to your embedded form from the "Audience name and defaults" page, which you can read more about here: https://mailchimp.com/help/about-fake-signups/#How_we_prevent_it
Sean Hull
ok. that is helpful. for the time being i enabled double opt-in.
but I also see that mailchimp has a WARNING.
about recent activity on my account, and possibly shutting it down. do you see that?
Neo
Are you referring to the "Account issue" that is referenced in the bar at the top of the screen?
Sean Hull
Warning

We noticed a 0.55% abuse rate on your campaign "Welcome Message". This is above industry standards, so we strongly recommend you review your collection process, audience management, and sending frequency.

Internet service providers set strict limits on unsubscribe rates, undeliverable mail, and abuse complaints. Mailchimp is required to observe these limits. If your emails continue to generate high rates of unsubscribes, bounces, or abuse complaints, we may need to review or restrict your account. Please take the opportunity to address this now.
This is what it says...
so to explain more...
first off this is fraudulent activity.
so I'm concerned that mailchimp would just charge my account, without warning of some problem.
and further, it seems that mailchimp *monitors* to INCREASE BILLING and monitors to DISABLE YOUR ACCOUNT, but they don't monitor to protect their customers.
Is that correct? Because if there is some type of monitoring I can enable, that would certainly be very helpful.
Also is it possible to DISABLE AUTO PAYMENT on my credit card?
Neo
For the sake of clarity, let's tackle your questions one at a time. I'm getting some more information for you at the moment and will follow up with you shortly. Thank you for your patience.
Sean Hull
thank you Neo.
you're awesome !
do i need *both* double-opt-in and RECAPTCHA? or is RECAPTCHA enough?
Neo
It certainly couldn't hurt to use both. Using double op-t in will help ensure higher engagement rates overall and less likely to present warnings such as what you've seen. Here is some more information on double opt-in: About Double Opt-In: https://eepurl.com/dyij4v
Sean Hull
i mean the warning is a mistake from mailchimp isn't it?
because these automated systems just sent that because of the hacking.
i feel mailchimp should be protecting me, so I'm confused by that.
I am a paying subscriber of the service. and the price has gone up in recent months. so i think we can agree there should be protection from spambots.
is it possible to disable AUTOPAY on my credit card? BC i don't want to get further fraudulent charges from mailchimp, because of a spam problem.
does that make sense?

Neo
The method of protecting your account would be through tools provided to avoid spam signups, which would be double opt-in and ReCAPTCHA. Additionally, our teams, such as Compliance and Billing, would be happy to look into your account with you to help resolve any issues you may be experiencing.

I would also like to let you know that I understand the situation you are facing is frustrating, so I will be submitting feedback on your behalf internally.
Sean Hull
anyway, could you scan my account for further spam signups? I think around july 19th there was a bump in signups of 80 people. I tried the segment method but couldn't find which day they were from.
Neo
Sure thing, allow me a moment to take a look.
Sean Hull
thank you Neo, i do appreciate that.
i mean at the end of the day I'm not an email spam expert, so that's why I pay for a service like mailchimp. to avoid problems and run a really clean list.
so for example that RECAPTCHA thing should be ON BY DEFAULT. probably that would have avoided all this to begin with.
also mailchimp should SCAN FOR SPAM FIRST. not charge customers first, then realize there is spam and charge back. Because that is a fraudulent charge. which i find super frustrating. I do realize these are all automated systems. But mailchimp should be more sophisticated to protect good customers like me.
Neo
I can certainly see how that is frustrating and would be useful to users such as yourself, so I would highly recommend leaving feedback on the matter via the "Feedback" tab at the right side of your screen.
Additionally, I’ll certainly be routing your concerns and feedback to our internal teams.
Sean Hull
Is that part of "chat comments"?
i left a good review of your help :)
okay thx again Neo.
I'll see if I can find that feedback tab
have a nice night :)

Neo
You can find it within your Mailchimp account on any page on the right side of the screen. Please don't hesitate to reach back out if you have any further issues, and have a great rest of your night.
Sean Hull
thx
laters

The remedy as you can see above was the enable recaptcha and also double-opt-in. That was fairly easy once I knew where to look.

From there I created a "segment" which is a collection of emails. And I selected the date range for the three days where I got spambot hit. And then clicked UNSUBSCRIBE for all those.

Why didn't Mailchimp do this for me? Read more to find out why I think they don't automatically fix this.

Read: How do I migrate my skills to the cloud

6. What Mailchimp did wrong

o They monitored the account to increase a SALE.
o They monitored my account to warn me about shutdown.
o They did not warn me about RECAPTCHA.
o They did not alert me when I got aberrant signups. When you see a 100x increase in signups, it doesn't take a rocket scientist to see that's a hacker of some kind.

o Then Mailchimp fraudulently charged my account!

Read: How to hire a developer that doesn't suck?

7. The sinister side

This charge could be innocent. A left over part of an automation system that hasn't evolved with the spambots. But I wonder.

From the help forums, they are clearly AWARE of the problem.

And the company *could* go and FORCE enable RECAPTHA for all lists. They could then email customers about the change, and for those who this poses and problem, the user could then go and manually DISABLE it.

They haven't done that. And certainly RECAPTCHA was not enabled by default.

Please don't call me paranoid when I say that there is a *huge* revenue stream to be had for users who fail to notice, this, get charged, and don't even know their own negligence. Think of all that revenue. My list has under 1500 subscribers, but others have 5000 or 10,000. Imagine how easily a *higher billing tier* could get overlooked.

Yes folks, it's dark.

And I'm not happy with Mailchimp now.

I'm happy to pay for a service, when it is done well. But I'm not a fan of these dirty tricks.

Read: 5 things toxic to Scalability

Get more. Grab our exclusive monthly Scalable Startups. We share tips and special content. Our latest Why I don't work with recruiters