How we coped when our business was hacked

Any security breach can cause worry and stress, even if it only does minor damage to your business. But when you’re a growing technology company, how you handle the issue can really affect your company’s long-term prospects

Just ask the team at Buffer, a service that helps people and companies share content through social channels like Twitter and Facebook.

In October 2013, Buffer was attacked by hackers. As a result, many of their service users had spam messages posted to their Facebook and Twitter accounts.

We spoke to Carolyn Kopprasch, the company’s chief happiness officer (yes, really), to find out how they handled the issue and to see what other businesses can learn from it.

No early warning

With over a million users posting more than 200,000 updates a day through the service, Buffer has grown fast since it was founded in 2010, receiving positive reviews and building a loyal user base. Yet when the service was attacked, there was no warning at all.

“We didn’t know anything was wrong until everyone knew,” explains Carolyn. “We found out from our community of users, which obviously wasn’t ideal. We had an absolute avalanche of tweets and messages from people saying they thought we’d been hacked.”

As soon as the Buffer team realised something was wrong, they acted quickly. “The first thing we did was pause all updates running through the service,” continues Carolyn. “That stopped the spam from spreading.”

Reacting to the breach

Although the company hadn’t planned for this eventuality, the team went into a kind of crisis mode. Its 16 people work remotely and are based in locations as far apart as San Francisco, London and Australia.

“We were lucky that it happened when most of our people were awake,” says Carolyn. “Our team is very connected and used to working remotely, so everyone came online and was ready to go within a few minutes.”

As the company scrambled to understand how hackers had gained access, employees in different locations stayed in touch through a Google+ Hangout group video call. This ensured everyone knew what was happening.

Open communication

Buffer takes an unusually open approach to business, even documenting employee salaries and company performance on its website. When users and technology blogs demanded information, the company decided to be as open as possible.

“We were very honest,” confirms Carolyn. “People who were affected were angry, especially to begin with, but they were pleased that they could see what we were doing. We didn’t delay in tweeting about what was going on, which surprised people in a positive way.”

There was some initial uncertainty as to the cause of the hack, and the Buffer team weren’t afraid to admit it when they didn’t know the answer. “The only thing we were worried about was sharing anything we didn’t know for sure,” says Carolyn. “We were working around the clock to figure it all out and that’s what we said.”

Investigating the problem

With everyone focused on fixing the problem, Buffer soon worked out what had happened. Hackers had targeted the company that hosts Buffer’s main database, using an employee’s account to obtain Buffer’s Twitter and Facebook access tokens. This enabled the criminals to post messages to users’ social media accounts.

The attack had been well planned. The hackers also managed to access Buffer’s source code by breaking into a staff member’s Github account.

“For our developers, that’s like someone rifling through their closet,” describes Carolyn, recalling how Buffer’s staff felt in the aftermath of the breach. “However, it was worse because they went after our customers. We were really saddened that we had caused this for our users.”

Thankfully, Buffer’s users are an understanding bunch. Although the company saw a spike in customers downgrading packages in the week after the breach, this didn’t translate into a sustained trend.

“We were lucky to have really forgiving users,” says Carolyn, who ascribes this both to the company’s honest approach, and to the fact that its customers tend to be familiar with the issues facing technology firms. “You get that a bit in the tech community, because people have seen it happen and they know it’s a risk that every business runs.”

Closing the breach

Once they’d figured out what had happened, fixing the problem and locking the hackers out was relatively straightforward for Buffer’s developers.

However, the incident highlighted some of the service’s vulnerabilities, sparking a rethink in Buffer’s approach to security. “We learned so much,” continues Carolyn. “As we investigated, we found other ways we could have been hurt. We’ve fixed that hole and we’ve gone through many other steps too.”

“For instance, everyone in our company had access to our code in Github, so we removed that.” The company has also introduced two-factor authentication for its service and requires employees to enable it on any services they use for work, too.

As Carolyn explains, the company’s efforts to bolster security continue: “We’ve put more firewalls in place, and our security audit is ongoing. Experts from some of our partners have helped us and there’s more that we’ll do, because there’s always more you can do.”

What Buffer learnt

A combination of good management, sympathetic customers and a little bit of luck means this breach had a relatively small impact on Buffer’s long-term prospects. The incident could have been a PR disaster, but the company’s open approach seems to have paid dividends.

Carolyn believes good communication is vital during a security breach, especially if customer data has been directly affected: “Don’t be afraid to apologise and to be transparent. Talking to people is the way to keep ownership of the problem.”

“Have a way to contact all your customers if you need to. Email is good, but messages don’t always get opened, so use Twitter and other channels too. We put a banner at the top of our website.”

However, she also thinks Buffer’s experience shows the importance of monitoring for issues. “If we had been keeping a closer eye on our systems, we would have known there was a problem sooner. We didn’t have a way to spot it early — we should have had an alert set up to flag suspicious behaviour.”

Of course, the demands of a growing company often make it hard to prioritise security over customer service or adding new features. But Carolyn believes the incident has triggered a bit of a shift in focus.

“We’re now thinking about all the possible things that people might want to do that are bad. It’s opened our eyes to all the damage that could have been done.”

If you would like to read more on IT security, see:

• Your security plan
• Perform an IT security risk assessment
• Q&A: Using security software to protect your business