It can be difficult to plan for an unanticipated spike in Web traffic. One of the most amusing parts of reading Macolyte John Gruber's Daring Fireball blog is the near certainty that at least once per day, one of the poor saps he links to goes down due to the resulting crush of curious onlookers. It happens so frequently that it's practically second nature to see him update a post saying the link has been "Fireballed", opting instead to point to Google's cache of the original. This phenomenon, of course, is not a Gruber exclusive. For years, otherwise innocent bloggers have been Slashdotted, TechCrunched or even Scobleized, seeing their sites crack under pressure. But from what I can tell, if you're hosted on Blogger, this isn't an issue at all. The company was just recognized for perfect uptime during a survey by Royal Pingdom, and I've seen my own site spike in traffic following controversial posts with no ill effects.

Blogger was not always known for a stellar uptime record. In late 2006, the service practically had to write "a novel" about continued outages. In fall of 2007, I railed against Google for ignoring users during another major outage. But services mature and times change.


With the backdrop of Tumblr's highly-publicized downtime of more than a day earlier this month, Royal Pingdom also spotted occasional outages at WordPress, TypePad and Posterous, each of whom looked great compared to Tumblr's unfortunate blip, but not as spotless as the often overlooked Blogger.


While escaping system-wide downtime is a major win in itself, that doesn't speak to the services' ability to scale under pressure. Gruber recommends the WP Super Cache Plugin for WordPress, letting users serve more than a page a second, as it's self-hosted WordPress blogs that often get crushed with one of his links. But I've been linked from Daring Fireball a few times and lived to tell the tale. Same for Slashdot, Techmeme, Hacker News and the Huffington Post, all of which are capable of sending solid and sustained traffic on good days.


Similar to how Google is well known for its Web site never going down, I have to assume Blogger has mastered Google's distributed architecture, and makes no single blog, or its hotspotting, an instrument for failure. This blog didn't slow or go down when tens of thousands of visitors dropped by this July to tell me I was an idiot for switching from iPhone to Android. It also didn't go down when I questioned the usefulness of Google Wave last year, or when the Huffington Post liked my recap of David Kirkpatrick's Facebook Effect.

Even the largest services have occasional downtime. Facebook was down last week, intentionally, due to a code push ahead of schedule. Twitter has had its share of bumps, though the site has gotten much better in the last year, and even Facebook subsidiary FriendFeed went down this weekend. But Blogger users are sitting pretty knowing their blogs are going to be up and responsive, even under pressure.

Over the last two days, the popular location based service and king of check-ins, Foursquare, has been suffering from intermittent outages related to database overload and storage troubles. And this is wonderful news. Not because I wish them ill will, but it seems this is a necessary strain practically all successful companies go through at some point as they achieve rapid user growth. How users react to what are hopefully temporary bumps, and how the service reacts after the fact can help determine if the company will continue to be embraced, like a Twitter, or dumped, like a Technorati.

Foursquare's rise from its SXSW 2009 debut to near pervasiveness among the digerati, becoming synonymous with the location check-in movement, despite big players like Facebook and Google participating, and more direct competition like Gowalla and Mytown, is a testament to the service's viral nature, including game mechanics, close ties to friending and following, and a casual attitude that eliminated the dark feeling of insecurity many people once had about announcing their locations, and replacing it with something one not just accepts, but expects.


Foursquare seems to be in the position to choose their own destiny - to forge ahead as an independent company, flush with cash from a summer VC round, or to court willing suitors at a later date, hopefully at a higher valuation than deals they famously turned down in a public negotiation for control. While two days of shakiness are no doubt unwelcome by the company or its addicted userbase, it seems a rite of passage that the biggest Web services go through in a circuitous route to the top.

Twitter is by far the most famous for continued bouts with downtime, closely tied to scalability issues, and its fail whale mascot is the icon of downtime that all other interruption pages wish to be when they grow up. The company's status blog at http://status.twitter.com/ has actually seen 15 different posts since September 1st, contrasted with only 10 posts from the company's official blog. (http://blog.twitter.com/)

But the blue bird and white whale on a teal background are not the first to document downtime doom. Amazon Web Services promotes a service health dashboard, while Salesforce.com displays a system status at Trust.Salesforce.com. Google provides a status dashboard for Google Apps, with dedicated pages for Blogger status and Feedburner status and others. Even pokey old Web 1.0 dinosaur Hotmail has a status dashboard.

Common to all these services? Previous downtime issues that hit the community hard. Truth is, until there is a major downtime incident, most companies do not offer insight into their status. They may feel like they are teflon, and they can't be taken down, or they simply didn't think about what could happen if something went wrong. It takes a public shakeup that calls the service's stability and viability into question to push for improved system integrity and redundancy, and despite best efforts, some never quite solve the problem.

Back in 2007 and 2008, Technorati was as much known for its Technorat Monster escaping as anything else. This cutesy downtime message foretold the demise of the service, which had to strip out much of its functionality simply to stay up, and was pretty much abandoned by all but a core of users who watched the company evolve its business model and reduce their technical demands. And so far as I know, Technorati never launched a status page. They didn't do what the many other successful companies (above) did when push came to crash. Foursquare, it looks like, is working hard to push this recent event and will come out of this stronger because they are following the proven path - of ownership, late hours, and public transparency.

Tonight may suck for the company's employees, who are taking a public flogging that cans of Red Bull and pizza won't fix. But they should wear the fatigue as a badge of pride. In time, they could look back at this week's events with warm and fuzzy feelings, remembering the outage, which resulted from "too many checkins", as a clear tell they were graduating from Web curiosity to Web titan.

After an initial wave of attacks brought the popular micromessaging service to a standstill, as well as a host of other social networking sites, on August 7th, the Twitter team has been fighting to turn the tide and restore access for the service's customers and developers to near-normalcy. Ten days into the attacks, they have not disappeared, according to postings from Twitter, but it looks like the company may have turned a corner in dealing with the additional stress from the distributed denial of service.

In a post to the company's Twitter Development Talk forum titled "Platform Status Update" entered at 4:30 this afternoon, Twitter's Ryan Sarver said:

"Over the past ten days we have been dealing with a lot of stress on our network and that has caused a number of our partners to be knocked offline for extended periods of time. This is obviously not something we want to happen and the platform and ops teams have been working hard throughout that time to address the needs of our ecosystem while protecting the system as a whole. We have made some great progress today in tuning the system to a point that should allow our partners to operate as they were before the recent issue began."

He added "the system is still under stress", and asked developers who might be continuing to have issues to work with them and provide information around what IP addresses their machines use to access the Twitter API, the method they are using to make requests, and any other information - including data as granular as their host operating system, browser, cookies and network connection.

Ryan's comments followed a note from earlier in the afternoon by Alex Payne, the company's platorm lead, where he also said the company continues to "fend off" the attack, saying that without specifics from developers (like those listed by Ryan), they could not adequately troubleshoot any issues.

While the majority of Twitter users are not experiencing the widespread outages that we saw a week and a half ago, it is still not uncommon for applications utilizing the Twitter API, and specifically OAuth, to see failures. There's also no doubt the ongoing attacks have led to a fair share of fail whale sightings. (I've encountered a few myself) Twitter posted a note to the company's status blog Sunday acknowledging the problems, and said they are "working to resolve it".

It has been tempting at times to poke at Twitter for its struggles during this period, but as I mentioned on August 7th, when the troubles first arose, the company is being much more visible and fast to respond to developers and to provide status, as they show signs of maturing. Their operations teams look to have been busy around the clock for the last two weeks, and they will need to keep fighting - as it looks like this battle may go on for some time.

The much-discussed distributed denial of service attacks over the last two days which hobbled Twitter and also impacted a number of other sites, including Facebook and LiveJournal, has also had a tremendous negative impact downstream, bringing many third party applications and services to a halt. With the sluggishness and non-responsiveness stretching into what will be the third calendar day tomorrow, Twitter told developers, heading into the weekend, that not only was the attack ongoing, and not lessening, but they had no ETA for anything to get better.

While many developers are understandably sympathetic toward Twitter's troubles, even as the popular microblogging service's network architecture is being analyzed for weaknesses, the strain many of them were feeling without receiving answers started to be on display this evening as hours went on without an update - while at the same time, key Twitter employees happily updated that they were going out for sushi or enjoying tonight's Underworld concert.

One developer, in the company's Google Groups forums, wrote:

"There's been lots of posts from Twitter saying they plan for better communication. Okay, cool. But it's Friday night and I am pulling my hair out with no new information. I am sure they are doing all they can... but many third-party apps have been broken for 2 days now. And just as Twitter is getting bombarded with emails from developers... developers, too, are getting bombarded with complaints/questions from customers."

After the same developer noted that employees had gone out to catch dinner, he exclaimed frustratedly, "While food is important, would be helpful to also update devs what's going on, too, before a night on the town?!?!?!" This led to another developer's stating he too agreed with the outrage, adding:

"Thanks for your outrage. I thought i was one of the only few that have issues with how things are being handled."

But shortly after the conversation threatened to get ugly, Twitter's Chad Etzel posted an update, around 8 p.m. Pacific time, catching up the community, and he didn't have much good news.

His major points:

• "The DDoS attack is still ongoing"
• "the intensity has not decreased at all"
• "interaction with the site and with the API will continue to be shaky"
• *There is no ETA on fixing any of this* (repeated twice)

He told developers that the situation at Twitter will "continue to be rocky" and may actually get worse, but also said that the company's operations team will be working "around the clock" this weekend, adding, "as much as you want it to be fixed, we want it to be fixed more."

With past outages, Twitter has had a mixed bag in terms of communicating to the developer community. Had the initial complaints gone unanswered, it's likely Twitter would have transitioned further away from being the victim to being seen as part of the problem. Yes, applications are broken. Yes, access is still slow, and as the site has become a major piece of infrastructure in today's social Web, its downtime is more than trivial. But this isn't a situation of the company's employees partying on a sinking ship. It sounds like the team is strained, and being realistic about what will continue to be a long process of restoring normality.

TweetStats: Closed Since Sunday Afternoon
On Sunday, we mentioned Twitter had run into a bug that masked the display of third party clients on the service, erroneously reporting all updates as coming from "Web", whether they were, or if they were instead from the mobile interface or any of the growing array of applications that interact with the microblogging leader. Now into Tuesday, the bug hasn't yet been fixed, and in the meantime, popular statistics tracker TweetStats has been shut down - incapable of operating correctly with bad data pouring in.

Twitter's API team is telling developers that the "from Web" issue is the result of large growth in the company's database, thanks to a recent increase in API developers and their registering applications to work with OAuth. The database object reportedly grew to a size incapable of being cached, dramatically impacting performance.

Still a small company, rather than push for Twitter's engineers to come in over the weekend to resolve the issue, the company said they chose the "quick solution", opting to "disable source parameters", according to postings on their Twitter API forum here.

Initial guidance was that the issue would be resolved "likely early in June 1 workday" (sic), but more issues have cropped up. The company followed up with a second note saying, "Due to problems with other *critical* code we've had to delay deployment of this fix until tomorrow."


TweetStats Reports The Trouble Sunday
TweetStats shut down mid-day Sunday after the issue had impacted the statistics gathering service dramatically. And while Twitter has again offered a new date of resolution, the developer, Damon Cortesi, has said he'll find a way to get rebooted, with or without help from Twitter.


TweetStats Hopes for Best, But Prepares...
He noted late Monday night, "If not fixed tomorrow AM, will re-open and deal with the consequences."

Jesse Stay has more around the details of the API issue here: Where is Twitter’s Emergency Response System?

Though it may end up being a temporary blip, at this moment FriendFeed is down, following a scheduled outage at Twitter this afternoon. And while that's not really news, it comes on the heels of many discussing the potential for failure that third-party URL shorteners bring to the Web. For every fan of TinyURL or bit.ly, there are others who say relying on another service to be a go-between between the user and the intended data is just begging for trouble. But the truth is that in a network, when there are multiple items with potential to fail between the user and the data, any one of those pieces in many cases can bring the entire system down to its knees.

• Storage can fail.
• Servers can fail.
• Networks can fail.
• Routers can fail.
• Lines can be cut.
• Services can close down.
• Users can delete images or pages.

It happens, and until we control all aspects of the system, there will be outages.

On Wednesday, in the middle of testing a third-party Twitter service, I linked to the Guardian using a URL shortener called tr.im, required to get the service to work. Later that night, tr.im failed, and it broke all links that were being used.


The conversation (in Google cache)
In response, Paul Buchheit, co-founder of FriendFeed, with a long history at Google, Microsoft and Intel prior to his latest efforts, referenced the break, calling it "another reason why url shorteners are annoying."

But FriendFeed itself has a URL shortener, called ff.im, which it uses when sending updates to Twitter. Paul added in the thread, "Except ff.im of course :)"

But guess what? Because FriendFeed is down (for now), also down are the ff.im links, making them as likely to fail as any other third party shortener. I could rant up and down saying that FriendFeed and ff.im should be served from different data centers, or offer better redundancy, but I won't. Nobody loves downtime, and FriendFeed by and large has had a fantastic track record of staying up. But as they become a more integrated part of the ecosystem, they too will get more opportunities to fail and need to take the same safeguards to protect the infrastructure as do all the other players.

Things will fail. We will live, but we know that there is no such thing as a fully redundant failsafe machine. Every hop delivers the potential to turn into a skip, and not in a good way.

I didn't make any blog posts on Thursday, after a full Wednesday which included a visit to Google headquarters to meet with the Google Reader team. And even though I made a few posts on Friday, the first day of the SXSW conference, many people still think I'm on a temporary hiatus, thanks to a tag-team failure between FeedBurner and Blogger, who have significantly impacted many users by zeroing out their feeds, stopping their posts from getting out of their domain. It looks like I should have spent more time in Mountain View after chatting up the Reader team, to see just what the heck is going on elsewhere on campus.

As the resurgent Kent Newsome of Newsome.org noted today, in his post, "FeedBurner & Blogger Conspire to Assassinate My Joy", the XML file that Blogger generates to distribute RSS feeds was completely wiped out - and I have been impacted as well. No matter if I had 20 or 2,000 posts historically, the file reports it has zero kilobytes, and no amount of trouble-shooting thus far has been of any help.

In this world of RSS-enabled services, a failure of this level means that my posts aren't getting out to the previously-mentioned Google Reader. They aren't populating social networking sites like FriendFeed, Socialmedian and Facebook. And that will no doubt cripple visits for those affected.

As always, I at first considered the issue was my fault. I noticed FriendFeed this morning wasn't picking up my posts, and tried to delete and add my blog six ways from Sunday, but with no success to speak of. While Kent recounts his searches for real tech support, I was in the air during much of the day, and came back to find I was not the only one hearing the silence from Google, as others assumed silence from me.

Complaints are wide-spread in Blogger's help forums - from RSS Feeds are EMPTY today when publishing via FTP. to atom.xml is empty and simply empty feeds.

I'm concerned on quite a few levels here. FeedBurner and Blogger are among the most mocked Google products because of their presumed neglect. Yet they are two of the major foundations for my blog. Also, I am concerned because we are seeing this issue as a weekend starts and a major tech conference is starting up. There's a huge chance this issue won't be resolved if Google is asleep at the wheel. There's nothing I can do except accelerate my move to Wordpress. This doesn't make me feel all that lovey-dovey with Blogger at the moment.

Just before 7:30 this evening, as we were putting the twins down to sleep, the lights fluttered and went out. They whirred to life again, twice, but soon dropped again, and we've pretty much been in the dark for the better part of two hours. No TV. No WiFi. Not even the background noises of the refrigerator and heater. With temperatures in the low 40s outside, our home is cooling, and we've unpacked the flashlights and candles we could find.

Did you know you can combine a C battery and a D battery in a flashlight, and it will still work? By necessity, I found out tonight that it does.

Preferring to be constantly connected, my iPhone 3G is keeping me sane. The laptops are fairly useless, but not my ubergadget. It still lets me post updates to Twitter and FriendFeed, browse bookmarks, and read e-mail. Nobody told it to shut down, after all.

Losing power is really no big deal, for the short term. Everyone is safe, and if this goes longer, we could pluck our twins from the cooling crib, and warm them up ourselves. But it's got me thinking about being better prepared for something bigger. It's now clear we need more batteries. And the whole hubbub about Twitter being a good news hub during emergencies doesn't hold too well if you lack power. And it doesn't translate well to the small screen. If a neighbor has discovered the source or reach of this outage, I haven't seen it. My network is too diverse and too noisy to get data on local happenings.

So for now, we're a little disconnected. It's dark. It's getting colder. And we don't have answers. I'm lucky to have the iPhone 3G around, but that aside, the infrastructure holding our power and Web together looks pretty flaky - even in Sunnyvale, smack dab in Silicon Valley proper.

Update: Power was restored shortly after 10 p.m., having been out for just under three hours. No cause has yet been determined.

By Mona Nomura of Pixel Bits (FriendFeed/Twitter)


Reading Search Engine Journal Loren Baker's Gmail horror story brought back my Gmail nightmare.

Way back in 2005, I tried logging into Gmail as usual. But Gmail kept redirecting me to this odd error screen with the message: "Sorry... account maintenance underway" and would not let me sign in. (Above image taken from my 2005 Japanese blog.)

After trying (and failing to log in) for two full days, I contacted gmail-maintenance@google.com, and even posted in Google Groups, but did not receive a resolution or even a response. I tried logging in twice a day, everyday, for four months, and finally my persistence paid off. Out of the blue my access had been restored and I haven't had problems since. But to this day, I still have no idea how or why my account was under maintenance -- for four months.

• Yes, I know GMail is free.
  
• Yes, I know GMail is still in beta.
• Yes, I am aware I should not be complaining... but it's... Google.

Even if it's free and in beta, Google isn't supposed to... break. As embarrassed as I am to admit this, I quickly got over the trauma, and continue to use Gmail. But when Google nightmare stories catch my eye, it brings me back to 2005, and the panic of when I couldn't access my e-mail, compelling me to go out of my way and remind my friends the same thing could happen to them.

Fortunately, Gmail has two great backup features that takes only a few seconds to set-up. My peers were extremely thankful I shared, so hopefully they'll help you too. :)

Two Features Every Gmail user should have enabled:

E-mail forwarding.

I created a backup e-mail account for my main e-mail, and have a copy of everything sent to my inbox to my backup. To set this up:

1. Create a back up e-mail (ie: mye-mailaddress.backup@gmail.com).
2. Settings
3. Forwarding and POP/IMAP -> Forwarding -> Foward a copy of incoming mail to "mye-mailaddress.backup@gmail.com" -- or whatever your backup e-mail address is.

Gmail's "Send mail as:"

Gmail enables adding custom 'From' addresses for free. learn more here.
So in case my main e-mail is disabled for one reason or another, I can always send e-mail as my e-mail address from my backup. For free. Pretty neat.

With the above, I have some peace of mind, though I truly hope I will never ever get locked out of my account again. Bonus: check out techradar.com's "40 Brilliant Gmail hints, hacks, and secrets" it may have some more useful tips.

Have any of the nightmare Gmail stories happened to you? Is Gmail your primary e-mail address? Do you have any preventative tips or tricks I don't know?

Read more by Mona Nomura at Pixel Bits

Apple's replacement for .Mac, MobileMe, has been roundly mocked for its spotty uptime since rollout last month, drawing the company's CEO, Steve Jobs to apologize for the lack of quality in an internal memo. But even following an internal reorganization and the public thrashing, users, including me, were unable to access their e-mail for a good portion of the afternoon - even as the company's MobileMe Status page shows no updates since the end of July.

Not to be outdone, the most popularly cited alternative to MobileMe, Google's GMail, has also suffered outages this afternoon, locking its many users out of their e-mail, again, including me.


At the beginning of the issues with MobileMe Mail, Apple famously said the outages were only impacting a small 1 percent of users, despite widespread complaints throughout the Web. Today's outage, which Apple reported lasted about a half hour, cited only that "MobileMe members were unable to access MobileMe mail", so that indicates a full outage.

GMail, on the other hand, says, "We’re sorry, but your Gmail account is currently experiencing errors," without going into detail as to how widespread the issues are. GMail even goes the extra mile to promise "your account data and messages are safe." A discussion sparked by Shey Smith on FriendFeed shows the outages don't appear to have hit everyone.


The 1-2 punch of the outages has made discussion of the downtime the top conversation starters on Twitter, even higher than the Beijing Olympics or the Russia/Georgia skirmish. People must really hate having their e-mail interrupted!

Much of the impact of Twitter's frequent downtime has been anecdotal. Amid some saying they're leaving the service for greener pastures, or developers pulling up their stakes in the Twitter community, statistics show that some of Twitter's most prominent and active users have dramatically reduced their activity on the site over the last two months.

The drop in total tweets by virtually every top user who was measured could be part technical - due to their simply being unable to login, or psychological, a result of lower activity and lower conversations which became a self-fulfilling prophecy. While none abandoned the site altogether, what could have been an "up and to the right" activity graph has largely stalled, and in many cases, reversed.


My own activity, rising month by month after I finally started using the service in January, stalled in June, and is still well below what it likely would have been had stability not been impacted.

Using TweetStats, a site which can show your total tweeting activity, who you most frequently message, and which hours and days you use the service, I polled ten top Tweeters to see how their June and July activity compared with April and May. Here's what I found:



Chris Brogan / @chrisbrogan
April and May Tweets: 2,896
June and July Tweets: 1,070
Change in Tweeting: Down 63%


Corvida Raven / @corvida
April and May Tweets: 2,669
June and July Tweets: 1,065
Change in Tweeting: Down 60%


Danny Sullivan / @dannysullivan
April and May Tweets: 1,281
June and July Tweets: 551
Change in Tweeting: Down 57%


Dave Winer / @davewiner
April and May Tweets: 1,535
June and July Tweets: 527
Change in Tweeting: Down 66%


Drew Olanoff / @drewolanoff
April and May Tweets: 2,131
June and July Tweets: 909
Change in Tweeting: Down 57%


GeekMommy / @geekmommy
April and May Tweets: 6,030
June and July Tweets: 1,419
Change in Tweeting: Down 76%


Jason Calacanis / @jasoncalacanis
April and May Tweets: 1,017
June and July Tweets: 562
Change in Tweeting: Down 45%


Leo Laporte / @leolaporte
April and May Tweets: 363
June and July Tweets: 237
Change in Tweeting: Down 35%


Robert Scoble / @scobleizer
April and May Tweets: 3,579
June and July Tweets: 746
Change in Tweeting: Down 79%


Michael Arrington / @techcrunch
April and May Tweets: 1,587
June and July Tweets: 1,079
Change in Tweeting: Down 32%

Across the board, Twitter's issues cut activity to the site by about half or more for some of the most visible users of the site. Others, like Kevin Rose of Digg (TweetStats) and Pete Cashmore of Mashable (TweetStats) saw only a less than 20 percent reduction in their Twittering activity between the two time periods. While there's no doubt many people, like Steve Rubel and Allen Stern, wish discussion of Twitter's problems would just go away, the impact it had on the site over the lsat few months has been very real, and we're just now able to take a step back and measure its impact.

Just when you thought it was safe to Tweet again, Twitter ran into yet another database problem, which not only resulted in sporadic "Fail Whale" sightings, but dramatically impacted the roster of those following one's updates, as well as those each of us were subscribed to. The latest snafu comes at a time when the growing number of microblogging addicts are seeking alternatives, moving to FriendFeed, Plurk, and increasingly, Identi.ca.

According to a post on the Twitter Status blog, the issue first showed a reduced number of followers on the service, and later, in order to solve the issue, Twitter went into a maintenance mode, warning of lower counts across the board. Amusingly, they claimed that some of the lower counts could be due to the removal of spammers, but in my experience, it's been more than 9 hours since the problem was first identified, and I've seen the number of people I'm following drastically cut, from more than 1,500, down to "only" 672, less than half.


On Monday, when I said "The Talk About Rules for Social Following Is Getting Out of Hand", I had taken a screenshot of my current Twitter ratio, at 1,534 to 1,441, after having worked for a good part of the previous week with Twitter Karma to get my ratio synchronized. Just a few days later, that data is carved to 672 and 1,236, prompting some to try and refollow me, and even more to flock to identi.ca.

Twitter's gotten a lot of abuse on this blog in the past few weeks, as we've gone over issues with developers, uptime and changes to the API, but every time I think they've captured the market on a single route to failure, they find another way.

The team's employees are talking a good game about getting this resolved, but seriously, Twitter, why should we believe you now?

See also:

Why Does Everything Suck: The Nightmare Twitter Scenario May Be Upon Us
Profy: So You Thought Nothing Could Be Worse Than Fail Whale? Now Get Your Followers Back