When You Put Data In, You Should Be Able to Get It Out

July 17, 2009

Input/OutputOne of the most basic features a good software application offers is a way for users to get the data they put into the system out. Sounds reasonable, no? But that isn't the way it works, especially for a a few notable webapps.

For example, Twitter only lets its users retrieve the last 3,200 updates they've entered into the system. If you've tweeted over 3,200 times? The status updates starting with number 3,201 are simply lost to you. Poof! Gone. You can't retrieve them using Twitter's API or search engine. And yeah, I know, Twitter's supposed to be ephemeral, represent the NOW, be the current "pulse of the planet" or whatever. But for its most loyal users, who log their lives and thoughts and links on Twitter, that update back in early 2007 matters. The cap can feel like someone has burned your diary. Most likely this is a scaling issue, but let's get serious here: if I've been kind enough to enter data into your system over 3,200 times, the least you can do is let me export, backup, page back through all my status updates always.

Flickr does a similar thing with non-Pro accounts: you lose access to sets you've created beyond a certain number. At least Flickr gives you the option to upgrade to a Pro account in order to still get them (and I have), but it still wigs me out that I would create something on a system and then get denied access to it without paying. I could see denying me access to other people's data, but my own? Really?

Most software works pretty simply: you enter data into it, and it outputs something useful. But at the most base level, you should be able to get back out what you've put in.

If your Twitter update count is nearing or over 3,200, and you want to save a copy to your computer, this simple (but geeky/XML) method using cURL will work. Photo by mary.w.e.

Hi Gina,

I agree that its frustrating. I think the problem is that it costs these services money to store data and if its infinite and they get many users the pennies start to add up.

The right thing to do in my mind, is to store everything but then charge users beyond certain point, re: classic freemium, which is what Flickr does.


Alex Iskold
Jul 17 09 at 9:01 am

If you want a little bit less geeky way to backup your tweets you should try out the (aptly named) BackupMyTweets.

David Giesberg [+2]
Jul 17 09 at 9:16 am

The worst culprit of not being able to get out what you put in is the iPod

Fletcher Fowler
Jul 17 09 at 9:17 am

These are all interesting points. Another webapp I’ve often wondered about getting data out of is Google Reader.

For instance, it says I’ve shared something like a total of 6,000+ items. Now, can I really go back through and see all of those? Or do they only let me see the past 30 days or some such?

I thought somewhere it said that when you search, it’s only searching the past 30 days, but what about manually accessing it all?

mrhaydel [+7]
Jul 17 09 at 9:34 am

I’ve been thinking the same thing about Facebook. There is that “Older Posts” link at the bottom, but I wonder how far back that goes.

Giannicolus Jones [+2]
Jul 17 09 at 9:38 am

Just curious, if you are using FriendFeed (FF), won’t the FF archive help you out? I am at about 2250 updates, and so it will be some time before I can test this using FF. Thanks.

People shouldn’t trust the cloud. If you have information or data that you care about, always keep a copy for yourself, and keep it somewhere where you have control of it.

Offline storage is so cheap. Even though you may post every picture you take to Flickr, why would you not keep a copy for yourself on-site?

If you’re going to post to Twitter, why not cross-post everything to a WordPress blog that you control, and just categorize it as a “tweet”? That way, you’ll always have access to your content, even after the fad dies down.

I’m not someone who hates the idea of “cloud computer”. However, users should be aware of what they’re giving up by putting someone else in charge of their data. Most won’t care, but those who do should keep copies for themselves.

Mike Cerm [+21]
Jul 17 09 at 11:56 am

People back up their twitter stream? Seriously? For what? If you tweet a link, bookmark it in your browser. If you say something witty, or insightful, or noteworthy, put it in a proper blog or in a text file on your computer. If you’re over 3200 tweets – and I am – how many of those do you *really* think are worth saving? And why would you expect a free service to do it for you?

Flickr, I understand. You’ve got photos that you’ve uploaded, and it’s outrageous that they would deny you access unless you pay. In that case, they’re keeping your content and holding it hostage.

But Twitter? Seriously?

Daniel Clark
Jul 17 09 at 8:19 pm

Did Twitter post a sign out front saying “dead content storage”? No? Because storing dead content isn’t their business.

My expectation is that everything I post to Twitter is ephemeral. If it survives for a week, I’m grateful. (It’s not like anybody is paying Twitter anything for its service.) If I have something to write that I want to preserve, I post it to my blog (or, if it’s a link, I save it to Delicious).

And this is not about “trusting the cloud”. It’s about using the right tool for the job.

Jeffrey McManus
Jul 18 09 at 8:40 am

I’m with the first commenter.

You’re confusing “software applications” with “free webapps”. With software running on your computer, you can get any data out because you’ve paid for the resources for data in, data out: your computer’s RAM and HDD, software, ISP, etc. With online services, somebody else has to pay the bills for storing, transporting, and displaying your bits.

It totally makes sense that webapp for which you are not paying limits the amount of data you can put in or take out.

Jul 19 09 at 1:29 am

@mrhaydel We’ve tried to make it as easy as possible to get data out of Reader. If you go to the “Your stuff” -> “Shared items” link (or http://www.google.com/reader/view/user/-/state/com.google/broadcast) you’ll be able to see all your shared items. Similarly, search will act over all of the items you’ve ever read, shared, etc. Additionally, shared items are available as an Atom feed (see the auto-discovery tag on the shared items page), so it should be easy to do automated exports too.

Mihai Parparita
Google Reader Engineer

Mihai Parparita
Jul 19 09 at 4:52 am

Twitter doesn’t let you see updates older than your 3,200th post? I can’t believe it. I mean, I believe you Gina, but this just seems contrary to the Twitter ethos. Wow, I’m now very mad at Twitter.

Eric Geller [+2]
Jul 19 09 at 12:47 pm

For those of you with Evernote, there is an option to essentially backup your tweets. I’ve not used it yet because I don’t really have anything on Twitter worth saving, but I can see why someone may want to archive certain messages or conversations.

> Giannicolus Jones:

I recently went back through my older posts in Facebook and it does seem to go back to the start of your profile. At least for me, but keep in mind I’m not a heavy user. It is frustrating though to have to page back through all of it a little bit at a time.

Editor: fixed broken link up above, sorry there’s no preview!

JD [+5]
Jul 19 09 at 5:27 pm

I feel for the flickr users, they might actually be losing something important and useful if they were silly enough to not save the photos locally as well…

but who cares about twitter posts that old? Heck, who cares about twitter posts from more than a week ago?

Jul 20 09 at 4:49 pm

Given the number of persons commenting here signed up through Facebook, I am surprised FB has not been frowned upon yet for not allowing to backup your contacts details.

I am a nomad user not always using my own computers and I consider quite harmful to have to deal with these “software apps” / “webapps” (we all heard about convergence, right?0 which does not allow me to backup reliably data produced from anywhere else than the comfort of my “sweet home”!!

There is definitely something to do with cloud computing for domestic individual users :)

There is a work around for flickr.

Any user can create a group. Any photo posted to a group is visible, so long as it hasn’t been deleted.

Groups access can be fine tuned; at least as well as account access.

Which means one can make a group (before the pumpkin limit of 200) and still get at the pictures. I don’t know (because I have a “pro” account, if one can use this workaround to edit photos, behind the magi 200, but I know they can still be seen.

Terry Karney [+1]
Jul 21 09 at 5:43 am

Fletcher Fowler: The only thing about the iPod which is, “can’t get back what you put in” is the renaming. I have used mine as a disk drive to move music. The file names are all hashed (AABA AADG ASEF, etc. ) but iTunes recognised them, so I have all the names on the iPod. I could go and play them in a different player (they are still mp3s) and rename them, but right now I don’t need to, and I’m lazy.

Terry Karney [+1]
Jul 21 09 at 5:46 am

Comments are closed. Thanks for reading!