Twitalytic Alpha Preview: Archiving, Curating, and Threading Tweets

August 20, 2009

Twitalytic preview Update, July 2010: Twitalytic has been renamed ThinkUp.

Been hacking away at a personal programming project, tentatively named Twitalytic, a web application that archives your tweets and mentions and gives you useful data about your friends and followers on Twitter. (Here's some background on why the heck I'd ever want to archive the ephemeral chatter that is on the Twitter.)

Anyway, after a few months I've collected a teensy bit of data, so I wanted to make sharing replies to my lazyweb tweets easier. To that end, today I added a public face to the app. Here's the Twitalytic public timeline, which lists my most recent tweets and the number of replies each has. Click on the reply total to drill down into the responses to a particular tweet, like my tweet asking what people eat first thing in the morning. (Ha, I made over 200 people Twitter about breakfast!)

That listing is the most basic thing Twitalytic can do. When you set it up and log into the app, you can curate replies and see other views of your data, like your least likely followers, your least and most active friends, your most replied-to tweets and more. For each of your friends or followers, you can see mutual friends, conversations you've had, and even what Twitter client that person uses most. If you're approaching the 3,200 tweet mark, Twitalytic will also back up your tweets and let you export them into a simple text file. Here are some screenshots of the internal, logged-in only views.

List your tweets by most recent, most replied-to, or back-and-forth conversations.

Twitalytic - All Tweets

Here's the conversation view.

Twitalytic - Conversations

File replies as "standalone"--i.e., someone just said something to you out of the blue, or associate mentions with a specific tweet (if the user didn't use the "reply" button in their Twitter client).

Twitalytic - Curate Mentions

Your "least likely" followers are those who have the largest follower to followee ratio.

Twitalytic - Least Likely Followers

See which of your friends are most active and least active.

Twitalytic - Friends

Click on any tweet's reply count to view its responses, "likely" reponses that happened around the time of that tweet, and public/republishable replies (those from users's whose tweets are not protected).

Twitalytic - Tweet replies

Click on a username to see recent tweets, conversations, mutual friends, and what client that person uses.

Twitaltyic - User Stats


I realize there are ten dozen Twitter applications out there that do much of this, and possibly better. Creating yet another Twitter app is probably the most cliché development undertaking of the year, but I've had a ball using this little thing to teach myself a few things about the Twitter API, OAuth, and jQuery and brush up on my PHP and MySQL skills.

To use Twitalytic you've got to install it on your own server and run the crawler to start gathering your data. As far as I know, only one other brave soul beside me has done this; I don't recommend it unless you're well-versed in PHP development and LAMP environments. This is still very alpha stuff--and I haven't built in any easy way to update an installation either, which means as the code changes, manual re-installation will be required. That said, I welcome PHP hackers to check out the GitHub repository and start tooling with the code. It's been a very long time since I've developed a webapp of this size, and the code is full of //TODO's and //Must be a better way to do this's, too.

At some point I'll set up some sort of dedicated web page and perhaps a mailing list for the app. Until then, post your questions and thoughts here.

Update: Wow, I didn't ever think this little spare-time data-wanking PHP thing would be TechCrunch-worthy. Thanks for the post, TC!

This is really, really cool!

Unfortunately, I don’t have the time to install and play with it, but I wish I did! :-)

AJ Robins [+5]
Aug 20 09 at 7:26 pm

I’m digging all of the features that you’re building into this, I’d love to try it out once it gets a little further along (with an easy way to update).
Keep up the good work Gina, this is coming along really nicely.

ffff00 [+1]
Aug 20 09 at 8:56 pm

Awesome. I am going to install it today.

Thejesh GN [+5]
Aug 20 09 at 9:00 pm

I like your code! I am also curating twitter data, but to detect stuff happening in cities.
I detect city names, dates and event keywords and aggregate tweets. It’s kind of cool to see where people are going (festivals, conferences, tweetups, etc.) in a given city:
http://www.twillage.com
and
http://www.twillage.com/today

Laurent Denoue
Aug 20 09 at 9:15 pm

Looks really cool – I like that you’ve created lots of unique ways to display information. I’m looking forward to trying it out.

Steven Rossi [+2]
Aug 20 09 at 9:18 pm

I tried this on my local web server. (Manually entered the oauth_token in the GET url).

There was a problem in the crawler logger. The realpath(‘logs/’) evaluated to null it tried to save logs in my root directory. Why did you commented “global config” parameters? I took the log path from it.

Gina,
This is great stuff! I got it up and running in less than 15 minutes with a few minor changes.

By putting the code up on GitHub, are you releasing the code under some sort of opensource license such as the GPL, MIT, MSD, etc.?

Also, since you’re getting back into coding PHP, I’d highly recommend taking a look at CodeIgniter. I found CI to be a great framework for implementing MVC in PHP.
Cheers,
Pankaj

I have a doubt.. by default.. in twitalytic/common/config.inc.php the path for logs goes to /advanced/logs/ while there is no such directory.. am I supposed to replace it to crawler/log/ along with rest of the path..??

Sandeep Bali
Aug 21 09 at 3:21 am

I own the domain (www.twittalytic.com) and Twitter handle (tiwttalytic) and am planning to release something relatively soon. But I wasn’t sold on “Twittalytic” because it’s a little too geeky.

In any case, I’m approaching some of these problems slightly differently. Good to see some validation here though. Congrats. :-)

Ken Yarmosh
Aug 21 09 at 5:33 am

@Sandeep and @Sandaruwan: the sample config file line to the log should be:

$TWITALYTIC_CFG['log_location'] = ‘/your-path-totwitalytic/crawler/logs/’;

Sorry about that.

Gina Trapani [+195]
Aug 21 09 at 8:14 am

I like this a lot. I would definitely use it, if for no other reason than I’m intrigued by the stats (’cause I’m just a regular gal with a small number of followers).

@Ken it’s *supposed* to be geeky :P

macpug
Aug 21 09 at 9:05 am

Hey Gina, long time, no talk!

Love what you’re doing. I’ve also been working on a Twitter project, which I’m calling GraphEdge (http://graphedge.com). I’m looking at some of the same metrics, but focused more on follows and drops. More a B2B play, but certainly valuable to obsessed Twitter consumers. Will be adding something close to “least likely” and “most likely” soon, plus a lot more.

Drop me a note and we’ll talk.

- Waldron

waldron [at] graphedge.com

It seems that the current code doesn’t use the path mentioned in $TWITALYTIC_CFG['log_location']. In line 14 of “common/class.Logger.php”, you are using “realpath(‘logs/’);”.

Small suggestion : use “dirname(__FILE__)”.

~ Sandaruwan

Awesome! I have just installed it and it’s working great!

One thing I did notice is that if I view a users page (like click on someone else’s username to see mutual friends ect.) it screws up the session. Then every page I go to after that gives me an error like “Error, selection query failed: SELECT * , round(total_replies_in_system/(datediff(curdate(), earliest_reply_in_system)), 2) as avg_replies_per_day FROM instances i INNER JOIN owner_instances oi ON i.id = oi.instance_id WHERE oi.owner_id = ORDER BY crawler_last_run DESC”

wee493
Aug 23 09 at 6:37 pm

I’ve managed to install this on dreamhost and got the front end working succesfully but every time I try to run the crawler I get the following error.

$php crawler/crawl.php

Parse error: syntax error, unexpected T_STRING, expecting T_OLD_FUNCTION or T_FUNCTION or T_VAR or ‘}’ in /home/.amelda/me/my.domain.cc/common/class.User.php on line 54

Any thoughts?

undisclosedp [+2]
Aug 25 09 at 1:35 am

@undisclosedp: Are you sure you’re using PHP5 to crawl? Check this out: Installing Twitalytic on Dreamhost

Gina Trapani [+195]
Aug 25 09 at 7:56 am

@Gina Doh… its the simple things. The domain was set to php5 but calling php on the cli uses 4! I’ve moved things on to a permissions error now.

$usr/local/php5/bin/php /home/me/my.domain.cc/crawler/crawl.php

Warning: fopen(/crawler.log): failed to open stream: Permission denied in /home/.amelda/me/my.domain.cc/common/class.Logger.php on line 41
can’t open file /crawler.log

Which is peculiar. I’ve even done the old chmod 777 on the whole directory structure (bad I know) and its still erroring. Thoughts? And thanks for your help!

undisclosedp [+2]
Aug 25 09 at 8:50 am

Aha, reading the comments above looks like the logger doesnt log to the path in the config. Having edited class.Logger.php to read the correct value all is well! Brilliant. Crawling my tweets now. Thanks Gina!

undisclosedp [+2]
Aug 25 09 at 8:56 am

I installed this today and love it. Thanks Gina!

I’m a total novice! I hope I can get this set up. Couple ideas though:
1) When Do you Twitter Most? (Day of Week, Time of Day, etc.)
2) What sites do you link to the most (NOT Bit.ly)

Looks awesome!
Will post back!

snyderxc [+8]
Aug 25 09 at 5:44 pm

Thanks for this wonderful open source contribution Gina! I installed this on my Ubuntu server at home and really like the threading features and the fact that I can keep a local copy of my tweets. I can’t wait to see how it influences the movement towards a federated Twitter.

P.S. Keep up the good work on TWiG. You, Jeff and Leo always have great insights that I enjoy thinking about throughout the week.

Stephen Zeidner [+1]
Aug 27 09 at 8:22 am

Gina,

Looks like David Pogue is looking for something like this, so I told him about your code.

http://pogue.blogs.nytimes.com/2009/08/27/tools-for-twitter/#comment-216057

We’ll see if he’s interested.

Chris

cmcraft
Aug 28 09 at 4:25 am

Just heard about this on the TWIG podcast. Sounds pretty cool, though I have no webapp background to even attempt to use it.

Mike Carlucci [+3]
Aug 28 09 at 4:42 am

Got it up and running on http://ival.us just a project for now.. not the main purpose for purchasing the domain… (ival.us = visual)

got80s [+1]
Aug 30 09 at 6:12 pm

I’m a noob, so I have to ask: What do I do when PuTTY tells me “ln: creating symbolic link ‘twitalytic/webapp’ to ‘alyxportur.com/tweets’: No such file or directory”? (pls email me via capryor@alyxportur.com so I can get this fixed fast. Tks!)

capryor
Aug 30 09 at 7:39 pm

Gina, great work – thanks!

Quick suggestion: have a suffix at the beginning of your table names. For example, “owners” becomes “ta_owners”, “tweets” becomes “ta_tweets”, etc…

My host only gives me one MySQL database that all my apps can access. Other apps have this suffix, such as “mt_*” for Movable Type tables, “g2_*” for Gallery2 tables, etc… so it makes it much easier to work with the tables.

ehuna
Aug 31 09 at 8:16 pm

Gina,

For those of us not quite at the level of installing our own servers and then this software onto the servers, do you have any advice or recommendations? I would love to first and foremost archive and have access to my tweets. I would love the ability to run some analytics as well, but more importantly would be having access to my whole history.

Jeff Jarvis gave one solution using Twistory and Google Calendars, but it only goes back 1 week’s time. While I haven’t been tweeting for years, it has been a few months and 1 week’s worth is much less than I’d like.

So, any tips or suggestions?

Thanks,

mwgrl5

mwgrl5
Sep 12 09 at 5:59 pm

Hi Gina,

Great idea here, I stumbled across the project via the Mine!Project blog.

I’ve installed the app locally and got as far as trying to add my Twitter account into the system – for some reason the app isn’t generating an oAuth token, I’m getting some php Warnings. I haven’t got much more time to spend on it now, but it looks very useful, will try and see if I can figure out the token problem

@ketan

Ketan
Oct 10 09 at 11:08 am


Comments are closed. Thanks for reading!