Update, July 2010: Twitalytic has been renamed ThinkUp.
Been hacking away at a personal programming project, tentatively named Twitalytic, a web application that archives your tweets and mentions and gives you useful data about your friends and followers on Twitter. (Here’s some background on why the heck I’d ever want to archive the ephemeral chatter that is on the Twitter.)
Anyway, after a few months I’ve collected a teensy bit of data, so I wanted to make sharing replies to my lazyweb tweets easier. To that end, today I added a public face to the app. Here’s the Twitalytic public timeline, which lists my most recent tweets and the number of replies each has. Click on the reply total to drill down into the responses to a particular tweet, like my tweet asking what people eat first thing in the morning. (Ha, I made over 200 people Twitter about breakfast!)
That listing is the most basic thing Twitalytic can do. When you set it up and log into the app, you can curate replies and see other views of your data, like your least likely followers, your least and most active friends, your most replied-to tweets and more. For each of your friends or followers, you can see mutual friends, conversations you’ve had, and even what Twitter client that person uses most. If you’re approaching the 3,200 tweet mark, Twitalytic will also back up your tweets and let you export them into a simple text file. Here are some screenshots of the internal, logged-in only views.
List your tweets by most recent, most replied-to, or back-and-forth conversations.
Here’s the conversation view.
File replies as “standalone”–i.e., someone just said something to you out of the blue, or associate mentions with a specific tweet (if the user didn’t use the “reply” button in their Twitter client).
Your “least likely” followers are those who have the largest follower to followee ratio.
See which of your friends are most active and least active.
Click on any tweet’s reply count to view its responses, “likely” reponses that happened around the time of that tweet, and public/republishable replies (those from users’s whose tweets are not protected).
Click on a username to see recent tweets, conversations, mutual friends, and what client that person uses.
I realize there are ten dozen Twitter applications out there that do much of this, and possibly better. Creating yet another Twitter app is probably the most cliché development undertaking of the year, but I’ve had a ball using this little thing to teach myself a few things about the Twitter API, OAuth, and jQuery and brush up on my PHP and MySQL skills.
To use Twitalytic you’ve got to install it on your own server and run the crawler to start gathering your data. As far as I know, only one other brave soul beside me has done this; I don’t recommend it unless you’re well-versed in PHP development and LAMP environments. This is still very alpha stuff–and I haven’t built in any easy way to update an installation either, which means as the code changes, manual re-installation will be required. That said, I welcome PHP hackers to check out the GitHub repository and start tooling with the code. It’s been a very long time since I’ve developed a webapp of this size, and the code is full of //TODO‘s and //Must be a better way to do this‘s, too.
At some point I’ll set up some sort of dedicated web page and perhaps a mailing list for the app. Until then, post your questions and thoughts here.
Update: Wow, I didn’t ever think this little spare-time data-wanking PHP thing would be TechCrunch-worthy. Thanks for the post, TC!
28 Comments
AJ Robins
This is really, really cool!
Unfortunately, I don’t have the time to install and play with it, but I wish I did! 🙂
ffff00
I’m digging all of the features that you’re building into this, I’d love to try it out once it gets a little further along (with an easy way to update).
Keep up the good work Gina, this is coming along really nicely.
Thejesh GN
Awesome. I am going to install it today.
Laurent Denoue
I like your code! I am also curating twitter data, but to detect stuff happening in cities.
I detect city names, dates and event keywords and aggregate tweets. It’s kind of cool to see where people are going (festivals, conferences, tweetups, etc.) in a given city:
http://www.twillage.com
and
http://www.twillage.com/today
Steven Rossi
Looks really cool – I like that you’ve created lots of unique ways to display information. I’m looking forward to trying it out.
Sandaruwan Gunathilake
I tried this on my local web server. (Manually entered the oauth_token in the GET url).
There was a problem in the crawler logger. The realpath(‘logs/’) evaluated to null it tried to save logs in my root directory. Why did you commented “global config” parameters? I took the log path from it.
openid.teknatus.com/pjain
Gina,
This is great stuff! I got it up and running in less than 15 minutes with a few minor changes.
By putting the code up on GitHub, are you releasing the code under some sort of opensource license such as the GPL, MIT, MSD, etc.?
Also, since you’re getting back into coding PHP, I’d highly recommend taking a look at CodeIgniter. I found CI to be a great framework for implementing MVC in PHP.
Cheers,
Pankaj
Sandeep Bali
I have a doubt.. by default.. in twitalytic/common/config.inc.php the path for logs goes to /advanced/logs/ while there is no such directory.. am I supposed to replace it to crawler/log/ along with rest of the path..??
Ken Yarmosh
I own the domain (www.twittalytic.com) and Twitter handle (tiwttalytic) and am planning to release something relatively soon. But I wasn’t sold on “Twittalytic” because it’s a little too geeky.
In any case, I’m approaching some of these problems slightly differently. Good to see some validation here though. Congrats. 🙂
Gina Trapani
@Sandeep and @Sandaruwan: the sample config file line to the log should be:
$TWITALYTIC_CFG[‘log_location’] = ‘/your-path-totwitalytic/crawler/logs/’;
Sorry about that.
macpug
I like this a lot. I would definitely use it, if for no other reason than I’m intrigued by the stats (’cause I’m just a regular gal with a small number of followers).
@Ken it’s *supposed* to be geeky 😛
Waldron Faulkner
Hey Gina, long time, no talk!
Love what you’re doing. I’ve also been working on a Twitter project, which I’m calling GraphEdge (http://graphedge.com). I’m looking at some of the same metrics, but focused more on follows and drops. More a B2B play, but certainly valuable to obsessed Twitter consumers. Will be adding something close to “least likely” and “most likely” soon, plus a lot more.
Drop me a note and we’ll talk.
– Waldron
waldron [at] graphedge.com
Sandaruwan Gunathilake
It seems that the current code doesn’t use the path mentioned in $TWITALYTIC_CFG[‘log_location’]. In line 14 of “common/class.Logger.php”, you are using “realpath(‘logs/’);”.
Small suggestion : use “dirname(__FILE__)”.
~ Sandaruwan
wee493
Awesome! I have just installed it and it’s working great!
One thing I did notice is that if I view a users page (like click on someone else’s username to see mutual friends ect.) it screws up the session. Then every page I go to after that gives me an error like “Error, selection query failed: SELECT * , round(total_replies_in_system/(datediff(curdate(), earliest_reply_in_system)), 2) as avg_replies_per_day FROM instances i INNER JOIN owner_instances oi ON i.id = oi.instance_id WHERE oi.owner_id = ORDER BY crawler_last_run DESC”
undisclosedp
I’ve managed to install this on dreamhost and got the front end working succesfully but every time I try to run the crawler I get the following error.
$php crawler/crawl.php
Parse error: syntax error, unexpected T_STRING, expecting T_OLD_FUNCTION or T_FUNCTION or T_VAR or ‘}’ in /home/.amelda/me/my.domain.cc/common/class.User.php on line 54
Any thoughts?
Gina Trapani
@undisclosedp: Are you sure you’re using PHP5 to crawl? Check this out: Installing Twitalytic on Dreamhost
undisclosedp
@Gina Doh… its the simple things. The domain was set to php5 but calling php on the cli uses 4! I’ve moved things on to a permissions error now.
$usr/local/php5/bin/php /home/me/my.domain.cc/crawler/crawl.php
Warning: fopen(/crawler.log): failed to open stream: Permission denied in /home/.amelda/me/my.domain.cc/common/class.Logger.php on line 41
can’t open file /crawler.log
Which is peculiar. I’ve even done the old chmod 777 on the whole directory structure (bad I know) and its still erroring. Thoughts? And thanks for your help!
undisclosedp
Aha, reading the comments above looks like the logger doesnt log to the path in the config. Having edited class.Logger.php to read the correct value all is well! Brilliant. Crawling my tweets now. Thanks Gina!
Michael Doornbos
I installed this today and love it. Thanks Gina!
snyderxc
I’m a total novice! I hope I can get this set up. Couple ideas though:
1) When Do you Twitter Most? (Day of Week, Time of Day, etc.)
2) What sites do you link to the most (NOT Bit.ly)
Looks awesome!
Will post back!
Stephen Zeidner
Thanks for this wonderful open source contribution Gina! I installed this on my Ubuntu server at home and really like the threading features and the fact that I can keep a local copy of my tweets. I can’t wait to see how it influences the movement towards a federated Twitter.
P.S. Keep up the good work on TWiG. You, Jeff and Leo always have great insights that I enjoy thinking about throughout the week.
cmcraft
Gina,
Looks like David Pogue is looking for something like this, so I told him about your code.
http://pogue.blogs.nytimes.com/2009/08/27/tools-for-twitter/#comment-216057
We’ll see if he’s interested.
Chris
Mike Carlucci
Just heard about this on the TWIG podcast. Sounds pretty cool, though I have no webapp background to even attempt to use it.
got80s
Got it up and running on http://ival.us just a project for now.. not the main purpose for purchasing the domain… (ival.us = visual)
capryor
I’m a noob, so I have to ask: What do I do when PuTTY tells me “ln: creating symbolic link ‘twitalytic/webapp’ to ‘alyxportur.com/tweets’: No such file or directory”? (pls email me via capryor@alyxportur.com so I can get this fixed fast. Tks!)
ehuna
Gina, great work – thanks!
Quick suggestion: have a suffix at the beginning of your table names. For example, “owners” becomes “ta_owners”, “tweets” becomes “ta_tweets”, etc…
My host only gives me one MySQL database that all my apps can access. Other apps have this suffix, such as “mt_*” for Movable Type tables, “g2_*” for Gallery2 tables, etc… so it makes it much easier to work with the tables.
mwgrl5
Gina,
For those of us not quite at the level of installing our own servers and then this software onto the servers, do you have any advice or recommendations? I would love to first and foremost archive and have access to my tweets. I would love the ability to run some analytics as well, but more importantly would be having access to my whole history.
Jeff Jarvis gave one solution using Twistory and Google Calendars, but it only goes back 1 week’s time. While I haven’t been tweeting for years, it has been a few months and 1 week’s worth is much less than I’d like.
So, any tips or suggestions?
Thanks,
mwgrl5
Ketan
Hi Gina,
Great idea here, I stumbled across the project via the Mine!Project blog.
I’ve installed the app locally and got as far as trying to add my Twitter account into the system – for some reason the app isn’t generating an oAuth token, I’m getting some php Warnings. I haven’t got much more time to spend on it now, but it looks very useful, will try and see if I can figure out the token problem
@ketan