Twitter data– the endless stream of tweets, the user network, and the rise and fall of hashtags– offers a flood of insight into the minute-by-minute state of the society. Or at least one self-selecting part of it. A lot of people want to use it for research, and it turns out to be pretty easy to do so.<\/p>\n
You can either purchase twitter data, or collect it in real-time. If you purchase twitter data, it’s all organized for you and available historically, but it basically isn’t anything that you can’t get yourself by monitoring twitter in real-time. I’ve used GNIP<\/a>, where the going rate was about $500 per million tweets in 2013.<\/p>\n There are two main ways to collect data directly from twitter: “queries” and the “stream”. Queries let you get up to 1000 tweets at any point in time– whichever the most recent tweets that match your search criteria. The stream gives you a fraction of a percent of tweets continuously, which very quickly adds up, based on filtering criteria.<\/p>\n Scripts for doing these two options are below, but you need to decide on the search\/streaming criteria. Typically, these are search terms and geographical constraints. See Twitter’s API documentation<\/a> to decide on your search options.<\/p>\n Twitter uses an athentication system to identify both the individual collecting the data, and what tool is helping them do it. It is easy to register a new tool, whereby you pretend that you’re a startup with a great new app. Here are the steps:<\/p>\n Here is what a simple script looks like:<\/p>\n For automating twitter collection, I’ve put together scripts for queries (search.py<\/tt>), streaming (filter.py<\/tt>), and bash scripts that run them repeatedly (repsearch.sh<\/tt> and repfilter.sh<\/tt>). Download the scripts<\/a>.<\/p>\n To use the repetition scripts, make the repetition scripts executable by running “chmod a+x repsearch.sh repfilter.sh<\/tt>“. Then run them, by typing .\/repfilter.sh<\/tt> or .\/repsearch.sh<\/tt>. Note that these will create many many files over time, which you’ll have to merge together.<\/p>\n","protected":false},"excerpt":{"rendered":" Twitter data– the endless stream of tweets, the user network, and the rise and fall of hashtags– offers a flood of insight into the minute-by-minute state of the society. Or at least one self-selecting part of it. A lot of people want to use it for research, and it turns out to be pretty easy … Continue reading Scripts for Twitter Data<\/span> \n
\nimport os, twitter\n\nAPP_NAME = "Your app name"\nCONSUMER_KEY = 'Your consumer key'\nCONSUMER_SECRET = 'Your consumer token'\n\n# Do we already have a token saved?\nMY_TWITTER_CREDS = os.path.expanduser('~\/.class_credentials')\nif not os.path.exists(MY_TWITTER_CREDS):\n # This will ask you to accept the permissions and save the token\n twitter.oauth_dance(APP_NAME, CONSUMER_KEY, CONSUMER_SECRET,\n MY_TWITTER_CREDS)\n\n# Read the token\noauth_token, oauth_secret = twitter.read_token_file(MY_TWITTER_CREDS)\n\n# Open up an API object, with the OAuth token\napi = twitter.Twitter(api_version="1.1", auth=twitter.OAuth(oauth_token, oauth_secret, CONSUMER_KEY, CONSUMER_SECRET))\n\n# Perform our query\ntweets = api.search.tweets(q="risky business")\n\n# Print the results\nfor tweet in tweets['statuses']:\n if not 'text' in tweet:\n continue\n\n print tweet\n break\n<\/pre>\n