froki / basic_twitter_scraper_533

Twitter Scraper with geocoding and date


This is an example scraper of how to get tweets within a special area and time frame.

Made as a response to this Stackoverflow Question:
http://stackoverflow.com/questions/16472463/scraperwiki-twitter-query/16473317#16473317

Forked from ScraperWiki

Contributors froki

Last run failed with status code 1.

Console output of last run

Traceback (most recent call last): File "/repo/scraper.py", line 22, in <module> fh = urllib.urlopen(basic_url + query) File "/usr/lib/python2.7/urllib.py", line 86, in urlopen return opener.open(url) File "/usr/lib/python2.7/urllib.py", line 207, in open return getattr(self, name)(url) File "/usr/lib/python2.7/urllib.py", line 450, in open_https return self.http_error(url, fp, errcode, errmsg, headers) File "/usr/lib/python2.7/urllib.py", line 371, in http_error result = method(url, fp, errcode, errmsg, headers) File "/usr/lib/python2.7/urllib.py", line 682, in http_error_401 errcode, errmsg, headers) File "/usr/lib/python2.7/urllib.py", line 380, in http_error_default raise IOError, ('http error', errcode, errmsg, headers) IOError: ('http error', 401, 'Unauthorized', <httplib.HTTPMessage instance at 0x27dbcb0>)

Data

Downloaded 2 times by dyakhnov MikeRalphson

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (18 KB) Use the API

rows 10 / 15

text to_user_id_str in_reply_to_status_id id profile_image_url_https source geocode id_str from_user_name metadata profile_image_url to_user from_user from_user_id to_user_id iso_language_code to_user_name created_at location in_reply_to_status_id_str from_user_id_str place geo
@WillMcAvoyACN vs. Europe (exl Eastern Europe) & a few areas of Asia. But what is your guess as to WHY violent crime for us is worse?
617825073
332643922750869505
332644719144337408
<a href="http://twitter.com/download/iphone">Twitter for iPhone</a>
37.781157,-122.398720,25mi
332644719144337408
James Wilder
{'result_type': 'recent'}
WillMcAvoyACN
jamesdwilder
55937627
617825073
en
Will McAvoy
Thu, 09 May 2013 23:54:15 +0000
San Francisco, CA
332643922750869505
55937627
Have you taught abroad in #Vienna?! Help us out and write a teach guide for @GoOverseas! #TEFL #ESL #Austria #Europe http://t.co/rqP3sIOgdY
332643913460510720
<a href="http://bufferapp.com">Buffer</a>
37.781157,-122.398720,25mi
332643913460510720
Rachel Overseas
{'result_type': 'recent'}
TeachReviews
371015517
en
Thu, 09 May 2013 23:51:03 +0000
Berkeley, CA
371015517
@AutoPublication Are you in Canada or Europe?
152360033
332630126565158912
332639936018776064
<a href="http://itunes.apple.com/us/app/twitter/id409789998?mt=12">Twitter for Mac</a>
37.781157,-122.398720,25mi
332639936018776064
aerbook
{'result_type': 'recent'}
AutoPublication
aerbook
183055043
152360033
en
AutoPublication
Thu, 09 May 2013 23:35:15 +0000
San Francisco, CA
332630126565158912
183055043
RT @Crunchyroll: News: "Dead or Alive 5 Ultimate" Confirmed for North America and Europe http://t.co/utfovVcNJ0
332639402784337920
<a href="http://twitter.com/download/iphone">Twitter for iPhone</a>
37.781157,-122.398720,25mi
332639402784337920
Ryan Gendreau
{'result_type': 'recent'}
evilshadowx
36159823
en
Thu, 09 May 2013 23:33:08 +0000
Wishing I was in Japan
36159823
RT @Crunchyroll: News: "Dead or Alive 5 Ultimate" Confirmed for North America and Europe http://t.co/utfovVcNJ0
332639387978436608
<a href="http://levelupstudio.com/plume">Plume for Android</a>
37.781157,-122.398720,25mi
332639387978436608
Kristina
{'result_type': 'recent'}
ChibiUFO
35598437
en
Thu, 09 May 2013 23:33:04 +0000
Canada-Land
35598437
News: "Dead or Alive 5 Ultimate" Confirmed for North America and Europe http://t.co/utfovVcNJ0
332639128321667072
<a href="http://www.crunchyroll.com">Crunchyroll</a>
37.781157,-122.398720,25mi
332639128321667072
Crunchyroll
{'result_type': 'recent'}
Crunchyroll
28023330
en
Thu, 09 May 2013 23:32:02 +0000
San Francisco, CA
28023330
Europe stocks end higher after US data http://t.co/oCUF0K6ZRL
332638394066808832
<a href="http://www.snsanalytics.com">SNS Analytics</a>
37.781157,-122.398720,25mi
332638394066808832
MarketWatch News
{'result_type': 'recent'}
MarketWatchNews
200896018
en
Thu, 09 May 2013 23:29:07 +0000
San Francisco
200896018
Happy Europe Day! Register .EU, your European Identity, for just $2.99 during our #sale! http://t.co/2PbPYmZoVH @Euregistry
332636914261831680
<a href="http://twitter.com/">web</a>
37.781157,-122.398720,25mi
332636914261831680
Dynadot
{'result_type': 'recent'}
Dynadot
89060745
en
Thu, 09 May 2013 23:23:15 +0000
San Mateo, CA
89060745
I think I might go to Central Europe next summer :o
332635594591174657
<a href="http://twitter.com/download/iphone">Twitter for iPhone</a>
37.781157,-122.398720,25mi
332635594591174657
CarouselSniperVictim
{'result_type': 'recent'}
EricParkerAsc
870195673
en
Thu, 09 May 2013 23:18:00 +0000
870195673
{'type': 'CITY', 'id': '51d80bae0d6cb453', 'full_name': 'Fairview'}
{'type': 'Point', 'coordinates': [37.6866996929, -122.0547432772]}
Yet another brazen robbery. - Focus Turns to 3 Suspects in Belgian Diamond Robbery, via @nytimes http://t.co/sbJ5y2j11P
332634567586488320
<a href="http://itunes.apple.com/us/app/nytimes/id284862083?mt=8&uo=4">NYTimes on iOS</a>
37.781157,-122.398720,25mi
332634567586488320
Dushka Zapata
{'result_type': 'recent'}
DushkaAmateur
20890666
en
Thu, 09 May 2013 23:13:55 +0000
San Francisco
20890666

Statistics

Total run time: less than 10 seconds

Total cpu time used: less than 5 seconds

Total disk space used: 36 KB

History

  • Manually ran revision 31f6da5a and failed .
    nothing changed in the database
  • Manually ran revision 31f6da5a and failed .
    nothing changed in the database
  • Manually ran revision 31f6da5a and failed .
    nothing changed in the database
  • Forked from ScraperWiki

Scraper code

Python

basic_twitter_scraper_533 / scraper.py