everypolitician-scrapers / twitter-mexico-diputados-friends

Scraping member accounts of Mexican Chamber of Deputies, who are all friends of the official Chamber on twitter

Scrapes api.twitter.com


Mexican Chamber of Deputies twitter scraper

The official account of Mexico's Chamber of Deputies

follows each of its members.

This wee scraper gets the list of its friends for the EveryPolitician project.

Intended to run on morph.io.

Contributors davewhiteland

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.2.4 -----> Installing dependencies using bundler 1.11.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler is older than the version that created the lockfile. We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/.........  Fetching version metadata from https://rubygems.org/...  Fetching dependency metadata from https://rubygems.org/..  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Installing addressable 2.4.0  Installing coderay 1.1.1  Installing buftok 0.2.0  Installing dotenv 2.1.1  Installing multipart-post 2.0.0  Installing equalizer 0.0.10  Installing unf_ext 0.0.7.2 with native extensions  Installing http-form_data 1.0.1  Installing http_parser.rb 0.6.0 with native extensions  Installing httpclient 2.8.0  Installing json 1.8.3 with native extensions  Installing thread_safe 0.3.5  Installing method_source 0.8.2  Installing naught 1.1.0  Installing slop 3.6.0  Installing sqlite3 1.3.11 with native extensions  Installing simple_oauth 0.3.1  Using bundler 1.11.2  Installing faraday 0.9.2  Installing unf 0.1.4  Installing memoizable 0.4.2  Installing pry 0.10.3  Installing domain_name 0.5.20160615  Installing http-cookie 1.0.2  Installing http 1.0.4  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Installing twitter 5.16.0  Bundle complete! 4 Gemfile dependencies, 28 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (17.32s)  Cleaning up the bundler cache.   ###### WARNING:  You have not declared a Ruby version in your Gemfile.  To set your Ruby version add this line to your Gemfile:  ruby '2.2.4'  # See https://devcenter.heroku.com/articles/ruby-versions for more information.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... /app/vendor/bundle/ruby/2.2.0/gems/twitter-5.16.0/lib/twitter/rest/response/raise_error.rb:15:in `on_complete': Rate limit exceeded (Twitter::Error::TooManyRequests) from /app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/response.rb:9:in `block in call' from /app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/response.rb:57:in `on_complete' from /app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/response.rb:8:in `call' from /app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/request/url_encoded.rb:15:in `call' from /app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/request/multipart.rb:14:in `call' from /app/vendor/bundle/ruby/2.2.0/gems/twitter-5.16.0/lib/twitter/rest/request/multipart_with_file.rb:19:in `call' from /app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:139:in `build_response' from /app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/connection.rb:377:in `run_request' from /app/vendor/bundle/ruby/2.2.0/gems/faraday-0.9.2/lib/faraday/connection.rb:140:in `get' from /app/vendor/bundle/ruby/2.2.0/gems/twitter-5.16.0/lib/twitter/rest/request.rb:33:in `perform' from /app/vendor/bundle/ruby/2.2.0/gems/twitter-5.16.0/lib/twitter/cursor.rb:47:in `fetch_next_page' from /app/vendor/bundle/ruby/2.2.0/gems/twitter-5.16.0/lib/twitter/enumerable.rb:13:in `each' from /app/vendor/bundle/ruby/2.2.0/gems/twitter-5.16.0/lib/twitter/enumerable.rb:14:in `each' from /app/vendor/bundle/ruby/2.2.0/gems/twitter-5.16.0/lib/twitter/enumerable.rb:14:in `each' from /app/vendor/bundle/ruby/2.2.0/gems/twitter-5.16.0/lib/twitter/enumerable.rb:14:in `each' from /app/vendor/bundle/ruby/2.2.0/gems/twitter-5.16.0/lib/twitter/enumerable.rb:14:in `each' from /app/vendor/bundle/ruby/2.2.0/gems/twitter-5.16.0/lib/twitter/enumerable.rb:14:in `each' from /app/vendor/bundle/ruby/2.2.0/gems/twitter-5.16.0/lib/twitter/enumerable.rb:14:in `each' from /app/vendor/bundle/ruby/2.2.0/gems/twitter-5.16.0/lib/twitter/enumerable.rb:14:in `each' from /app/vendor/bundle/ruby/2.2.0/gems/twitter-5.16.0/lib/twitter/enumerable.rb:14:in `each' from /app/vendor/bundle/ruby/2.2.0/gems/twitter-5.16.0/lib/twitter/enumerable.rb:14:in `each' from /app/vendor/bundle/ruby/2.2.0/gems/twitter-5.16.0/lib/twitter/enumerable.rb:14:in `each' from /app/vendor/bundle/ruby/2.2.0/gems/twitter-5.16.0/lib/twitter/enumerable.rb:14:in `each' from /app/vendor/bundle/ruby/2.2.0/gems/twitter-5.16.0/lib/twitter/enumerable.rb:14:in `each' from /app/vendor/bundle/ruby/2.2.0/gems/twitter-5.16.0/lib/twitter/enumerable.rb:14:in `each' from /app/vendor/bundle/ruby/2.2.0/gems/twitter-5.16.0/lib/twitter/enumerable.rb:14:in `each' from scraper.rb:22:in `<main>'

Data

Downloaded 2 times by octopusinvitro antoniomilanese

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (58 KB) Use the API

rows 10 / 331

Statistics

Total run time: about 2 hours

Total cpu time used: 5 minutes

Total disk space used: 80.5 KB

History

  • Auto ran revision 78084771 and failed .
    300 records added, 300 records removed in the database
    16 pages scraped
  • Auto ran revision 78084771 and failed .
    nothing changed in the database
    16 pages scraped
  • Auto ran revision 78084771 and failed .
    300 records added, 300 records removed in the database
    16 pages scraped
  • Auto ran revision 78084771 and failed .
    300 records added, 300 records removed in the database
    16 pages scraped
  • Auto ran revision 78084771 and failed .
    300 records added, 300 records removed in the database
    16 pages scraped
  • ...
  • Created on morph.io

Show complete history