everypolitician-scrapers / malaysian_parliament-wikidata

Scrapes api.morph.io, en.wikipedia.org, ms.wikipedia.org, and 1 other domain

Get structured data out of the web. Code collaboration through GitHub. Run your scrapers in the cloud.


Contributors tmtmtmtm ondenman

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.0.0 -----> Installing dependencies using bundler 1.11.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler is older than the version that created the lockfile. We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/.........  Fetching version metadata from https://rubygems.org/...  Fetching dependency metadata from https://rubygems.org/..  Fetching https://github.com/openc/sqlite_magic.git  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Fetching https://github.com/everypolitician/wikidata-fetcher.git  Fetching https://github.com/everypolitician/wikisnakker  Rubygems 2.0.14.1 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation.  Installing coderay 1.1.0  Installing colorize 0.8.1  Installing diskcached 1.1.2  Installing unf_ext 0.0.7.2 with native extensions  Installing excon 0.53.0  Installing multipart-post 2.0.0  Installing hashie 3.4.6  Installing httpclient 2.8.2.4  Installing json 2.0.2 with native extensions  Installing method_source 0.8.2  Installing mime-types-data 3.2016.0521  Installing mini_portile2 2.1.0  Installing netrc 0.11.0  Installing pkg-config 1.1.7  Installing open-uri-cached 0.0.5  Installing slop 3.6.0  Installing sqlite3 1.3.11 with native extensions  Using wikisnakker 0.7.0 from https://github.com/everypolitician/wikisnakker (at master@cc55a4f)  Using bundler 1.11.2  Installing unf 0.1.4  Installing faraday 0.9.2  Installing mime-types 3.1  Installing nokogiri 1.6.8 with native extensions  Installing pry 0.10.3  Using sqlite_magic 0.0.6 from https://github.com/openc/sqlite_magic.git (at master@4df975e)  Installing domain_name 0.5.20160826  Installing faraday_middleware 0.10.0  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Installing http-cookie 1.0.3  Installing wikidata-client 0.0.10  Installing faraday-cookie_jar 0.0.6  Installing rest-client 2.0.0  Installing mediawiki_api 0.7.0  Using wikidata-fetcher 0.17.1 from https://github.com/everypolitician/wikidata-fetcher.git (at master@2ab5ce3)  Bundle complete! 10 Gemfile dependencies, 34 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (33.88s)  Cleaning up the bundler cache.  Warning: the running version of Bundler is older than the version that created the lockfile. We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Can't find Wikidata IDs for: Mohamed Said Keruak in en Can't find Wikidata IDs for: Mohamed Said Keruak in en Can't find Wikidata IDs for: Mohamed Said Keruak in en /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:878:in `initialize': Network is unreachable - connect(2) (Errno::ENETUNREACH) from /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:878:in `open' from /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:878:in `block in connect' from /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/timeout.rb:52:in `timeout' from /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:877:in `connect' from /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:862:in `do_start' from /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:851:in `start' from /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:313:in `open_http' from /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:709:in `buffer_open' from /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:210:in `block in open_loop' from /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:208:in `catch' from /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:208:in `open_loop' from /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:149:in `open_uri' from /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:689:in `open' from /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:34:in `open' from /app/vendor/bundle/ruby/2.0.0/bundler/gems/wikisnakker-cc55a4f45290/lib/wikisnakker.rb:53:in `get' from /app/vendor/bundle/ruby/2.0.0/bundler/gems/wikisnakker-cc55a4f45290/lib/wikisnakker.rb:21:in `block in initialize' from /app/vendor/bundle/ruby/2.0.0/bundler/gems/wikisnakker-cc55a4f45290/lib/wikisnakker.rb:20:in `each' from /app/vendor/bundle/ruby/2.0.0/bundler/gems/wikisnakker-cc55a4f45290/lib/wikisnakker.rb:20:in `each_slice' from /app/vendor/bundle/ruby/2.0.0/bundler/gems/wikisnakker-cc55a4f45290/lib/wikisnakker.rb:20:in `each' from /app/vendor/bundle/ruby/2.0.0/bundler/gems/wikisnakker-cc55a4f45290/lib/wikisnakker.rb:20:in `map' from /app/vendor/bundle/ruby/2.0.0/bundler/gems/wikisnakker-cc55a4f45290/lib/wikisnakker.rb:20:in `initialize' from /app/vendor/bundle/ruby/2.0.0/bundler/gems/wikisnakker-cc55a4f45290/lib/wikisnakker.rb:11:in `new' from /app/vendor/bundle/ruby/2.0.0/bundler/gems/wikisnakker-cc55a4f45290/lib/wikisnakker.rb:11:in `find' from /app/vendor/bundle/ruby/2.0.0/bundler/gems/wikisnakker-cc55a4f45290/lib/wikisnakker.rb:121:in `find' from /app/vendor/bundle/ruby/2.0.0/bundler/gems/wikidata-fetcher-2ab5ce3068a7/lib/wikidata/fetcher.rb:204:in `find' from /app/vendor/bundle/ruby/2.0.0/bundler/gems/wikidata-fetcher-2ab5ce3068a7/lib/wikidata/fetcher.rb:79:in `block in scrape_wikidata' from /app/vendor/bundle/ruby/2.0.0/bundler/gems/wikidata-fetcher-2ab5ce3068a7/lib/wikidata/fetcher.rb:77:in `each_slice' from /app/vendor/bundle/ruby/2.0.0/bundler/gems/wikidata-fetcher-2ab5ce3068a7/lib/wikidata/fetcher.rb:77:in `scrape_wikidata' from scraper.rb:10:in `<main>'

Data

Downloaded 77 times by everypolitician ondenman

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (467 KB) Use the API

rows 0 / 0

Statistics

Average successful run time: 4 minutes

Total run time: 3 days

Total cpu time used: about 1 hour

Total disk space used: 517 KB

History

  • Auto ran revision 0bf94504 and failed .
    506 records removed in the database
    16 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    506 records updated in the database
    547 pages scraped
  • Auto ran revision 0bf94504 and failed .
    nothing changed in the database
  • Auto ran revision 0bf94504 and completed successfully .
    1 record added, 505 records updated in the database
    547 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    505 records updated in the database
    546 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    505 records updated in the database
    546 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    503 records updated in the database
    546 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    505 records updated in the database
    546 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    1 record added, 504 records updated in the database
    546 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    504 records updated in the database
    545 pages scraped
  • Auto ran revision 0bf94504 and failed .
  • Auto ran revision 0bf94504 and failed .
    nothing changed in the database
  • Auto ran revision 0bf94504 and failed .
    nothing changed in the database
  • Auto ran revision 0bf94504 and failed .
    nothing changed in the database
  • Auto ran revision 0bf94504 and completed successfully .
    nothing changed in the database
  • Auto ran revision 0bf94504 and completed successfully .
    503 records updated in the database
    545 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    504 records updated in the database
    545 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    504 records updated in the database
    545 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    504 records updated in the database
    545 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    504 records updated in the database
    545 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
    545 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    nothing changed in the database
    545 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    nothing changed in the database
  • Auto ran revision 0bf94504 and completed successfully .
    504 records updated in the database
    545 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
    545 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    1 record added, 501 records updated in the database
    545 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
    544 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    503 records added in the database
    544 pages scraped
  • Auto ran revision 0bf94504 and failed .
    nothing changed in the database
  • Auto ran revision 0bf94504 and failed .
    503 records removed in the database
    402 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    503 records updated in the database
    544 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
    544 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
    544 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    503 records added in the database
    544 pages scraped
  • Auto ran revision 0bf94504 and failed .
    503 records removed in the database
    138 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    503 records updated in the database
    544 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
    544 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    501 records updated in the database
  • Auto ran revision 0bf94504 and failed .
  • Auto ran revision 0bf94504 and completed successfully .
    501 records updated in the database
    544 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    1 record added, 502 records updated in the database
  • Auto ran revision 0bf94504 and failed .
    nothing changed in the database
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
    543 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    nothing changed in the database
  • Auto ran revision 0bf94504 and completed successfully .
    nothing changed in the database
    543 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
    543 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    nothing changed in the database
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
  • Auto ran revision 0bf94504 and completed successfully .
    500 records updated in the database
  • Auto ran revision 0bf94504 and completed successfully .
    501 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    501 records updated in the database
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
    542 pages scraped
  • Auto ran and failed .
  • Auto ran revision 0bf94504 and completed successfully .
    nothing changed in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    498 records updated in the database
  • Auto ran revision 0bf94504 and completed successfully .
    501 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    501 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    498 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    501 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    501 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
  • Auto ran revision 0bf94504 and completed successfully .
    501 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    500 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    501 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    501 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    nothing changed in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    nothing changed in the database
  • Auto ran revision 0bf94504 and completed successfully .
    nothing changed in the database
    543 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    nothing changed in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    nothing changed in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    nothing changed in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    499 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    501 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    502 records updated in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and completed successfully .
    502 records added in the database
    542 pages scraped
  • Auto ran revision 0bf94504 and failed .
    502 records removed in the database
    185 pages scraped
  • Manually ran revision 0bf94504 and completed successfully .
    502 records added in the database
    542 pages scraped
  • Manually ran revision 0bf94504 and failed .
    nothing changed in the database
    1 page scraped
  • Manually ran revision 0bf94504 and failed .
    nothing changed in the database
    1 page scraped
  • Created on morph.io