everypolitician-scrapers / russia-duma-2016

Scrapes github.com and www.duma.gov.ru

Build software better, together.


Contributors tmtmtmtm chrismytton ondenman

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.3.3 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/.........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/everypolitician/combine_popolo_memberships.git  Fetching https://github.com/everypolitician/scraped.git  Fetching https://github.com/everypolitician/scraped_page_archive.git  Fetching https://github.com/everypolitician/scraper_test.git  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Fetching https://github.com/everypolitician/table_unspanner.git  Fetching rake 12.0.0  Fetching public_suffix 2.0.5  Fetching ast 2.3.0  Installing ast 2.3.0  Installing rake 12.0.0  Using bundler 1.15.2  Fetching coderay 1.1.1  Installing public_suffix 2.0.5  Using combine_popolo_memberships 0.2.0 from https://github.com/everypolitician/combine_popolo_memberships.git (at master@5769841)  Fetching safe_yaml 1.0.4  Fetching unf_ext 0.0.7.4  Installing safe_yaml 1.0.4  Fetching execjs 2.7.0  Installing unf_ext 0.0.7.4 with native extensions  Installing coderay 1.1.1  Fetching field_serializer 0.3.0  Installing execjs 2.7.0  Fetching git 1.3.0  Installing git 1.3.0  Fetching hashdiff 0.3.4  Installing hashdiff 0.3.4  Fetching httpclient 2.8.3  Installing httpclient 2.8.3  Installing field_serializer 0.3.0  Fetching method_source 0.8.2  Installing method_source 0.8.2  Fetching mime-types-data 3.2016.0521  Fetching mini_portile2 2.2.0  Installing mime-types-data 3.2016.0521  Installing mini_portile2 2.2.0  Fetching minitest 5.10.3  Fetching vcr 3.0.3  Installing minitest 5.10.3  Installing vcr 3.0.3  Fetching netrc 0.11.0  Installing netrc 0.11.0  Fetching open-uri-cached 0.0.5  Installing open-uri-cached 0.0.5  Fetching parallel 1.12.0  Installing parallel 1.12.0  Fetching powerpack 0.1.1  Installing powerpack 0.1.1  Fetching slop 3.6.0  Installing slop 3.6.0  Fetching require_all 1.4.0  Fetching ruby-progressbar 1.8.1  Installing require_all 1.4.0  Installing ruby-progressbar 1.8.1  Fetching unicode-display_width 1.3.0  Installing unicode-display_width 1.3.0  Fetching sqlite3 1.3.13  Fetching parser 2.4.0.0  Installing sqlite3 1.3.13 with native extensions  Installing parser 2.4.0.0  Fetching rainbow 2.2.2  Installing rainbow 2.2.2 with native extensions  Fetching addressable 2.5.1  Installing addressable 2.5.1  Fetching crack 0.4.3  Installing crack 0.4.3  Fetching nokogiri 1.8.0  Installing nokogiri 1.8.0 with native extensions  Fetching mime-types 3.1  Installing mime-types 3.1  Fetching minispec-metadata 2.0.0  Installing minispec-metadata 2.0.0  Fetching minitest-around 0.4.0  Fetching pry 0.10.4  Installing pry 0.10.4  Installing minitest-around 0.4.0  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching webmock 2.0.3  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Installing webmock 2.0.3  Fetching minitest-vcr 1.4.0  Fetching rubocop 0.49.1  Installing minitest-vcr 1.4.0  Fetching domain_name 0.5.20170404  Installing rubocop 0.49.1  Installing domain_name 0.5.20170404  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching vcr-archive 0.3.0  Installing vcr-archive 0.3.0  Using scraper_test 0.1.0 from https://github.com/everypolitician/scraper_test.git (at master@9b4326c)  Fetching http-cookie 1.0.3  Using scraped_page_archive 0.5.0 from https://github.com/everypolitician/scraped_page_archive.git (at master@3b67c31)  Using scraped 0.6.2 from https://github.com/everypolitician/scraped.git (at master@58c88c1)  Using table_unspanner 0.1.0 from https://github.com/everypolitician/table_unspanner.git (at master@a70a98a)  Installing http-cookie 1.0.3  Fetching rest-client 2.0.2  Installing rest-client 2.0.2  Bundle complete! 18 Gemfile dependencies, 50 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Post-install message from webmock:  WebMock 2.0 has some breaking changes. Please check the CHANGELOG: https://goo.gl/piDGLu  Bundle completed (26.94s)  Cleaning up the bundler cache. -----> Installing node-v6.11.1-linux-x64 -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Found 449 members Found 449 members Found 449 members /app/vendor/ruby-2.3.3/lib/ruby/2.3.0/open-uri.rb:233:in `open_loop': HTTP redirection loop: http://www.duma.gov.ru/structure/deputies/1756690/ (RuntimeError) from /app/vendor/ruby-2.3.3/lib/ruby/2.3.0/open-uri.rb:151:in `open_uri' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/scraped_page_archive-3b67c315ddab/lib/scraped_page_archive/open-uri.rb:9:in `block in open_uri' from /app/vendor/bundle/ruby/2.3.0/gems/vcr-3.0.3/lib/vcr/util/variable_args_block_caller.rb:9:in `call_block' from /app/vendor/bundle/ruby/2.3.0/gems/vcr-3.0.3/lib/vcr.rb:189:in `use_cassette' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/scraped_page_archive-3b67c315ddab/lib/scraped_page_archive.rb:36:in `record' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/scraped_page_archive-3b67c315ddab/lib/scraped_page_archive/open-uri.rb:9:in `open_uri' from /app/vendor/ruby-2.3.3/lib/ruby/2.3.0/open-uri.rb:717:in `open' from /app/vendor/ruby-2.3.3/lib/ruby/2.3.0/open-uri.rb:35:in `open' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/scraped-58c88c135f96/lib/scraped/request/strategy/live_request.rb:10:in `response' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/scraped-58c88c135f96/lib/scraped/request.rb:29:in `block in first_successful_response' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/scraped-58c88c135f96/lib/scraped/request.rb:30:in `each' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/scraped-58c88c135f96/lib/scraped/request.rb:30:in `each' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/scraped-58c88c135f96/lib/scraped/request.rb:30:in `each' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/scraped-58c88c135f96/lib/scraped/request.rb:30:in `each' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/scraped-58c88c135f96/lib/scraped/request.rb:30:in `each' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/scraped-58c88c135f96/lib/scraped/request.rb:30:in `each' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/scraped-58c88c135f96/lib/scraped/request.rb:30:in `first' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/scraped-58c88c135f96/lib/scraped/request.rb:30:in `first_successful_response' from /app/vendor/bundle/ruby/2.3.0/bundler/gems/scraped-58c88c135f96/lib/scraped/request.rb:13:in `response' from scraper.rb:18:in `scrape' from scraper.rb:31:in `block in <main>' from scraper.rb:30:in `each' from scraper.rb:30:in `<main>'

Data

Downloaded 472 times by everypolitician chrismytton

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (193 KB) Use the API

rows 10 / 11

id name image source faction area birth_date start_date end_date area_id
1756684
Абрамов Иван Николаевич
ЛДПР
Амурская область
1978-06-16
2016-09-18
0071
1756998
Авдеев Александр Александрович
«ЕДИНАЯРОССИЯ»
Калужская область
1975-08-12
2016-09-18
0099
1756685
Агаев Ваха Абуевич
КПР
все субъекты Российской Федерации
1953-03-15
2016-09-18
1756730
Адучиев Батор Канурович
«ЕДИНАЯРОССИЯ»
Республика КалмыкияСтавропольский крайАстраханская областьРостовская область
1963-01-27
2016-09-18
1756688
Азимов Рахим Азизбоевич
«ЕДИНАЯРОССИЯ»
Кировская область
1964-08-16
2016-09-18
0105
1756975
Аксаков Анатолий Геннадьевич
«СПРАВЕДЛИВАЯРОССИЯ»
Чувашская Республика - Чувашия
1957-11-28
2016-09-18
0037
1756689
Алексеева Татьяна Олеговна
«ЕДИНАЯРОССИЯ»
Кемеровская область
1962-12-16
2016-09-18
0101
1756686
Алферов Жорес Иванович
КПР
все субъекты Российской Федерации
1930-03-15
2016-09-18
1756731
Альшевских Андрей Геннадьевич
«ЕДИНАЯРОССИЯ»
Свердловская область
1972-05-14
2016-09-18
0168
1756942
Ананских Игорь Александрович
«СПРАВЕДЛИВАЯРОССИЯ»
Республика КарелияЛенинградская областьМурманская область
1966-09-06
2016-09-18

Statistics

Average successful run time: about 6 hours

Total run time: about 2 months

Total cpu time used: 27 days

Total disk space used: 192 MB

History

  • Auto ran revision 2f58a1f9 and failed .
    46 records removed in the database
    52 pages scraped
  • Auto ran revision 2f58a1f9 and failed .
    58 records removed in the database
    237 pages scraped
  • Auto ran revision 2f58a1f9 and failed .
    144 records removed in the database
    470 pages scraped
  • Auto ran revision 2f58a1f9 and failed .
    256 records added in the database
    1043 pages scraped
  • Auto ran revision 2f58a1f9 and failed .
    2 records added in the database
    20 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

russia-duma-2016 / scraper.rb