This is a scraper that runs on Morph. To get started see the documentation

Contributors tmtmtmtm chrismytton

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.3.3 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/.........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/everypolitician/combine_popolo_memberships.git  Fetching https://github.com/everypolitician/scraped.git  Fetching https://github.com/everypolitician/scraped_page_archive.git  Fetching https://github.com/everypolitician/scraper_test.git  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Fetching https://github.com/everypolitician/table_unspanner.git  Fetching rake 12.0.0  Fetching public_suffix 2.0.5  Fetching ast 2.3.0  Installing ast 2.3.0  Installing rake 12.0.0  Installing public_suffix 2.0.5  Using bundler 1.15.2  Fetching coderay 1.1.1  Installing coderay 1.1.1  Using combine_popolo_memberships 0.2.0 from https://github.com/everypolitician/combine_popolo_memberships.git (at master@5769841)  Fetching safe_yaml 1.0.4  Fetching unf_ext 0.0.7.4  Installing safe_yaml 1.0.4  Fetching execjs 2.7.0  Installing execjs 2.7.0  Installing unf_ext 0.0.7.4 with native extensions  Fetching field_serializer 0.3.0  Fetching git 1.3.0  Installing field_serializer 0.3.0  Installing git 1.3.0  Fetching hashdiff 0.3.4  Fetching httpclient 2.8.3  Installing hashdiff 0.3.4  Fetching method_source 0.8.2  Installing method_source 0.8.2  Fetching mime-types-data 3.2016.0521  Installing mime-types-data 3.2016.0521  Installing httpclient 2.8.3  Fetching mini_portile2 2.2.0  Installing mini_portile2 2.2.0  Fetching minitest 5.10.3  Fetching vcr 3.0.3  Installing minitest 5.10.3  Installing vcr 3.0.3  Fetching netrc 0.11.0  Fetching open-uri-cached 0.0.5  Installing open-uri-cached 0.0.5  Installing netrc 0.11.0  Fetching parallel 1.12.0  Fetching powerpack 0.1.1  Installing parallel 1.12.0  Installing powerpack 0.1.1  Fetching slop 3.6.0  Installing slop 3.6.0  Fetching require_all 1.4.0  Fetching ruby-progressbar 1.8.1  Installing ruby-progressbar 1.8.1  Installing require_all 1.4.0  Fetching unicode-display_width 1.3.0  Fetching sqlite3 1.3.13  Installing unicode-display_width 1.3.0  Installing sqlite3 1.3.13 with native extensions  Fetching parser 2.4.0.0  Installing parser 2.4.0.0  Fetching addressable 2.5.1  Installing addressable 2.5.1  Fetching rainbow 2.2.2  Installing rainbow 2.2.2 with native extensions  Fetching crack 0.4.3  Installing crack 0.4.3  Fetching mime-types 3.1  Installing mime-types 3.1  Fetching nokogiri 1.8.0  Installing nokogiri 1.8.0 with native extensions  Fetching minispec-metadata 2.0.0  Installing minispec-metadata 2.0.0  Fetching minitest-around 0.4.0  Installing minitest-around 0.4.0  Fetching pry 0.10.4  Installing pry 0.10.4  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching webmock 2.0.3  Fetching sqlite_magic 0.0.6  Installing webmock 2.0.3  Installing sqlite_magic 0.0.6  Fetching minitest-vcr 1.4.0  Installing minitest-vcr 1.4.0  Fetching domain_name 0.5.20170404  Fetching rubocop 0.49.1  Installing rubocop 0.49.1  Installing domain_name 0.5.20170404  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching vcr-archive 0.3.0  Installing vcr-archive 0.3.0  Using scraper_test 0.1.0 from https://github.com/everypolitician/scraper_test.git (at master@9b4326c)  Fetching http-cookie 1.0.3  Using scraped_page_archive 0.5.0 from https://github.com/everypolitician/scraped_page_archive.git (at master@3b67c31)  Using scraped 0.6.2 from https://github.com/everypolitician/scraped.git (at master@58c88c1)  Using table_unspanner 0.1.0 from https://github.com/everypolitician/table_unspanner.git (at master@a70a98a)  Installing http-cookie 1.0.3  Fetching rest-client 2.0.2  Installing rest-client 2.0.2  Bundle complete! 18 Gemfile dependencies, 50 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Post-install message from webmock:  WebMock 2.0 has some breaking changes. Please check the CHANGELOG: https://goo.gl/piDGLu  Bundle completed (36.09s)  Cleaning up the bundler cache. -----> Installing node-v6.11.1-linux-x64 -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... http://api.parldata.eu/kv/kuvendi/organizations?where=%7B%22classification%22%3A%22chamber%22%7D http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=2 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=3 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=4 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=5 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=6 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=7 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=8 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=9 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=10 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=11 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=12 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=13 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=14 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=15 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=16 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=17 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=18 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=19 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=20 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=21 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=22 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=23 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=24 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=25 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=26 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=27 http://api.parldata.eu/kv/kuvendi/people?max_results=50&embed=%5B%22memberships.organization%22%5D&page=28 http://api.parldata.eu/kv/kuvendi/organizations?where=%7B%22classification%22%3A%22parliamentary_group%22%7D http://api.parldata.eu/kv/kuvendi/organizations?where=%7B%22classification%22%3A%22parliamentary_group%22%7D&page=2

Data

Downloaded 727 times by everypolitician MikeRalphson

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (283 KB) Use the API

rows 5 / 5

id name start_date end_date source
chamber_2001-11-17
Periudha e parë Legjislative
2001-11-17
2004-11-23
chamber_2004-11-23
Periudha e dytë Legjislative
2004-11-23
2007-12-12
chamber_2007-12-13
Periudha e tretë Legjislative
2007-12-13
2010-11-03
chamber_2010-12-12
Periudha e katërt Legjislative
2010-12-12
2014-05-07
chamber_2014-07-17
Periudha e pestë Legjislative
2014-07-17
2017-05-10

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (283 KB) Use the API

rows 10 / 595

id name sort_name family_name given_name source start_date end_date faction_id term faction image birth_date
mp_AbdylYmeri
Kuvendit të Kosovës - Periudha e tretë Legjislative
Ymeri Abdyl
Ymeri
Abdyl
2007-12-13
2010-11-03
parliamentary_group_25
chamber_2007-12-13
Grupi i PDK-së
mp_AdemGrabovci
Kuvendit të Kosovës - Periudha e tretë Legjislative
Grabovci Adem
Grabovci
Adem
2007-12-13
2010-11-03
parliamentary_group_25
chamber_2007-12-13
Grupi i PDK-së
mp_AdemGrabovci
Kuvendit të Kosovës - Periudha e katërt Legjislative
Grabovci Adem
Grabovci
Adem
2010-12-12
2014-05-07
parliamentary_group_43
chamber_2010-12-12
PDK
mp_AdemGrabovci
Kuvendit të Kosovës - Periudha e pestë Legjislative
Grabovci Adem
Grabovci
Adem
2014-07-17
2017-05-10
parliamentary_group_54
chamber_2014-07-17
PDK
mp_AdemHajdaraj
Kuvendit të Kosovës - Periudha e tretë Legjislative
Hajdaraj Adem
Hajdaraj
Adem
2007-12-13
2010-11-03
parliamentary_group_26
chamber_2007-12-13
Grupi i LDK-së
1944-12-13
mp_AdemHodžA
Kuvendit të Kosovës - Periudha e pestë Legjislative
Hodža Adem
Hodža
Adem
2014-07-17
2017-05-10
parliamentary_group_57
chamber_2014-07-17
Lista serbe
mp_AdemSalihaj
Kuvendit të Kosovës - Periudha e tretë Legjislative
Salihaj Adem
Salihaj
Adem
2007-12-13
2010-11-03
parliamentary_group_26
chamber_2007-12-13
Grupi i LDK-së
mp_AdemSalihaj
Kuvendit të Kosovës - Periudha e pestë Legjislative
Salihaj Adem
Salihaj
Adem
2014-07-17
2017-05-10
parliamentary_group_55
chamber_2014-07-17
LDK
mp_AfrimArzullaxhiu
Kuvendit të Kosovës - Periudha e dytë Legjislative
Arzullaxhiu Afrim
Arzullaxhiu
Afrim
2004-11-23
2007-12-12
parliamentary_group_8
chamber_2004-11-23
Grupi Parlamentar i PDK-së
1954-04-03
mp_AfrimHoti
Kuvendit të Kosovës - Periudha e katërt Legjislative
Hoti Afrim
Hoti
Afrim
2010-12-12
2014-05-07
parliamentary_group_40
chamber_2010-12-12
Grupi i Vetëvendosjes
1962-06-05

Statistics

Average successful run time: 3 minutes

Total run time: about 1 month

Total cpu time used: about 1 hour

Total disk space used: 327 KB

History

  • Auto ran revision bec97265 and completed successfully .
    600 records added, 600 records removed in the database
    31 pages scraped
  • Auto ran revision bec97265 and completed successfully .
    600 records added, 600 records removed in the database
    31 pages scraped
  • Auto ran revision bec97265 and completed successfully .
    600 records added, 600 records removed in the database
  • Auto ran revision bec97265 and completed successfully .
    600 records added, 600 records removed in the database
  • Auto ran revision bec97265 and completed successfully .
    600 records added, 600 records removed in the database
    31 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

kosovo-parldata / scraper.rb