tmtmtmtm / denmark-folketing-wp

Scrapes da.wikipedia.org

Wikipedia, den frie encyklopædi


Contributors tmtmtmtm ondenman

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.3.3 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/.........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/everypolitician/scraped.git  Fetching https://github.com/everypolitician/scraped_page_archive.git  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Fetching https://github.com/everypolitician/table_unspanner.git  Fetching rake 12.0.0  Fetching public_suffix 2.0.5  Fetching ast 2.3.0  Installing ast 2.3.0  Installing public_suffix 2.0.5  Installing rake 12.0.0  Using bundler 1.15.2  Fetching coderay 1.1.1  Installing coderay 1.1.1  Fetching safe_yaml 1.0.4  Installing safe_yaml 1.0.4  Fetching unf_ext 0.0.7.2  Fetching field_serializer 0.3.0  Fetching git 1.3.0  Installing field_serializer 0.3.0  Fetching hashdiff 0.3.2  Installing unf_ext 0.0.7.2 with native extensions  Installing hashdiff 0.3.2  Installing git 1.3.0  Fetching httpclient 2.8.3  Fetching method_source 0.8.2  Installing method_source 0.8.2  Fetching mime-types-data 3.2016.0521  Installing mime-types-data 3.2016.0521  Installing httpclient 2.8.3  Fetching mini_portile2 2.1.0  Installing mini_portile2 2.1.0  Fetching minitest 5.10.1  Installing minitest 5.10.1  Fetching vcr 3.0.3  Fetching netrc 0.11.0  Installing netrc 0.11.0  Fetching open-uri-cached 0.0.5  Installing open-uri-cached 0.0.5  Installing vcr 3.0.3  Fetching powerpack 0.1.1  Installing powerpack 0.1.1  Fetching slop 3.6.0  Fetching rainbow 2.2.1  Installing slop 3.6.0  Installing rainbow 2.2.1 with native extensions  Fetching require_all 1.4.0  Installing require_all 1.4.0  Fetching ruby-progressbar 1.8.1  Installing ruby-progressbar 1.8.1  Fetching unicode-display_width 1.1.3  Installing unicode-display_width 1.1.3  Fetching sqlite3 1.3.13  Installing sqlite3 1.3.13 with native extensions  Fetching parser 2.4.0.0  Installing parser 2.4.0.0  Fetching addressable 2.5.1  Installing addressable 2.5.1  Fetching crack 0.4.3  Fetching mime-types 3.1  Installing crack 0.4.3  Installing mime-types 3.1  Fetching nokogiri 1.7.1  Fetching minispec-metadata 2.0.0  Installing minispec-metadata 2.0.0  Fetching minitest-around 0.4.0  Installing minitest-around 0.4.0  Fetching pry 0.10.4  Installing pry 0.10.4  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching rubocop 0.48.0  Installing rubocop 0.48.0  Installing nokogiri 1.7.1 with native extensions  Fetching webmock 2.0.3  Installing webmock 2.0.3  Fetching minitest-vcr 1.4.0  Installing minitest-vcr 1.4.0  Fetching domain_name 0.5.20170223  Installing domain_name 0.5.20170223  Fetching vcr-archive 0.3.0  Installing vcr-archive 0.3.0  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Using scraped_page_archive 0.5.0 from https://github.com/everypolitician/scraped_page_archive.git (at master@28f93d7)  Fetching rest-client 2.0.1  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Installing rest-client 2.0.1  Using scraped 0.6.0 from https://github.com/everypolitician/scraped.git (at master@0e6f657)  Using table_unspanner 0.1.0 from https://github.com/everypolitician/table_unspanner.git (at master@a70a98a)  Bundle complete! 15 Gemfile dependencies, 46 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Post-install message from webmock:  WebMock 2.0 has some breaking changes. Please check the CHANGELOG: https://goo.gl/piDGLu  Bundle completed (35.28s)  Cleaning up the bundler cache. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Added 188 for 2011 Can't find date in Anita Christensen (DF) (Overtog Mogens Camres mandat) Can't find date in Mia Falkenberg (DF) (Udtrådt af Folketinget. Mandatet er overtaget af René Christensen oktober 2008) Can't find date in Meta Fuglsang (SF) (Overtog Flemming Bonnes mandat) Can't find date in Rikke Hvilshøj (V) (Udtrådt af Folketinget. Mandatet er overtaget af Marion Pedersen) Can't find date in Maja Panduro (S) (Overtog Svend Aukens mandat) Can't find date in Marion Pedersen (V) (Overtog Rikke Hvilshøjs mandat) Added 186 for 2007 Added 178 for 2005 Can't find date in Mariann Fischer Boel (V) (Udtrådt 2004) Added 188 for 2001 Added 175 for 1998 Added 175 for 1994 Added 179 for 1990

Data

Downloaded 2228 times by everypolitician ondenman chrismytton MikeRalphson tmtmtmtm Protesuiq

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (175 KB) Use the API

rows 10 / 1269

name party_id wikiname term party end_date start_date
Pia Adelsteen
DF
Pia Adelsteen
2011
Dansk Folkeparti
Karina Adsbøl
DF
Karina Adsbøl
2011
Dansk Folkeparti
Alex Ahrendtsen
DF
Alex Ahrendtsen
2011
Dansk Folkeparti
Simon Emil Ammitzbøll
LA
Simon Emil Ammitzbøll
2011
Liberal Alliance
Eigil Andersen
SF
Eigil Andersen
2011
Socialistisk Folkeparti
Hans Andersen
V
Hans Andersen (politiker)
2011
Venstre
Kim Andersen
V
Kim Andersen (politiker)
2011
Venstre
Liv Holm Andersen
R
Liv Holm Andersen
2011
Det Radikale Venstre
Sophie Hæstorp Andersen
S
Sophie Hæstorp Andersen
2011
Socialdemokraterne
2014-01-01
Christine Antorini
S
Christine Antorini
2011
Socialdemokraterne

Statistics

Average successful run time: less than a minute

Total run time: about 12 hours

Total cpu time used: 32 minutes

Total disk space used: 213 KB

History

  • Auto ran revision 924bdf5e and completed successfully .
    nothing changed in the database
    7 pages scraped
  • Auto ran revision 924bdf5e and completed successfully .
    717 records added in the database
    7 pages scraped
  • Auto ran revision 924bdf5e and failed .
    717 records removed in the database
  • Auto ran revision 924bdf5e and completed successfully .
    nothing changed in the database
    10 pages scraped
  • Auto ran revision 924bdf5e and completed successfully .
    nothing changed in the database
    7 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

denmark-folketing-wp / scraper.rb