Scraper for https://ratsinfo.leipzig.de/

The scraped data is eventually displayed by stadtratmonitor.leipzig.codefor.de which is developed here.

This is a scraper that runs on Morph. To get started, see the documentation.

Install docker: curl -fsSL https://get.docker.com/ | sh

If you encounter any troubles, follow the installation guide at the official Docker site.

Build docker image: docker build -t leipzig_scraper .

Use docker image to execute scraper: Calling docker run leipzig_scraper will process https://ratsinfo.leipzig.de/bi/vo020.asp?VOLFDNR=1003952 by default, with docker run leipzig_scraper scraper.rb https://ratsinfo.leipzig.de/bi/vo020.asp?VOLFDNR=1003952 you can process any other paper and with docker run leipzig_scraper person_scraper.rb https://ratsinfo.leipzig.de/bi/pa021.asp you can process council members.

Related projects: * Scraper for http://ratsinfo.dresden.de, scraped data is stored here, used by democropticon to visualize it here * scraper used by Politik bei uns * scraper used by München Transparent * Offener Rat Münster * Old Leipzig ERIS scraper * OParl Specification

Contributors joergreichert jrlover

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.7 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/......  Fetching version metadata from https://rubygems.org/.  Fetching concurrent-ruby 1.1.6  Fetching minitest 5.14.0  Fetching thread_safe 0.3.6  Installing minitest 5.14.0  Installing thread_safe 0.3.6  Installing concurrent-ruby 1.1.6  Fetching zeitwerk 2.2.2  Installing zeitwerk 2.2.2  Using bundler 1.15.2  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching httpclient 2.8.3  Fetching sqlite3 1.4.2  Installing sqlite3 1.4.2 with native extensions  Installing httpclient 2.8.3  Fetching tzinfo 1.2.6  Installing tzinfo 1.2.6  Fetching nokogiri 1.10.8  Fetching i18n 1.8.2  Installing i18n 1.8.2  Fetching activesupport 6.0.2.1  Installing activesupport 6.0.2.1  Installing nokogiri 1.10.8 with native extensions  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching scraperwiki 3.0.2  Installing scraperwiki 3.0.2  Fetching html_to_plain_text 1.0.5  Installing html_to_plain_text 1.0.5  Bundle complete! 6 Gemfile dependencies, 15 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Post-install message from i18n:  HEADS UP! i18n 1.1 changed fallbacks to exclude default locale.  But that may break your application.  If you are upgrading your Rails application from an older version of Rails:  Please check your Rails app for 'config.i18n.fallbacks = true'.  If you're using I18n (>= 1.1.0) and Rails (< 5.2.2), this should be  'config.i18n.fallbacks = [I18n.default_locale]'.  If not, fallbacks will be broken in your app by I18n 1.1.x.  If you are starting a NEW Rails application, you can ignore this notice.  For more info see:  https://github.com/svenfuchs/i18n/releases/tag/v1.1.0  Bundle completed (20.02s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Load PaperIndex from https://ratsinfo.leipzig.de/bi/vo040.asp?

Statistics

Average successful run time: 15 minutes

Total run time: 16 days

Total cpu time used: about 9 hours

Total disk space used: 103 KB

History

  • Manually ran revision 9b78d7a9 and completed successfully .
    nothing changed in the database
  • Auto ran revision 9b78d7a9 and completed successfully .
    nothing changed in the database
  • Auto ran revision 9b78d7a9 and completed successfully .
    nothing changed in the database
  • Auto ran revision 9b78d7a9 and completed successfully .
    nothing changed in the database
  • Auto ran revision 9b78d7a9 and completed successfully .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history