openaustralia / alaveteli_versions

Scrapes the version of every Alaveteli deployment

Scrapes alaveteli.org, www.whatdotheyknow.com, kimittud.atlatszo.hu, and 23 other domains

Welcome | Alaveteli | mySociety


This is a scraper that runs on Morph.

To get started see the documentation

Development

Set up a local development copy of the scraper by running:

bash git clone https://github.com/openaustralia/alaveteli_versions cd alaveteli_versions bundle

Then run regression tests and audit the scraper style with:

bash bundle exec rake

Contributors henare auxesis

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.3.4 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.15.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching unf_ext 0.0.7.1  Fetching httpclient 2.7.0.1  Fetching mime-types 2.6.2  Installing mime-types 2.6.2  Installing httpclient 2.7.0.1  Installing unf_ext 0.0.7.1 with native extensions  Fetching net-http-digest_auth 1.4  Installing net-http-digest_auth 1.4  Fetching net-http-persistent 2.9.4  Fetching mini_portile 0.6.2  Installing mini_portile 0.6.2  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.1  Installing net-http-persistent 2.9.4  Installing webrobots 0.1.1  Fetching sqlite3 1.3.11  Fetching nokogiri 1.6.6.3  Installing sqlite3 1.3.11 with native extensions  Installing nokogiri 1.6.6.3 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.25  Installing domain_name 0.5.25  Fetching http-cookie 1.0.2  Installing http-cookie 1.0.2  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.3  Installing mechanize 2.7.3  Bundle complete! 10 Gemfile dependencies, 17 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (27.76s)  Cleaning up the bundler cache. -----> Detecting rake tasks   ###### WARNING:  You have not declared a Ruby version in your Gemfile.  To set your Ruby version add this line to your Gemfile:  ruby '2.3.4'  # See https://devcenter.heroku.com/articles/ruby-versions for more information.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Discovering deployments... Getting version information for: https://www.whatdotheyknow.com/ Getting version information for: http://kimittud.atlatszo.hu/ Getting version information for: https://fyi.org.nz/ Getting version information for: http://www.asktheeu.org/ Getting version information for: https://dostup.pravda.com.ua/ Getting version information for: http://www.infoprovsechny.cz/ Getting version information for: http://imamopravoznati.org/ Getting version information for: https://www.righttoknow.org.au/ Getting version information for: http://foi.sinarproject.org/ Discovering deployments... Getting version information for: https://www.whatdotheyknow.com/ Getting version information for: http://kimittud.atlatszo.hu/ Getting version information for: https://fyi.org.nz/ Getting version information for: http://www.asktheeu.org/ Getting version information for: https://dostup.pravda.com.ua/ Getting version information for: http://www.infoprovsechny.cz/ Getting version information for: http://imamopravoznati.org/ Getting version information for: https://www.righttoknow.org.au/ Getting version information for: http://foi.sinarproject.org/ Getting version information for: http://www.quesabes.org/ Getting version information for: http://nuvasuparati.info/ Getting version information for: http://askdata.org.il/ Getting version information for: http://www.slobodenpristap.mk/ Getting version information for: https://sobanukirwa.rw/ Getting version information for: https://chiedi.dirittodisapere.it/ Getting version information for: https://accessinfo.hk/ Getting version information for: http://askyourgov.ug/ Getting version information for: https://www.mimesbronn.no/ Getting version information for: http://queremosaber.org.py/ Getting version information for: https://derechoapreguntar.org/ Getting version information for: http://infolib.org.lr/ Getting version information for: https://transparencia.be/ Getting version information for: https://www.queremosdatos.co/ Getting version information for: https://fragastaten.se/ Getting version information for: https://opramachine.com/ Getting version information for: http://bilmehakki.org/ Getting version information for: http://informini.org/ar/ Saving 27 records Done.

Data

Downloaded 45 times by gavinrozzi henare n-mouse tuttinator crowbot schlos handelaar skylark1848 garethrees paullenz auxesis

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (12 KB) Use the API

rows 10 / 36

alaveteli_version ruby_version visible_request_count alaveteli_git_commit confirmed_user_count name url version_url error visible_public_body_count visible_comment_count track_thing_count widget_vote_count public_body_change_request_count request_classification_count visible_followup_message_count date_checked
0.20.0.14
1.8.7
29
823e58dc69960c600230b10604a0051359173f85
50
Sobunakirwa
0.20.0.14
1.9.3
17
48f07a68cb7373a9a032009ab760b5f4d156d6cd
39
Diritto Di Sapere
Je Veux Savoir
404 => Net::HTTPNotFound for http://www.nordouvert.ca/version.json/2015/06/15/je-veux-savoir-fin-d-une-belle-aventure.html -- unhandled response
0.19
1.8.7
991
d12c236fbd606fb38c01aebb39694a568740de06
939
Queremos Saber
0.21.0.26
1.8.7
10
5e2f00176f73736929fde4e1ab09f594ba5252cf
4
Marsoum 41
0.12
1.8.7
ad820340ad8bebffbe453ae337a4ec49dec6022f
Tu Derecho A Saber
Da Zanamo Svi
404 => Net::HTTPNotFound for http://daznamosvi.rs/sr/version.json -- unhandled response
Nós Queremos Saber
Didn't get JSON from API.
Pravo Da Znam
Net::HTTP::Persistent::Error: too many connection resets (due to Net::ReadTimeout - Net::ReadTimeout) after 0 requests on 47044091722280, last used 1531865782.47403 seconds ago
2018-07-17
0.31.0.3
2.3.5
498645
1b9ae4249ea2ba09be03983449b0b905f8ebef1d
135194
WhatDoTheyKnow
23506
79041
109291
0
3279
64938
298038
2018-09-21

Statistics

Average successful run time: 3 minutes

Total run time: about 1 month

Total cpu time used: 38 minutes

Total disk space used: 131 KB

History

  • Auto ran revision 4a0c1181 and completed successfully .
    27 records added, 27 records removed in the database
    31 pages scraped
  • Auto ran revision 4a0c1181 and completed successfully .
    27 records added, 27 records removed in the database
    31 pages scraped
  • Auto ran revision 4a0c1181 and completed successfully .
    27 records added, 27 records removed in the database
    31 pages scraped
  • Auto ran revision 4a0c1181 and completed successfully .
    27 records added, 27 records removed in the database
    31 pages scraped
  • Auto ran revision 4a0c1181 and completed successfully .
    27 records added, 27 records removed in the database
    31 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

alaveteli_versions / scraper.rb