openaustralia / alaveteli_versions

Scrapes the version of every Alaveteli deployment

Scrapes alaveteli.org, www.whatdotheyknow.com, kimittud.atlatszo.hu, and 24 other domains

Welcome | Alaveteli | mySociety


This is a scraper that runs on Morph.

To get started see the documentation

Development

Set up a local development copy of the scraper by running:

bash git clone https://github.com/openaustralia/alaveteli_versions cd alaveteli_versions bundle

Then run regression tests and audit the scraper style with:

bash bundle exec rake

Contributors henare auxesis

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.3.4 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.15.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching unf_ext 0.0.7.1  Fetching httpclient 2.7.0.1  Fetching mime-types 2.6.2  Installing unf_ext 0.0.7.1 with native extensions  Installing mime-types 2.6.2  Installing httpclient 2.7.0.1  Fetching net-http-digest_auth 1.4  Installing net-http-digest_auth 1.4  Fetching net-http-persistent 2.9.4  Fetching mini_portile 0.6.2  Installing mini_portile 0.6.2  Fetching ntlm-http 0.1.1  Installing net-http-persistent 2.9.4  Fetching webrobots 0.1.1  Installing ntlm-http 0.1.1  Fetching sqlite3 1.3.11  Installing webrobots 0.1.1  Fetching nokogiri 1.6.6.3  Installing sqlite3 1.3.11 with native extensions  Installing nokogiri 1.6.6.3 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.25  Installing domain_name 0.5.25  Fetching http-cookie 1.0.2  Installing http-cookie 1.0.2  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.3  Installing mechanize 2.7.3  Bundle complete! 10 Gemfile dependencies, 17 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (34.60s)  Cleaning up the bundler cache. -----> Detecting rake tasks   ###### WARNING:  You have not declared a Ruby version in your Gemfile.  To set your Ruby version add this line to your Gemfile:  ruby '2.3.4'  # See https://devcenter.heroku.com/articles/ruby-versions for more information.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Discovering deployments... Getting version information for: https://www.whatdotheyknow.com/ Getting version information for: http://kimittud.atlatszo.hu/ Getting version information for: https://fyi.org.nz/ Getting version information for: http://www.asktheeu.org/ Getting version information for: https://dostup.pravda.com.ua/ Getting version information for: http://www.infoprovsechny.cz/ Getting version information for: http://imamopravoznati.org/ Getting version information for: https://www.righttoknow.org.au/ Getting version information for: http://foi.sinarproject.org/ Discovering deployments... Getting version information for: https://www.whatdotheyknow.com/ Getting version information for: http://kimittud.atlatszo.hu/ Getting version information for: https://fyi.org.nz/ Getting version information for: http://www.asktheeu.org/ Getting version information for: https://dostup.pravda.com.ua/ Getting version information for: http://www.infoprovsechny.cz/ Getting version information for: http://imamopravoznati.org/ Getting version information for: https://www.righttoknow.org.au/ Getting version information for: http://foi.sinarproject.org/ Getting version information for: http://www.quesabes.org/ Getting version information for: http://nuvasuparati.info/ Getting version information for: http://askdata.org.il/ Getting version information for: http://www.slobodenpristap.mk/ Getting version information for: https://sobanukirwa.rw/ Getting version information for: https://chiedi.dirittodisapere.it/ Getting version information for: https://accessinfo.hk/ Getting version information for: http://askyourgov.ug/ Getting version information for: https://www.mimesbronn.no/ Getting version information for: http://queremosaber.org.py/ Getting version information for: https://derechoapreguntar.org/ Getting version information for: http://infolib.org.lr/ Getting version information for: https://transparencia.be/ Getting version information for: https://www.queremosdatos.co/ Getting version information for: https://fragastaten.se/ Getting version information for: https://opramachine.com/ Getting version information for: http://bilmehakki.org/ Getting version information for: http://informini.org/ar/ Saving 27 records Done.

Data

Downloaded 44 times by gavinrozzi henare n-mouse tuttinator schlos handelaar crowbot skylark1848 garethrees paullenz auxesis

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (12 KB) Use the API

rows 10 / 36

alaveteli_version ruby_version visible_request_count alaveteli_git_commit confirmed_user_count name url version_url error visible_public_body_count visible_comment_count track_thing_count widget_vote_count public_body_change_request_count request_classification_count visible_followup_message_count date_checked
0.20.0.14
1.8.7
29
823e58dc69960c600230b10604a0051359173f85
50
Sobunakirwa
0.20.0.14
1.9.3
17
48f07a68cb7373a9a032009ab760b5f4d156d6cd
39
Diritto Di Sapere
Je Veux Savoir
404 => Net::HTTPNotFound for http://www.nordouvert.ca/version.json/2015/06/15/je-veux-savoir-fin-d-une-belle-aventure.html -- unhandled response
0.19
1.8.7
991
d12c236fbd606fb38c01aebb39694a568740de06
939
Queremos Saber
0.21.0.26
1.8.7
10
5e2f00176f73736929fde4e1ab09f594ba5252cf
4
Marsoum 41
0.12
1.8.7
ad820340ad8bebffbe453ae337a4ec49dec6022f
Tu Derecho A Saber
Da Zanamo Svi
404 => Net::HTTPNotFound for http://daznamosvi.rs/sr/version.json -- unhandled response
Nós Queremos Saber
Didn't get JSON from API.
Pravo Da Znam
Net::HTTP::Persistent::Error: too many connection resets (due to Net::ReadTimeout - Net::ReadTimeout) after 0 requests on 47044091722280, last used 1531865782.47403 seconds ago
2018-07-17
0.31.0.3
2.3.5
485933
cc26e963be2eb1e37a5119ad2dcaefc1ab6cf4f5
131740
WhatDoTheyKnow
23304
78373
107725
0
3150
63799
289834
2018-07-20

Statistics

Average successful run time: 3 minutes

Total run time: about 1 month

Total cpu time used: 35 minutes

Total disk space used: 131 KB

History

  • Auto ran revision 4a0c1181 and completed successfully .
    27 records added, 27 records removed in the database
    31 pages scraped
  • Auto ran revision 4a0c1181 and completed successfully .
    27 records added, 27 records removed in the database
    31 pages scraped
  • Auto ran revision 4a0c1181 and completed successfully .
    27 records added, 26 records removed in the database
    31 pages scraped
  • Auto ran revision 4a0c1181 and completed successfully .
    27 records added, 27 records removed in the database
    30 pages scraped
  • Auto ran revision 4a0c1181 and completed successfully .
    27 records added, 27 records removed in the database
    30 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

alaveteli_versions / scraper.rb