openaustralia / right_to_know_requests

Data about requests on Right To Know for analysis


This is a scraper that runs on Morph. To get started see the documentation

Contributors henare

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.0.0 -----> Installing dependencies using bundler 1.11.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/.........  Fetching version metadata from https://rubygems.org/..  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Rubygems 2.0.14.1 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation.  Installing unf_ext 0.0.7.1 with native extensions  Installing httpclient 2.6.0.1  Installing mime-types 2.5  Installing net-http-digest_auth 1.4  Installing net-http-persistent 2.9.4  Installing mini_portile 0.6.2  Installing ntlm-http 0.1.1  Installing webrobots 0.1.1  Installing sqlite3 1.3.10 with native extensions  Using bundler 1.11.2  Installing unf 0.1.4  Installing nokogiri 1.6.6.2 with native extensions  Installing sqlite_magic 0.0.3  Installing domain_name 0.5.24  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Installing http-cookie 1.0.2  Installing mechanize 2.7.3  Bundle complete! 2 Gemfile dependencies, 17 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (24.24s)  Cleaning up the bundler cache.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Fetching request ID: 2246... Fetching request ID: 2247... Fetching request ID: 2248... Fetching request ID: 2249... Fetching request ID: 2250... Fetching request ID: 2251... Fetching request ID: 2252... Fetching request ID: 2253... Fetching request ID: 2254... Fetching request ID: 2255... Skipping missing request https://www.righttoknow.org.au/request/2255.json Fetching request ID: 2256... Fetching request ID: 2257... Skipping missing request https://www.righttoknow.org.au/request/2257.json Fetching request ID: 2258... Fetching request ID: 2259... Fetching request ID: 2260... Fetching request ID: 2261... Fetching request ID: 2262... Fetching request ID: 2263... Fetching request ID: 2264... Fetching request ID: 2265... Fetching request ID: 2266... Fetching request ID: 2267... Fetching request ID: 2268... Fetching request ID: 2269... Fetching request ID: 2270... Fetching request ID: 2271... Fetching request ID: 2272... Fetching request ID: 2273... Skipping missing request https://www.righttoknow.org.au/request/2273.json Fetching request ID: 2274... Fetching request ID: 2275... Fetching request ID: 2276... Skipping missing request https://www.righttoknow.org.au/request/2276.json Fetching request ID: 2277... Fetching request ID: 2278... Fetching request ID: 2279... Skipping missing request https://www.righttoknow.org.au/request/2279.json Fetching request ID: 2280... Fetching request ID: 2281... Skipping missing request https://www.righttoknow.org.au/request/2281.json Fetching request ID: 2282... Fetching request ID: 2283... Fetching request ID: 2284... Fetching request ID: 2285... Fetching request ID: 2286... Fetching request ID: 2287... Fetching request ID: 2288... Fetching request ID: 2289... Fetching request ID: 2290... Fetching request ID: 2291... Skipping missing request https://www.righttoknow.org.au/request/2291.json Fetching request ID: 2292... Fetching request ID: 2293... Fetching request ID: 2294... Fetching request ID: 2295... Skipping missing request https://www.righttoknow.org.au/request/2295.json Fetching request ID: 2296... Skipping missing request https://www.righttoknow.org.au/request/2296.json Fetching request ID: 2297... Fetching request ID: 2298... Fetching request ID: 2299... Fetching request ID: 2300... Fetching request ID: 2301... Skipping missing request https://www.righttoknow.org.au/request/2301.json Fetching request ID: 2302... Fetching request ID: 2303... Skipping missing request https://www.righttoknow.org.au/request/2303.json Fetching request ID: 2304... Fetching request ID: 2305... Fetching request ID: 2306... Fetching request ID: 2307... Skipping missing request https://www.righttoknow.org.au/request/2307.json Fetching request ID: 2308... Fetching request ID: 2309... Skipping missing request https://www.righttoknow.org.au/request/2309.json Fetching request ID: 2310... Fetching request ID: 2311... Fetching request ID: 2312... Skipping missing request https://www.righttoknow.org.au/request/2312.json Fetching request ID: 2313... Fetching request ID: 2314... Fetching request ID: 2315... Fetching request ID: 2316... Skipping missing request https://www.righttoknow.org.au/request/2316.json Fetching request ID: 2317... Skipping missing request https://www.righttoknow.org.au/request/2317.json Fetching request ID: 2318... Skipping missing request https://www.righttoknow.org.au/request/2318.json Fetching request ID: 2319... Fetching request ID: 2320... Fetching request ID: 2321... Skipping missing request https://www.righttoknow.org.au/request/2321.json Fetching request ID: 2322... Fetching request ID: 2323... Skipping missing request https://www.righttoknow.org.au/request/2323.json Fetching request ID: 2324... Fetching request ID: 2325... Fetching request ID: 2326... Fetching request ID: 2327... Fetching request ID: 2328... Fetching request ID: 2329... Fetching request ID: 2330... Fetching request ID: 2331... Fetching request ID: 2332... Fetching request ID: 2333... Fetching request ID: 2334... Fetching request ID: 2335... Fetching request ID: 2336... Fetching request ID: 2337... Skipping missing request https://www.righttoknow.org.au/request/2337.json Fetching request ID: 2338... Fetching request ID: 2339... Fetching request ID: 2340... Skipping missing request https://www.righttoknow.org.au/request/2340.json Fetching request ID: 2341... Skipping missing request https://www.righttoknow.org.au/request/2341.json Fetching request ID: 2342... Skipping missing request https://www.righttoknow.org.au/request/2342.json Fetching request ID: 2343... Skipping missing request https://www.righttoknow.org.au/request/2343.json Fetching request ID: 2344... Fetching request ID: 2345... Fetching request ID: 2346... Fetching request ID: 2347... Skipping missing request https://www.righttoknow.org.au/request/2347.json Fetching request ID: 2348... Fetching request ID: 2349... Fetching request ID: 2350... Fetching request ID: 2351... Fetching request ID: 2352... Fetching request ID: 2353... Fetching request ID: 2354... Fetching request ID: 2355... Fetching request ID: 2356... Fetching request ID: 2357... Fetching request ID: 2358... Fetching request ID: 2359... Fetching request ID: 2360... Fetching request ID: 2361... Fetching request ID: 2362... Skipping missing request https://www.righttoknow.org.au/request/2362.json Fetching request ID: 2363... Fetching request ID: 2364... Fetching request ID: 2365... Skipping missing request https://www.righttoknow.org.au/request/2365.json Fetching request ID: 2366... Fetching request ID: 2367... Fetching request ID: 2368... Fetching request ID: 2369... Fetching request ID: 2370... Skipping missing request https://www.righttoknow.org.au/request/2370.json Fetching request ID: 2371... Fetching request ID: 2372... Fetching request ID: 2373... Finished.

Data

Downloaded 3 times by henare

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (1.25 MB) Use the API

rows 10 / 2138

request_url title created_at display_status described_state user_name user_url public_body_name public_body_url public_body_tags response_count
Holding pen
2012-10-15T19:15:13+11:00
Awaiting classification.
vexatious
Internal admin user
Internal admin authority
0
AWB Inquiry
2012-10-16T17:36:21+11:00
Partially successful.
partially_successful
Evelyn Doyle
Australian Federal Police
attorney_general oaic_report_2011_2012 directory_gov_au federal foi_yes
9
Uranium Sales To India
2012-10-16T18:00:04+11:00
Successful.
successful
Evelyn Doyle
Department of Foreign Affairs and Trade
federal foi_yes directory_gov_au oaic_report_2011_2012 foreign_affairs_and_trade
16
List of Government Domain Names
2012-10-17T12:15:42+11:00
Successful.
successful
Alex Sadleir
Department of Finance
federal foi_yes directory_gov_au oaic_report_2011_2012 finance
1
The Open Government Partnership
2012-10-23T22:22:02+11:00
Successful.
successful
Peter Timmins
Attorney-General's Department
attorney_general oaic_report_2011_2012 directory_gov_au federal foi_yes
4
Documents relating to data retention proposals
2012-10-24T18:08:06+11:00
Partially successful.
partially_successful
Brendan Molloy
Attorney-General's Department
attorney_general oaic_report_2011_2012 directory_gov_au federal foi_yes
11
Irregular maritime arrivals who are stateless
2012-10-26T11:29:06+11:00
Long overdue.
waiting_response
Kieran Ingrey
Department of Immigration and Border Protection
immigration_and_border_protection foi_yes federal directory_gov_au oaic_report_2011_2012 immigration_&_citizenship
2
Document Verification Service — extension to the private sector
2012-10-30T07:58:07+11:00
Waiting clarification.
waiting_clarification
Simon Frew
Attorney-General's Department
attorney_general oaic_report_2011_2012 directory_gov_au federal foi_yes
2
Palestinian Authority upgrade to UN non-member status, 2012
2012-11-02T17:32:53+11:00
Refused.
rejected
Aidan Archer
Department of Foreign Affairs and Trade
federal foi_yes directory_gov_au oaic_report_2011_2012 foreign_affairs_and_trade
8
Pros and cons of removing the River Murray barrages
2012-11-07T16:20:23+11:00
Successful.
successful
Alan Noble
Murray-Darling Basin Authority
federal foi_yes directory_gov_au oaic_report_2011_2012 environment
3

Statistics

Average successful run time: about 1 hour

Total run time: about 2 hours

Total cpu time used: 7 minutes

Total disk space used: 1.27 MB

History

  • Manually ran revision 2e58431c and completed successfully .
    100 records added, 1 record removed in the database
  • Manually ran revision cabe959c and completed successfully .
    2039 records added in the database
    4490 pages scraped
  • Created on morph.io

Scraper code

Ruby

right_to_know_requests / scraper.rb