Redland City Council Scraper

This Council involves the followings

  • Terms to Agreed - Yes
  • Cookie - Yes
  • Scrape detail page - No
  • Clearly defined data within a row - Yes

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default)
  • thismonth
  • lastmonth

Enjoy

Contributors henare LoveMyData equivalentideas MichaelBone

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.3.4 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Fetching unf_ext 0.0.7.4  Using bundler 1.15.2  Fetching httpclient 2.8.3  Fetching mime-types-data 3.2016.0521  Installing mime-types-data 3.2016.0521  Installing httpclient 2.8.3  Installing unf_ext 0.0.7.4 with native extensions  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching net-http-persistent 2.9.4  Installing net-http-persistent 2.9.4  Fetching mini_portile2 2.1.0  Installing mini_portile2 2.1.0  Fetching ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing ntlm-http 0.1.1  Installing webrobots 0.1.2  Fetching sqlite3 1.3.13  Fetching mime-types 3.1  Installing sqlite3 1.3.13 with native extensions  Installing mime-types 3.1  Fetching nokogiri 1.6.8.1  Installing nokogiri 1.6.8.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20170404  Installing domain_name 0.5.20170404  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.5  Installing mechanize 2.7.5  Bundle complete! 2 Gemfile dependencies, 18 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (27.17s)  Cleaning up the bundler cache. -----> Detecting rake tasks   ###### WARNING:  You have not declared a Ruby version in your Gemfile.  To set your Ruby version add this line to your Gemfile:  ruby '2.3.4'  # See https://devcenter.heroku.com/articles/ruby-versions for more information.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting 'thisweek' data, changable via MORPH_PERIOD environment Skipping already saved record BX318167 Skipping already saved record BX318171 Skipping already saved record PD237599 Skipping already saved record PD237600 Skipping already saved record PD237602 Skipping already saved record PD237601 Skipping already saved record BX318168 Skipping already saved record BX318170 Skipping already saved record PD237603 Skipping already saved record PD237604 Skipping already saved record BX318138 Skipping already saved record BX318143 Skipping already saved record PD237606 Skipping already saved record BX318147 Skipping already saved record BX318154 Skipping already saved record BX318156 Skipping already saved record PD237611 Skipping already saved record PD237607 Skipping already saved record PD237609 Skipping already saved record PD237608 Skipping already saved record PD237605 Skipping already saved record PD237512 Skipping already saved record BX318158 Skipping already saved record BX318160 Skipping already saved record BX318163 Skipping already saved record BX318164 Skipping already saved record BX318165 Skipping already saved record PD237614 Skipping already saved record PD237615 Skipping already saved record PD237610 Skipping already saved record PD237613 Skipping already saved record PD237616

Data

Downloaded 4063 times by slav123 openaustralia LoveMyData jesse-a-reilly TheZepto dsccramb Tomt2u MikeRalphson spatialsteve jamezpolley MichaelBone Jedwards157

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (2.08 MB) Use the API

rows 10 / 6395

info_url comment_url council_reference date_received address description date_scraped
mailto:rcc@redland.qld.gov.au?subject=Development+Application+Enquiry%3A+APS000367
APS000367
2015-08-03
4 Zephyr Court, Birkdale, QLD
Standard Format - 1 into 2
2015-08-24
mailto:rcc@redland.qld.gov.au?subject=Development+Application+Enquiry%3A+APS000368
APS000368
2015-08-04
271 Railway Parade, Birkdale, QLD
Standard Format - 1 into 2
2015-08-24
mailto:rcc@redland.qld.gov.au?subject=Development+Application+Enquiry%3A+APS000369
APS000369
2015-08-05
12-16 Wilson Esplanade, Redland Bay, QLD
Standard Format 1 into 3
2015-08-24
mailto:rcc@redland.qld.gov.au?subject=Development+Application+Enquiry%3A+APS000370
APS000370
2015-08-11
38-40 School Road, Capalaba, QLD
8 Lots & 1 Standard Format lot
2015-08-24
mailto:rcc@redland.qld.gov.au?subject=Development+Application+Enquiry%3A+APS000371
APS000371
2015-08-12
184 Thorneside Road, Thorneside, QLD
Multiple Dwelling X 7 Town Houses
2015-08-24
mailto:rcc@redland.qld.gov.au?subject=Development+Application+Enquiry%3A+APS000373
APS000373
2015-08-12
223-231 Wellington Street, Ormiston, QLD
Standard Format - 1 into 24
2015-08-24
mailto:rcc@redland.qld.gov.au?subject=Development+Application+Enquiry%3A+APS000366
APS000366
2015-08-13
281 Old Cleveland Road East, Capalaba, QLD
Building Format- 1 into 17
2015-08-24
mailto:rcc@redland.qld.gov.au?subject=Development+Application+Enquiry%3A+APS000365
APS000365
2015-08-13
38 Booran Street, Point Lookout, QLD
Plan of Lots 1 & 2 and Common Property
2015-08-24
mailto:rcc@redland.qld.gov.au?subject=Development+Application+Enquiry%3A+APS000374
APS000374
2015-08-18
43 Dawson Road, Alexandra Hills, QLD
Standard Format - 1 into 3
2015-08-24
mailto:rcc@redland.qld.gov.au?subject=Development+Application+Enquiry%3A+APS000372
APS000372
2015-08-19
76 Old Cleveland Road, Capalaba, QLD
Building Format 1 into 4
2015-08-24

Statistics

Average successful run time: 2 minutes

Total run time: 1 day

Total cpu time used: 30 minutes

Total disk space used: 2.11 MB

History

  • Auto ran revision 0607eee3 and completed successfully .
    nothing changed in the database
  • Auto ran revision 0607eee3 and completed successfully .
    9 records added in the database
  • Auto ran revision 0607eee3 and completed successfully .
    14 records added in the database
  • Auto ran revision 0607eee3 and completed successfully .
    7 records added in the database
  • Auto ran revision 0607eee3 and completed successfully .
    4 records added in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

redland / scraper.rb