planningalerts-scrapers / redland

Scrapes pdonline.redland.qld.gov.au

Welcome to MasterView 2.0


Redland City Council Scraper

This Council involves the followings * Terms to Agreed - Yes * Cookie - Yes * Scrape detail page - No * Clearly defined data within a row - Yes

Setup MORPH_PERIOD for data recovery, available options are * thisweek (default) * thismonth * lastmonth

Enjoy

Contributors henare LoveMyData equivalentideas

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.3.4 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching unf_ext 0.0.7.4  Fetching httpclient 2.8.3  Fetching mime-types-data 3.2016.0521  Installing mime-types-data 3.2016.0521  Fetching net-http-digest_auth 1.4.1  Installing httpclient 2.8.3  Installing net-http-digest_auth 1.4.1  Installing unf_ext 0.0.7.4 with native extensions  Fetching net-http-persistent 2.9.4  Installing net-http-persistent 2.9.4  Fetching mini_portile2 2.1.0  Installing mini_portile2 2.1.0  Fetching ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Installing ntlm-http 0.1.1  Fetching sqlite3 1.3.13  Fetching mime-types 3.1  Installing mime-types 3.1  Installing sqlite3 1.3.13 with native extensions  Fetching nokogiri 1.6.8.1  Installing nokogiri 1.6.8.1 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20170404  Installing domain_name 0.5.20170404  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.5  Installing mechanize 2.7.5  Bundle complete! 2 Gemfile dependencies, 18 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (28.91s)  Cleaning up the bundler cache. -----> Detecting rake tasks   ###### WARNING:  You have not declared a Ruby version in your Gemfile.  To set your Ruby version add this line to your Gemfile:  ruby '2.3.4'  # See https://devcenter.heroku.com/articles/ruby-versions for more information.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting 'thisweek' data, changable via MORPH_PERIOD environment Skipping already saved record BX316241 Saving record BX316251, 42 Luke Street, Thornlands QLD 4164 Skipping already saved record PD236871 Skipping already saved record PD236874 Skipping already saved record PD236875 Skipping already saved record PD236876 Skipping already saved record PD236878 Saving record PD236879, 12 Browning Street, Russell Island QLD 4184 Skipping already saved record PD236880 Skipping already saved record PD236865 Skipping already saved record PD236867 Skipping already saved record PD236869 Skipping already saved record PD236873 Skipping already saved record PD236872 Skipping already saved record BX316233 Skipping already saved record BX316234 Skipping already saved record BX316235 Saving record BX316256, 2 Glen Street, Ormiston QLD 4160 Saving record BX316268, 29 Windsong Circuit, Cleveland QLD 4163 Skipping already saved record PD236881 Skipping already saved record PD236882 Skipping already saved record PD236877 Skipping already saved record PD236883 Skipping already saved record OPW002271 Skipping already saved record PD236803 Skipping already saved record PD236884 Skipping already saved record PD236885 Saving record BX316267, 23 Berkingham Street, Thornlands QLD 4164 Saving record BX316253, 54 Redland Bay Road, Capalaba QLD 4157 Saving record PD236886, 8 Berkingham Street, Thornlands QLD 4164 Saving record PD236887, 10 Berkingham Street, Thornlands QLD 4164 Saving record PD236889, 27 Affinity Way, Thornlands QLD 4164 Saving record PD236890, 15 Berkingham Street, Thornlands QLD 4164 Saving record PD236891, 25 Berkingham Street, Thornlands QLD 4164 Saving record PD236894, 15 Leven Street, Thornlands QLD 4164 Saving record PD236892, 7 Berkingham Street, Thornlands QLD 4164 Saving record PD236895, 9 Highgrove Street, Thornlands QLD 4164 Saving record PD236897, 47 Lungren Drive, Thornlands QLD 4164 Saving record PD236896, 37 Highfield Avenue, Thornlands QLD 4164 Saving record PD236898, 13 Berkingham Street, Thornlands QLD 4164

Data

Downloaded 3159 times by slav123 openaustralia LoveMyData jesse-a-reilly TheZepto Tomt2u dsccramb MikeRalphson spatialsteve

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (1.75 MB) Use the API

rows 10 / 5371

info_url comment_url council_reference date_received address description date_scraped
mailto:rcc@redland.qld.gov.au?subject=Development+Application+Enquiry%3A+APS000367
APS000367
2015-08-03
4 Zephyr Court, Birkdale, QLD
Standard Format - 1 into 2
2015-08-24
mailto:rcc@redland.qld.gov.au?subject=Development+Application+Enquiry%3A+APS000368
APS000368
2015-08-04
271 Railway Parade, Birkdale, QLD
Standard Format - 1 into 2
2015-08-24
mailto:rcc@redland.qld.gov.au?subject=Development+Application+Enquiry%3A+APS000369
APS000369
2015-08-05
12-16 Wilson Esplanade, Redland Bay, QLD
Standard Format 1 into 3
2015-08-24
mailto:rcc@redland.qld.gov.au?subject=Development+Application+Enquiry%3A+APS000370
APS000370
2015-08-11
38-40 School Road, Capalaba, QLD
8 Lots & 1 Standard Format lot
2015-08-24
mailto:rcc@redland.qld.gov.au?subject=Development+Application+Enquiry%3A+APS000371
APS000371
2015-08-12
184 Thorneside Road, Thorneside, QLD
Multiple Dwelling X 7 Town Houses
2015-08-24
mailto:rcc@redland.qld.gov.au?subject=Development+Application+Enquiry%3A+APS000373
APS000373
2015-08-12
223-231 Wellington Street, Ormiston, QLD
Standard Format - 1 into 24
2015-08-24
mailto:rcc@redland.qld.gov.au?subject=Development+Application+Enquiry%3A+APS000366
APS000366
2015-08-13
281 Old Cleveland Road East, Capalaba, QLD
Building Format- 1 into 17
2015-08-24
mailto:rcc@redland.qld.gov.au?subject=Development+Application+Enquiry%3A+APS000365
APS000365
2015-08-13
38 Booran Street, Point Lookout, QLD
Plan of Lots 1 & 2 and Common Property
2015-08-24
mailto:rcc@redland.qld.gov.au?subject=Development+Application+Enquiry%3A+APS000374
APS000374
2015-08-18
43 Dawson Road, Alexandra Hills, QLD
Standard Format - 1 into 3
2015-08-24
mailto:rcc@redland.qld.gov.au?subject=Development+Application+Enquiry%3A+APS000372
APS000372
2015-08-19
76 Old Cleveland Road, Capalaba, QLD
Building Format 1 into 4
2015-08-24

Statistics

Average successful run time: 2 minutes

Total run time: 1 day

Total cpu time used: 24 minutes

Total disk space used: 1.78 MB

History

  • Auto ran revision 4ea55a88 and completed successfully .
    17 records added in the database
    4 pages scraped
  • Auto ran revision 4ea55a88 and completed successfully .
    12 records added in the database
    4 pages scraped
  • Auto ran revision 4ea55a88 and completed successfully .
    9 records added in the database
    4 pages scraped
  • Auto ran revision 4ea55a88 and completed successfully .
    1 record added in the database
    4 pages scraped
  • Auto ran revision 4ea55a88 and completed successfully .
    nothing changed in the database
    154 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

redland / scraper.rb