City of Ipswich Council scraper

  • Accept Terms - Yes
  • Cookie tracking - Yes
  • Request actual DA page for data - No
  • Clearly defined data within a row - No but reasonable
  • Limited to 10 pages only

Setup MORPH_PERIOD for data recovery, available options are

  • not set = past 14 days (default)
  • thismonth
  • lastmonth
  • custom [201808] for 2018 August

Contributors mlandauer LoveMyData MichaelBone jamezpolley

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.4.1 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Fetching mime-types-data 3.2018.0812  Installing httpclient 2.8.3  Installing unf_ext 0.0.7.5 with native extensions  Installing mime-types-data 3.2018.0812  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.3.0  Fetching ntlm-http 0.1.1  Installing mini_portile2 2.3.0  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Fetching sqlite3 1.3.13  Installing webrobots 0.1.2  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Installing sqlite3 1.3.13 with native extensions  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.8.5  Installing nokogiri 1.8.5 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 2 Gemfile dependencies, 19 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (26.97s)  Cleaning up the bundler cache. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Collecting data from thisweek Date: 2019-04-05 Scraping page 1... Skipping already saved record ADP-2679/2019 Skipping already saved record ADP-2691/2019 Skipping already saved record CA-2778/2019 Skipping already saved record RAL-2601/2019 Skipping already saved record PDA-2834/2019 Skipping already saved record MAMC-6329/2017A Date: 2019-04-06 Scraping page 1... Date: 2019-04-07 Scraping page 1... Date: 2019-04-08 Scraping page 1... Skipping already saved record MAEXT-1835/2017A Date: 2019-04-09 Scraping page 1... Skipping already saved record OW-2846/2019 Skipping already saved record OD-2854/2019 Skipping already saved record MAEXT-2128/2017A Date: 2019-04-10 Scraping page 1... Skipping already saved record MCU-2895/2019 Skipping already saved record MCU-2945/2019 Skipping already saved record OD-3005/2019 Skipping already saved record PDA-2954/2019 Skipping already saved record MAEXT-2646/2015A Skipping already saved record MAEXT-3372/2014A Date: 2019-04-11 Scraping page 1... Skipping already saved record MCU-3007/2019 Skipping already saved record OW-2951/2019 Skipping already saved record OW-2955/2019 Skipping already saved record OW-2975/2019 Skipping already saved record OW-2987/2019 Skipping already saved record OW-3008/2019 Skipping already saved record OD-2985/2019 Skipping already saved record MAEXT-4358/2017A Date: 2019-04-12 Scraping page 1... Skipping already saved record OD-3004/2019 Skipping already saved record OD-3006/2019 Skipping already saved record MAEXT-1837/2007B Skipping already saved record MAMC-1721/2018A Skipping already saved record EXC-3052/2019 Skipping already saved record SPSR-3010/2019 Skipping already saved record SPSR-3054/2019 Date: 2019-04-13 Scraping page 1... Date: 2019-04-14 Scraping page 1... Date: 2019-04-15 Scraping page 1... Skipping already saved record OW-3122/2019 Skipping already saved record MAMC-2668/2008A Skipping already saved record MAPDA-40/2015E Date: 2019-04-16 Scraping page 1... Skipping already saved record CA-3153/2019 Skipping already saved record RAL-3129/2019 Skipping already saved record SPSR-3126/2019 Date: 2019-04-17 Scraping page 1... Date: 2019-04-18 Scraping page 1... Saving record MAEXT-6244/2012B, 56 Robertson Road EASTERN HEIGHTS QLD 4305 Saving record MAEXT-7236/2014A, 7 Monique Court RACEVIEW QLD 4305 Saving record MAEXT-8174/2004B, 108-132 Kruger Parade REDBANK QLD 4301 Date: 2019-04-19 Scraping page 1...

Data

Downloaded 5279 times by slav123 openaustralia LoveMyData geolocarta jesse-a-reilly stevebrencorp TheZepto spatialsteve dsccramb MikeRalphson rimonhm benrfairless Jedwards157 timmh84

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (1.79 MB) Use the API

rows 10 / 5100

info_url council_reference date_received description address date_scraped comment_url
MCU-1548/2014
2014-04-01
Busines Use - Medical Premises
84 Brisbane Road EAST IPSWICH QLD 4305
2014-04-08
mailto:plandev@ipswich.qld.gov.au
MCU-1669/2014
2014-04-07
Dual Ocupancy
19 Navickas Circuit REDBANK PLAINS QLD 4301
2014-04-08
mailto:plandev@ipswich.qld.gov.au
MA-2484/2012A
2014-04-04
Permisable Change - Smart eDA
10 Brisbane Stret IPSWICH QLD 4305
2014-04-08
mailto:plandev@ipswich.qld.gov.au
OW-1577/2014
2014-04-02
Roadworks, Stormwater, Earthworks & Landscaping
51 Bary Alexander Drive SPRINGFIELD CENTRAL QLD 430
2014-04-08
mailto:plandev@ipswich.qld.gov.au
OW-1617/2014
2014-04-03
Stormwater and Earthworks
1 Mary Stret WODEND QLD 4305
2014-04-08
mailto:plandev@ipswich.qld.gov.au
MCU-1603/2014
2014-04-03
Busines Use - Extension to Childcare Centre
7 Leonard Stret YAMANTO QLD 4305
2014-04-08
mailto:plandev@ipswich.qld.gov.au
ADP-1670/2014
2014-04-07
Area Development Plan - Coles Expres Service Station & Asociated signage
1 Main Stret SPRINGFIELD CENTRAL QLD 430
2014-04-08
mailto:plandev@ipswich.qld.gov.au
OW-1567/2014
2014-04-01
Stormwater
9 Equestrian Drive YAMANTO QLD 4305
2014-04-08
mailto:plandev@ipswich.qld.gov.au
OW-1591/2014
2014-04-02
Roadworks, Stormwater, Water Infrastructure, Drainage Works, Earthworks, Sewerage Infrastructure and Landscaping
10 Brisbane Stret IPSWICH QLD 4305
2014-04-08
mailto:plandev@ipswich.qld.gov.au
OW-1639/2014
2014-04-04
Stormwater & Drainage Works
16 Waterford Road GAILES QLD 430
2014-04-08
mailto:plandev@ipswich.qld.gov.au

Statistics

Average successful run time: 1 minute

Total run time: 9 days

Total cpu time used: 44 minutes

Total disk space used: 1.83 MB

History

  • Auto ran revision bb58819e and completed successfully .
    3 records added in the database
  • Auto ran revision bb58819e and completed successfully .
    1 record added in the database
  • Auto ran revision bb58819e and completed successfully .
    4 records added in the database
  • Auto ran revision bb58819e and completed successfully .
    4 records added in the database
  • Auto ran revision bb58819e and completed successfully .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

ipswich / scraper.rb