jamezpolley / ipswich

README.md

City of Ipswich Council scraper

Accept Terms - Yes
Cookie tracking - Yes
Request actual DA page for data - No
Clearly defined data within a row - No but reasonable
Limited to 10 pages only

Setup MORPH_PERIOD for data recovery, available options are

not set = past 14 days (default)
thismonth
lastmonth
custom [201808] for 2018 August

Contributors

Last run failed 2025-04-21 with status code 1.

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... Collecting data from custom year and month: 201810 /app/vendor/ruby-2.4.1/lib/ruby/2.4.0/net/http.rb:904:in `initialize': execution expired (Net::OpenTimeout) from /app/vendor/ruby-2.4.1/lib/ruby/2.4.0/net/http.rb:904:in `open' from /app/vendor/ruby-2.4.1/lib/ruby/2.4.0/net/http.rb:904:in `block in connect' from /app/vendor/ruby-2.4.1/lib/ruby/2.4.0/timeout.rb:103:in `timeout' from /app/vendor/ruby-2.4.1/lib/ruby/2.4.0/net/http.rb:902:in `connect' from /app/vendor/ruby-2.4.1/lib/ruby/2.4.0/net/http.rb:887:in `do_start' from /app/vendor/ruby-2.4.1/lib/ruby/2.4.0/net/http.rb:882:in `start' from /app/vendor/bundle/ruby/2.4.0/gems/net-http-persistent-3.0.0/lib/net/http/persistent.rb:692:in `start' from /app/vendor/bundle/ruby/2.4.0/gems/net-http-persistent-3.0.0/lib/net/http/persistent.rb:622:in `connection_for' from /app/vendor/bundle/ruby/2.4.0/gems/net-http-persistent-3.0.0/lib/net/http/persistent.rb:927:in `request' from /app/vendor/bundle/ruby/2.4.0/gems/mechanize-2.7.6/lib/mechanize/http/agent.rb:280:in `fetch' from /app/vendor/bundle/ruby/2.4.0/gems/mechanize-2.7.6/lib/mechanize.rb:464:in `get' from scraper.rb:83:in `<main>'

Data

Downloaded 1 time by

data

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (61 KB) Use the API

rows 10 / 148

info_url	council_reference	date_received	description	address	date_scraped	comment_url
http://pdonline.ipswich.qld.gov.au/pdonline/modules/applicationmaster/default.aspx?page=wrapper&key=771784	MCU-8599/2018	2018-11-02	Single Residential in a Character Zone and affected by a Development Constraints Overlay	11-13 Horan Street WOODEND QLD 4305	2018-11-13	mailto:plandev@ipswich.qld.gov.au
http://pdonline.ipswich.qld.gov.au/pdonline/modules/applicationmaster/default.aspx?page=wrapper&key=772155	MCU-8623/2018	2018-11-02	Material Change of Use - General Industry (Storage and Truck Depot) and Service/Trades Use (Builders/Contractors Depot, Repair Station and Storage)	136 Chalk Street WULKURAKA QLD 4305	2018-11-13	mailto:plandev@ipswich.qld.gov.au
http://pdonline.ipswich.qld.gov.au/pdonline/modules/applicationmaster/default.aspx?page=wrapper&key=772175	MCU-8625/2018	2018-11-02	Material Change of Use - Single Residential in a Rural Zone	287-355 Sippels Road CALVERT QLD 4340	2018-11-13	mailto:plandev@ipswich.qld.gov.au
http://pdonline.ipswich.qld.gov.au/pdonline/modules/applicationmaster/default.aspx?page=wrapper&key=773614	MCU-8715/2018	2018-11-07	Material Change of Use - Single Residential - Building Setback Plan	7001 Essex Street KARALEE QLD 4306	2018-11-13	mailto:plandev@ipswich.qld.gov.au
http://pdonline.ipswich.qld.gov.au/pdonline/modules/applicationmaster/default.aspx?page=wrapper&key=774064	MCU-8747/2018	2018-11-08	Material Change of Use - Single Residential (including Auxiliary Unit) in a Character Zone	14 George Street BLACKSTONE QLD 4304	2018-11-13	mailto:plandev@ipswich.qld.gov.au
http://pdonline.ipswich.qld.gov.au/pdonline/modules/applicationmaster/default.aspx?page=wrapper&key=774336	MCU-8766/2018	2018-11-08	General Industry - Concrete Column Manufacturing Plant	3 Mary Street BLACKSTONE QLD 4304	2018-11-13	mailto:plandev@ipswich.qld.gov.au
http://pdonline.ipswich.qld.gov.au/pdonline/modules/applicationmaster/default.aspx?page=wrapper&key=772257	CA-8631/2018	2018-11-02	Material Change of Use - Temporary Sales Office Advertising Device - One (1) Billboard and Four (4) Wall Signs	7001 Karrabin Rosewood Road ROSEWOOD QLD 4340	2018-11-13	mailto:plandev@ipswich.qld.gov.au
http://pdonline.ipswich.qld.gov.au/pdonline/modules/applicationmaster/default.aspx?page=wrapper&key=771594	OW-8583/2018	2018-11-01	Earthworks and Clearing Vegetation	48 Austin Street NEW CHUM QLD 4303	2018-11-13	mailto:plandev@ipswich.qld.gov.au
http://pdonline.ipswich.qld.gov.au/pdonline/modules/applicationmaster/default.aspx?page=wrapper&key=771644	OW-8586/2018	2018-11-02	Rate 3 Street Lighting	7001 Baird Circuit REDBANK PLAINS QLD 4301	2018-11-13	mailto:plandev@ipswich.qld.gov.au
http://pdonline.ipswich.qld.gov.au/pdonline/modules/applicationmaster/default.aspx?page=wrapper&key=772914	OW-8674/2018	2018-11-06	Earthworks - Edens Crossing Stages 10 to 14	Lot 89 Unnamed Road REDBANK PLAINS QLD 4301	2018-11-13	mailto:plandev@ipswich.qld.gov.au

Statistics

Average successful run time: half a minute

Total run time: about 14 hours

Total cpu time used: 27 minutes

Total disk space used: 100 KB

History

Auto ran revision e1757d7a and failed 2025-04-21.

run time 1 minute

nothing changed in the database
Auto ran revision e1757d7a and failed 2025-04-20.

run time 1 minute

nothing changed in the database
Auto ran revision e1757d7a and failed 2025-04-19.

run time 1 minute

nothing changed in the database
Auto ran revision e1757d7a and failed 2025-04-18.

run time 1 minute

nothing changed in the database
Auto ran revision e1757d7a and failed 2025-04-17.

run time 1 minute

nothing changed in the database
...
Created on morph.io 2018-11-13

Show complete history

Scraper code

Ruby

ipswich / scraper.rb

git clone URL