Brisbane Council scraper

  • Accept Terms - Yes
  • Cookie tracking - Yes
  • Request actual DA page for data - Yes
  • Clearly defined data within a row - No but reasonable

Setup MORPH_PERIOD for data recovery, available options are

  • not set = past 14 days (default)
  • thismonth
  • lastmonth

Contributors equivalentideas ljnicol mlandauer henare LoveMyData

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.4.1 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/......  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Fetching mime-types-data 3.2018.0812  Installing httpclient 2.8.3  Installing unf_ext 0.0.7.5 with native extensions  Installing mime-types-data 3.2018.0812  Fetching net-http-digest_auth 1.4.1  Fetching mini_portile2 2.3.0  Installing mini_portile2 2.3.0  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.3.13  Installing sqlite3 1.3.13 with native extensions  Installing net-http-digest_auth 1.4.1  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.8.4  Installing nokogiri 1.8.4 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 2 Gemfile dependencies, 19 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (22.46s)  Cleaning up the bundler cache. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Collecting data from 10/05/2022&2=24/05/2022 /app/vendor/ruby-2.4.1/lib/ruby/2.4.0/net/http.rb:904:in `initialize': execution expired (Net::OpenTimeout) from /app/vendor/ruby-2.4.1/lib/ruby/2.4.0/net/http.rb:904:in `open' from /app/vendor/ruby-2.4.1/lib/ruby/2.4.0/net/http.rb:904:in `block in connect' from /app/vendor/ruby-2.4.1/lib/ruby/2.4.0/timeout.rb:103:in `timeout' from /app/vendor/ruby-2.4.1/lib/ruby/2.4.0/net/http.rb:902:in `connect' from /app/vendor/ruby-2.4.1/lib/ruby/2.4.0/net/http.rb:887:in `do_start' from /app/vendor/ruby-2.4.1/lib/ruby/2.4.0/net/http.rb:882:in `start' from /app/vendor/bundle/ruby/2.4.0/gems/net-http-persistent-3.0.0/lib/net/http/persistent.rb:692:in `start' from /app/vendor/bundle/ruby/2.4.0/gems/net-http-persistent-3.0.0/lib/net/http/persistent.rb:622:in `connection_for' from /app/vendor/bundle/ruby/2.4.0/gems/net-http-persistent-3.0.0/lib/net/http/persistent.rb:927:in `request' from /app/vendor/bundle/ruby/2.4.0/gems/mechanize-2.7.6/lib/mechanize/http/agent.rb:280:in `fetch' from /app/vendor/bundle/ruby/2.4.0/gems/mechanize-2.7.6/lib/mechanize.rb:464:in `get' from scraper.rb:89:in `<main>'

Data

Downloaded 974 times by LoveMyData MichaelBone yourneighbourhood EMBLAPROJECTS Stephen-bmrk spatialsteve rimonhm

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (7.64 MB) Use the API

rows 10 / 14182

info_url council_reference date_received description address date_scraped comment_url lot property_description
A004990622
2018-08-08
Extension, Dwelling House, Amenity and aesthetics
56 BARON ST GREENSLOPES QLD 4120
2018-08-22
394RP61920
L.394 RP.61920 PAR Y'PILLY
A004991047
2018-08-08
Extension, Special Industry
398 TINGIRA ST PINKENBA QLD 4008
2018-08-22
846SP135226
A004991142
2018-08-08
Food and Drink Outlet, Low Impact Industry, Medium Impact Industry, Shop, Food and Drink Outlet, Low Impact Industry, Medium Impact Industry, Shop
655 TOOHEY RD SALISBURY QLD 4107
2018-08-22
1RP118653
L.1 RP.118653 PAR Y'PILLY
A004990975
2018-08-08
Volumetric Plan
38 WHARF ST BRISBANE CITY QLD 4000
2018-08-22
11B118227
L.11 B.118227 PAR NTH BRISBANE
A004990740
2018-08-08
Vegetation Management Plan
20 BROOKSIDE ST DOOLANDELLA QLD 4077
2018-08-22
9SP125112
L.9 SP.125112 PAR OXLEY
A004990826
2018-08-08
Construction Management Plans
278 GYMPIE RD KEDRON QLD 4031
2018-08-22
1RP125416
L.1 RP.125416 PAR KEDRON
A004991408
2018-08-09
Subdivision of Land
70 AGNES ST SUNNYBANK QLD 4109
2018-08-22
37RP78607
L.37 RP.78607 PAR Y'PILLY
A004991436
2018-08-09
Low Impact Industry, Warehouse
94 OLD TOOMBUL RD NORTHGATE QLD 4013
2018-08-22
7RP60267
L.7 RP.60267 PAR TOOMBUL
A004991586
2018-08-09
Dwelling House, Dwelling House
144 MOREHEAD AVE NORMAN PARK QLD 4170
2018-08-22
177RP13150
L.177 RP.13150 PAR BULIMBA
A004991666
2018-08-09
Stormwater Drainage
37 MOLONGA TCE GRACEVILLE QLD 4075
2018-08-22
1RP96414
L.2 RP.72478 & L.1 RP.96414 PAR OXLEY

Statistics

Average successful run time: 8 minutes

Total run time: 4 days

Total cpu time used: about 2 hours

Total disk space used: 7.69 MB

History

  • Auto ran revision f6647683 and failed .
    nothing changed in the database
  • Auto ran revision f6647683 and failed .
    nothing changed in the database
  • Auto ran revision f6647683 and failed .
    nothing changed in the database
  • Auto ran revision f6647683 and failed .
    nothing changed in the database
  • Auto ran revision f6647683 and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

brisbane / scraper.rb