Brisbane Council scraper

  • Accept Terms - Yes
  • Cookie tracking - Yes
  • Request actual DA page for data - Yes
  • Clearly defined data within a row - No but reasonable

Setup MORPH_PERIOD for data recovery, available options are

  • not set = past 14 days (default)
  • thismonth
  • lastmonth

Contributors mlandauer equivalentideas henare ljnicol LoveMyData MichaelBone

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.4.1 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Fetching mime-types-data 3.2018.0812  Installing httpclient 2.8.3  Installing unf_ext 0.0.7.5 with native extensions  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.3.0  Installing mime-types-data 3.2018.0812  Fetching ntlm-http 0.1.1  Installing mini_portile2 2.3.0  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.3.13  Installing sqlite3 1.3.13 with native extensions  Installing ntlm-http 0.1.1  Fetching net-http-persistent 3.0.0  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.8.4  Installing nokogiri 1.8.4 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 2 Gemfile dependencies, 19 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (26.80s)  Cleaning up the bundler cache. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Collecting data from 05/04/2019&2=19/04/2019 Skipping already saved record A005166134 Skipping already saved record A005166029 Skipping already saved record A005166158 Skipping already saved record A005168560 Skipping already saved record A005168728 Skipping already saved record A005168642 Skipping already saved record A005168757 Skipping already saved record A005168853 Skipping already saved record A005168762 Skipping already saved record A005168702 Skipping already saved record A005169206 Skipping already saved record A005169457 Skipping already saved record A005169492 - Skipping already saved record A005169700 Skipping already saved record A005169757 Skipping already saved record A005169719 Skipping already saved record A005169871 Skipping already saved record A005170300 Skipping already saved record A005170378 Skipping already saved record A005170470 Skipping already saved record A005170284 Skipping already saved record A005170449 Skipping already saved record A005170544 Skipping already saved record A005170338 Skipping already saved record A005171007 Skipping already saved record A005170784 Skipping already saved record A005170808 Skipping already saved record A005173522 Skipping already saved record A005173465 Skipping already saved record A005173606 Skipping already saved record A005173300 Skipping already saved record A005173684 Skipping already saved record A005173654 Skipping already saved record A005173688 Skipping already saved record A005173852 Skipping already saved record A005173964 Skipping already saved record A005174224 Skipping already saved record A005174206 Skipping already saved record A005173862 Skipping already saved record A005174055 Skipping already saved record A005174122 Skipping already saved record A005174238 Skipping already saved record A005174556 Skipping already saved record A005174753 Skipping already saved record A005174646 Skipping already saved record A005174817 Skipping already saved record A005174600 Skipping already saved record A005174831 Skipping already saved record A005175416 Skipping already saved record A005175174 Skipping already saved record A005175241 Skipping already saved record A005166224 Skipping already saved record A005166242 Skipping already saved record A005166032 Skipping already saved record A005166066 Skipping already saved record A005166076 Skipping already saved record A005168916 Skipping already saved record A005168638 Skipping already saved record A005168737 Skipping already saved record A005169303 Skipping already saved record A005169978 Skipping already saved record A005169719 Skipping already saved record A005169951 Skipping already saved record A005170055 Skipping already saved record A005169987 Skipping already saved record A005170286 Skipping already saved record A005170320 Skipping already saved record A005170337 Skipping already saved record A005170229 Skipping already saved record A005170414 Skipping already saved record A005170790 Skipping already saved record A005170732 Skipping already saved record A005170906 Skipping already saved record A005170881 Skipping already saved record A005170689 Skipping already saved record A005170899 Skipping already saved record A005170841 Skipping already saved record A005173522 Skipping already saved record A005173690 Skipping already saved record A005173429 Skipping already saved record A005173521 Skipping already saved record A005173300 Skipping already saved record A005173612 Skipping already saved record A005173688 Skipping already saved record A005173441 Skipping already saved record A005174184 Skipping already saved record A005173958 Skipping already saved record A005173875 Skipping already saved record A005174166 Skipping already saved record A005174556 Skipping already saved record A005174753 Skipping already saved record A005174850 Skipping already saved record A005174548 Skipping already saved record A005174802 Skipping already saved record A005174572 Skipping already saved record A005174820 Skipping already saved record A005174831 Skipping already saved record A005175110 Skipping already saved record A005175302 Skipping already saved record A005175416 Skipping already saved record A005175246 Skipping already saved record A005175339 Skipping already saved record A005175092 Skipping already saved record A005175232 Skipping already saved record A005166169 Skipping already saved record A005166243 Skipping already saved record A005166098 Skipping already saved record A005165973 Skipping already saved record A005166384 Skipping already saved record A005168560 Skipping already saved record A005168619 Skipping already saved record A005168638 Skipping already saved record A005168895 Skipping already saved record A005169179 Skipping already saved record A005169159 Skipping already saved record A005169282 Skipping already saved record A005169905 Skipping already saved record A005170378 Skipping already saved record A005170467 Skipping already saved record A005170491 Skipping already saved record A005170320 Skipping already saved record A005170268 Skipping already saved record A005170544 Skipping already saved record A005170414 Skipping already saved record A005170571 Skipping already saved record A005170934 Skipping already saved record A005170904 Skipping already saved record A005171015 Skipping already saved record A005170938 Skipping already saved record A005170745 Skipping already saved record A005170958 Skipping already saved record A005170803 Skipping already saved record A005170957 Skipping already saved record A005170854 Skipping already saved record A005170887 Skipping already saved record A005173459 Skipping already saved record A005173522 Skipping already saved record A005173300 Skipping already saved record A005173684 Skipping already saved record A005173688 Skipping already saved record A005174101 Skipping already saved record A005174206 Skipping already saved record A005173995 Skipping already saved record A005174214 Skipping already saved record A005174122 Skipping already saved record A005174486 Skipping already saved record A005174387 Skipping already saved record A005174594 Skipping already saved record A005174676 Skipping already saved record A005175198 Skipping already saved record A005175240 Skipping already saved record A005175225 Skipping already saved record A005175341 Skipping already saved record A005175358 Skipping already saved record A005166104 Skipping already saved record A005166287 Skipping already saved record A005168727 Skipping already saved record A005168894 Skipping already saved record A005168601 Skipping already saved record A005168697 Skipping already saved record A005168728 Skipping already saved record A005168737 Skipping already saved record A005169541 Skipping already saved record A005169309 Skipping already saved record A005169162 Skipping already saved record A005169757 Skipping already saved record A005169754 Skipping already saved record A005170008 Skipping already saved record A005170378 Skipping already saved record A005170399 Skipping already saved record A005170414 Skipping already saved record A005170571 Skipping already saved record A005171014 Skipping already saved record A005170803 Skipping already saved record A005170785 Skipping already saved record A005173326 Skipping already saved record A005173465 Skipping already saved record A005173606 Skipping already saved record A005173632 Skipping already saved record A005173300 Skipping already saved record A005173639 Skipping already saved record A005173654 Skipping already saved record A005173688 Skipping already saved record A005173303 Skipping already saved record A005174167 Skipping already saved record A005173964 Skipping already saved record A005173987 Skipping already saved record A005174178 Skipping already saved record A005173880 Skipping already saved record A005174179 Skipping already saved record A005173859 Skipping already saved record A005174411 Skipping already saved record A005174556 Skipping already saved record A005174837 Skipping already saved record A005174451 Skipping already saved record A005174596 Skipping already saved record A005174802 Skipping already saved record A005174655 Skipping already saved record A005174727 Skipping already saved record A005174788 Skipping already saved record A005174807 Skipping already saved record A005174820 Skipping already saved record A005175160 Skipping already saved record A005175302 Skipping already saved record A005175316 Skipping already saved record A005175296 Skipping already saved record A005175183 Skipping already saved record A005175395 Skipping already saved record A005175384 Skipping already saved record A005165989 Skipping already saved record A005166136 Skipping already saved record A005166333 Skipping already saved record A005166204 Skipping already saved record A005166124 Skipping already saved record A005168819 Skipping already saved record A005168601 Skipping already saved record A005168697 Skipping already saved record A005168842 Skipping already saved record A005168615 Skipping already saved record A005168637 Skipping already saved record A005169217 Skipping already saved record A005169978 Skipping already saved record A005169840 Skipping already saved record A005169988 Skipping already saved record A005170352 Skipping already saved record A005170503 Skipping already saved record A005170320 Skipping already saved record A005170339 Skipping already saved record A005170254 Skipping already saved record A005171014 Skipping already saved record A005170904 Skipping already saved record A005173501 Skipping already saved record A005173614 Skipping already saved record A005173429 Skipping already saved record A005173606 Skipping already saved record A005173532 Skipping already saved record A005173591 Skipping already saved record A005173513 Skipping already saved record A005173688 Skipping already saved record A005173441 Skipping already saved record A005173627 Skipping already saved record A005174184 Skipping already saved record A005173966 Skipping already saved record A005173964 Skipping already saved record A005174224 Skipping already saved record A005173921 Skipping already saved record A005174055 Skipping already saved record A005174556 Skipping already saved record A005174753 Skipping already saved record A005174844 - Skipping already saved record A005174450 Skipping already saved record A005174817 Skipping already saved record A005174594 Skipping already saved record A005174582 Skipping already saved record A005174807 Skipping already saved record A005175341 Skipping already saved record A005175412 Skipping already saved record A005166369 Skipping already saved record A005166370 Skipping already saved record A005166253 Skipping already saved record A005168754 Skipping already saved record A005168580 Skipping already saved record A005168697 Skipping already saved record A005169215 Skipping already saved record A005169438 Skipping already saved record A005169156 Skipping already saved record A005169272 Skipping already saved record A005169378 Skipping already saved record A005169074 Skipping already saved record A005169206 Skipping already saved record A005169249 Skipping already saved record A005169986 Skipping already saved record A005169690 Skipping already saved record A005169858 Skipping already saved record A005170378 Skipping already saved record A005170320 Skipping already saved record A005170449 Skipping already saved record A005170537 Skipping already saved record A005170544 Skipping already saved record A005170229 Skipping already saved record A005170584 Skipping already saved record A005170652 Skipping already saved record A005170882 Skipping already saved record A005170987 Skipping already saved record A005171006 Skipping already saved record A005170983 Skipping already saved record A005170747 Skipping already saved record A005170863 Skipping already saved record A005173522 Skipping already saved record A005173614 Skipping already saved record A005173366 Skipping already saved record A005173429 Skipping already saved record A005173684 Skipping already saved record A005173513 Skipping already saved record A005173612 Skipping already saved record A005173688 Skipping already saved record A005173389 Skipping already saved record A005173964 Skipping already saved record A005174556 Skipping already saved record A005174850 Skipping already saved record A005174817 Skipping already saved record A005174449 Skipping already saved record A005175070 Skipping already saved record A005175110 Skipping already saved record A005175441 Skipping already saved record A005174967 Skipping already saved record A005166318 Skipping already saved record A005166096 Skipping already saved record A005165997 Skipping already saved record A005168773 - Skipping already saved record A005168845 Skipping already saved record A005168851 Skipping already saved record A005168560 Skipping already saved record A005168728 Skipping already saved record A005168642 Skipping already saved record A005168954 Skipping already saved record A005168909 Skipping already saved record A005169182 Skipping already saved record A005169286 Skipping already saved record A005169541 Skipping already saved record A005169315 Skipping already saved record A005169114 Skipping already saved record A005169700 Skipping already saved record A005170352 Skipping already saved record A005170347 Skipping already saved record A005170500 Skipping already saved record A005170544 Skipping already saved record A005170584 Skipping already saved record A005171014 Skipping already saved record A005170950 Skipping already saved record A005170660 Skipping already saved record A005170744 Skipping already saved record A005170890 Skipping already saved record A005171006 Skipping already saved record A005170793 Skipping already saved record A005171011 Skipping already saved record A005170984 Skipping already saved record A005170808 Skipping already saved record A005170762 Skipping already saved record A005170801 Skipping already saved record A005170887 Skipping already saved record A005173273 Skipping already saved record A005173632 Skipping already saved record A005173521 Skipping already saved record A005173559 Skipping already saved record A005173612 Skipping already saved record A005173688 Skipping already saved record A005173594 Skipping already saved record A005173957 Skipping already saved record A005174105 Skipping already saved record A005173958 Skipping already saved record A005174118 Skipping already saved record A005173987 Skipping already saved record A005174224 Skipping already saved record A005174206 Skipping already saved record A005174021 Skipping already saved record A005174753 Skipping already saved record A005174451 Skipping already saved record A005174642 Skipping already saved record A005174523 Skipping already saved record A005174600 Skipping already saved record A005174831 Skipping already saved record A005175082 Skipping already saved record A005175110 Skipping already saved record A005175160 Skipping already saved record A005175359 Skipping already saved record A005175108 Skipping already saved record A005175413 Skipping already saved record A005175299 Skipping already saved record A005175441 Skipping already saved record A005175254 Skipping already saved record A005175282 Skipping already saved record A005175430 Skipping already saved record A005175115 Skipping already saved record A005165976 Skipping already saved record A005166378 Skipping already saved record A005166338 Skipping already saved record A005168714 Skipping already saved record A005168601 Skipping already saved record A005168619 Skipping already saved record A005168577 Skipping already saved record A005168710 Skipping already saved record A005168840 Skipping already saved record A005168645 Skipping already saved record A005169437 Skipping already saved record A005169719 Skipping already saved record A005170041 Skipping already saved record A005169899 Skipping already saved record A005170470 Skipping already saved record A005170328 Skipping already saved record A005170478 Skipping already saved record A005170518 Skipping already saved record A005170580 Skipping already saved record A005170770 Skipping already saved record A005170939 Skipping already saved record A005173599 Skipping already saved record A005173570 Skipping already saved record A005173593 Skipping already saved record A005173614 Skipping already saved record A005173516 - Skipping already saved record A005173429 Skipping already saved record A005173517 Skipping already saved record A005173686 Skipping already saved record A005173639 Skipping already saved record A005173688 Skipping already saved record A005173402 Skipping already saved record A005174112 Skipping already saved record A005174213 Skipping already saved record A005174411 Skipping already saved record A005174704 Skipping already saved record A005174813 Skipping already saved record A005174755 Skipping already saved record A005174523 Skipping already saved record A005175160 Skipping already saved record A005175402 Skipping already saved record A005175225 Skipping already saved record A005175341

Data

Downloaded 8377 times by slav123 openaustralia LoveMyData jesse-a-reilly TheZepto spatialsteve stevebrencorp yourneighbourhood srsmonty ljnicol dsccramb peteclowes primin2 MikeRalphson rimonhm darylrae mlandauer Jedwards157 GitHubMLee n1k0p0l ShawInnes newmana thinkpropertyjason bigsmaug jamezpolley

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (3.88 MB) Use the API

rows 10 / 7490

info_url council_reference date_received description address date_scraped comment_url lot property_description
A004868622
2018-03-01
Signs and Line Marking for Minor Roads
106 CONSIDINE ST ELLEN GROVE QLD 4078
2018-03-09
A004868759
2018-03-01
Multiple Dwelling, Extension, Multiple Dwelling, Demolition
14 FIFTH AVE SANDGATE QLD 4017
2018-03-09
A004871923
2018-03-01
Standard Format Plan
79 BROOKBENT RD PALLARA QLD 4110
2018-03-09
A004869372
2018-03-01
Standard Format Plan
7 HOOLAN ST STAFFORD QLD 4053
2018-03-09
A004872047
2018-03-01
Standard Format Plan
261 GOODERHAM RD WILLAWONG QLD 4110
2018-03-09
A004868803
2018-03-01
Outdoor Sport and Recreation, Extension
338 GOODERHAM RD N WILLAWONG QLD 4110
2018-03-09
A004868586
2018-03-01
Food and Drink Outlet, Office, Food and Drink Outlet, Office
22 ROSS ST NEWSTEAD QLD 4006
2018-03-09
A004869102
2018-03-02
Dwelling House, Extension, Dwelling House, Extension, Dwelling House, Demolition
66 PRIMROSE ST GRANGE QLD 4051
2018-03-09
A004869404
2018-03-02
Educational Establishment, Extension
109 NORTH RD WYNNUM WEST QLD 4178
2018-03-09
A004869449
2018-03-02
Filling and/or Excavation
59 DENHAM ST BRACKEN RIDGE QLD 4017
2018-03-09

Statistics

Average successful run time: 4 minutes

Total run time: 15 days

Total cpu time used: about 2 hours

Total disk space used: 3.92 MB

History

  • Auto ran revision 3b80c5bd and completed successfully .
    nothing changed in the database
  • Auto ran revision 3b80c5bd and completed successfully .
    34 records added in the database
  • Auto ran revision 3b80c5bd and completed successfully .
    32 records added in the database
  • Auto ran revision 3b80c5bd and completed successfully .
    29 records added in the database
  • Auto ran revision 3b80c5bd and completed successfully .
    35 records added in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

brisbane / scraper.rb