Brisbane Council scraper

  • Accept Terms - Yes
  • Cookie tracking - Yes
  • Request actual DA page for data - Yes
  • Clearly defined data within a row - No but reasonable

Setup MORPH_PERIOD for data recovery, available options are

  • not set = past 14 days (default)
  • thismonth
  • lastmonth

Contributors mlandauer equivalentideas henare ljnicol LoveMyData MichaelBone

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.4.1 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Fetching mime-types-data 3.2018.0812  Installing unf_ext 0.0.7.5 with native extensions  Installing httpclient 2.8.3  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.3.0  Installing mime-types-data 3.2018.0812  Installing mini_portile2 2.3.0  Fetching ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing ntlm-http 0.1.1  Installing webrobots 0.1.2  Fetching sqlite3 1.3.13  Fetching net-http-persistent 3.0.0  Installing sqlite3 1.3.13 with native extensions  Installing net-http-persistent 3.0.0  Fetching nokogiri 1.8.4  Installing nokogiri 1.8.4 with native extensions  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 2 Gemfile dependencies, 19 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (26.08s)  Cleaning up the bundler cache. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Collecting data from 03/02/2019&2=17/02/2019 Skipping already saved record A005123536 Skipping already saved record A005123589 Skipping already saved record A005123372 Skipping already saved record A005123228 Skipping already saved record A005123521 Skipping already saved record A005123430 Skipping already saved record A005123969 Skipping already saved record A005123942 Skipping already saved record A005123947 Skipping already saved record A005124718 Skipping already saved record A005124324 Skipping already saved record A005124512 Skipping already saved record A005124570 Skipping already saved record A005124327 Skipping already saved record A005124380 Skipping already saved record A005125104 Skipping already saved record A005125184 Skipping already saved record A005125208 Skipping already saved record A005125127 Skipping already saved record A005125223 Skipping already saved record A005124832 Skipping already saved record A005124943 Skipping already saved record A005124904 Skipping already saved record A005125389 Skipping already saved record A005125577 Skipping already saved record A005125586 Skipping already saved record A005125491 Skipping already saved record A005125557 Skipping already saved record A005125560 Skipping already saved record A005125590 Skipping already saved record A005127638 Skipping already saved record A005127743 Skipping already saved record A005127875 Skipping already saved record A005128425 Skipping already saved record A005128554 Skipping already saved record A005128255 Skipping already saved record A005128300 Skipping already saved record A005128579 Skipping already saved record A005128614 Skipping already saved record A005128482 Skipping already saved record A005128900 Skipping already saved record A005128901 Skipping already saved record A005128945 Skipping already saved record A005129013 Skipping already saved record A005129246 Skipping already saved record A005128848 Skipping already saved record A005129710 Skipping already saved record A005129598 Skipping already saved record A005129547 Skipping already saved record A005129713 Skipping already saved record A005129788 Skipping already saved record A005130113 Skipping already saved record A005130240 Skipping already saved record A005129865 Skipping already saved record A005130129 Skipping already saved record A005123615 Skipping already saved record A005123395 Skipping already saved record A005123515 Skipping already saved record A005123389 Skipping already saved record A005123257 Skipping already saved record A005123412 Skipping already saved record A005123772 Skipping already saved record A005123932 Skipping already saved record A005123969 Skipping already saved record A005123942 Skipping already saved record A005124400 Skipping already saved record A005124327 Skipping already saved record A005124380 Skipping already saved record A005124872 Skipping already saved record A005125012 Skipping already saved record A005124943 Skipping already saved record A005125489 Skipping already saved record A005125551 Skipping already saved record A005125450 Skipping already saved record A005125503 Skipping already saved record A005125590 Skipping already saved record A005127743 Skipping already saved record A005127800 Skipping already saved record A005128016 Skipping already saved record A005127840 Skipping already saved record A005127891 Skipping already saved record A005128539 Skipping already saved record A005128554 Skipping already saved record A005128384 Skipping already saved record A005128414 Skipping already saved record A005128945 Skipping already saved record A005129132 Skipping already saved record A005129246 Skipping already saved record A005128990 Skipping already saved record A005129535 Skipping already saved record A005129547 Skipping already saved record A005129441 Skipping already saved record A005129577 Skipping already saved record A005129788 Skipping already saved record A005130113 Skipping already saved record A005130173 Skipping already saved record A005129917 Skipping already saved record A005129945 Skipping already saved record A005130124 Skipping already saved record A005123509 Skipping already saved record A005123419 Skipping already saved record A005123857 Skipping already saved record A005123932 Skipping already saved record A005123969 Skipping already saved record A005124291 Skipping already saved record A005124456 Skipping already saved record A005124648 Skipping already saved record A005124380 Skipping already saved record A005125104 Skipping already saved record A005125184 Skipping already saved record A005124890 Skipping already saved record A005125165 Skipping already saved record A005125196 Skipping already saved record A005125609 Skipping already saved record A005125347 Skipping already saved record A005125491 Skipping already saved record A005125591 Skipping already saved record A005125412 Skipping already saved record A005125449 Skipping already saved record A005125668 - Skipping already saved record A005125590 Skipping already saved record A005125698 Skipping already saved record A005127743 Skipping already saved record A005127685 Skipping already saved record A005127891 Skipping already saved record A005128641 Skipping already saved record A005128554 Skipping already saved record A005128342 Skipping already saved record A005128515 Skipping already saved record A005128614 Skipping already saved record A005128901 Skipping already saved record A005129216 Skipping already saved record A005129114 Skipping already saved record A005129747 Skipping already saved record A005129588 Skipping already saved record A005129498 Skipping already saved record A005129547 Skipping already saved record A005129696 Skipping already saved record A005129441 Skipping already saved record A005130012 Skipping already saved record A005130027 Skipping already saved record A005130243 Skipping already saved record A005129934 Skipping already saved record A005123262 Skipping already saved record A005123392 Skipping already saved record A005123463 Skipping already saved record A005123215 Skipping already saved record A005123806 Skipping already saved record A005123842 Skipping already saved record A005124105 Skipping already saved record A005123742 Skipping already saved record A005123942 Skipping already saved record A005124559 Skipping already saved record A005124647 Skipping already saved record A005124291 Skipping already saved record A005124300 Skipping already saved record A005124512 Skipping already saved record A005124689 Skipping already saved record A005124380 Skipping already saved record A005124643 Skipping already saved record A005124965 Skipping already saved record A005125104 Skipping already saved record A005125043 Skipping already saved record A005124925 Skipping already saved record A005124969 Skipping already saved record A005125586 Skipping already saved record A005125551 Skipping already saved record A005125559 Skipping already saved record A005125624 Skipping already saved record A005125650 Skipping already saved record A005125599 Skipping already saved record A005125674 Skipping already saved record A005125590 Skipping already saved record A005127743 Skipping already saved record A005127772 Skipping already saved record A005127782 Skipping already saved record A005127891 Skipping already saved record A005128188 Skipping already saved record A005128539 Skipping already saved record A005128632 Skipping already saved record A005128245 Skipping already saved record A005128492 Skipping already saved record A005128859 Skipping already saved record A005128945 Skipping already saved record A005129013 Skipping already saved record A005129214 Skipping already saved record A005129262 Skipping already saved record A005129114 Skipping already saved record A005129202 Skipping already saved record A005128990 Skipping already saved record A005129650 Skipping already saved record A005129660 Skipping already saved record A005129710 Skipping already saved record A005129588 Skipping already saved record A005129444 Skipping already saved record A005129582 Skipping already saved record A005129732 Skipping already saved record A005129480 Skipping already saved record A005129964 Skipping already saved record A005129934 Skipping already saved record A005130002 Skipping already saved record A005129953 Skipping already saved record A005123265 Skipping already saved record A005123316 Skipping already saved record A005123515 Skipping already saved record A005123387 Skipping already saved record A005123315 Skipping already saved record A005123392 Skipping already saved record A005123497 Skipping already saved record A005123533 Skipping already saved record A005123283 Skipping already saved record A005123412 Skipping already saved record A005123849 Skipping already saved record A005123822 Skipping already saved record A005123751 Skipping already saved record A005123807 Skipping already saved record A005124090 Skipping already saved record A005124660 Skipping already saved record A005124300 Skipping already saved record A005124380 Skipping already saved record A005124459 Skipping already saved record A005124643 Skipping already saved record A005124705 Skipping already saved record A005125104 Skipping already saved record A005125147 Skipping already saved record A005124817 Skipping already saved record A005124934 Skipping already saved record A005125019 Skipping already saved record A005125196 Skipping already saved record A005124894 Skipping already saved record A005125599 Skipping already saved record A005125446 Skipping already saved record A005125664 Skipping already saved record A005127638 Skipping already saved record A005127743 Skipping already saved record A005127747 Skipping already saved record A005127842 Skipping already saved record A005127890 Skipping already saved record A005128077 Skipping already saved record A005127759 Skipping already saved record A005128336 Skipping already saved record A005128441 Skipping already saved record A005128554 Skipping already saved record A005128342 Skipping already saved record A005128579 Skipping already saved record A005128267 Skipping already saved record A005128576 Skipping already saved record A005128615 Skipping already saved record A005128976 Skipping already saved record A005129549 Skipping already saved record A005129547 Skipping already saved record A005129528 Skipping already saved record A005129787 Skipping already saved record A005130113 Skipping already saved record A005130118 Skipping already saved record A005129903 Skipping already saved record A005123212 Skipping already saved record A005123932 Skipping already saved record A005123942 Skipping already saved record A005123850 Skipping already saved record A005123875 Skipping already saved record A005124090 Skipping already saved record A005124300 Skipping already saved record A005124607 Skipping already saved record A005124413 Skipping already saved record A005124468 Skipping already saved record A005124380 Skipping already saved record A005124508 Skipping already saved record A005124672 Skipping already saved record A005125104 Skipping already saved record A005125214 Skipping already saved record A005124901 Skipping already saved record A005124905 Skipping already saved record A005124832 Skipping already saved record A005124958 Skipping already saved record A005125336 Skipping already saved record A005125491 Skipping already saved record A005125619 Skipping already saved record A005125597 Skipping already saved record A005125557 Skipping already saved record A005125418 Skipping already saved record A005127747 Skipping already saved record A005127772 Skipping already saved record A005128058 Skipping already saved record A005127795 Skipping already saved record A005127923 Skipping already saved record A005127843 Skipping already saved record A005128491 Skipping already saved record A005128595 Skipping already saved record A005128632 Skipping already saved record A005128384 Skipping already saved record A005128908 Skipping already saved record A005128981 Skipping already saved record A005129246 Skipping already saved record A005129016 Skipping already saved record A005129487 Skipping already saved record A005129660 Skipping already saved record A005129376 Skipping already saved record A005130030 Skipping already saved record A005130092 Skipping already saved record A005129917 Skipping already saved record A005129992 Skipping already saved record A005123274 Skipping already saved record A005123326 Skipping already saved record A005123395 Skipping already saved record A005123510 Skipping already saved record A005123283 Skipping already saved record A005123562 Skipping already saved record A005123698 Skipping already saved record A005123930 Skipping already saved record A005123942 Skipping already saved record A005123917 Skipping already saved record A005124327 Skipping already saved record A005124380 Skipping already saved record A005125006 Skipping already saved record A005125104 Skipping already saved record A005124853 Skipping already saved record A005124857 Skipping already saved record A005124958 Skipping already saved record A005125521 Skipping already saved record A005125559 Skipping already saved record A005125419 Skipping already saved record A005125485 Skipping already saved record A005125557 Skipping already saved record A005125393 Skipping already saved record A005125484 Skipping already saved record A005127638 Skipping already saved record A005127743 Skipping already saved record A005128017 Skipping already saved record A005127643 Skipping already saved record A005128233 Skipping already saved record A005128632 Skipping already saved record A005128554 Skipping already saved record A005128255 Skipping already saved record A005128300 Skipping already saved record A005128414 Skipping already saved record A005128579 Skipping already saved record A005128614 Skipping already saved record A005128615 Skipping already saved record A005128964 Skipping already saved record A005128770 Skipping already saved record A005129547 Skipping already saved record A005129582 Skipping already saved record A005129441 Skipping already saved record A005129517 Skipping already saved record A005130255 Skipping already saved record A005129939 Skipping already saved record A005129903 Skipping already saved record A005123861 Skipping already saved record A005123857 Skipping already saved record A005124105 Skipping already saved record A005123980 Skipping already saved record A005123877 Skipping already saved record A005123933 Skipping already saved record A005124461 Skipping already saved record A005124538 Skipping already saved record A005124646 Skipping already saved record A005124327 Skipping already saved record A005124380 Skipping already saved record A005124437 Skipping already saved record A005124459 Skipping already saved record A005125104 Skipping already saved record A005125043 Skipping already saved record A005125196 Skipping already saved record A005124997 Skipping already saved record A005124943 Skipping already saved record A005125577 Skipping already saved record A005125521 Skipping already saved record A005125557 Skipping already saved record A005125590 Skipping already saved record A005125647 Skipping already saved record A005127743 Skipping already saved record A005127891 Skipping already saved record A005128564 Skipping already saved record A005128554 Skipping already saved record A005128245 Skipping already saved record A005128202 Skipping already saved record A005128900 Skipping already saved record A005129013 Skipping already saved record A005129055 Skipping already saved record A005128813 Skipping already saved record A005128761 Skipping already saved record A005129016 Skipping already saved record A005129004 Skipping already saved record A005129487 Skipping already saved record A005129595 Skipping already saved record A005129655 Skipping already saved record A005129769 Skipping already saved record A005129441 Skipping already saved record A005129934 Skipping already saved record A005129986

Data

Downloaded 8118 times by slav123 openaustralia LoveMyData TheZepto jesse-a-reilly spatialsteve stevebrencorp yourneighbourhood srsmonty ljnicol dsccramb peteclowes primin2 MikeRalphson ShawInnes newmana thinkpropertyjason bigsmaug jamezpolley rimonhm darylrae mlandauer Jedwards157 n1k0p0l

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (3.26 MB) Use the API

rows 10 / 6296

info_url council_reference date_received description address date_scraped comment_url lot property_description
A004868622
2018-03-01
Signs and Line Marking for Minor Roads
106 CONSIDINE ST ELLEN GROVE QLD 4078
2018-03-09
A004868759
2018-03-01
Multiple Dwelling, Extension, Multiple Dwelling, Demolition
14 FIFTH AVE SANDGATE QLD 4017
2018-03-09
A004871923
2018-03-01
Standard Format Plan
79 BROOKBENT RD PALLARA QLD 4110
2018-03-09
A004869372
2018-03-01
Standard Format Plan
7 HOOLAN ST STAFFORD QLD 4053
2018-03-09
A004872047
2018-03-01
Standard Format Plan
261 GOODERHAM RD WILLAWONG QLD 4110
2018-03-09
A004868803
2018-03-01
Outdoor Sport and Recreation, Extension
338 GOODERHAM RD N WILLAWONG QLD 4110
2018-03-09
A004868586
2018-03-01
Food and Drink Outlet, Office, Food and Drink Outlet, Office
22 ROSS ST NEWSTEAD QLD 4006
2018-03-09
A004869102
2018-03-02
Dwelling House, Extension, Dwelling House, Extension, Dwelling House, Demolition
66 PRIMROSE ST GRANGE QLD 4051
2018-03-09
A004869404
2018-03-02
Educational Establishment, Extension
109 NORTH RD WYNNUM WEST QLD 4178
2018-03-09
A004869449
2018-03-02
Filling and/or Excavation
59 DENHAM ST BRACKEN RIDGE QLD 4017
2018-03-09

Statistics

Average successful run time: 4 minutes

Total run time: 14 days

Total cpu time used: about 2 hours

Total disk space used: 3.3 MB

History

  • Auto ran revision 3b80c5bd and completed successfully .
    nothing changed in the database
  • Auto ran revision 3b80c5bd and completed successfully .
    nothing changed in the database
    824 pages scraped
  • Auto ran revision 3b80c5bd and completed successfully .
    23 records added in the database
    876 pages scraped
  • Auto ran revision 3b80c5bd and completed successfully .
    27 records added in the database
    908 pages scraped
  • Auto ran revision 3b80c5bd and completed successfully .
    24 records added in the database
    916 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

brisbane / scraper.rb