This is a scraper that runs on Morph. To get started see the documentation

Contributors mlandauer equivalentideas henare ljnicol

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby/Rack -----> Using Ruby version: ruby-1.9.3 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/.........  Fetching version metadata from https://rubygems.org/...  Fetching dependency metadata from https://rubygems.org/..  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Rubygems 1.8.23.2 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation.  Fetching Ascii85 1.0.1  Installing Ascii85 1.0.1  Fetching aasm 3.0.8  Installing aasm 3.0.8  Fetching i18n 0.6.0  Installing i18n 0.6.0  Fetching multi_json 1.1.0  Installing multi_json 1.1.0  Fetching builder 3.0.0  Installing builder 3.0.0  Fetching arel 3.0.2  Installing arel 3.0.2  Fetching tzinfo 0.3.33  Installing tzinfo 0.3.33  Fetching addressable 2.2.7  Installing addressable 2.2.7  Fetching mime-types 1.17.2  Installing mime-types 1.17.2  Fetching awesome_print 1.0.2  Installing awesome_print 1.0.2  Fetching multi_xml 0.5.1  Installing multi_xml 0.5.1  Fetching json 1.6.5  Installing json 1.6.5 with native extensions  Fetching nokogiri 1.5.0  Installing nokogiri 1.5.0 with native extensions  Fetching uuidtools 2.1.3  Installing uuidtools 2.1.3  Using bundler 1.15.2  Fetching choice 0.1.4  Installing choice 0.1.4  Fetching chronic 0.6.7  Installing chronic 0.6.7  Fetching fastercsv 1.5.4  Installing fastercsv 1.5.4  Fetching oauth 0.4.5  Installing oauth 0.4.5  Fetching multipart-post 1.1.5  Installing multipart-post 1.1.5  Fetching ruby-ole 1.2.11.2  Installing ruby-ole 1.2.11.2  Fetching log4r 1.1.9  Installing log4r 1.1.9  Fetching posix-spawn 0.3.6  Installing posix-spawn 0.3.6 with native extensions  Fetching to_regexp 0.1.1  Installing to_regexp 0.1.1  Fetching upsert 0.3.4  Installing upsert 0.3.4  Fetching unf_ext 0.0.4  Installing unf_ext 0.0.4 with native extensions  Fetching fast-stemmer 1.0.1  Installing fast-stemmer 1.0.1 with native extensions  Fetching hashie 1.2.0  Installing hashie 1.2.0  Fetching gdata 1.1.2  Installing gdata 1.1.2  Fetching hpricot 0.8.4  Installing hpricot 0.8.4 with native extensions  Fetching htmlentities 4.3.1  Installing htmlentities 4.3.1  Fetching httpauth 0.1  Installing httpauth 0.1  Fetching httpclient 2.2.4  Installing httpclient 2.2.4  Fetching libxml-ruby 2.8.0  Installing libxml-ruby 2.8.0 with native extensions  Fetching net-http-digest_auth 1.1.1  Installing net-http-digest_auth 1.1.1  Fetching net-http-persistent 2.5.2  Installing net-http-persistent 2.5.2  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching ruby-rc4 0.1.5  Installing ruby-rc4 0.1.5  Fetching shoulda-context 1.0.0  Installing shoulda-context 1.0.0  Fetching shoulda-matchers 1.0.0  Installing shoulda-matchers 1.0.0  Fetching polylines 0.1.0  Installing polylines 0.1.0  Fetching rack 1.4.1  Installing rack 1.4.1  Fetching rubyzip 0.9.4  Installing rubyzip 0.9.4  Fetching sqlite3 1.3.9  Installing sqlite3 1.3.9 with native extensions  Fetching simple_oauth 0.2.0  Installing simple_oauth 0.2.0  Fetching stringex 1.4.0  Installing stringex 1.4.0  Fetching trollop 1.16.2  Installing trollop 1.16.2  Fetching stackdeck 0.2.0  Installing stackdeck 0.2.0  Fetching tmail 1.2.7.1  Installing tmail 1.2.7.1 with native extensions  Fetching activesupport 3.2.6  Installing activesupport 3.2.6  Fetching typhoeus 0.3.3  Installing typhoeus 0.3.3 with native extensions  Fetching httparty 0.8.3  Installing httparty 0.8.3  Fetching money 5.0.0  Installing money 5.0.0  Fetching jwt 0.1.4  Installing jwt 0.1.4  Fetching rfgraph 0.3  Installing rfgraph 0.3  Fetching webrobots 0.0.12  Installing webrobots 0.0.12  Fetching sanitize 2.0.3  Installing sanitize 2.0.3  Fetching faraday 0.8.1  Installing faraday 0.8.1  Fetching spreadsheet 0.6.5.7  Installing spreadsheet 0.6.5.7  Fetching todonotes 0.1.0  Installing todonotes 0.1.0  Fetching unix_utils 0.0.14  Installing unix_utils 0.0.14  Fetching unf 0.0.4  Installing unf 0.0.4  Fetching pdf-reader 1.0.0  Installing pdf-reader 1.0.0  Fetching shoulda 3.0.1  Installing shoulda 3.0.1  Fetching sqlite_magic 0.0.3  Installing sqlite_magic 0.0.3  Fetching activemodel 3.2.6  Installing activemodel 3.2.6  Fetching fixed_width-multibyte 0.2.3  Installing fixed_width-multibyte 0.2.3  Fetching hash_digest 1.0.0  Installing hash_digest 1.0.0  Fetching sinew 1.0.3  Installing sinew 1.0.3  Fetching monster_mash 0.3.0  Installing monster_mash 0.3.0  Fetching aws-sdk 1.6.1  Installing aws-sdk 1.6.1  Fetching fletcher 0.4.1  Installing fletcher 0.4.1  Fetching oauth2 0.5.2  Installing oauth2 0.5.2  Fetching twitter 4.5.0  Installing twitter 4.5.0  Fetching domain_name 0.5.2  Installing domain_name 0.5.2  Fetching pismo 0.7.2  Installing pismo 0.7.2  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching activerecord 3.2.6  Installing activerecord 3.2.6  Fetching activeresource 3.2.6  Installing activeresource 3.2.6  Fetching alchemy_api 0.1.2  Installing alchemy_api 0.1.2  Fetching google-spreadsheet-ruby 0.1.8  Installing google-spreadsheet-ruby 0.1.8  Fetching google_drive 0.3.1  Installing google_drive 0.3.1  Fetching mechanize 2.5.1  Installing mechanize 2.5.1  Fetching active_record_inline_schema 0.5.7  Installing active_record_inline_schema 0.5.7  Fetching highrise 3.0.3  Installing highrise 3.0.3  Fetching roo 1.10.0  Installing roo 1.10.0  Fetching remote_table 2.0.2  Installing remote_table 2.0.2  Fetching errata 1.1.1  Installing errata 1.1.1  Fetching data_miner 2.3.4  Installing data_miner 2.3.4  Bundle complete! 86 Gemfile dependencies, 89 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Post-install message from httparty:  When you HTTParty, you must party hard!  Post-install message from pdf-reader:  ********************************************  v1.0.0 of PDF::Reader introduced a new page-based API. There are extensive  examples showing how to use it in the README and examples directory.  For detailed documentation, check the rdocs for the PDF::Reader,  PDF::Reader::Page and PDF::Reader::ObjectHash classes.  The old API is marked as deprecated but will continue to work with no  visible warnings for now.  ********************************************  Bundle completed (192.51s)  Cleaning up the bundler cache. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Skipping already saved record A004937350 Skipping already saved record A004937253 Skipping already saved record A004937403 Skipping already saved record A004937508 Skipping already saved record A004937531 Skipping already saved record A004937396 Skipping already saved record A004940199 Skipping already saved record A004940397 Skipping already saved record A004940370 Skipping already saved record A004940133 Skipping already saved record A004940378 Skipping already saved record A004940279 Skipping already saved record A004940303 Skipping already saved record A004940743 Skipping already saved record A004940805 Skipping already saved record A004940867 Skipping already saved record A004940715 Skipping already saved record A004940490 Skipping already saved record A004940742 Skipping already saved record A004940806 Skipping already saved record A004942083 Skipping already saved record A004941447 Skipping already saved record A004941187 Skipping already saved record A004941295 Skipping already saved record A004941758 Skipping already saved record A004947034 Skipping already saved record A004941874 Skipping already saved record A004941871 Skipping already saved record A004941889 Skipping already saved record A004941909 Skipping already saved record A004942005 Skipping already saved record A004941897 Skipping already saved record A004942478 Skipping already saved record A004942196 Skipping already saved record A004942464 Skipping already saved record A004942639 Skipping already saved record A004942299 Skipping already saved record A004942449 Skipping already saved record A004942585 Skipping already saved record A004944918 Skipping already saved record A004944954 Skipping already saved record A004944992 Skipping already saved record A004945139 Skipping already saved record A004945823 Skipping already saved record A004945385 Skipping already saved record A004945604 Skipping already saved record A004945587 Skipping already saved record A004945543 Skipping already saved record A004945311 Skipping already saved record A004946227 Skipping already saved record A004946553 Skipping already saved record A004946610 Skipping already saved record A004946952 Skipping already saved record A004946647 Skipping already saved record A004946830 Skipping already saved record A004946995 Skipping already saved record A004947009 Skipping already saved record A004947179 Skipping already saved record A004947222 Skipping already saved record A004947412 Skipping already saved record A004941056 Skipping already saved record A004937228 Skipping already saved record A004937560 Skipping already saved record A004937253 Skipping already saved record A004940252 Skipping already saved record A004940370 Skipping already saved record A004940258 Skipping already saved record A004940940 Skipping already saved record A004940583 Skipping already saved record A004941502 Skipping already saved record A004941511 Skipping already saved record A004941376 Skipping already saved record A004941304 Skipping already saved record A004941417 Skipping already saved record A004941816 Skipping already saved record A004941771 Skipping already saved record A004941845 Skipping already saved record A004942007 Skipping already saved record A004941897 Skipping already saved record A004942395 Skipping already saved record A004942575 Skipping already saved record A004942160 Skipping already saved record A004942226 Skipping already saved record A004942432 Skipping already saved record A004942527 Skipping already saved record A004942566 Skipping already saved record A004942585 Skipping already saved record A004942185 Skipping already saved record A004945162 Skipping already saved record A004944869 Skipping already saved record A004944992 Skipping already saved record A004945127 scraper.rb:10:in `scrape_property_details': undefined method `text' for nil:NilClass (NoMethodError) from scraper.rb:36:in `block in scrape_page' from /app/vendor/bundle/ruby/1.9.1/gems/nokogiri-1.5.0/lib/nokogiri/xml/node_set.rb:239:in `block in each' from /app/vendor/bundle/ruby/1.9.1/gems/nokogiri-1.5.0/lib/nokogiri/xml/node_set.rb:238:in `upto' from /app/vendor/bundle/ruby/1.9.1/gems/nokogiri-1.5.0/lib/nokogiri/xml/node_set.rb:238:in `each' from scraper.rb:22:in `scrape_page' from scraper.rb:51:in `scrape_and_follow_next_link' from scraper.rb:67:in `scrape_and_follow_next_link' from scraper.rb:67:in `scrape_and_follow_next_link' from scraper.rb:67:in `scrape_and_follow_next_link' from scraper.rb:67:in `scrape_and_follow_next_link' from scraper.rb:67:in `scrape_and_follow_next_link' from scraper.rb:67:in `scrape_and_follow_next_link' from scraper.rb:67:in `scrape_and_follow_next_link' from scraper.rb:67:in `scrape_and_follow_next_link' from scraper.rb:67:in `scrape_and_follow_next_link' from scraper.rb:86:in `<main>'

Data

Downloaded 6927 times by slav123 openaustralia LoveMyData TheZepto jesse-a-reilly spatialsteve stevebrencorp yourneighbourhood srsmonty ljnicol dsccramb peteclowes primin2 MikeRalphson ShawInnes newmana thinkpropertyjason bigsmaug n1k0p0l

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (925 KB) Use the API

rows 10 / 1748

info_url council_reference date_received description address date_scraped comment_url lot property_description
A004868622
2018-03-01
Signs and Line Marking for Minor Roads
106 CONSIDINE ST ELLEN GROVE QLD 4078
2018-03-09
A004868759
2018-03-01
Multiple Dwelling, Extension, Multiple Dwelling, Demolition
14 FIFTH AVE SANDGATE QLD 4017
2018-03-09
A004871923
2018-03-01
Standard Format Plan
79 BROOKBENT RD PALLARA QLD 4110
2018-03-09
A004869372
2018-03-01
Standard Format Plan
7 HOOLAN ST STAFFORD QLD 4053
2018-03-09
A004872047
2018-03-01
Standard Format Plan
261 GOODERHAM RD WILLAWONG QLD 4110
2018-03-09
A004868803
2018-03-01
Outdoor Sport and Recreation, Extension
338 GOODERHAM RD N WILLAWONG QLD 4110
2018-03-09
A004868586
2018-03-01
Food and Drink Outlet, Office, Food and Drink Outlet, Office
22 ROSS ST NEWSTEAD QLD 4006
2018-03-09
A004869102
2018-03-02
Dwelling House, Extension, Dwelling House, Extension, Dwelling House, Demolition
66 PRIMROSE ST GRANGE QLD 4051
2018-03-09
A004869404
2018-03-02
Educational Establishment, Extension
109 NORTH RD WYNNUM WEST QLD 4178
2018-03-09
A004869449
2018-03-02
Filling and/or Excavation
59 DENHAM ST BRACKEN RIDGE QLD 4017
2018-03-09

Statistics

Average successful run time: 3 minutes

Total run time: 13 days

Total cpu time used: about 1 hour

Total disk space used: 961 KB

History

  • Auto ran revision a84c0485 and failed .
    1 record added in the database
    200 pages scraped
  • Auto ran revision a84c0485 and failed .
    1 record added in the database
    198 pages scraped
  • Auto ran revision a84c0485 and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision a84c0485 and failed .
    17 records added in the database
    196 pages scraped
  • Auto ran revision a84c0485 and failed .
    8 records added in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

brisbane / scraper.rb