planningalerts-scrapers / gold_coast

Gold Coast City Council Development Applications

Scrapes pdonline.goldcoast.qld.gov.au and datracker.begavalley.nsw.gov.au

PD Online is part of a program of work funded by the federal government through AusIndustry and endorsed by the Council of Mayors (SEQ).


City of Gold Coast Council Scraper

This Council involves the followings

  • Server - Masterview, ApplicationMaster
  • Cookie - Yes
  • Scrape detail page - No
  • Clearly defined data within a row - Yes
  • JavaScript - Yes - doPostBack

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default)
  • thismonth
  • lastmonth

Enjoy

Forked from ScraperWiki

Contributors LoveMyData henare

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... Getting 'thisweek' data, changable via MORPH_PERIOD environment scraper.rb:18:in `clean_whitespace': undefined method `gsub' for nil:NilClass (NoMethodError) from scraper.rb:38:in `block in scrape_table' from /app/vendor/bundle/ruby/2.4.0/gems/nokogiri-1.8.1/lib/nokogiri/xml/node_set.rb:190:in `block in each' from /app/vendor/bundle/ruby/2.4.0/gems/nokogiri-1.8.1/lib/nokogiri/xml/node_set.rb:189:in `upto' from /app/vendor/bundle/ruby/2.4.0/gems/nokogiri-1.8.1/lib/nokogiri/xml/node_set.rb:189:in `each' from scraper.rb:31:in `scrape_table' from scraper.rb:61:in `<main>'

Data

Downloaded 3386 times by slav123 openaustralia LoveMyData TheZepto equivalentideas primin2 dsccramb thinkpropertyjason thomas-deneux

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (13.9 MB) Use the API

rows 10 / 33185

date_scraped description info_url date_received council_reference address comment_url
2012-04-25
Description: Class: CODE Work Type:
2012-04-12
MCU201200186
7 KATTA AVENUE CURRUMBIN 4223
mailto:gcccmail@goldcoast.qld.gov.au?subject=Development+Application+Enquiry%3A+MCU201200186
2012-04-25
Description: ELEVATE EXISTING DWELLING AND RENOVATE Class: HOUSE Work Type: ALTERATIONS
2012-04-11
BLD201202570
27 RUTLEDGE STREET COOLANGATTA 4225
mailto:gcccmail@goldcoast.qld.gov.au?subject=Development+Application+Enquiry%3A+BLD201202570
2012-04-25
Description: BATHROOM & LAUNDRY RENOVATION Class: HOUSE Work Type: ALTERATIONS
2012-04-11
BLD201202551
4 STATION STREET TUGUN 4224
mailto:gcccmail@goldcoast.qld.gov.au?subject=Development+Application+Enquiry%3A+BLD201202551
2012-04-25
Description: DUPLEX Class: DUPLEX Work Type: CONSTRUCT
2012-04-19
BLD201202815
26 HOOPER DRIVE CURRUMBIN 4223
mailto:gcccmail@goldcoast.qld.gov.au?subject=Development+Application+Enquiry%3A+BLD201202815
2012-04-25
Description: CARPORT + DECK WITHIN SETBACKS Class: REFERRAL AGENCY ASSESSMENT Work Type: DESIGN AND SITING
2012-04-17
BLD201202737
37 MITCHELL AVENUE CURRUMBIN 4223
mailto:gcccmail@goldcoast.qld.gov.au?subject=Development+Application+Enquiry%3A+BLD201202737
2012-04-25
Description: Class: CODE Work Type: PERMISSIBLE CHANGE
2012-04-24
MCU201200212
27 MILLERS DRIVE CURRUMBIN 4223
mailto:gcccmail@goldcoast.qld.gov.au?subject=Development+Application+Enquiry%3A+MCU201200212
2012-04-25
Description: NEW DWELLING Class: HOUSE Work Type: CONSTRUCT
2012-04-04
BLD201202446
71 DIXON STREET COOLANGATTA 4225
mailto:gcccmail@goldcoast.qld.gov.au?subject=Development+Application+Enquiry%3A+BLD201202446
2012-04-25
Description: REPLACE AWNINGS AND UMBRELLA STRUCTURE Class: RETAIL/DINING/SERVICES Work Type: ALTERATIONS
2012-04-13
BLD201202640
48 MUSGRAVE STREET COOLANGATTA 4225
mailto:gcccmail@goldcoast.qld.gov.au?subject=Development+Application+Enquiry%3A+BLD201202640
2012-04-25
Description: COURTHOUSE - ENCLOSED CARPARKING & STOREROOM Class: OFFICE FITOUT Work Type: ALTERATIONS
2012-04-23
BLD201202912
MUSGRAVE STREET COOLANGATTA 4225
mailto:gcccmail@goldcoast.qld.gov.au?subject=Development+Application+Enquiry%3A+BLD201202912
2012-04-25
Description: HOUSE DEMOLITION Class: DEMOLITION Work Type: DEMOLISH
2012-04-04
BLD201202435
161 GOLDEN FOUR DRIVE BILINGA 4225
mailto:gcccmail@goldcoast.qld.gov.au?subject=Development+Application+Enquiry%3A+BLD201202435

Statistics

Average successful run time: less than a minute

Total run time: 16 days

Total cpu time used: about 1 hour

Total disk space used: 13.9 MB

History

  • Auto ran revision 159650d6 and failed .
    nothing changed in the database
    9 pages scraped
  • Auto ran revision 159650d6 and completed successfully .
    nothing changed in the database
    29 pages scraped
  • Auto ran revision 159650d6 and completed successfully .
    122 records added in the database
    29 pages scraped
  • Auto ran revision 159650d6 and failed .
    nothing changed in the database
  • Auto ran revision 159650d6 and completed successfully .
    51 records added in the database
    13 pages scraped
  • ...
  • Forked from ScraperWiki

Show complete history

Scraper code

Ruby

gold_coast / scraper.rb