planningalerts-scrapers / port_stephens-1

Port Stephens Development Applications

Scrapes datracker.portstephens.nsw.gov.au, www.google-analytics.com, maps.googleapis.com, and 1 other domain

The official Port Stephens Council Development Application Tracker (DA Tracker)


This is a scraper that runs on Morph. To get started see the documentation

Contributors otherchirps ishmagel mlandauer henare

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Python app detected  ! The latest version of Python 2 is python-2.7.14 (you are using python-2.7.6, which is unsupported).  ! We recommend upgrading by specifying the latest version (python-2.7.14).  Learn More: https://devcenter.heroku.com/articles/python-runtimes -----> Installing python-2.7.6 -----> Installing pip -----> Installing requirements with pip  Collecting beautifulsoup4==4.3.2 (from -r /tmp/build/requirements.txt (line 1))  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.  SNIMissingWarning  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.  InsecurePlatformWarning  Downloading beautifulsoup4-4.3.2.tar.gz (143kB)  Collecting python-dateutil==2.4.2 (from -r /tmp/build/requirements.txt (line 2))  Downloading python_dateutil-2.4.2-py2.py3-none-any.whl (188kB)  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 4))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to morph_defaults) to /app/.heroku/src/scraperwiki  Collecting selenium==2.46.0 (from -r /tmp/build/requirements.txt (line 5))  Downloading selenium-2.46.0.tar.gz (2.6MB)  Collecting six>=1.5 (from python-dateutil==2.4.2->-r /tmp/build/requirements.txt (line 2))  Downloading six-1.11.0-py2.py3-none-any.whl  Collecting dumptruck>=0.1.2 (from scraperwiki->-r /tmp/build/requirements.txt (line 4))  Downloading dumptruck-0.1.6.tar.gz  Collecting requests (from scraperwiki->-r /tmp/build/requirements.txt (line 4))  Downloading requests-2.18.4-py2.py3-none-any.whl (88kB)  Collecting chardet<3.1.0,>=3.0.2 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 4))  Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB)  Collecting certifi>=2017.4.17 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 4))  Downloading certifi-2017.11.5-py2.py3-none-any.whl (330kB)  Collecting urllib3<1.23,>=1.21.1 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 4))  Downloading urllib3-1.22-py2.py3-none-any.whl (132kB)  Collecting idna<2.7,>=2.5 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 4))  Downloading idna-2.6-py2.py3-none-any.whl (56kB)  Installing collected packages: beautifulsoup4, six, python-dateutil, dumptruck, chardet, certifi, urllib3, idna, requests, scraperwiki, selenium  Running setup.py install for beautifulsoup4: started  Running setup.py install for beautifulsoup4: finished with status 'done'  Running setup.py install for dumptruck: started  Running setup.py install for dumptruck: finished with status 'done'  Running setup.py develop for scraperwiki  Running setup.py install for selenium: started  Running setup.py install for selenium: finished with status 'done'  Successfully installed beautifulsoup4-4.3.2 certifi-2017.11.5 chardet-3.0.4 dumptruck-0.1.6 idna-2.6 python-dateutil-2.4.2 requests-2.18.4 scraperwiki selenium-2.46.0 six-1.11.0 urllib3-1.22   ! Hello! It looks like your application is using an outdated version of Python.  ! This caused the security warning you saw above during the 'pip install' step.  ! We recommend 'python-3.6.2', which you can specify in a 'runtime.txt' file.  ! -- Much Love, Heroku.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Saving: 016-2018-00000029-001 Skipping: 016-2017-00000466-002 Skipping: 016-2018-00000026-001 Skipping: 016-2018-00000027-001 Skipping: 016-2018-00000028-001 Skipping: 016-2018-00006000-001 Skipping: 016-2018-00000025-001 Skipping: 016-2018-00000023-001 Skipping: 016-2018-00000022-001 Skipping: 016-2018-00000024-001 Skipping: 009-2018-00000002-001 Skipping: 016-2018-00000020-001 Skipping: 016-2018-00000021-001 Skipping: 016-2018-00000019-001 Skipping: 016-2017-00000477-002 Skipping: 016-2018-00000018-001 Skipping: 016-2018-00000016-001 Skipping: 016-2018-00000017-001 Skipping: 016-2018-00000015-001 Skipping: 016-2018-00000014-001 Skipping: 016-2018-00000011-001 Skipping: 016-2017-00000592-002 Skipping: 016-2018-00000013-001 Skipping: 016-2018-00000010-001 Skipping: 016-2018-00000012-001 Skipping: 016-2018-00000009-001 Skipping: 016-2018-00000007-001 Skipping: 016-2013-00000439-002 Skipping: 016-2018-00000006-001 Skipping: 016-2018-00000008-001 Skipping: 016-2017-00000470-002 Skipping: 016-2018-00000005-001 Skipping: 016-2018-00000004-001 Skipping: 016-2018-00000003-001 Skipping: 016-2018-00000002-001 Skipping: 016-2018-00000001-001

Data

Downloaded 1499 times by openaustralia LoveMyData TheZepto gdmhunter MikeRalphson Ray-White-Project-Marketing

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (1.38 MB) Use the API

rows 10 / 3319

council_reference address description info_url comment_url date_scraped date_received
016-2015-00000370-001
9 Scott St, Wallalong 2320 NSW
Dwelling House
2015-06-21
2015-06-19
016-2015-00000374-001
3 Semillon Cl, Hinton 2321 NSW
Storage Shed
2015-06-21
2015-06-19
016-2015-00000373-001
19 Tomaree Rd, Shoal Bay 2315 NSW
Single Storey Dwelling & Secondary Dwelling ( Granny Flat)
2015-06-21
2015-06-19
016-2015-00000372-001
12 Heather St, Heatherbrae 2324 NSW
Commercial Development - Showroom & Caretakers Cottage & Shed
2015-06-21
2015-06-19
016-2015-00000371-001
48 Argyle Ave, Anna Bay 2316 NSW
Storage & Tool Shed
2015-06-21
2015-06-19
016-2015-00000369-001
7 Cecilia Cl, Salt Ash 2318 NSW
Demolish Dwelling & Garage
2015-06-21
2015-06-19
016-2015-00000367-001
1057 Lemon Tree Passage Rd, Tanilba Bay 2319 NSW
Commercial Premises - Demolish Existing
2015-06-21
2015-06-19
016-2015-00000368-001
47 Osprey Cct, Medowie 2318 NSW
Dwelling House
2015-06-21
2015-06-19
016-2004-00000709-002
137 Sandy Point Rd, Corlette 2315 NSW
Section 96(1) to Approved Three (3) Lot Torrens Title Subdivision - Amend Condition 3
2015-06-21
2015-06-19
016-2009-00000709-002
8 Hank St, Heatherbrae 2324 NSW
S.96 Amendment - amendement to condition
2015-06-21
2015-06-19

Statistics

Average successful run time: 2 minutes

Total run time: 5 days

Total cpu time used: 33 minutes

Total disk space used: 1.41 MB

History

  • Auto ran revision bc1a0db4 and completed successfully .
    1 record added in the database
    53 pages scraped
  • Auto ran revision bc1a0db4 and completed successfully .
    nothing changed in the database
  • Auto ran revision bc1a0db4 and completed successfully .
    12 records added in the database
    53 pages scraped
  • Auto ran revision bc1a0db4 and failed .
    nothing changed in the database
  • Auto ran revision bc1a0db4 and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

port_stephens-1 / scraper.py