stanle / cityperthscraper

scraper for the perth city building applications

Scrapes www.perth.wa.gov.au and www.perth.wa.gov.au:443

City of Perth | Welcome to the City of Perth, the local government authority for the capital of Western Australia.


Contributors stanle

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Python app detected -----> Installing python-3.6.2 -----> Installing pip -----> Installing requirements with pip  Collecting pandas (from -r /tmp/build/requirements.txt (line 1))  Downloading https://files.pythonhosted.org/packages/e1/d8/feeb346d41f181e83fba45224ab14a8d8af019b48af742e047f3845d8cff/pandas-0.23.4-cp36-cp36m-manylinux1_x86_64.whl (8.9MB)  Collecting tabula-py (from -r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/1f/f6/20e43951738260cff0005ea175b83ada43e0a74a12e729d0bcabc03b129a/tabula_py-1.3.1-py2.py3-none-any.whl (20.4MB)  Collecting SQLAlchemy (from -r /tmp/build/requirements.txt (line 3))  Downloading https://files.pythonhosted.org/packages/0c/7d/769c5fc22c0cdefd097b91cc525b6d8c88bf2afd8b0315b1e7ca088956b4/SQLAlchemy-1.2.15.tar.gz (5.6MB)  Collecting bs4 (from -r /tmp/build/requirements.txt (line 4))  Downloading https://files.pythonhosted.org/packages/10/ed/7e8b97591f6f456174139ec089c769f89a94a1a4025fe967691de971f314/bs4-0.0.1.tar.gz  Collecting requests (from -r /tmp/build/requirements.txt (line 5))  Downloading https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl (57kB)  Collecting numpy>=1.9.0 (from pandas->-r /tmp/build/requirements.txt (line 1))  Downloading https://files.pythonhosted.org/packages/ff/7f/9d804d2348471c67a7d8b5f84f9bc59fd1cefa148986f2b74552f8573555/numpy-1.15.4-cp36-cp36m-manylinux1_x86_64.whl (13.9MB)  Collecting pytz>=2011k (from pandas->-r /tmp/build/requirements.txt (line 1))  Downloading https://files.pythonhosted.org/packages/61/28/1d3920e4d1d50b19bc5d24398a7cd85cc7b9a75a490570d5a30c57622d34/pytz-2018.9-py2.py3-none-any.whl (510kB)  Collecting python-dateutil>=2.5.0 (from pandas->-r /tmp/build/requirements.txt (line 1))  Downloading https://files.pythonhosted.org/packages/74/68/d87d9b36af36f44254a8d512cbfc48369103a3b9e474be9bdfe536abfc45/python_dateutil-2.7.5-py2.py3-none-any.whl (225kB)  Collecting distro (from tabula-py->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/f6/b1/ba5a96bccd3496241d8908164b9502a129156443cdd5acbdbf04a90b7a09/distro-1.3.0-py2.py3-none-any.whl  Collecting urllib3 (from tabula-py->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)  Collecting beautifulsoup4 (from bs4->-r /tmp/build/requirements.txt (line 4))  Downloading https://files.pythonhosted.org/packages/1d/5d/3260694a59df0ec52f8b4883f5d23b130bc237602a1411fa670eae12351e/beautifulsoup4-4.7.1-py3-none-any.whl (94kB)  Collecting certifi>=2017.4.17 (from requests->-r /tmp/build/requirements.txt (line 5))  Downloading https://files.pythonhosted.org/packages/9f/e0/accfc1b56b57e9750eba272e24c4dddeac86852c2bebd1236674d7887e8a/certifi-2018.11.29-py2.py3-none-any.whl (154kB)  Collecting chardet<3.1.0,>=3.0.2 (from requests->-r /tmp/build/requirements.txt (line 5))  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)  Collecting idna<2.9,>=2.5 (from requests->-r /tmp/build/requirements.txt (line 5))  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)  Collecting six>=1.5 (from python-dateutil>=2.5.0->pandas->-r /tmp/build/requirements.txt (line 1))  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl  Collecting soupsieve>=1.2 (from beautifulsoup4->bs4->-r /tmp/build/requirements.txt (line 4))  Downloading https://files.pythonhosted.org/packages/ef/06/53edcae4edea76b38a325980dd35aed3b39f9bd0ef27b9d33f2e6dc4c7f6/soupsieve-1.6.2-py2.py3-none-any.whl  Installing collected packages: numpy, pytz, six, python-dateutil, pandas, distro, urllib3, tabula-py, SQLAlchemy, soupsieve, beautifulsoup4, bs4, certifi, chardet, idna, requests  Running setup.py install for SQLAlchemy: started  Running setup.py install for SQLAlchemy: finished with status 'done'  Running setup.py install for bs4: started  Running setup.py install for bs4: finished with status 'done'  Successfully installed SQLAlchemy-1.2.15 beautifulsoup4-4.7.1 bs4-0.0.1 certifi-2018.11.29 chardet-3.0.4 distro-1.3.0 idna-2.8 numpy-1.15.4 pandas-0.23.4 python-dateutil-2.7.5 pytz-2018.9 requests-2.21.0 six-1.12.0 soupsieve-1.6.2 tabula-py-1.3.1 urllib3-1.24.1   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Building Permits Issued 02-01-19 to 08-01-19 App Year/NumberApplication DescriptionPrimary Property AddressEst Value Date ApprovedDecisionDecision Date\r220 St Georges Terrace\r2018/787Internal bank fitout to existing building - BankwestPERTH WA 6000$976,963 4/01/19Approved - Building Permit Certified 3/01/19\rRoyal hotel refurbishment and modification to"RAINE SQUARE"\rservices design for Wentworth and Murray Street300 Murray Street\r2018/830Retail buildingPERTH WA 6000$900,500 8/01/19Approved - Building Permit Certified 7/01/19\r"EASTPOINT PLAZA"\r233 Adelaide Terrace\r2018/894Office FitoutPERTH WA 6000$223,000 3/01/19Approved - Building Permit Certified 2/01/19\r"ST GEORGE'S COLLEGE"\rForward works - earthworks, in-ground services,44 Mounts Bay Road\r2018/907retaining walls, footings, ground floor slabCRAWLEY WA 6009$230,000 7/01/19Approved - Building Permit Certified 7/01/19\rEnabling works for future data centre comprising of105 Lord Street\r2018/910sheet piling and bulk excavationPERTH WA 6000$1,600,000 8/01/19Approved - Building Permit Certified 7/01/19\r"THE WILLS BUILDING"\rUnit 1-37/82 King Street\r2018/931Plant Enclosure ScreeningPERTH WA 6000$10,000 4/01/19Approved - Building Permit Certified 4/01/19\r"TAXATION DEPT"\r45 Francis Street\r2018/932Tenancy Fitout - Unit 2, National ArchivesNORTHBRIDGE WA 6003$800,000 8/01/19Approved - Building Permit Certified 7/01/19\rInternal Fitout to Existing Tenancy - 3/4 Topped Up30 Fielder Street\r2018/935CoffeeEAST PERTH WA 6004$60,000 8/01/19Approved - Building Permit Certified 8/01/19\rUnit 2/297-301 William Street\r2018/937Fitout to existing shop - #TGIFNORTHBRIDGE WA 6003$95,000 8/01/19Approved - Building Permit Certified 8/01/19\rProposed external alterations to existing building (Re-\rpaint building facade, replace balcony balustrades,\rreplace window awnings, installing timberUnit 1-13/192 Hampden Road\r2018/940architectural screen to entrance)NEDLANDS WA 6009$290,178 8/01/19Approved - Building Permit Certified 7/01/19\r"GOVERNOR STIRLING TOWER"\rInternal Office Refurbishment to Level 7 - Knightcorp197 St Georges Terrace\r2019/2OfficesPERTH WA 6000$180,000 3/01/19Approved - Building Permit Certified 3/01/19\r11\rSum:$5,365,641 ... Decision Date Traceback (most recent call last): File "scraper.py", line 42, in <module> resultTable['description'] = df['Application Description'] + ", Value: " + df['Est Value'] + ", Descision: " + df.Descision File "/app/.heroku/python/lib/python3.6/site-packages/pandas/core/generic.py", line 4376, in __getattr__ return object.__getattribute__(self, name) AttributeError: 'DataFrame' object has no attribute 'Descision' 0 2018/787 ... NaN 1 2018/830 ... NaN 2 2018/894 ... NaN 3 2018/907 ... NaN 4 2018/910 ... NaN [5 rows x 8 columns] ['App Year/NumberApplication DescriptionPrimary Property AddressEst Value Date ApprovedDecisionDecision Date\r220 St Georges Terrace\r2018/787Internal bank fitout to existing building - BankwestPERTH WA 6000$976,963 4/01/19Approved - Building Permit Certified 3/01/19\rRoyal hotel refurbishment and modification to"RAINE SQUARE"\rservices design for Wentworth and Murray Street300 Murray Street\r2018/830Retail buildingPERTH WA 6000$900,500 8/01/19Approved - Building Permit Certified 7/01/19\r"EASTPOINT PLAZA"\r233 Adelaide Terrace\r2018/894Office FitoutPERTH WA 6000$223,000 3/01/19Approved - Building Permit Certified 2/01/19\r"ST GEORGE\'S COLLEGE"\rForward works - earthworks, in-ground services,44 Mounts Bay Road\r2018/907retaining walls, footings, ground floor slabCRAWLEY WA 6009$230,000 7/01/19Approved - Building Permit Certified 7/01/19\rEnabling works for future data centre comprising of105 Lord Street\r2018/910sheet piling and bulk excavationPERTH WA 6000$1,600,000 8/01/19Approved - Building Permit Certified 7/01/19\r"THE WILLS BUILDING"\rUnit 1-37/82 King Street\r2018/931Plant Enclosure ScreeningPERTH WA 6000$10,000 4/01/19Approved - Building Permit Certified 4/01/19\r"TAXATION DEPT"\r45 Francis Street\r2018/932Tenancy Fitout - Unit 2, National ArchivesNORTHBRIDGE WA 6003$800,000 8/01/19Approved - Building Permit Certified 7/01/19\rInternal Fitout to Existing Tenancy - 3/4 Topped Up30 Fielder Street\r2018/935CoffeeEAST PERTH WA 6004$60,000 8/01/19Approved - Building Permit Certified 8/01/19\rUnit 2/297-301 William Street\r2018/937Fitout to existing shop - #TGIFNORTHBRIDGE WA 6003$95,000 8/01/19Approved - Building Permit Certified 8/01/19\rProposed external alterations to existing building (Re-\rpaint building facade, replace balcony balustrades,\rreplace window awnings, installing timberUnit 1-13/192 Hampden Road\r2018/940architectural screen to entrance)NEDLANDS WA 6009$290,178 8/01/19Approved - Building Permit Certified 7/01/19\r"GOVERNOR STIRLING TOWER"\rInternal Office Refurbishment to Level 7 - Knightcorp197 St Georges Terrace\r2019/2OfficesPERTH WA 6000$180,000 3/01/19Approved - Building Permit Certified 3/01/19\r11\rSum:$5,365,641' 'App Year/Number' 'Application Description' 'Primary Property Address' 'Est Value' 'Date Approved' 'Decision' 'Decision Date']

Data

Downloaded 0 times

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (3 KB) Use the API

rows 0 / 0

Statistics

Total run time: 8 minutes

Total cpu time used: half a minute

Total disk space used: 43.9 KB

History

  • Manually ran revision 3dbcbff8 and failed .
    nothing changed in the database
    2 pages scraped
  • Manually ran revision 9ec6939a and failed .
    nothing changed in the database
    99 pages scraped
  • Manually ran revision d4f743ee and failed .
    nothing changed in the database
  • Manually ran revision e62efd2c and failed .
  • Manually ran revision cfa7a5a2 and failed .
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

cityperthscraper / scraper.py