jacksongs / lobbybot

This chirpy fellow checks the Australian lobbyist register every day.


Contributors jacksongs

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Python app detected -----> Installing python-2.7.14 -----> Installing pip -----> Installing requirements with pip  DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support  Collecting requests  Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)  Collecting tweepy  Downloading tweepy-3.9.0-py2.py3-none-any.whl (30 kB)  Collecting BeautifulSoup4  Downloading beautifulsoup4-4.9.1-py2-none-any.whl (111 kB)  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 4))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to revision morph_defaults) to /app/.heroku/src/scraperwiki  Collecting lxml  Downloading lxml-4.5.2-cp27-cp27mu-manylinux1_x86_64.whl (5.5 MB)  Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1  Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)  Collecting certifi>=2017.4.17  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)  Collecting chardet<4,>=3.0.2  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)  Collecting idna<3,>=2.5  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)  Collecting requests-oauthlib>=0.7.0  Downloading requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)  Collecting six>=1.10.0  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)  Collecting soupsieve<2.0  Downloading soupsieve-1.9.6-py2.py3-none-any.whl (33 kB)  Collecting dumptruck>=0.1.2  Downloading dumptruck-0.1.6.tar.gz (15 kB)  Collecting oauthlib>=3.0.0  Downloading oauthlib-3.1.0-py2.py3-none-any.whl (147 kB)  Collecting backports.functools-lru-cache; python_version < "3"  Downloading backports.functools_lru_cache-1.6.1-py2.py3-none-any.whl (5.7 kB)  Building wheels for collected packages: dumptruck  Building wheel for dumptruck (setup.py): started  Building wheel for dumptruck (setup.py): finished with status 'done'  Created wheel for dumptruck: filename=dumptruck-0.1.6-py2-none-any.whl size=11842 sha256=5d33d31dad0839dcbe7a2da5f85f78e28c248de570828376ad83da84dbf1c6ff  Stored in directory: /tmp/pip-ephem-wheel-cache-S73pJF/wheels/dc/75/e9/1e61c4080c73e7bda99614549591f83b53bcc2d682f26fce62  Successfully built dumptruck  Installing collected packages: urllib3, certifi, chardet, idna, requests, oauthlib, requests-oauthlib, six, tweepy, backports.functools-lru-cache, soupsieve, BeautifulSoup4, dumptruck, scraperwiki, lxml  Running setup.py develop for scraperwiki  Successfully installed BeautifulSoup4-4.9.1 backports.functools-lru-cache-1.6.1 certifi-2020.6.20 chardet-3.0.4 dumptruck-0.1.6 idna-2.10 lxml-4.5.2 oauthlib-3.1.0 requests-2.24.0 requests-oauthlib-1.3.0 scraperwiki six-1.15.0 soupsieve-1.9.6 tweepy-3.9.0 urllib3-1.25.10 DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support    -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Yesterday the current record count for agencies was: 258 Yesterday the current record count for lobbyists was: 560 Yesterday the current record count for client was: 1759 Traceback (most recent call last): File "scraper.py", line 46, in <module> response = requests.get("http://lobbyists.pmc.gov.au/export/export_%s.cfm"%kind) File "/app/.heroku/python/lib/python2.7/site-packages/requests/api.py", line 76, in get return request('get', url, params=params, **kwargs) File "/app/.heroku/python/lib/python2.7/site-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/app/.heroku/python/lib/python2.7/site-packages/requests/sessions.py", line 530, in request resp = self.send(prep, **send_kwargs) File "/app/.heroku/python/lib/python2.7/site-packages/requests/sessions.py", line 643, in send r = adapter.send(request, **kwargs) File "/app/.heroku/python/lib/python2.7/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='lobbyists.pmc.gov.au', port=80): Max retries exceeded with url: /export/export_agencies.cfm (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3f7ba0ab10>: Failed to establish a new connection: [Errno -2] Name or service not known',))

Data

Downloaded 11 times by jacksongs MikeRalphson

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (2.17 MB) Use the API

rows 10 / 1430

Former Government Representative End Lobbyist Name Lobbyist Position Start Cessation Date Agency Name
Yes
Adam Kilgour
Managing Director
20-Jan-2016
July 1993
Diplomacy Pty Limited
No
Adrienne Day-Hodge
Director
20-Jan-2016
Day & Hodge Associates Pty Ltd
Yes
Alan Evers-Buckland
Owner / Former Adviser
20-Jan-2016
October 2007
A D Evers-Buckland
Yes
Alexander Cramb
Partner
20-Jan-2016
January 2008
GRACosway Pty Ltd
Yes
Alexandra Brook Kennedy
Director
20-Jan-2016
April 1992
The Write Connection Pty Ltd
No
Andres Puig
Director
20-Jan-2016
The Civic Group Holdings Pty Ltd and Highchair Pty Ltd
No
Andrew Butler
Managing Director
20-Jan-2016
Etched Communications Pty Ltd
No
Andrew Graham Farrant
Principal
20-Jan-2016
One Degree Advisory
No
Andrew James King
Director
20-Jan-2016
BroadSpectrum Consultants Pty Ltd
No
Andrew Plumley
Director
20-Jan-2016
Willard Public Affairs Pty Limited

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (2.17 MB) Use the API

rows 10 / 4224

End Trading Name Start Updated ABN Agency Name
20-Jan-2016
Knowminister
20-Jan-2016
28-Sep-2015
26140833772
KnowMinister.com Pty Ltd
21-Jan-2016
LESL Services Pty Ltd
20-Jan-2016
21-Jun-2015
19096984440
LESL Services Pty Ltd
21-Jan-2016
MacGregor Public Relations
20-Jan-2016
30-Sep-2015
18104844626
MacGregor Public Relations
21-Jan-2016
Parker & Partners Public Affairs
20-Jan-2016
08-Oct-2015
22083475683
Parker & Partners Pty Ltd
21-Jan-2016
Strategic Partnership Group (Aust) Pty Limited
20-Jan-2016
16-Jun-2015
92 089 461 450
Strategic Partnership Group (Aust) Pty Limited
21-Jan-2016
FTI Consulting
20-Jan-2016
24-Nov-2015
69102531457
FTI Consulting (Sydney) Pty Ltd as Trustee for the Third Person Communication Trust
21-Jan-2016
Visionary Enterprising Management Services Pty Ltd
20-Jan-2016
23-Jun-2015
13099902459
Visionary Enterprising Management Services Pty Ltd
21-Jan-2016
WELLS HASLEM STRATEGIC PUBLIC AFFAIRS PTY. LTD.
20-Jan-2016
02-Sep-2015
52159456685
WELLS HASLEM STRATEGIC PUBLIC AFFAIRS PTY. LTD.
26-Jan-2016
A D EVERS-BUCKLAND
20-Jan-2016
09-Nov-2015
72963316425
A D Evers-Buckland
26-Jan-2016
assetpr
20-Jan-2016
18-Jun-2015
37134426716
assetpr

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (2.17 MB) Use the API

rows 10 / 4431

Start Client Name End Agency Name
20-Jan-2016
AA Crown Holdings
The Premier Communications Group Pty Ltd.
20-Jan-2016
AECOM Australia
GRACosway Pty Ltd
20-Jan-2016
Aeroflex a Cobham Company
Vicom Australia Pty Ltd
20-Jan-2016
Aeronautics System Ltd
ABAKUS - Aircraft and Aviation Specialists Pty Limited
20-Jan-2016
AHA
SPIER CONSULTING P/L
20-Jan-2016
AIR CONDITIONING AND REFRIGERATION EQUIPMENT MANUFACTURERS ASSOCIATION OF AUSTRALIA INCORPORATED
Global Issues Group
20-Jan-2016
Air New Zealand
Diplomacy Pty Limited
20-Jan-2016
ALDI Stores
GRACosway Pty Ltd
20-Jan-2016
Alinta Energy
Megan Anwyl
20-Jan-2016
Allergan Australia Pty Limited
Health Communincations Australia Pty Limited

Statistics

Average successful run time: 5 minutes

Total run time: 5 days

Total cpu time used: about 5 hours

Total disk space used: 2.2 MB

History

  • Auto ran revision a3cc62da and failed .
    nothing changed in the database
  • Auto ran revision a3cc62da and failed .
    nothing changed in the database
  • Auto ran revision a3cc62da and failed .
    nothing changed in the database
  • Auto ran revision a3cc62da and failed .
    nothing changed in the database
  • Auto ran revision a3cc62da and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

lobbybot / scraper.py