danozgriff / AB_Scaper_1

Miscellaneous


This is a scraper that runs on Morph. To get started see the documentation

Contributors danozgriff

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Python app detected -----> Installing python-2.7.14 -----> Installing pip -----> Installing requirements with pip  DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 6))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to revision morph_defaults) to /app/.heroku/src/scraperwiki  Collecting lxml==3.4.4  Downloading lxml-3.4.4.tar.gz (3.5 MB)  Collecting cssselect==0.9.1  Downloading cssselect-0.9.1.tar.gz (32 kB)  Collecting mechanize==0.2.5  Downloading mechanize-0.2.5.tar.gz (383 kB)  Collecting beautifulsoup4==4.9.1  Downloading beautifulsoup4-4.9.1-py2-none-any.whl (111 kB)  Collecting pytz==2020.1  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)  Collecting dumptruck>=0.1.2  Downloading dumptruck-0.1.6.tar.gz (15 kB)  Collecting requests  Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)  Collecting soupsieve<2.0  Downloading soupsieve-1.9.6-py2.py3-none-any.whl (33 kB)  Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1  Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)  Collecting certifi>=2017.4.17  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)  Collecting chardet<4,>=3.0.2  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)  Collecting idna<3,>=2.5  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)  Collecting backports.functools-lru-cache; python_version < "3"  Downloading backports.functools_lru_cache-1.6.1-py2.py3-none-any.whl (5.7 kB)  Building wheels for collected packages: lxml, cssselect, mechanize, dumptruck  Building wheel for lxml (setup.py): started  Building wheel for lxml (setup.py): still running...  Building wheel for lxml (setup.py): finished with status 'done'  Created wheel for lxml: filename=lxml-3.4.4-cp27-cp27mu-linux_x86_64.whl size=2987167 sha256=cb929f3c3b1a6670dcce4af8bcd1ac8a02b6c8734ca39a22bdcdcab17108004d  Stored in directory: /tmp/pip-ephem-wheel-cache-HYvzYm/wheels/d6/de/81/11ae6edd05c75aac677e67dd154c85da758ba6f3e8e80e962e  Building wheel for cssselect (setup.py): started  Building wheel for cssselect (setup.py): finished with status 'done'  Created wheel for cssselect: filename=cssselect-0.9.1-py2-none-any.whl size=26993 sha256=50e109c949936c91ce835c2a98d4bcdbfd48f1d29e435279c09652135ea76fd6  Stored in directory: /tmp/pip-ephem-wheel-cache-HYvzYm/wheels/85/fe/00/b94036d8583cec9791d8cda24c184f2d2ac1397822f7f0e8d4  Building wheel for mechanize (setup.py): started  Building wheel for mechanize (setup.py): finished with status 'done'  Created wheel for mechanize: filename=mechanize-0.2.5-py2-none-any.whl size=150109 sha256=d7745555ac76a7b266b113b1a9555322f2ebeb4892158b9e653228f19a86861b  Stored in directory: /tmp/pip-ephem-wheel-cache-HYvzYm/wheels/fe/20/00/f8e9ca5a8b8c75eec22f5d63730b66e95be06274f04e9babf6  Building wheel for dumptruck (setup.py): started  Building wheel for dumptruck (setup.py): finished with status 'done'  Created wheel for dumptruck: filename=dumptruck-0.1.6-py2-none-any.whl size=11843 sha256=d2b636bedfab35fdf3bbff49120bef9a32fce14a464174ba8f1a29a07cd38982  Stored in directory: /tmp/pip-ephem-wheel-cache-HYvzYm/wheels/dc/75/e9/1e61c4080c73e7bda99614549591f83b53bcc2d682f26fce62  Successfully built lxml cssselect mechanize dumptruck  Installing collected packages: dumptruck, urllib3, certifi, chardet, idna, requests, scraperwiki, lxml, cssselect, mechanize, backports.functools-lru-cache, soupsieve, beautifulsoup4, pytz  Running setup.py develop for scraperwiki  Successfully installed backports.functools-lru-cache-1.6.1 beautifulsoup4-4.9.1 certifi-2020.6.20 chardet-3.0.4 cssselect-0.9.1 dumptruck-0.1.6 idna-2.10 lxml-3.4.4 mechanize-0.2.5 pytz-2020.1 requests-2.24.0 scraperwiki soupsieve-1.9.6 urllib3-1.25.10 DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.    -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running...

Data

Downloaded 0 times

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (2.11 MB) Use the API

rows 10 / 11

Start_DateTime End_DateTime Company_EOD_Date
2020-09-15 22:19:47
2020-09-15 22:19:51
2020-09-15
2020-09-17 04:26:48
2020-09-17 04:26:52
2020-09-16
2020-09-17 21:36:12
2020-09-17 21:36:16
2020-09-17
2020-09-18 23:24:09
2020-09-18 23:24:13
2020-09-18
2020-09-19 16:31:14
2020-09-19 16:31:18
2020-09-18
2020-09-20 15:41:51
2020-09-20 15:41:55
2020-09-18
2020-09-22 00:35:27
2020-09-22 00:35:31
2020-09-21
2020-09-23 05:20:11
2020-09-23 05:20:14
2020-09-22
2020-09-24 04:55:33
2020-09-24 04:55:37
2020-09-23
2020-09-24 22:43:26
2020-09-24 22:43:29
2020-09-24

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (2.11 MB) Use the API

rows 10 / 19619

Rank Code Company Price Change Perc_Change Perc_Change_1_Year Market_Cap EOD_Date
1
CSL
CSL Ltd
287.24
4.1
0.0145
0.2468
130660000000
2020-09-15
2
CBA
Commonwealth Bank of Australia
65.03
-1.14
-0.0172
-0.2039
115120000000
2020-09-15
3
BHP
BHP Group Ltd
37.49
0.25
0.0067
-0.0212
110440000000
2020-09-15
4
WBC
Westpac Banking Corporation
16.72
-0.25
-0.0147
-0.434
60390000000
2020-09-15
5
NAB
National Australia Bank Ltd
17.16
-0.24
-0.0138
-0.4109
56460000000
2020-09-15
6
FMG
Fortescue Metals Group Ltd
17.7
-0.01
-0.0006
0.9645
54500000000
2020-09-15
7
WES
Wesfarmers Ltd
45.01
0.09
0.002
0.1453
51030000000
2020-09-15
8
ANZ
Australia and New Zealand Banking Group Ltd
17.35
-0.33
-0.0187
-0.3714
49210000000
2020-09-15
9
WOW
Woolworths Group Ltd
36.34
-0.15
-0.0041
0.0028
45900000000
2020-09-15
10
MQG
Macquarie Group Ltd
118.69
-1.51
-0.0126
-0.083
42910000000
2020-09-15

Statistics

Average successful run time: 3 minutes

Total run time: 33 minutes

Total cpu time used: less than a minute

Total disk space used: 2.14 MB

History

  • Auto ran revision 184cb814 and completed successfully .
    2183 records added in the database
  • Auto ran revision 184cb814 and completed successfully .
    2180 records added in the database
  • Auto ran revision 184cb814 and completed successfully .
    2180 records added in the database
  • Auto ran revision 184cb814 and completed successfully .
    2182 records added in the database
  • Auto ran revision 184cb814 and completed successfully .
    2181 records added in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

AB_Scaper_1 / scraper.py