markbrough / exchangerates-scraper

Scraper to collect and provide API for historical exchangerates data on morph.io


Exchange rates scraper

Scraper to collect and provide API for historical exchangerates data on morph.io

This simple scraper uses the exchangerates pypi package to generate a rolling set of exchangerates data, sourced from FRED and the OECD. For more information, see the exchangerates repository.

Contributors markbrough

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Python app detected -----> Installing python-3.6.2 -----> Installing pip -----> Installing requirements with pip  Collecting urllib3==1.24.2  Downloading urllib3-1.24.2-py2.py3-none-any.whl (131 kB)  Collecting exchangerates==0.3.1  Downloading exchangerates-0.3.1-py3-none-any.whl (6.6 kB)  Collecting requests==2.22.0  Downloading requests-2.22.0-py2.py3-none-any.whl (57 kB)  Collecting scraperwiki==0.5.1  Downloading scraperwiki-0.5.1.tar.gz (7.7 kB)  Collecting six==1.12.0  Downloading six-1.12.0-py2.py3-none-any.whl (10 kB)  Collecting lxml==4.4.0  Downloading lxml-4.4.0-cp36-cp36m-manylinux1_x86_64.whl (5.7 MB)  Collecting chardet<3.1.0,>=3.0.2  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)  Collecting certifi>=2017.4.17  Downloading certifi-2019.11.28-py2.py3-none-any.whl (156 kB)  Collecting idna<2.9,>=2.5  Downloading idna-2.8-py2.py3-none-any.whl (58 kB)  Collecting sqlalchemy  Downloading SQLAlchemy-1.3.13.tar.gz (6.0 MB)  Collecting alembic  Downloading alembic-1.4.0.tar.gz (1.1 MB)  Collecting Mako  Downloading Mako-1.1.1.tar.gz (468 kB)  Collecting python-editor>=0.3  Downloading python_editor-1.0.4-py3-none-any.whl (4.9 kB)  Collecting python-dateutil  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)  Collecting MarkupSafe>=0.9.2  Downloading MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl (27 kB)  Building wheels for collected packages: scraperwiki, sqlalchemy, alembic, Mako  Building wheel for scraperwiki (setup.py): started  Building wheel for scraperwiki (setup.py): finished with status 'done'  Created wheel for scraperwiki: filename=scraperwiki-0.5.1-py3-none-any.whl size=6543 sha256=5d5f92bf4e2cb600ce7c93863879bb84947ccdfc6a5fb060e38034e53262d5be  Stored in directory: /tmp/pip-ephem-wheel-cache-62480_98/wheels/cd/f8/ac/cd66eb1c557ab40d35c1ed852da3e9b37baa3e21b61906a5cf  Building wheel for sqlalchemy (setup.py): started  Building wheel for sqlalchemy (setup.py): finished with status 'done'  Created wheel for sqlalchemy: filename=SQLAlchemy-1.3.13-cp36-cp36m-linux_x86_64.whl size=1203895 sha256=a31f9fa218aa1be3a8fcfb62bf41f8a772b6d14f9bf73fdddcd7a10c00275799  Stored in directory: /tmp/pip-ephem-wheel-cache-62480_98/wheels/28/3e/f9/8eca04781258bb6956ffba37e4e6e6951e5b3a16d4494b91cb  Building wheel for alembic (setup.py): started  Building wheel for alembic (setup.py): finished with status 'done'  Created wheel for alembic: filename=alembic-1.4.0-py2.py3-none-any.whl size=157561 sha256=18857c7e1b32fd944744b3dcae5996115e6a010bb1eb173d22a411448d8931de  Stored in directory: /tmp/pip-ephem-wheel-cache-62480_98/wheels/54/88/28/d771a55dfb3c62af8b3358c60f8034edcb5c9a57d44a9024cf  Building wheel for Mako (setup.py): started  Building wheel for Mako (setup.py): finished with status 'done'  Created wheel for Mako: filename=Mako-1.1.1-py3-none-any.whl size=75409 sha256=f18a8c8b2ba8f6e03b3a6394870db96368f3f26117e01903d7672622b5317fe1  Stored in directory: /tmp/pip-ephem-wheel-cache-62480_98/wheels/43/b1/7c/f14ef20f4683e5087ae684c6447194e09695315f20b9c45575  Successfully built scraperwiki sqlalchemy alembic Mako  Installing collected packages: urllib3, six, lxml, chardet, certifi, idna, requests, exchangerates, sqlalchemy, MarkupSafe, Mako, python-editor, python-dateutil, alembic, scraperwiki  Successfully installed Mako-1.1.1 MarkupSafe-1.1.1 alembic-1.4.0 certifi-2019.11.28 chardet-3.0.4 exchangerates-0.3.1 idna-2.8 lxml-4.4.0 python-dateutil-2.8.1 python-editor-1.0.4 requests-2.22.0 scraperwiki-0.5.1 six-1.12.0 sqlalchemy-1.3.13 urllib3-1.24.2   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Retrieving rates from FRED for DEXBZUS Getting data for currency BRL Retrieving rates from FRED for DEXCAUS Getting data for currency CAD Retrieving rates from FRED for DEXCHUS Getting data for currency CNY Retrieving rates from FRED for DEXDNUS Getting data for currency DKK Retrieving rates from FRED for DEXHKUS Getting data for currency HKD Retrieving rates from FRED for DEXINUS Getting data for currency INR Retrieving rates from FRED for DEXJPUS Getting data for currency JPY Retrieving rates from FRED for DEXKOUS Getting data for currency KRW Retrieving rates from FRED for DEXMAUS Getting data for currency MYR Retrieving rates from FRED for DEXMXUS Getting data for currency MXN Retrieving rates from FRED for DEXNOUS Getting data for currency NOK Retrieving rates from FRED for DEXSDUS Getting data for currency SEK Retrieving rates from FRED for DEXSFUS Getting data for currency ZAR Retrieving rates from FRED for DEXSIUS Getting data for currency SGD Retrieving rates from FRED for DEXSLUS Getting data for currency LKR Retrieving rates from FRED for DEXSZUS Getting data for currency CHF Retrieving rates from FRED for DEXTAUS Getting data for currency TWD Retrieving rates from FRED for DEXTHUS Getting data for currency THB Retrieving rates from FRED for DEXUSAL Getting data for currency AUD Retrieving rates from FRED for DEXUSEU Getting data for currency EUR Retrieving rates from FRED for DEXUSNZ Getting data for currency NZD Retrieving rates from FRED for DEXUSUK Getting data for currency GBP Retrieving rates from FRED for DEXVZUS Getting data for currency VEF Getting rates from OECD Downloaded data, loading existing DB data! Loaded existing data (259288 rows), parsing! Retrieving rates from FRED for DEXBZUS Getting data for currency BRL Retrieving rates from FRED for DEXCAUS Getting data for currency CAD Retrieving rates from FRED for DEXCHUS Getting data for currency CNY Retrieving rates from FRED for DEXDNUS Getting data for currency DKK Retrieving rates from FRED for DEXHKUS Getting data for currency HKD Retrieving rates from FRED for DEXINUS Getting data for currency INR Retrieving rates from FRED for DEXJPUS Getting data for currency JPY Retrieving rates from FRED for DEXKOUS Getting data for currency KRW Retrieving rates from FRED for DEXMAUS Getting data for currency MYR Retrieving rates from FRED for DEXMXUS Getting data for currency MXN Retrieving rates from FRED for DEXNOUS Getting data for currency NOK Retrieving rates from FRED for DEXSDUS Getting data for currency SEK Retrieving rates from FRED for DEXSFUS Getting data for currency ZAR Retrieving rates from FRED for DEXSIUS Getting data for currency SGD Retrieving rates from FRED for DEXSLUS Getting data for currency LKR Retrieving rates from FRED for DEXSZUS Getting data for currency CHF Retrieving rates from FRED for DEXTAUS Getting data for currency TWD Retrieving rates from FRED for DEXTHUS Getting data for currency THB Retrieving rates from FRED for DEXUSAL Getting data for currency AUD Retrieving rates from FRED for DEXUSEU Getting data for currency EUR Retrieving rates from FRED for DEXUSNZ Getting data for currency NZD Retrieving rates from FRED for DEXUSUK Getting data for currency GBP Retrieving rates from FRED for DEXVZUS Getting data for currency VEF Getting rates from OECD Downloaded data, loading existing DB data! Loaded existing data (259288 rows), parsing! Retrieving rates from FRED for DEXBZUS Getting data for currency BRL Retrieving rates from FRED for DEXCAUS Getting data for currency CAD Retrieving rates from FRED for DEXCHUS Getting data for currency CNY Retrieving rates from FRED for DEXDNUS Getting data for currency DKK Retrieving rates from FRED for DEXHKUS Getting data for currency HKD Retrieving rates from FRED for DEXINUS Getting data for currency INR Retrieving rates from FRED for DEXJPUS Getting data for currency JPY Retrieving rates from FRED for DEXKOUS Getting data for currency KRW Retrieving rates from FRED for DEXMAUS Getting data for currency MYR Retrieving rates from FRED for DEXMXUS Getting data for currency MXN Retrieving rates from FRED for DEXNOUS Getting data for currency NOK Retrieving rates from FRED for DEXSDUS Getting data for currency SEK Retrieving rates from FRED for DEXSFUS Getting data for currency ZAR Retrieving rates from FRED for DEXSIUS Getting data for currency SGD Retrieving rates from FRED for DEXSLUS Getting data for currency LKR Retrieving rates from FRED for DEXSZUS Getting data for currency CHF Retrieving rates from FRED for DEXTAUS Getting data for currency TWD Retrieving rates from FRED for DEXTHUS Getting data for currency THB Retrieving rates from FRED for DEXUSAL Getting data for currency AUD Retrieving rates from FRED for DEXUSEU Getting data for currency EUR Retrieving rates from FRED for DEXUSNZ Getting data for currency NZD Retrieving rates from FRED for DEXUSUK Getting data for currency GBP Retrieving rates from FRED for DEXVZUS Getting data for currency VEF Getting rates from OECD Downloaded data, loading existing DB data! Loaded existing data (259288 rows), parsing! Retrieving rates from FRED for DEXBZUS Getting data for currency BRL Retrieving rates from FRED for DEXCAUS Getting data for currency CAD Retrieving rates from FRED for DEXCHUS Getting data for currency CNY Retrieving rates from FRED for DEXDNUS Getting data for currency DKK Retrieving rates from FRED for DEXHKUS Getting data for currency HKD Retrieving rates from FRED for DEXINUS Getting data for currency INR Retrieving rates from FRED for DEXJPUS Getting data for currency JPY Retrieving rates from FRED for DEXKOUS Getting data for currency KRW Retrieving rates from FRED for DEXMAUS Getting data for currency MYR Retrieving rates from FRED for DEXMXUS Getting data for currency MXN Retrieving rates from FRED for DEXNOUS Getting data for currency NOK Retrieving rates from FRED for DEXSDUS Getting data for currency SEK Retrieving rates from FRED for DEXSFUS Getting data for currency ZAR Retrieving rates from FRED for DEXSIUS Getting data for currency SGD Retrieving rates from FRED for DEXSLUS Getting data for currency LKR Retrieving rates from FRED for DEXSZUS Getting data for currency CHF Retrieving rates from FRED for DEXTAUS Getting data for currency TWD Retrieving rates from FRED for DEXTHUS Getting data for currency THB Retrieving rates from FRED for DEXUSAL Getting data for currency AUD Retrieving rates from FRED for DEXUSEU Getting data for currency EUR Retrieving rates from FRED for DEXUSNZ Getting data for currency NZD Retrieving rates from FRED for DEXUSUK Getting data for currency GBP Retrieving rates from FRED for DEXVZUS Getting data for currency VEF Getting rates from OECD Downloaded data, loading existing DB data! Loaded existing data (259288 rows), parsing!

Data

Downloaded 4596 times by andylolz markbrough siemvaessen matmaxgeds

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (24.9 MB) Use the API

rows 10 / 226

started_at finished_at success
2019-04-24T00:24:50+00:00
false
2019-04-24T01:13:00+00:00
2019-04-24 01:35:48.378022
true
2019-04-25T12:23:10+00:00
2019-04-25 12:46:06.486504
true
2019-05-15T15:36:50+00:00
2019-05-15 15:59:41.450950
true
2019-07-12T18:20:44+00:00
2019-07-12 18:43:33.062638
true
2019-07-13T18:05:44+00:00
2019-07-13 18:28:39.063152
true
2019-07-14T03:44:05+00:00
2019-07-14 04:06:54.002352
true
2019-07-15T11:25:25+00:00
2019-07-15 11:48:05.282473
true
2019-07-16T13:55:44+00:00
2019-07-16 14:18:32.582658
true
2019-07-17T20:03:04+00:00
2019-07-17 20:25:44.737203
true

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (24.9 MB) Use the API

rows 10 / 259288

Date Rate Currency Frequency Source RateFirstSeen
1995-01-02
0.8440
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-03
0.8450
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-04
0.8450
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-05
0.8430
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-06
0.8400
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-09
0.8440
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-10
0.8470
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-11
0.8510
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-12
0.8450
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-13
0.8470
BRL
D
FRED
2019-04-24T00:25:04+00:00

Statistics

Average successful run time: 28 minutes

Total run time: 4 days

Total cpu time used: 4 days

Total disk space used: 24.9 MB

History

  • Auto ran revision aded1370 and completed successfully .
    1 record added in the database
  • Auto ran revision aded1370 and completed successfully .
    1 record added in the database
  • Auto ran revision aded1370 and completed successfully .
    1 record added in the database
  • Auto ran revision aded1370 and completed successfully .
    116 records added in the database
  • Auto ran revision aded1370 and completed successfully .
    1 record added in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

exchangerates-scraper / scraper.py