markbrough / exchangerates-scraper

Scraper to collect and provide API for historical exchangerates data on morph.io


Exchange rates scraper

Scraper to collect and provide API for historical exchangerates data on morph.io

This simple scraper uses the exchangerates pypi package to generate a rolling set of exchangerates data, sourced from FRED and the OECD. For more information, see the exchangerates repository.

Contributors markbrough

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Python app detected -----> Installing python-3.6.2 -----> Installing pip -----> Installing requirements with pip  Collecting urllib3==1.24.2  Downloading urllib3-1.24.2-py2.py3-none-any.whl (131 kB)  Collecting exchangerates==0.3.1  Downloading exchangerates-0.3.1-py3-none-any.whl (6.6 kB)  Collecting requests==2.22.0  Downloading requests-2.22.0-py2.py3-none-any.whl (57 kB)  Collecting scraperwiki==0.5.1  Downloading scraperwiki-0.5.1.tar.gz (7.7 kB)  Collecting lxml==4.4.0  Downloading lxml-4.4.0-cp36-cp36m-manylinux1_x86_64.whl (5.7 MB)  Collecting six==1.12.0  Downloading six-1.12.0-py2.py3-none-any.whl (10 kB)  Collecting certifi>=2017.4.17  Downloading certifi-2020.4.5.1-py2.py3-none-any.whl (157 kB)  Collecting chardet<3.1.0,>=3.0.2  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)  Collecting idna<2.9,>=2.5  Downloading idna-2.8-py2.py3-none-any.whl (58 kB)  Collecting sqlalchemy  Downloading SQLAlchemy-1.3.17-cp36-cp36m-manylinux2010_x86_64.whl (1.2 MB)  Collecting alembic  Downloading alembic-1.4.2.tar.gz (1.1 MB)  Installing build dependencies: started  Installing build dependencies: finished with status 'done'  Getting requirements to build wheel: started  Getting requirements to build wheel: finished with status 'done'  Preparing wheel metadata: started  Preparing wheel metadata: finished with status 'done'  Collecting python-editor>=0.3  Downloading python_editor-1.0.4-py3-none-any.whl (4.9 kB)  Collecting Mako  Downloading Mako-1.1.3.tar.gz (477 kB)  Collecting python-dateutil  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)  Collecting MarkupSafe>=0.9.2  Downloading MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl (27 kB)  Building wheels for collected packages: scraperwiki, alembic, Mako  Building wheel for scraperwiki (setup.py): started  Building wheel for scraperwiki (setup.py): finished with status 'done'  Created wheel for scraperwiki: filename=scraperwiki-0.5.1-py3-none-any.whl size=6543 sha256=34093cf930b057b311159239736cb4c3e1752c7133b87ac2fdd1bc7b3fab3edf  Stored in directory: /tmp/pip-ephem-wheel-cache-q63hqfk_/wheels/cd/f8/ac/cd66eb1c557ab40d35c1ed852da3e9b37baa3e21b61906a5cf  Building wheel for alembic (PEP 517): started  Building wheel for alembic (PEP 517): finished with status 'done'  Created wheel for alembic: filename=alembic-1.4.2-py2.py3-none-any.whl size=159543 sha256=39949db2fdff2ab2e8cd44be267aef55b6d2439a63bd0ff14c6a7b1a8725124f  Stored in directory: /tmp/pip-ephem-wheel-cache-q63hqfk_/wheels/f2/50/61/5cc491b0ca39be60dfb4dce940b389ff91b847d62e0eb2d680  Building wheel for Mako (setup.py): started  Building wheel for Mako (setup.py): finished with status 'done'  Created wheel for Mako: filename=Mako-1.1.3-py2.py3-none-any.whl size=75517 sha256=2769acc28d8d81ac8fb4d3684558bba118118ecb57e4bab43c95b1e23108059d  Stored in directory: /tmp/pip-ephem-wheel-cache-q63hqfk_/wheels/2c/c6/4c/0d6db5a180111a4cd162c6b7bac4c2d13d20e901a62938e4d5  Successfully built scraperwiki alembic Mako  Installing collected packages: urllib3, lxml, certifi, chardet, idna, requests, six, exchangerates, sqlalchemy, python-editor, MarkupSafe, Mako, python-dateutil, alembic, scraperwiki  Successfully installed Mako-1.1.3 MarkupSafe-1.1.1 alembic-1.4.2 certifi-2020.4.5.1 chardet-3.0.4 exchangerates-0.3.1 idna-2.8 lxml-4.4.0 python-dateutil-2.8.1 python-editor-1.0.4 requests-2.22.0 scraperwiki-0.5.1 six-1.12.0 sqlalchemy-1.3.17 urllib3-1.24.2   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Retrieving rates from FRED for DEXBZUS Getting data for currency BRL Retrieving rates from FRED for DEXCAUS Getting data for currency CAD Retrieving rates from FRED for DEXCHUS Getting data for currency CNY Retrieving rates from FRED for DEXDNUS Getting data for currency DKK Retrieving rates from FRED for DEXHKUS Getting data for currency HKD Retrieving rates from FRED for DEXINUS Getting data for currency INR Retrieving rates from FRED for DEXJPUS Getting data for currency JPY Retrieving rates from FRED for DEXKOUS Getting data for currency KRW Retrieving rates from FRED for DEXMAUS Getting data for currency MYR Retrieving rates from FRED for DEXMXUS Getting data for currency MXN Retrieving rates from FRED for DEXNOUS Getting data for currency NOK Retrieving rates from FRED for DEXSDUS Getting data for currency SEK Retrieving rates from FRED for DEXSFUS Getting data for currency ZAR Retrieving rates from FRED for DEXSIUS Getting data for currency SGD Retrieving rates from FRED for DEXSLUS Getting data for currency LKR Retrieving rates from FRED for DEXSZUS Getting data for currency CHF Retrieving rates from FRED for DEXTAUS Getting data for currency TWD Retrieving rates from FRED for DEXTHUS Getting data for currency THB Retrieving rates from FRED for DEXUSAL Getting data for currency AUD Retrieving rates from FRED for DEXUSEU Getting data for currency EUR Retrieving rates from FRED for DEXUSNZ Getting data for currency NZD Retrieving rates from FRED for DEXUSUK Getting data for currency GBP Retrieving rates from FRED for DEXVZUS Getting data for currency VEF Getting rates from OECD Downloaded data, loading existing DB data! Loaded existing data (260853 rows), parsing! Retrieving rates from FRED for DEXBZUS Getting data for currency BRL Retrieving rates from FRED for DEXCAUS Getting data for currency CAD Retrieving rates from FRED for DEXCHUS Getting data for currency CNY Retrieving rates from FRED for DEXDNUS Getting data for currency DKK Retrieving rates from FRED for DEXHKUS Getting data for currency HKD Retrieving rates from FRED for DEXINUS Getting data for currency INR Retrieving rates from FRED for DEXJPUS Getting data for currency JPY Retrieving rates from FRED for DEXKOUS Getting data for currency KRW Retrieving rates from FRED for DEXMAUS Getting data for currency MYR Retrieving rates from FRED for DEXMXUS Getting data for currency MXN Retrieving rates from FRED for DEXNOUS Getting data for currency NOK Retrieving rates from FRED for DEXSDUS Getting data for currency SEK Retrieving rates from FRED for DEXSFUS Getting data for currency ZAR Retrieving rates from FRED for DEXSIUS Getting data for currency SGD Retrieving rates from FRED for DEXSLUS Getting data for currency LKR Retrieving rates from FRED for DEXSZUS Getting data for currency CHF Retrieving rates from FRED for DEXTAUS Getting data for currency TWD Retrieving rates from FRED for DEXTHUS Getting data for currency THB Retrieving rates from FRED for DEXUSAL Getting data for currency AUD Retrieving rates from FRED for DEXUSEU Getting data for currency EUR Retrieving rates from FRED for DEXUSNZ Getting data for currency NZD Retrieving rates from FRED for DEXUSUK Getting data for currency GBP Retrieving rates from FRED for DEXVZUS Getting data for currency VEF Getting rates from OECD Downloaded data, loading existing DB data! Loaded existing data (260853 rows), parsing! Retrieving rates from FRED for DEXBZUS Getting data for currency BRL Retrieving rates from FRED for DEXCAUS Getting data for currency CAD Retrieving rates from FRED for DEXCHUS Getting data for currency CNY Retrieving rates from FRED for DEXDNUS Getting data for currency DKK Retrieving rates from FRED for DEXHKUS Getting data for currency HKD Retrieving rates from FRED for DEXINUS Getting data for currency INR Retrieving rates from FRED for DEXJPUS Getting data for currency JPY Retrieving rates from FRED for DEXKOUS Getting data for currency KRW Retrieving rates from FRED for DEXMAUS Getting data for currency MYR Retrieving rates from FRED for DEXMXUS Getting data for currency MXN Retrieving rates from FRED for DEXNOUS Getting data for currency NOK Retrieving rates from FRED for DEXSDUS Getting data for currency SEK Retrieving rates from FRED for DEXSFUS Getting data for currency ZAR Retrieving rates from FRED for DEXSIUS Getting data for currency SGD Retrieving rates from FRED for DEXSLUS Getting data for currency LKR Retrieving rates from FRED for DEXSZUS Getting data for currency CHF Retrieving rates from FRED for DEXTAUS Getting data for currency TWD Retrieving rates from FRED for DEXTHUS Getting data for currency THB Retrieving rates from FRED for DEXUSAL Getting data for currency AUD Retrieving rates from FRED for DEXUSEU Getting data for currency EUR Retrieving rates from FRED for DEXUSNZ Getting data for currency NZD Retrieving rates from FRED for DEXUSUK Getting data for currency GBP Retrieving rates from FRED for DEXVZUS Getting data for currency VEF Getting rates from OECD Downloaded data, loading existing DB data! Loaded existing data (260853 rows), parsing! Retrieving rates from FRED for DEXBZUS Getting data for currency BRL Retrieving rates from FRED for DEXCAUS Getting data for currency CAD Retrieving rates from FRED for DEXCHUS Getting data for currency CNY Retrieving rates from FRED for DEXDNUS Getting data for currency DKK Retrieving rates from FRED for DEXHKUS Getting data for currency HKD Retrieving rates from FRED for DEXINUS Getting data for currency INR Retrieving rates from FRED for DEXJPUS Getting data for currency JPY Retrieving rates from FRED for DEXKOUS Getting data for currency KRW Retrieving rates from FRED for DEXMAUS Getting data for currency MYR Retrieving rates from FRED for DEXMXUS Getting data for currency MXN Retrieving rates from FRED for DEXNOUS Getting data for currency NOK Retrieving rates from FRED for DEXSDUS Getting data for currency SEK Retrieving rates from FRED for DEXSFUS Getting data for currency ZAR Retrieving rates from FRED for DEXSIUS Getting data for currency SGD Retrieving rates from FRED for DEXSLUS Getting data for currency LKR Retrieving rates from FRED for DEXSZUS Getting data for currency CHF Retrieving rates from FRED for DEXTAUS Getting data for currency TWD Retrieving rates from FRED for DEXTHUS Getting data for currency THB Retrieving rates from FRED for DEXUSAL Getting data for currency AUD Retrieving rates from FRED for DEXUSEU Getting data for currency EUR Retrieving rates from FRED for DEXUSNZ Getting data for currency NZD Retrieving rates from FRED for DEXUSUK Getting data for currency GBP Retrieving rates from FRED for DEXVZUS Getting data for currency VEF Getting rates from OECD Downloaded data, loading existing DB data! Loaded existing data (260853 rows), parsing!

Data

Downloaded 4774 times by andylolz markbrough matmaxgeds siemvaessen

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (25.1 MB) Use the API

rows 10 / 320

started_at finished_at success
2019-04-24T00:24:50+00:00
false
2019-04-24T01:13:00+00:00
2019-04-24 01:35:48.378022
true
2019-04-25T12:23:10+00:00
2019-04-25 12:46:06.486504
true
2019-05-15T15:36:50+00:00
2019-05-15 15:59:41.450950
true
2019-07-12T18:20:44+00:00
2019-07-12 18:43:33.062638
true
2019-07-13T18:05:44+00:00
2019-07-13 18:28:39.063152
true
2019-07-14T03:44:05+00:00
2019-07-14 04:06:54.002352
true
2019-07-15T11:25:25+00:00
2019-07-15 11:48:05.282473
true
2019-07-16T13:55:44+00:00
2019-07-16 14:18:32.582658
true
2019-07-17T20:03:04+00:00
2019-07-17 20:25:44.737203
true

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (25.1 MB) Use the API

rows 10 / 260968

Date Rate Currency Frequency Source RateFirstSeen
1995-01-02
0.8440
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-03
0.8450
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-04
0.8450
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-05
0.8430
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-06
0.8400
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-09
0.8440
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-10
0.8470
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-11
0.8510
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-12
0.8450
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-13
0.8470
BRL
D
FRED
2019-04-24T00:25:04+00:00

Statistics

Average successful run time: 29 minutes

Total run time: 7 days

Total cpu time used: 6 days

Total disk space used: 25.1 MB

History

  • Auto ran revision aded1370 and completed successfully .
    116 records added in the database
  • Auto ran revision aded1370 and completed successfully .
    1 record added in the database
  • Auto ran revision aded1370 and completed successfully .
    1 record added in the database
  • Auto ran revision aded1370 and completed successfully .
    1 record added in the database
  • Auto ran revision aded1370 and completed successfully .
    1 record added in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

exchangerates-scraper / scraper.py