markbrough / exchangerates-scraper

Scraper to collect and provide API for historical exchangerates data on morph.io


Exchange rates scraper

Scraper to collect and provide API for historical exchangerates data on morph.io

This simple scraper uses the exchangerates pypi package to generate a rolling set of exchangerates data, sourced from FRED and the OECD. For more information, see the exchangerates repository.

Contributors markbrough

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Python app detected -----> Installing python-3.6.2 -----> Installing pip -----> Installing requirements with pip  Collecting urllib3==1.24.2  Downloading urllib3-1.24.2-py2.py3-none-any.whl (131 kB)  Collecting exchangerates==0.3.1  Downloading exchangerates-0.3.1-py3-none-any.whl (6.6 kB)  Collecting requests==2.22.0  Downloading requests-2.22.0-py2.py3-none-any.whl (57 kB)  Collecting scraperwiki==0.5.1  Downloading scraperwiki-0.5.1.tar.gz (7.7 kB)  Collecting six==1.12.0  Downloading six-1.12.0-py2.py3-none-any.whl (10 kB)  Collecting lxml==4.4.0  Downloading lxml-4.4.0-cp36-cp36m-manylinux1_x86_64.whl (5.7 MB)  Collecting idna<2.9,>=2.5  Downloading idna-2.8-py2.py3-none-any.whl (58 kB)  Collecting certifi>=2017.4.17  Downloading certifi-2020.11.8-py2.py3-none-any.whl (155 kB)  Collecting chardet<3.1.0,>=3.0.2  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)  Collecting sqlalchemy  Downloading SQLAlchemy-1.3.20-cp36-cp36m-manylinux2010_x86_64.whl (1.3 MB)  Collecting alembic  Downloading alembic-1.4.3-py2.py3-none-any.whl (159 kB)  Collecting python-dateutil  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)  Collecting Mako  Downloading Mako-1.1.3-py2.py3-none-any.whl (75 kB)  Collecting python-editor>=0.3  Downloading python_editor-1.0.4-py3-none-any.whl (4.9 kB)  Collecting MarkupSafe>=0.9.2  Downloading MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl (27 kB)  Building wheels for collected packages: scraperwiki  Building wheel for scraperwiki (setup.py): started  Building wheel for scraperwiki (setup.py): finished with status 'done'  Created wheel for scraperwiki: filename=scraperwiki-0.5.1-py3-none-any.whl size=6545 sha256=3a2d841a07ef3f6d35d40c5608886ff310c6bf025e94dd0b654d026879b50c88  Stored in directory: /tmp/pip-ephem-wheel-cache-8wbo00uk/wheels/cd/f8/ac/cd66eb1c557ab40d35c1ed852da3e9b37baa3e21b61906a5cf  Successfully built scraperwiki  Installing collected packages: urllib3, six, lxml, idna, certifi, chardet, requests, exchangerates, sqlalchemy, python-dateutil, MarkupSafe, Mako, python-editor, alembic, scraperwiki  Successfully installed Mako-1.1.3 MarkupSafe-1.1.1 alembic-1.4.3 certifi-2020.11.8 chardet-3.0.4 exchangerates-0.3.1 idna-2.8 lxml-4.4.0 python-dateutil-2.8.1 python-editor-1.0.4 requests-2.22.0 scraperwiki-0.5.1 six-1.12.0 sqlalchemy-1.3.20 urllib3-1.24.2   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Retrieving rates from FRED for DEXBZUS Getting data for currency BRL Retrieving rates from FRED for DEXCAUS Getting data for currency CAD Retrieving rates from FRED for DEXCHUS Getting data for currency CNY Retrieving rates from FRED for DEXDNUS Getting data for currency DKK Retrieving rates from FRED for DEXHKUS Getting data for currency HKD Retrieving rates from FRED for DEXINUS Getting data for currency INR Retrieving rates from FRED for DEXJPUS Getting data for currency JPY Retrieving rates from FRED for DEXKOUS Getting data for currency KRW Retrieving rates from FRED for DEXMAUS Getting data for currency MYR Retrieving rates from FRED for DEXMXUS Getting data for currency MXN Retrieving rates from FRED for DEXNOUS Getting data for currency NOK Retrieving rates from FRED for DEXSDUS Getting data for currency SEK Retrieving rates from FRED for DEXSFUS Getting data for currency ZAR Retrieving rates from FRED for DEXSIUS Getting data for currency SGD Retrieving rates from FRED for DEXSLUS Getting data for currency LKR Retrieving rates from FRED for DEXSZUS Getting data for currency CHF Retrieving rates from FRED for DEXTAUS Getting data for currency TWD Retrieving rates from FRED for DEXTHUS Getting data for currency THB Retrieving rates from FRED for DEXUSAL Getting data for currency AUD Retrieving rates from FRED for DEXUSEU Getting data for currency EUR Retrieving rates from FRED for DEXUSNZ Getting data for currency NZD Retrieving rates from FRED for DEXUSUK Getting data for currency GBP Retrieving rates from FRED for DEXVZUS Getting data for currency VEF Getting rates from OECD Downloaded data, loading existing DB data! Loaded existing data (263893 rows), parsing! Retrieving rates from FRED for DEXBZUS Getting data for currency BRL Retrieving rates from FRED for DEXCAUS Getting data for currency CAD Retrieving rates from FRED for DEXCHUS Getting data for currency CNY Retrieving rates from FRED for DEXDNUS Getting data for currency DKK Retrieving rates from FRED for DEXHKUS Getting data for currency HKD Retrieving rates from FRED for DEXINUS Getting data for currency INR Retrieving rates from FRED for DEXJPUS Getting data for currency JPY Retrieving rates from FRED for DEXKOUS Getting data for currency KRW Retrieving rates from FRED for DEXMAUS Getting data for currency MYR Retrieving rates from FRED for DEXMXUS Getting data for currency MXN Retrieving rates from FRED for DEXNOUS Getting data for currency NOK Retrieving rates from FRED for DEXSDUS Getting data for currency SEK Retrieving rates from FRED for DEXSFUS Getting data for currency ZAR Retrieving rates from FRED for DEXSIUS Getting data for currency SGD Retrieving rates from FRED for DEXSLUS Getting data for currency LKR Retrieving rates from FRED for DEXSZUS Getting data for currency CHF Retrieving rates from FRED for DEXTAUS Getting data for currency TWD Retrieving rates from FRED for DEXTHUS Getting data for currency THB Retrieving rates from FRED for DEXUSAL Getting data for currency AUD Retrieving rates from FRED for DEXUSEU Getting data for currency EUR Retrieving rates from FRED for DEXUSNZ Getting data for currency NZD Retrieving rates from FRED for DEXUSUK Getting data for currency GBP Retrieving rates from FRED for DEXVZUS Getting data for currency VEF Getting rates from OECD Downloaded data, loading existing DB data! Loaded existing data (263893 rows), parsing! Retrieving rates from FRED for DEXBZUS Getting data for currency BRL Retrieving rates from FRED for DEXCAUS Getting data for currency CAD Retrieving rates from FRED for DEXCHUS Getting data for currency CNY Retrieving rates from FRED for DEXDNUS Getting data for currency DKK Retrieving rates from FRED for DEXHKUS Getting data for currency HKD Retrieving rates from FRED for DEXINUS Getting data for currency INR Retrieving rates from FRED for DEXJPUS Getting data for currency JPY Retrieving rates from FRED for DEXKOUS Getting data for currency KRW Retrieving rates from FRED for DEXMAUS Getting data for currency MYR Retrieving rates from FRED for DEXMXUS Getting data for currency MXN Retrieving rates from FRED for DEXNOUS Getting data for currency NOK Retrieving rates from FRED for DEXSDUS Getting data for currency SEK Retrieving rates from FRED for DEXSFUS Getting data for currency ZAR Retrieving rates from FRED for DEXSIUS Getting data for currency SGD Retrieving rates from FRED for DEXSLUS Getting data for currency LKR Retrieving rates from FRED for DEXSZUS Getting data for currency CHF Retrieving rates from FRED for DEXTAUS Getting data for currency TWD Retrieving rates from FRED for DEXTHUS Getting data for currency THB Retrieving rates from FRED for DEXUSAL Getting data for currency AUD Retrieving rates from FRED for DEXUSEU Getting data for currency EUR Retrieving rates from FRED for DEXUSNZ Getting data for currency NZD Retrieving rates from FRED for DEXUSUK Getting data for currency GBP Retrieving rates from FRED for DEXVZUS Getting data for currency VEF Getting rates from OECD Downloaded data, loading existing DB data! Loaded existing data (263893 rows), parsing! Retrieving rates from FRED for DEXBZUS Getting data for currency BRL Retrieving rates from FRED for DEXCAUS Getting data for currency CAD Retrieving rates from FRED for DEXCHUS Getting data for currency CNY Retrieving rates from FRED for DEXDNUS Getting data for currency DKK Retrieving rates from FRED for DEXHKUS Getting data for currency HKD Retrieving rates from FRED for DEXINUS Getting data for currency INR Retrieving rates from FRED for DEXJPUS Getting data for currency JPY Retrieving rates from FRED for DEXKOUS Getting data for currency KRW Retrieving rates from FRED for DEXMAUS Getting data for currency MYR Retrieving rates from FRED for DEXMXUS Getting data for currency MXN Retrieving rates from FRED for DEXNOUS Getting data for currency NOK Retrieving rates from FRED for DEXSDUS Getting data for currency SEK Retrieving rates from FRED for DEXSFUS Getting data for currency ZAR Retrieving rates from FRED for DEXSIUS Getting data for currency SGD Retrieving rates from FRED for DEXSLUS Getting data for currency LKR Retrieving rates from FRED for DEXSZUS Getting data for currency CHF Retrieving rates from FRED for DEXTAUS Getting data for currency TWD Retrieving rates from FRED for DEXTHUS Getting data for currency THB Retrieving rates from FRED for DEXUSAL Getting data for currency AUD Retrieving rates from FRED for DEXUSEU Getting data for currency EUR Retrieving rates from FRED for DEXUSNZ Getting data for currency NZD Retrieving rates from FRED for DEXUSUK Getting data for currency GBP Retrieving rates from FRED for DEXVZUS Getting data for currency VEF Getting rates from OECD Downloaded data, loading existing DB data! Loaded existing data (263893 rows), parsing!

Data

Downloaded 7878 times by andylolz markbrough matmaxgeds siemvaessen xriss

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (25.4 MB) Use the API

rows 10 / 498

started_at finished_at success
2019-04-24T00:24:50+00:00
false
2019-04-24T01:13:00+00:00
2019-04-24 01:35:48.378022
true
2019-04-25T12:23:10+00:00
2019-04-25 12:46:06.486504
true
2019-05-15T15:36:50+00:00
2019-05-15 15:59:41.450950
true
2019-07-12T18:20:44+00:00
2019-07-12 18:43:33.062638
true
2019-07-13T18:05:44+00:00
2019-07-13 18:28:39.063152
true
2019-07-14T03:44:05+00:00
2019-07-14 04:06:54.002352
true
2019-07-15T11:25:25+00:00
2019-07-15 11:48:05.282473
true
2019-07-16T13:55:44+00:00
2019-07-16 14:18:32.582658
true
2019-07-17T20:03:04+00:00
2019-07-17 20:25:44.737203
true

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (25.4 MB) Use the API

rows 10 / 264008

Date Rate Currency Frequency Source RateFirstSeen
1995-01-02
0.8440
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-03
0.8450
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-04
0.8450
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-05
0.8430
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-06
0.8400
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-09
0.8440
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-10
0.8470
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-11
0.8510
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-12
0.8450
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-13
0.8470
BRL
D
FRED
2019-04-24T00:25:04+00:00

Statistics

Average successful run time: 32 minutes

Total run time: 11 days

Total cpu time used: 9 days

Total disk space used: 25.4 MB

History

  • Auto ran revision aded1370 and completed successfully .
    116 records added in the database
  • Auto ran revision aded1370 and completed successfully .
    1 record added in the database
  • Auto ran revision aded1370 and completed successfully .
    1 record added in the database
  • Auto ran revision aded1370 and completed successfully .
    1 record added in the database
  • Auto ran revision aded1370 and completed successfully .
    1 record added in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

exchangerates-scraper / scraper.py