markbrough / exchangerates-scraper

Scraper to collect and provide API for historical exchangerates data on morph.io


Exchange rates scraper

Scraper to collect and provide API for historical exchangerates data on morph.io

This simple scraper uses the exchangerates pypi package to generate a rolling set of exchangerates data, sourced from FRED and the OECD. For more information, see the exchangerates repository.

Contributors markbrough

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Python app detected -----> Installing python-3.6.2 -----> Installing pip -----> Installing requirements with pip  Collecting urllib3==1.24.2  Downloading urllib3-1.24.2-py2.py3-none-any.whl (131 kB)  Collecting exchangerates==0.3.1  Downloading exchangerates-0.3.1-py3-none-any.whl (6.6 kB)  Collecting requests==2.22.0  Downloading requests-2.22.0-py2.py3-none-any.whl (57 kB)  Collecting scraperwiki==0.5.1  Downloading scraperwiki-0.5.1.tar.gz (7.7 kB)  Collecting lxml==4.4.0  Downloading lxml-4.4.0-cp36-cp36m-manylinux1_x86_64.whl (5.7 MB)  Collecting six==1.12.0  Downloading six-1.12.0-py2.py3-none-any.whl (10 kB)  Collecting idna<2.9,>=2.5  Downloading idna-2.8-py2.py3-none-any.whl (58 kB)  Collecting chardet<3.1.0,>=3.0.2  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)  Collecting certifi>=2017.4.17  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)  Collecting sqlalchemy  Downloading SQLAlchemy-1.3.18-cp36-cp36m-manylinux2010_x86_64.whl (1.2 MB)  Collecting alembic  Downloading alembic-1.4.2.tar.gz (1.1 MB)  Installing build dependencies: started  Installing build dependencies: finished with status 'done'  Getting requirements to build wheel: started  Getting requirements to build wheel: finished with status 'done'  Preparing wheel metadata: started  Preparing wheel metadata: finished with status 'done'  Collecting python-editor>=0.3  Downloading python_editor-1.0.4-py3-none-any.whl (4.9 kB)  Collecting Mako  Downloading Mako-1.1.3-py2.py3-none-any.whl (75 kB)  Collecting python-dateutil  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)  Collecting MarkupSafe>=0.9.2  Downloading MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl (27 kB)  Building wheels for collected packages: scraperwiki, alembic  Building wheel for scraperwiki (setup.py): started  Building wheel for scraperwiki (setup.py): finished with status 'done'  Created wheel for scraperwiki: filename=scraperwiki-0.5.1-py3-none-any.whl size=6543 sha256=cefff4817c50d8908525844454ef717df4b59cfc16894fef1630c02de6382732  Stored in directory: /tmp/pip-ephem-wheel-cache-m5ipccgx/wheels/cd/f8/ac/cd66eb1c557ab40d35c1ed852da3e9b37baa3e21b61906a5cf  Building wheel for alembic (PEP 517): started  Building wheel for alembic (PEP 517): finished with status 'done'  Created wheel for alembic: filename=alembic-1.4.2-py2.py3-none-any.whl size=159543 sha256=f1dade0fa94d3e3d016e8a8bf2a838a5496bf732c1f5bad9def66008d680ccdd  Stored in directory: /tmp/pip-ephem-wheel-cache-m5ipccgx/wheels/f2/50/61/5cc491b0ca39be60dfb4dce940b389ff91b847d62e0eb2d680  Successfully built scraperwiki alembic  Installing collected packages: urllib3, idna, chardet, certifi, requests, lxml, six, exchangerates, sqlalchemy, python-editor, MarkupSafe, Mako, python-dateutil, alembic, scraperwiki  Successfully installed Mako-1.1.3 MarkupSafe-1.1.1 alembic-1.4.2 certifi-2020.6.20 chardet-3.0.4 exchangerates-0.3.1 idna-2.8 lxml-4.4.0 python-dateutil-2.8.1 python-editor-1.0.4 requests-2.22.0 scraperwiki-0.5.1 six-1.12.0 sqlalchemy-1.3.18 urllib3-1.24.2   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Retrieving rates from FRED for DEXBZUS Getting data for currency BRL Retrieving rates from FRED for DEXCAUS Getting data for currency CAD Retrieving rates from FRED for DEXCHUS Getting data for currency CNY Retrieving rates from FRED for DEXDNUS Getting data for currency DKK Retrieving rates from FRED for DEXHKUS Getting data for currency HKD Retrieving rates from FRED for DEXINUS Getting data for currency INR Retrieving rates from FRED for DEXJPUS Getting data for currency JPY Retrieving rates from FRED for DEXKOUS Getting data for currency KRW Retrieving rates from FRED for DEXMAUS Getting data for currency MYR Retrieving rates from FRED for DEXMXUS Getting data for currency MXN Retrieving rates from FRED for DEXNOUS Getting data for currency NOK Retrieving rates from FRED for DEXSDUS Getting data for currency SEK Retrieving rates from FRED for DEXSFUS Getting data for currency ZAR Retrieving rates from FRED for DEXSIUS Getting data for currency SGD Retrieving rates from FRED for DEXSLUS Getting data for currency LKR Retrieving rates from FRED for DEXSZUS Getting data for currency CHF Retrieving rates from FRED for DEXTAUS Getting data for currency TWD Retrieving rates from FRED for DEXTHUS Getting data for currency THB Retrieving rates from FRED for DEXUSAL Getting data for currency AUD Retrieving rates from FRED for DEXUSEU Getting data for currency EUR Retrieving rates from FRED for DEXUSNZ Getting data for currency NZD Retrieving rates from FRED for DEXUSUK Getting data for currency GBP Retrieving rates from FRED for DEXVZUS Getting data for currency VEF Getting rates from OECD Downloaded data, loading existing DB data! Loaded existing data (262150 rows), parsing! Retrieving rates from FRED for DEXBZUS Getting data for currency BRL Retrieving rates from FRED for DEXCAUS Getting data for currency CAD Retrieving rates from FRED for DEXCHUS Getting data for currency CNY Retrieving rates from FRED for DEXDNUS Getting data for currency DKK Retrieving rates from FRED for DEXHKUS Getting data for currency HKD Retrieving rates from FRED for DEXINUS Getting data for currency INR Retrieving rates from FRED for DEXJPUS Getting data for currency JPY Retrieving rates from FRED for DEXKOUS Getting data for currency KRW Retrieving rates from FRED for DEXMAUS Getting data for currency MYR Retrieving rates from FRED for DEXMXUS Getting data for currency MXN Retrieving rates from FRED for DEXNOUS Getting data for currency NOK Retrieving rates from FRED for DEXSDUS Getting data for currency SEK Retrieving rates from FRED for DEXSFUS Getting data for currency ZAR Retrieving rates from FRED for DEXSIUS Getting data for currency SGD Retrieving rates from FRED for DEXSLUS Getting data for currency LKR Retrieving rates from FRED for DEXSZUS Getting data for currency CHF Retrieving rates from FRED for DEXTAUS Getting data for currency TWD Retrieving rates from FRED for DEXTHUS Getting data for currency THB Retrieving rates from FRED for DEXUSAL Getting data for currency AUD Retrieving rates from FRED for DEXUSEU Getting data for currency EUR Retrieving rates from FRED for DEXUSNZ Getting data for currency NZD Retrieving rates from FRED for DEXUSUK Getting data for currency GBP Retrieving rates from FRED for DEXVZUS Getting data for currency VEF Getting rates from OECD Downloaded data, loading existing DB data! Loaded existing data (262150 rows), parsing! Retrieving rates from FRED for DEXBZUS Getting data for currency BRL Retrieving rates from FRED for DEXCAUS Getting data for currency CAD Retrieving rates from FRED for DEXCHUS Getting data for currency CNY Retrieving rates from FRED for DEXDNUS Getting data for currency DKK Retrieving rates from FRED for DEXHKUS Getting data for currency HKD Retrieving rates from FRED for DEXINUS Getting data for currency INR Retrieving rates from FRED for DEXJPUS Getting data for currency JPY Retrieving rates from FRED for DEXKOUS Getting data for currency KRW Retrieving rates from FRED for DEXMAUS Getting data for currency MYR Retrieving rates from FRED for DEXMXUS Getting data for currency MXN Retrieving rates from FRED for DEXNOUS Getting data for currency NOK Retrieving rates from FRED for DEXSDUS Getting data for currency SEK Retrieving rates from FRED for DEXSFUS Getting data for currency ZAR Retrieving rates from FRED for DEXSIUS Getting data for currency SGD Retrieving rates from FRED for DEXSLUS Getting data for currency LKR Retrieving rates from FRED for DEXSZUS Getting data for currency CHF Retrieving rates from FRED for DEXTAUS Getting data for currency TWD Retrieving rates from FRED for DEXTHUS Getting data for currency THB Retrieving rates from FRED for DEXUSAL Getting data for currency AUD Retrieving rates from FRED for DEXUSEU Getting data for currency EUR Retrieving rates from FRED for DEXUSNZ Getting data for currency NZD Retrieving rates from FRED for DEXUSUK Getting data for currency GBP Retrieving rates from FRED for DEXVZUS Getting data for currency VEF Getting rates from OECD Downloaded data, loading existing DB data! Loaded existing data (262150 rows), parsing! Retrieving rates from FRED for DEXBZUS Getting data for currency BRL Retrieving rates from FRED for DEXCAUS Getting data for currency CAD Retrieving rates from FRED for DEXCHUS Getting data for currency CNY Retrieving rates from FRED for DEXDNUS Getting data for currency DKK Retrieving rates from FRED for DEXHKUS Getting data for currency HKD Retrieving rates from FRED for DEXINUS Getting data for currency INR Retrieving rates from FRED for DEXJPUS Getting data for currency JPY Retrieving rates from FRED for DEXKOUS Getting data for currency KRW Retrieving rates from FRED for DEXMAUS Getting data for currency MYR Retrieving rates from FRED for DEXMXUS Getting data for currency MXN Retrieving rates from FRED for DEXNOUS Getting data for currency NOK Retrieving rates from FRED for DEXSDUS Getting data for currency SEK Retrieving rates from FRED for DEXSFUS Getting data for currency ZAR Retrieving rates from FRED for DEXSIUS Getting data for currency SGD Retrieving rates from FRED for DEXSLUS Getting data for currency LKR Retrieving rates from FRED for DEXSZUS Getting data for currency CHF Retrieving rates from FRED for DEXTAUS Getting data for currency TWD Retrieving rates from FRED for DEXTHUS Getting data for currency THB Retrieving rates from FRED for DEXUSAL Getting data for currency AUD Retrieving rates from FRED for DEXUSEU Getting data for currency EUR Retrieving rates from FRED for DEXUSNZ Getting data for currency NZD Retrieving rates from FRED for DEXUSUK Getting data for currency GBP Retrieving rates from FRED for DEXVZUS Getting data for currency VEF Getting rates from OECD Downloaded data, loading existing DB data! Loaded existing data (262150 rows), parsing!

Data

Downloaded 5525 times by andylolz markbrough matmaxgeds siemvaessen xriss

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (25.2 MB) Use the API

rows 10 / 391

started_at finished_at success
2019-04-24T00:24:50+00:00
false
2019-04-24T01:13:00+00:00
2019-04-24 01:35:48.378022
true
2019-04-25T12:23:10+00:00
2019-04-25 12:46:06.486504
true
2019-05-15T15:36:50+00:00
2019-05-15 15:59:41.450950
true
2019-07-12T18:20:44+00:00
2019-07-12 18:43:33.062638
true
2019-07-13T18:05:44+00:00
2019-07-13 18:28:39.063152
true
2019-07-14T03:44:05+00:00
2019-07-14 04:06:54.002352
true
2019-07-15T11:25:25+00:00
2019-07-15 11:48:05.282473
true
2019-07-16T13:55:44+00:00
2019-07-16 14:18:32.582658
true
2019-07-17T20:03:04+00:00
2019-07-17 20:25:44.737203
true

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (25.2 MB) Use the API

rows 10 / 262150

Date Rate Currency Frequency Source RateFirstSeen
1995-01-02
0.8440
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-03
0.8450
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-04
0.8450
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-05
0.8430
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-06
0.8400
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-09
0.8440
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-10
0.8470
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-11
0.8510
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-12
0.8450
BRL
D
FRED
2019-04-24T00:25:04+00:00
1995-01-13
0.8470
BRL
D
FRED
2019-04-24T00:25:04+00:00

Statistics

Average successful run time: 30 minutes

Total run time: 8 days

Total cpu time used: 7 days

Total disk space used: 25.2 MB

History

  • Auto ran revision aded1370 and completed successfully .
    1 record added in the database
  • Auto ran revision aded1370 and completed successfully .
    1 record added in the database
  • Auto ran revision aded1370 and completed successfully .
    1 record added in the database
  • Auto ran revision aded1370 and completed successfully .
    1 record added in the database
  • Auto ran revision aded1370 and completed successfully .
    1 record added in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

exchangerates-scraper / scraper.py