jeffreyliu / archivers-template

Generic template for Archivers projects


Contributors jeffreyliu

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Python app detected -----> Installing python-2.7.9 -----> Noticed cffi. Bootstrapping libffi.  $ pip install -r requirements.txt  Collecting asn1crypto==0.22.0 (from -r /tmp/build/requirements.txt (line 1))  Downloading asn1crypto-0.22.0-py2.py3-none-any.whl (97kB)  Collecting certifi==2017.4.17 (from -r /tmp/build/requirements.txt (line 2))  Downloading certifi-2017.4.17-py2.py3-none-any.whl (375kB)  Collecting cffi==1.10.0 (from -r /tmp/build/requirements.txt (line 3))  Downloading cffi-1.10.0-cp27-cp27m-manylinux1_x86_64.whl (394kB)  Collecting chardet==3.0.3 (from -r /tmp/build/requirements.txt (line 4))  Downloading chardet-3.0.3-py2.py3-none-any.whl (133kB)  Collecting cryptography==1.9 (from -r /tmp/build/requirements.txt (line 5))  Downloading cryptography-1.9.tar.gz (409kB)  Collecting cssselect==0.9.1 (from -r /tmp/build/requirements.txt (line 6))  Downloading cssselect-0.9.1.tar.gz  Collecting DateTime==4.2 (from -r /tmp/build/requirements.txt (line 7))  Downloading DateTime-4.2-py2.py3-none-any.whl (60kB)  Collecting dumptruck==0.1.6 (from -r /tmp/build/requirements.txt (line 8))  Downloading dumptruck-0.1.6.tar.gz  Collecting enum34==1.1.6 (from -r /tmp/build/requirements.txt (line 9))  Downloading enum34-1.1.6-py2-none-any.whl  Collecting idna==2.5 (from -r /tmp/build/requirements.txt (line 10))  Downloading idna-2.5-py2.py3-none-any.whl (55kB)  Collecting ipaddress==1.0.18 (from -r /tmp/build/requirements.txt (line 11))  Downloading ipaddress-1.0.18-py2-none-any.whl  Collecting lxml==3.4.4 (from -r /tmp/build/requirements.txt (line 12))  Downloading lxml-3.4.4.tar.gz (3.5MB)  Collecting pycparser==2.17 (from -r /tmp/build/requirements.txt (line 13))  Downloading pycparser-2.17.tar.gz (231kB)  Collecting pyOpenSSL==17.0.0 (from -r /tmp/build/requirements.txt (line 14))  Downloading pyOpenSSL-17.0.0-py2.py3-none-any.whl (51kB)  Collecting pytz==2017.2 (from -r /tmp/build/requirements.txt (line 15))  Downloading pytz-2017.2-py2.py3-none-any.whl (484kB)  Collecting requests==2.17.3 (from -r /tmp/build/requirements.txt (line 16))  Downloading requests-2.17.3-py2.py3-none-any.whl (87kB)  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@732dda1982a3b2073f6341a6a24f9df1bda77fa0#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 17))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to 732dda1982a3b2073f6341a6a24f9df1bda77fa0) to /app/.heroku/src/scraperwiki  Could not find a tag or branch '732dda1982a3b2073f6341a6a24f9df1bda77fa0', assuming commit.  Collecting six==1.10.0 (from -r /tmp/build/requirements.txt (line 18))  Downloading six-1.10.0-py2.py3-none-any.whl  Collecting urllib3==1.21.1 (from -r /tmp/build/requirements.txt (line 19))  Downloading urllib3-1.21.1-py2.py3-none-any.whl (131kB)  Collecting zope.interface==4.4.1 (from -r /tmp/build/requirements.txt (line 20))  Downloading zope.interface-4.4.1-cp27-cp27m-manylinux1_x86_64.whl (169kB)  Installing collected packages: asn1crypto, certifi, pycparser, cffi, chardet, idna, six, enum34, ipaddress, cryptography, cssselect, pytz, zope.interface, DateTime, dumptruck, lxml, pyOpenSSL, urllib3, requests, scraperwiki  Running setup.py install for pycparser: started  Running setup.py install for pycparser: finished with status 'done'  Running setup.py install for cryptography: started  Running setup.py install for cryptography: finished with status 'done'  Running setup.py install for cssselect: started  Running setup.py install for cssselect: finished with status 'done'  Running setup.py install for dumptruck: started  Running setup.py install for dumptruck: finished with status 'done'  Running setup.py install for lxml: started  Running setup.py install for lxml: still running...  Running setup.py install for lxml: finished with status 'done'  Running setup.py develop for scraperwiki  Successfully installed DateTime-4.2 asn1crypto-0.22.0 certifi-2017.4.17 cffi-1.10.0 chardet-3.0.3 cryptography-1.9 cssselect-0.9.1 dumptruck-0.1.6 enum34-1.1.6 idna-2.5 ipaddress-1.0.18 lxml-3.4.4 pyOpenSSL-17.0.0 pycparser-2.17 pytz-2017.2 requests-2.17.3 scraperwiki six-1.10.0 urllib3-1.21.1 zope.interface-4.4.1   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running...

Data

Downloaded 0 times

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (7 KB) Use the API

rows 2 / 2

run_id url UUID timestamp body_content body_SHA256 headers
1
0000
2017-06-07 21:13:30.107072
<!doctype html> <html> <head> <title>Example Domain</title> <meta charset="utf-8" /> <meta http-equiv="Content-type" content="text/html; charset=utf-8" /> <meta name="viewport" content="width=device-width, initial-scale=1" /> <style type="text/css"> body { background-color: #f0f0f2; margin: 0; padding: 0; font-family: "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif; } div { width: 600px; margin: 5em auto; padding: 50px; background-color: #fff; border-radius: 1em; } a:link, a:visited { color: #38488f; text-decoration: none; } @media (max-width: 700px) { body { background-color: #fff; } div { width: auto; margin: 0 auto; border-radius: 0; padding: 1em; } } </style> </head> <body> <div> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples in documents. You may use this domain in examples without prior coordination or asking for permission.</p> <p><a href="http://www.iana.org/domains/example">More information...</a></p> </div> </body> </html>
3587cb776ce0e4e8237f215800b7dffba0f25865cb84550e87ea8bbac838c423
{"Accept-Ranges": "bytes", "Content-Encoding": "gzip", "Etag": "\"359670651+gzip\"", "Date": "Wed, 07 Jun 2017 21:13:30 GMT", "Expires": "Wed, 14 Jun 2017 21:13:30 GMT", "Content-Length": "606", "Server": "ECS (rhv/818F)", "Vary": "Accept-Encoding", "Cache-Control": "max-age=604800", "X-Cache": "HIT", "Content-Type": "text/html", "Last-Modified": "Fri, 09 Aug 2013 23:54:35 GMT"}
2
0000
2017-06-08 22:00:19.666165
<!doctype html> <html> <head> <title>Example Domain</title> <meta charset="utf-8" /> <meta http-equiv="Content-type" content="text/html; charset=utf-8" /> <meta name="viewport" content="width=device-width, initial-scale=1" /> <style type="text/css"> body { background-color: #f0f0f2; margin: 0; padding: 0; font-family: "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif; } div { width: 600px; margin: 5em auto; padding: 50px; background-color: #fff; border-radius: 1em; } a:link, a:visited { color: #38488f; text-decoration: none; } @media (max-width: 700px) { body { background-color: #fff; } div { width: auto; margin: 0 auto; border-radius: 0; padding: 1em; } } </style> </head> <body> <div> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples in documents. You may use this domain in examples without prior coordination or asking for permission.</p> <p><a href="http://www.iana.org/domains/example">More information...</a></p> </div> </body> </html>
3587cb776ce0e4e8237f215800b7dffba0f25865cb84550e87ea8bbac838c423
{"Expires": "Thu, 15 Jun 2017 22:00:19 GMT", "Content-Type": "text/html", "Content-Encoding": "gzip", "Date": "Thu, 08 Jun 2017 22:00:19 GMT", "Etag": "\"359670651+gzip\"", "Accept-Ranges": "bytes", "Vary": "Accept-Encoding", "Server": "ECS (rhv/818F)", "Last-Modified": "Fri, 09 Aug 2013 23:54:35 GMT", "Cache-Control": "max-age=604800", "Content-Length": "606", "X-Cache": "HIT"}

Statistics

Average successful run time: 3 minutes

Total run time: 5 minutes

Total cpu time used: less than 5 seconds

Total disk space used: 30.9 KB

History

  • Manually ran revision c014fad7 and completed successfully .
    1 record added, 1 record updated in the database
  • Manually ran revision c014fad7 and completed successfully .
    2 records added in the database
  • Created on morph.io

Scraper code

archivers-template