This is a scraper that runs on Morph. To get started see the documentation

Contributors woodbine blablupcom

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Python app detected  ! The latest version of Python 2 is python-2.7.14 (you are using python-2.7.6, which is unsupported).  ! We recommend upgrading by specifying the latest version (python-2.7.14).  Learn More: https://devcenter.heroku.com/articles/python-runtimes -----> Installing python-2.7.6 -----> Installing pip -----> Installing requirements with pip  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 2))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to revision morph_defaults) to /app/.heroku/src/scraperwiki  Collecting BeautifulSoup==3.2.0 (from -r /tmp/build/requirements.txt (line 9))  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings  SNIMissingWarning  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings  InsecurePlatformWarning  /app/.heroku/python/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings  InsecurePlatformWarning  Downloading https://files.pythonhosted.org/packages/33/fe/15326560884f20d792d3ffc7fe8f639aab88647c9d46509a240d9bfbb6b1/BeautifulSoup-3.2.0.tar.gz  Collecting Creoleparser==0.7.4 (from -r /tmp/build/requirements.txt (line 10))  Downloading https://files.pythonhosted.org/packages/2a/0c/f442415eae7f0bb077fed28f6f8b1ddb07b32f216b2268dca9c267655f0d/Creoleparser-0.7.4.zip  Collecting Genshi==0.6 (from -r /tmp/build/requirements.txt (line 11))  Downloading https://files.pythonhosted.org/packages/3e/ef/64bd33c8acc94839db8e1b1df6110e0efcb495997c8578b68144886cdc82/Genshi-0.6.tar.gz (433kB)  Collecting Jinja2==2.6 (from -r /tmp/build/requirements.txt (line 12))  Downloading https://files.pythonhosted.org/packages/25/c8/212b1c2fd6df9eaf536384b6c6619c4e70a3afd2dffdd00e5296ffbae940/Jinja2-2.6.tar.gz (389kB)  Collecting Markdown==2.2.0 (from -r /tmp/build/requirements.txt (line 13))  Downloading https://files.pythonhosted.org/packages/ac/99/288a81a38526a42c98b5b9832c6e339ca8d5dd38b19a53abfac7c8037c7f/Markdown-2.2.0.tar.gz (236kB)  Collecting Pygments==1.4 (from -r /tmp/build/requirements.txt (line 14))  Downloading https://files.pythonhosted.org/packages/6c/0a/2174e016cf4c799fb30b37d0ab4329c99bc1bf5f949e1c0ec3aa0e5cf2ed/Pygments-1.4.tar.gz (3.5MB)  Collecting SQLAlchemy==0.6.6 (from -r /tmp/build/requirements.txt (line 15))  Downloading https://files.pythonhosted.org/packages/0e/a6/732b93ca53774a7a572477b5debc11ecd556ae685ca5bd2a073560765d18/SQLAlchemy-0.6.6.tar.gz (2.1MB)  Collecting Twisted==11.1.0 (from -r /tmp/build/requirements.txt (line 16))  Downloading https://files.pythonhosted.org/packages/c7/82/c71021c15625960e11b32cdba7c93bf9cdf79b9fe4f0a2dcde3a97ffcad3/Twisted-11.1.0.tar.bz2 (2.8MB)  Collecting Unidecode==0.04.9 (from -r /tmp/build/requirements.txt (line 17))  Downloading https://files.pythonhosted.org/packages/69/8a/de936836c087769b23d5937da47bc141fb302013fc7ca89599c3dd2f2f9f/Unidecode-0.04.9.tar.gz (196kB)  Collecting anyjson==0.3.3 (from -r /tmp/build/requirements.txt (line 18))  Downloading https://files.pythonhosted.org/packages/c3/4d/d4089e1a3dd25b46bebdb55a992b0797cff657b4477bc32ce28038fdecbc/anyjson-0.3.3.tar.gz  Collecting argparse==1.2.1 (from -r /tmp/build/requirements.txt (line 19))  Downloading https://files.pythonhosted.org/packages/6f/ad/86448942ad49c5fe05bfdf7ebc874807f521dfcca5ee543afaca2974ad5a/argparse-1.2.1.tar.gz (69kB)  Collecting beautifulsoup4==4.1.3 (from -r /tmp/build/requirements.txt (line 20))  Downloading https://files.pythonhosted.org/packages/8c/f3/e0e62c314c6f93f306415eec3fb1b665a710dc492879709c027db051db9d/beautifulsoup4-4.1.3.tar.gz (58kB)  Collecting bitlyapi==0.1.1 (from -r /tmp/build/requirements.txt (line 21))  Downloading https://files.pythonhosted.org/packages/2b/71/ae97330c791cb3c08ad479c64ad65fcb749bde5236dbe8ab86f3176d1d30/bitlyapi-0.1.1.tar.gz  Collecting blinker==1.2 (from -r /tmp/build/requirements.txt (line 22))  Downloading https://files.pythonhosted.org/packages/bf/92/b8c23de91e995d0f0245c5ebbae0e8a803bc1811be15921258a15efa1df5/blinker-1.2.tar.gz (66kB)  Collecting cartodb==0.6 (from -r /tmp/build/requirements.txt (line 23))  Downloading https://files.pythonhosted.org/packages/1c/a8/4d142bab9ab56142fbb3ad0e8268fd68213e8214219a649c0a19ad711211/cartodb-0.6.tar.gz  Collecting certifi==0.0.8 (from -r /tmp/build/requirements.txt (line 24))  Downloading https://files.pythonhosted.org/packages/38/70/d777da670969367780cb0cb66f43799e17e050dcdeb0fa4e26189519f9f2/certifi-0.0.8.tar.gz (118kB)  Collecting chardet==2.1.1 (from -r /tmp/build/requirements.txt (line 25))  Downloading https://files.pythonhosted.org/packages/f2/f1/2b5ab854299fe1ea312a9c10dda58421ea24af98a128ad1bff6b87c0c927/chardet-2.1.1.tar.gz (178kB)  Collecting ckanclient==0.10 (from -r /tmp/build/requirements.txt (line 26))  Downloading https://files.pythonhosted.org/packages/52/c3/8c9e69709811039d9e707ceaf9132dd7542eeb41c524ae41e17f89d8ec51/ckanclient-0.10.tar.gz  Collecting colormath==1.0.8 (from -r /tmp/build/requirements.txt (line 27))  Downloading https://files.pythonhosted.org/packages/5d/49/af6e9f9ef10b94be0d12da80895f2e5ae2b83d095c7f41343029d488e6b5/colormath-1.0.8.tar.gz  Collecting csvkit==0.3.0 (from -r /tmp/build/requirements.txt (line 28))  Downloading https://files.pythonhosted.org/packages/b4/2b/0c79ea25083f5400cf059b17d4ff7ff785bffb71a4a50435fa6a0ef46397/csvkit-0.3.0.tar.gz  Collecting dataset==0.5.2 (from -r /tmp/build/requirements.txt (line 29))  Downloading https://files.pythonhosted.org/packages/7a/41/0c27c5053131563d29ba03565edcb3e50f34fb3bee9261a4ccbfe6845d21/dataset-0.5.2.tar.gz  Collecting demjson==1.6 (from -r /tmp/build/requirements.txt (line 30))  Downloading https://files.pythonhosted.org/packages/2a/65/97c43d134641af8fed5d8d3dc3c9d87d445a3693829351a02d2e6cdbf35d/demjson-1.6.tar.gz (64kB)  Collecting dropbox==1.4 (from -r /tmp/build/requirements.txt (line 31))  Downloading https://files.pythonhosted.org/packages/66/b5/2a6255e63ea930c29c3d4f10fff49e601dbc05a9e520426c32f452647eb8/dropbox-1.4.tar.gz  Collecting errorhandler==1.1.1 (from -r /tmp/build/requirements.txt (line 32))  Downloading https://files.pythonhosted.org/packages/62/3a/f80955c4741a3b7fed9c7b621adb6d4997a28c5a1ddbb5a367601c95d1b2/errorhandler-1.1.1.tar.gz  Collecting feedparser==5.0.1 (from -r /tmp/build/requirements.txt (line 33))  Downloading https://files.pythonhosted.org/packages/90/8d/7818ed122854a8b338d6a52dfde1900c248c3c1fba0bf8f09f03bdff40cf/feedparser-5.0.1.tar.bz2 (204kB)  Collecting fluidinfo.py==1.1.2 (from -r /tmp/build/requirements.txt (line 34))  Downloading https://files.pythonhosted.org/packages/29/cb/79b44ba372ffc5be337ba99befd1f26fe20fa2a61e2dbd69a8dd7b0e1176/fluidinfo.py-1.1.2.tar.gz  Collecting gdata==2.0.15 (from -r /tmp/build/requirements.txt (line 35))  Downloading https://files.pythonhosted.org/packages/e3/80/87d95ec4729f46bf320d8b3936fbd4118935eb025ccb993b557a29f860fa/gdata-2.0.15.tar.gz (2.0MB)  Collecting geopy==0.94.1 (from -r /tmp/build/requirements.txt (line 36))  Downloading https://files.pythonhosted.org/packages/94/0d/07852a0560047f5a7aa12bc5a7b0b8c95feb5ab21aa571232ab228267cc3/geopy-0.94.1.tar.gz  Collecting gevent==0.13.6 (from -r /tmp/build/requirements.txt (line 37))  Downloading https://files.pythonhosted.org/packages/14/83/37f998c61406cb765264db8b68a24296e1f40d05a57b18dbfafa0883b5bd/gevent-0.13.6.tar.gz (289kB)  Collecting google-api-python-client==1.0beta8 (from -r /tmp/build/requirements.txt (line 38))  Downloading https://files.pythonhosted.org/packages/80/e4/95916a88cf92211949cea70c56c246afdd0885ad19e2cbf3bc005115b329/google-api-python-client-1.0beta8.tar.gz (348kB)  Collecting googlemaps==1.0.2 (from -r /tmp/build/requirements.txt (line 39))  Downloading https://files.pythonhosted.org/packages/a0/de/b8d19fac34a080e9c4db877db33fd5bbc6a98e19c2d9e70bf01c346a8655/googlemaps-1.0.2.tar.gz (60kB)  Collecting greenlet==0.3.2 (from -r /tmp/build/requirements.txt (line 40))  Downloading https://files.pythonhosted.org/packages/4a/a3/df6960827911eb281a9b86b12785e22c88e7d7df55e68ff4eeef0904a449/greenlet-0.3.2.zip (50kB)  Collecting html5lib==0.90 (from -r /tmp/build/requirements.txt (line 41))  Downloading https://files.pythonhosted.org/packages/5a/a8/2e264a1fc01e6e32c1d7583f22c83b4ac1582b5d7f35214a01960f0bc979/html5lib-0.90.tar.gz (86kB)  Collecting httplib2==0.7.4 (from -r /tmp/build/requirements.txt (line 42))  Downloading https://files.pythonhosted.org/packages/62/89/95df81f893da90744f0086d6841086953c41d25da4950e034dcfdf8cf334/httplib2-0.7.4.tar.gz (106kB)  Collecting imposm.parser==1.0.3 (from -r /tmp/build/requirements.txt (line 43))  Downloading https://files.pythonhosted.org/packages/1e/aa/20c79986749e15bdd6709e54db3945d3e20e135657fec4787e23a77d2c32/imposm.parser-1.0.3.tar.gz  Collecting jellyfish==0.2.0 (from -r /tmp/build/requirements.txt (line 44))  Downloading https://files.pythonhosted.org/packages/f0/d3/c78b33c6dac5d27f2e5aed7a0c25a80a9d615b635e1547d2d2b4a88e83fe/jellyfish-0.2.0.tar.gz  Collecting mechanize==0.2.5 (from -r /tmp/build/requirements.txt (line 45))  Downloading https://files.pythonhosted.org/packages/32/bc/d5b44fe4a3b5079f035240a7c76bd0c71a60c6082f4bfcb1c7585604aa35/mechanize-0.2.5.tar.gz (383kB)  Collecting mock==0.7.2 (from -r /tmp/build/requirements.txt (line 46))  Downloading https://files.pythonhosted.org/packages/6d/7f/3dff8eb00b040fd25235c5aec76d24d17553b36b817662140c50ca63e94f/mock-0.7.2.tar.gz (896kB)  Collecting networkx==1.6 (from -r /tmp/build/requirements.txt (line 47))  Downloading https://files.pythonhosted.org/packages/97/46/9014afb2ef7a450b32269805b736720324c398ae55edbc1824b49073beee/networkx-1.6.tar.gz (707kB)  Collecting ngram==3.3.0 (from -r /tmp/build/requirements.txt (line 48))  Downloading https://files.pythonhosted.org/packages/78/ff/4a7a88047fe50ab9806446488730ab3f74fc277be2357ac46f6f0c9b0227/ngram-3.3.0.tar.gz  Collecting nose==1.1.2 (from -r /tmp/build/requirements.txt (line 49))  Downloading https://files.pythonhosted.org/packages/38/96/7aa1c2583ddec558a230175d6aeddba796cde7191852bf3e6eb3cfb873e1/nose-1.1.2.tar.gz (729kB)  Collecting oauth2==1.5.170 (from -r /tmp/build/requirements.txt (line 50))  Downloading https://files.pythonhosted.org/packages/8b/d2/d9613db75252cee85ff2a5064436931c7f8b751ee46044ba54638ea7de52/oauth2-1.5.170.tar.gz  Collecting oauth==1.0.1 (from -r /tmp/build/requirements.txt (line 51))  Downloading https://files.pythonhosted.org/packages/e2/10/d7d6ae26ef7686109a10b3e88d345c4ec6686d07850f4ef7baefb7eb61e1/oauth-1.0.1.tar.gz  Collecting oauthlib==0.1.2 (from -r /tmp/build/requirements.txt (line 52))  Downloading https://files.pythonhosted.org/packages/86/15/acc8e6170fcaaadf0e2679c69f1acc108cd6a4dbb1f44ed42b173d0fa3cf/oauthlib-0.1.2.tar.gz  Collecting openpyxl==1.5.7 (from -r /tmp/build/requirements.txt (line 53))  Downloading https://files.pythonhosted.org/packages/cf/56/7a1414fafc30066ac1c3018fd5b89e2d1ab969b73890ec7c5ca0471a3c1d/openpyxl-1.5.7.tar.gz (67kB)  Collecting ordereddict==1.1 (from -r /tmp/build/requirements.txt (line 54))  Downloading https://files.pythonhosted.org/packages/53/25/ef88e8e45db141faa9598fbf7ad0062df8f50f881a36ed6a0073e1572126/ordereddict-1.1.tar.gz  Collecting pbkdf2==1.3 (from -r /tmp/build/requirements.txt (line 55))  Downloading https://files.pythonhosted.org/packages/02/c0/6a2376ae81beb82eda645a091684c0b0becb86b972def7849ea9066e3d5e/pbkdf2-1.3.tar.gz  Collecting pdfminer==20110515 (from -r /tmp/build/requirements.txt (line 56))  Downloading https://files.pythonhosted.org/packages/ce/f8/512bcd1a116d0332ab9fab84c3771d4699216db1086e120d581535665c31/pdfminer-20110515.tar.gz (4.1MB)  Collecting pexpect==2.4 (from -r /tmp/build/requirements.txt (line 57))  Downloading https://files.pythonhosted.org/packages/fa/e1/c1f8fce7e7d578ae69aff616cabd5e61b6cb734aade2486b2140853d0f26/pexpect-2.4.tar.gz (113kB)  Collecting pipe2py==0.9.2 (from -r /tmp/build/requirements.txt (line 58))  Downloading https://files.pythonhosted.org/packages/14/d7/f6d94e55e2c267dc0d39ba0e3d3829a20a469ec20a00afa38b91947fb967/pipe2py-0.9.2.tar.gz (57kB)  Collecting pyOpenSSL==0.13 (from -r /tmp/build/requirements.txt (line 59))  Downloading https://files.pythonhosted.org/packages/8b/20/8f4230b281a2a9d0ee9e24fd89aeded0b25d40c84b3d61100a96438e1626/pyOpenSSL-0.13.tar.gz (250kB)  Collecting pycrypto==2.5 (from -r /tmp/build/requirements.txt (line 60))  Downloading https://files.pythonhosted.org/packages/eb/0d/80b7706fa181128f55b34b2ed49bca24e1fecf25101c0364b602cfdd3f6c/pycrypto-2.5.tar.gz (426kB)  Collecting pycurl==7.19.0 (from -r /tmp/build/requirements.txt (line 61))  Downloading https://files.pythonhosted.org/packages/11/73/abcfbbb6e1dd7087fa53042c301c056c11264e8a737a4688f834162d731e/pycurl-7.19.0.tar.gz (70kB)  Collecting pyephem==3.7.5.1 (from -r /tmp/build/requirements.txt (line 62))  Downloading https://files.pythonhosted.org/packages/f9/62/4b486cec967357add6df1f24ef56e5bf0da5bc2110e4b0b3ce7264ce2ad7/pyephem-3.7.5.1.tar.gz (703kB)  Collecting pyparsing==1.5.6 (from -r /tmp/build/requirements.txt (line 63))  Downloading https://files.pythonhosted.org/packages/fa/fa/e063a194dd48b8e76c1ef77bda6be80e8f988dc111b29e5029127d324b72/pyparsing-1.5.6.tar.gz (1.4MB)  Collecting pyth==0.5.6 (from -r /tmp/build/requirements.txt (line 64))  Downloading https://files.pythonhosted.org/packages/9c/fb/489f35bd27074d02333e2e1c3a7ad511c63c56aa00c555ac9399f6637df4/pyth-0.5.6.tar.gz  Collecting python-Levenshtein==0.10.2 (from -r /tmp/build/requirements.txt (line 65))  Downloading https://files.pythonhosted.org/packages/32/3c/46cd4e5b41d46ad309372b9b5de70776aa66d5db02bafb3444782b86a23c/python-Levenshtein-0.10.2.tar.gz (45kB)  Collecting python-dateutil==1.5 (from -r /tmp/build/requirements.txt (line 66))  Downloading https://files.pythonhosted.org/packages/b4/7c/df59c89a753eb33c7c44e1dd42de0e9bc2ccdd5a4d576e0bfad97cc280cb/python-dateutil-1.5.tar.gz (233kB)  Collecting python-gflags==2.0 (from -r /tmp/build/requirements.txt (line 67))  Downloading https://files.pythonhosted.org/packages/46/47/12c17c3216c04a85e5ffd9163ad09f0c1661c2cc2ccc0faf70e39cb8dc96/python-gflags-2.0.tar.gz (65kB)  Collecting python-modargs==1.2 (from -r /tmp/build/requirements.txt (line 68))  Downloading https://files.pythonhosted.org/packages/a1/61/24d8587b069364de03dee98b20f90e0ad2e025ccb1db2ee16b3caf639b0e/python-modargs-1.2.tar.gz  Collecting python-stdnum==0.7 (from -r /tmp/build/requirements.txt (line 69))  Downloading https://files.pythonhosted.org/packages/40/01/c495a308c6fac2ab9419fb1be21165ec18b0ea68b9ac26e099c73ec57b83/python-stdnum-0.7.tar.gz (113kB)  Collecting pytz==2011k (from -r /tmp/build/requirements.txt (line 70))  Downloading https://files.pythonhosted.org/packages/9c/56/3813cd4d4ec4cd8d93388b8934e421122d8a89f19cf1f143a3c7ebc8827c/pytz-2011k.tar.bz2 (166kB)  Collecting rdflib==3.1.0 (from -r /tmp/build/requirements.txt (line 71))  Downloading https://files.pythonhosted.org/packages/30/f0/6c07b9639ed34fb0b5dea1d225864fc1b339d19fb5b06b2836508648db01/rdflib-3.1.0.tar.gz (249kB)  Collecting requests-foauth==0.1.1 (from -r /tmp/build/requirements.txt (line 72))  Downloading https://files.pythonhosted.org/packages/b7/6c/7291fa76577d0eb4530829a041f86d294b45aee6b36bae2b191c5bfd4994/requests-foauth-0.1.1.tar.gz  Collecting requests==1.0.4 (from -r /tmp/build/requirements.txt (line 73))  Downloading https://files.pythonhosted.org/packages/5d/e8/f27e0868b9a49946b3f800722e02b19efebde22ae534276df3e5f6cca41d/requests-1.0.4.tar.gz (336kB)  Collecting selenium==2.5.0 (from -r /tmp/build/requirements.txt (line 74))  Downloading https://files.pythonhosted.org/packages/21/0f/dbc8580df0eb4b2ea451f1901573ae09629e3135dacb70e504b950ec0cad/selenium-2.5.0.tar.gz (2.4MB)  Collecting simplejson==2.2.1 (from -r /tmp/build/requirements.txt (line 75))  Downloading https://files.pythonhosted.org/packages/08/aa/49ce621718cb55f27cc9bc85e38cc552bfb90e281889c155b0a59d2b01ec/simplejson-2.2.1.tar.gz (49kB)  Collecting suds==0.4 (from -r /tmp/build/requirements.txt (line 76))  Downloading https://files.pythonhosted.org/packages/bc/d6/960acce47ee6f096345fe5a7d9be7708135fd1d0713571836f073efc7393/suds-0.4.tar.gz (104kB)  Collecting tweepy==1.7.1 (from -r /tmp/build/requirements.txt (line 77))  Downloading https://files.pythonhosted.org/packages/09/21/2e87597c60fff537ecfff0533b634e1fdb09d5585990308354952a9370a9/tweepy-1.7.1.tar.gz  Collecting tweetstream==1.1.1 (from -r /tmp/build/requirements.txt (line 78))  Downloading https://files.pythonhosted.org/packages/60/a4/30b6d372e6bb0b3290b1012f5f84ee9d5183880d09ebb865e87200a55142/tweetstream-1.1.1.tar.gz  Collecting w3lib==1.0 (from -r /tmp/build/requirements.txt (line 79))  Downloading https://files.pythonhosted.org/packages/a2/07/b2c6767a26d473a7812ba7daa2b334ffbf2caed363b872cda0a5b0651fdd/w3lib-1.0.tar.gz  Collecting xlrd==0.7.1 (from -r /tmp/build/requirements.txt (line 81))  Downloading https://files.pythonhosted.org/packages/a8/b1/bf7e936d9ea1e68c5e3247fcafaa45fe3967282bb419a8c54312be0f64af/xlrd-0.7.1.tar.gz (118kB)  Collecting xlutils==1.4.1 (from -r /tmp/build/requirements.txt (line 82))  Downloading https://files.pythonhosted.org/packages/c2/fa/e264c0a2fdf3db9152b7b37551d4f8a8a432dbf9afef54f68c91da7a0233/xlutils-1.4.1.tar.gz (40kB)  Collecting xlwt==0.7.2 (from -r /tmp/build/requirements.txt (line 83))  Downloading https://files.pythonhosted.org/packages/98/e9/e9c551e993bde8eccfaf4fd3f991990850a3b1d2ddf9ba9e30f495462497/xlwt-0.7.2.tar.gz (114kB)  Collecting xmltodict==0.4 (from -r /tmp/build/requirements.txt (line 84))  Downloading https://files.pythonhosted.org/packages/dd/3b/14f5583a5128d4dde7a14738b554e7449d2f08d710af3fe1a3c49ca4b636/xmltodict-0.4.tar.gz  Collecting zope.interface==3.8.0 (from -r /tmp/build/requirements.txt (line 88))  Downloading https://files.pythonhosted.org/packages/a9/8d/cea179e663f9656f07d09b0b181299a2d8949fb6491ce3c5bc923ca9dd9f/zope.interface-3.8.0.tar.gz (111kB)  Collecting lxml==2.3.3 (from -r /tmp/build/requirements.txt (line 89))  Downloading https://files.pythonhosted.org/packages/09/70/9176a425aa436677dcc4ddd39b78ba598ca55683aefb47c6b7f617fa17cc/lxml-2.3.3.tar.gz (3.1MB)  Collecting chromium-compact-language-detector==0.031415 (from -r /tmp/build/requirements.txt (line 90))  Downloading https://files.pythonhosted.org/packages/90/9b/6754a53f6b622420e511e95d600ac7dbc90e459796009e2502f384666210/chromium_compact_language_detector-0.031415.tar.gz (2.2MB)  Collecting icalendar==3.0.1b1 (from -r /tmp/build/requirements.txt (line 93))  Downloading https://files.pythonhosted.org/packages/b3/1b/2dbc75d7b5b77dcc5e9c36eeae522a859ac3569bd22081bc02fce0e698e5/icalendar-3.0.1b1.tar.gz  Collecting pyquery==1.0 (from -r /tmp/build/requirements.txt (line 96))  Downloading https://files.pythonhosted.org/packages/92/43/4435ff3612477cbabf72d3688c510b5d031358ec5e9a6cef10f8babc4d33/pyquery-1.0.tar.gz  Collecting scrapely==0.9 (from -r /tmp/build/requirements.txt (line 98))  Downloading https://files.pythonhosted.org/packages/1d/16/892d94655015dc5e0dd1f0013e78b5c510a6da6b1f1d92b7b4ec7d43cdbd/scrapely-0.9.tar.gz  Collecting Fom==0.9.8 (from -r /tmp/build/requirements.txt (line 103))  Downloading https://files.pythonhosted.org/packages/52/cf/179866c56774cea643e9a8a046e612222eb6b3ed3f824de220d211c7b3e4/Fom-0.9.8.tar.gz (70kB)  Collecting PyYAML==3.10 (from -r /tmp/build/requirements.txt (line 105))  Downloading https://files.pythonhosted.org/packages/00/17/3b822893a1789a025d3f676a381338516a8f65e686d915b0834ecc9b4979/PyYAML-3.10.tar.gz (241kB)  Collecting Scrapy==0.14.1 (from -r /tmp/build/requirements.txt (line 107))  Downloading https://files.pythonhosted.org/packages/c1/7f/d898f6f3b19a3556c31224d137ec6864144d64d8b6a26a20f4096c3bee67/Scrapy-0.14.1.tar.gz (719kB)  Collecting adspygoogle.adwords==15.6.2 (from -r /tmp/build/requirements.txt (line 109))  Downloading https://files.pythonhosted.org/packages/88/f5/be287bdc6df013c571e2a0d0a4046d174d862cfd31751459efce5457dba3/adspygoogle.adwords-15.6.2.tar.gz (166kB)  Collecting nltk==3.0.2 (from -r /tmp/build/requirements.txt (line 111))  Downloading https://files.pythonhosted.org/packages/06/85/4ac5762ba85980b4250931d80d1d1ea3917de2f13c56d2c270a4b902ecd0/nltk-3.0.2.tar.gz (991kB)  Collecting pydot==1.0.2 (from -r /tmp/build/requirements.txt (line 113))  Downloading https://files.pythonhosted.org/packages/02/ff/cbd177256cfed9d0e6578a40ee74e1609d0532350f3cc8c66912831221dd/pydot-1.0.2.tar.gz  Collecting M2Crypto==0.22.3 (from -r /tmp/build/requirements.txt (line 115))  Downloading https://files.pythonhosted.org/packages/80/d4/09524cdccd88cb9a6ef99a1cf6a4996e2bb48dceb16a23530ca04f59f390/M2Crypto-0.22.3.tar.gz (74kB)  Collecting dumptruck>=0.1.2 (from scraperwiki->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/15/27/3330a343de80d6849545b6c7723f8c9a08b4b104de964ac366e7e6b318df/dumptruck-0.1.6.tar.gz  Collecting alembic>=0.6.2 (from dataset==0.5.2->-r /tmp/build/requirements.txt (line 29))  Downloading https://files.pythonhosted.org/packages/96/c7/a4129db460c3e0ea8fea0c9eb5de6680d38ea6b6dcffcb88898ae42e170a/alembic-1.0.0-py2.py3-none-any.whl (158kB)  Collecting python-slugify>=0.0.6 (from dataset==0.5.2->-r /tmp/build/requirements.txt (line 29))  Downloading https://files.pythonhosted.org/packages/70/c1/98bfb2c981787dcec4613c5da2c17d6f54613935b0e3a877e87a9fa974e4/python-slugify-1.2.5.tar.gz  Collecting distribute (from python-stdnum==0.7->-r /tmp/build/requirements.txt (line 69))  Downloading https://files.pythonhosted.org/packages/5f/ad/1fde06877a8d7d5c9b60eff7de2d452f639916ae1d48f0b8f97bf97e570a/distribute-0.7.3.zip (145kB)  Collecting numpy (from scrapely==0.9->-r /tmp/build/requirements.txt (line 98))  Downloading https://files.pythonhosted.org/packages/8d/2b/940739da20f24af313d6a6342ce9b6b4331a942eb212de42899524ae0378/numpy-1.14.5-cp27-cp27m-manylinux1_x86_64.whl (12.1MB)  Collecting Mako (from alembic>=0.6.2->dataset==0.5.2->-r /tmp/build/requirements.txt (line 29))  Downloading https://files.pythonhosted.org/packages/eb/f3/67579bb486517c0d49547f9697e36582cd19dafb5df9e687ed8e22de57fa/Mako-1.0.7.tar.gz (564kB)  Collecting python-editor>=0.3 (from alembic>=0.6.2->dataset==0.5.2->-r /tmp/build/requirements.txt (line 29))  Downloading https://files.pythonhosted.org/packages/65/1e/adf6e000ea5dc909aa420352d6ba37f16434c8a3c2fa030445411a1ed545/python-editor-1.0.3.tar.gz  Collecting MarkupSafe>=0.9.2 (from Mako->alembic>=0.6.2->dataset==0.5.2->-r /tmp/build/requirements.txt (line 29))  Downloading https://files.pythonhosted.org/packages/4d/de/32d741db316d8fdb7680822dd37001ef7a448255de9699ab4bfcbdf4172b/MarkupSafe-1.0.tar.gz  python-slugify 1.2.5 has requirement Unidecode>=0.04.16, but you'll have unidecode 0.4.9 which is incompatible.  alembic 1.0.0 has requirement SQLAlchemy>=0.9.0, but you'll have sqlalchemy 0.6.6 which is incompatible.  dataset 0.5.2 has requirement sqlalchemy>=0.9.1, but you'll have sqlalchemy 0.6.6 which is incompatible.  Installing collected packages: dumptruck, requests, scraperwiki, BeautifulSoup, Genshi, Creoleparser, Jinja2, Markdown, Pygments, SQLAlchemy, zope.interface, Twisted, Unidecode, anyjson, argparse, beautifulsoup4, bitlyapi, blinker, httplib2, oauth2, cartodb, certifi, chardet, ckanclient, colormath, xlrd, python-dateutil, csvkit, MarkupSafe, Mako, python-editor, alembic, python-slugify, PyYAML, dataset, demjson, oauth, simplejson, dropbox, errorhandler, feedparser, fluidinfo.py, gdata, geopy, greenlet, gevent, python-gflags, google-api-python-client, googlemaps, html5lib, imposm.parser, jellyfish, mechanize, mock, networkx, ngram, nose, pycrypto, oauthlib, openpyxl, ordereddict, pbkdf2, pdfminer, pexpect, pipe2py, pyOpenSSL, pycurl, pyephem, pyparsing, pyth, python-Levenshtein, python-modargs, distribute, python-stdnum, pytz, rdflib, requests-foauth, selenium, suds, tweepy, tweetstream, w3lib, xlwt, xlutils, xmltodict, lxml, chromium-compact-language-detector, icalendar, pyquery, numpy, scrapely, Fom, Scrapy, adspygoogle.adwords, nltk, pydot, M2Crypto  Running setup.py install for dumptruck: started  Running setup.py install for dumptruck: finished with status 'done'  Running setup.py install for requests: started  Running setup.py install for requests: finished with status 'done'  Running setup.py develop for scraperwiki  Running setup.py install for BeautifulSoup: started  Running setup.py install for BeautifulSoup: finished with status 'done'  Running setup.py install for Genshi: started  Running setup.py install for Genshi: finished with status 'done'  Running setup.py install for Creoleparser: started  Running setup.py install for Creoleparser: finished with status 'done'  Running setup.py install for Jinja2: started  Running setup.py install for Jinja2: finished with status 'done'  Running setup.py install for Markdown: started  Running setup.py install for Markdown: finished with status 'done'  Running setup.py install for Pygments: started  Running setup.py install for Pygments: finished with status 'done'  Running setup.py install for SQLAlchemy: started  Running setup.py install for SQLAlchemy: finished with status 'done'  Running setup.py install for zope.interface: started  Running setup.py install for zope.interface: finished with status 'done'  Running setup.py install for Twisted: started  Running setup.py install for Twisted: finished with status 'done'  Running setup.py install for Unidecode: started  Running setup.py install for Unidecode: finished with status 'done'  Running setup.py install for anyjson: started  Running setup.py install for anyjson: finished with status 'done'  Running setup.py install for argparse: started  Running setup.py install for argparse: finished with status 'done'  Running setup.py install for beautifulsoup4: started  Running setup.py install for beautifulsoup4: finished with status 'done'  Running setup.py install for bitlyapi: started  Running setup.py install for bitlyapi: finished with status 'done'  Running setup.py install for blinker: started  Running setup.py install for blinker: finished with status 'done'  Running setup.py install for httplib2: started  Running setup.py install for httplib2: finished with status 'done'  Running setup.py install for oauth2: started  Running setup.py install for oauth2: finished with status 'done'  Running setup.py install for cartodb: started  Running setup.py install for cartodb: finished with status 'done'  Running setup.py install for certifi: started  Running setup.py install for certifi: finished with status 'done'  Running setup.py install for chardet: started  Running setup.py install for chardet: finished with status 'done'  Running setup.py install for ckanclient: started  Running setup.py install for ckanclient: finished with status 'done'  Running setup.py install for colormath: started  Running setup.py install for colormath: finished with status 'done'  Running setup.py install for xlrd: started  Running setup.py install for xlrd: finished with status 'done'  Running setup.py install for python-dateutil: started  Running setup.py install for python-dateutil: finished with status 'done'  Running setup.py install for csvkit: started  Running setup.py install for csvkit: finished with status 'done'  Running setup.py install for MarkupSafe: started  Running setup.py install for MarkupSafe: finished with status 'done'  Running setup.py install for Mako: started  Running setup.py install for Mako: finished with status 'done'  Running setup.py install for python-editor: started  Running setup.py install for python-editor: finished with status 'done'  Running setup.py install for python-slugify: started  Running setup.py install for python-slugify: finished with status 'done'  Running setup.py install for PyYAML: started  Running setup.py install for PyYAML: finished with status 'done'  Running setup.py install for dataset: started  Running setup.py install for dataset: finished with status 'done'  Running setup.py install for demjson: started  Running setup.py install for demjson: finished with status 'done'  Running setup.py install for oauth: started  Running setup.py install for oauth: finished with status 'done'  Running setup.py install for simplejson: started  Running setup.py install for simplejson: finished with status 'done'  Running setup.py install for dropbox: started  Running setup.py install for dropbox: finished with status 'done'  Running setup.py install for errorhandler: started  Running setup.py install for errorhandler: finished with status 'done'  Running setup.py install for feedparser: started  Running setup.py install for feedparser: finished with status 'done'  Running setup.py install for fluidinfo.py: started  Running setup.py install for fluidinfo.py: finished with status 'done'  Running setup.py install for gdata: started  Running setup.py install for gdata: finished with status 'done'  Running setup.py install for geopy: started  Running setup.py install for geopy: finished with status 'done'  Running setup.py install for greenlet: started  Running setup.py install for greenlet: finished with status 'done'  Running setup.py install for gevent: started  Running setup.py install for gevent: finished with status 'done'  Running setup.py install for python-gflags: started  Running setup.py install for python-gflags: finished with status 'done'  Running setup.py install for google-api-python-client: started  Running setup.py install for google-api-python-client: finished with status 'done'  Running setup.py install for googlemaps: started  Running setup.py install for googlemaps: finished with status 'done'  Running setup.py install for html5lib: started  Running setup.py install for html5lib: finished with status 'done'  Running setup.py install for imposm.parser: started  Running setup.py install for imposm.parser: finished with status 'done'  Running setup.py install for jellyfish: started  Running setup.py install for jellyfish: finished with status 'done'  Running setup.py install for mechanize: started  Running setup.py install for mechanize: finished with status 'done'  Running setup.py install for mock: started  Running setup.py install for mock: finished with status 'done'  Running setup.py install for networkx: started  Running setup.py install for networkx: finished with status 'done'  Running setup.py install for ngram: started  Running setup.py install for ngram: finished with status 'done'  Running setup.py install for nose: started  Running setup.py install for nose: finished with status 'done'  Running setup.py install for pycrypto: started  Running setup.py install for pycrypto: finished with status 'done'  Running setup.py install for oauthlib: started  Running setup.py install for oauthlib: finished with status 'done'  Running setup.py install for openpyxl: started  Running setup.py install for openpyxl: finished with status 'done'  Running setup.py install for ordereddict: started  Running setup.py install for ordereddict: finished with status 'done'  Running setup.py install for pbkdf2: started  Running setup.py install for pbkdf2: finished with status 'done'  Running setup.py install for pdfminer: started  Running setup.py install for pdfminer: finished with status 'done'  Running setup.py install for pexpect: started  Running setup.py install for pexpect: finished with status 'done'  Running setup.py install for pipe2py: started  Running setup.py install for pipe2py: finished with status 'done'  Running setup.py install for pyOpenSSL: started  Running setup.py install for pyOpenSSL: finished with status 'done'  Running setup.py install for pycurl: started  Running setup.py install for pycurl: finished with status 'done'  Running setup.py install for pyephem: started  Running setup.py install for pyephem: finished with status 'done'  Running setup.py install for pyparsing: started  Running setup.py install for pyparsing: finished with status 'done'  Running setup.py install for pyth: started  Running setup.py install for pyth: finished with status 'done'  Running setup.py install for python-Levenshtein: started  Running setup.py install for python-Levenshtein: finished with status 'done'  Running setup.py install for python-modargs: started  Running setup.py install for python-modargs: finished with status 'done'  Running setup.py install for distribute: started  Running setup.py install for distribute: finished with status 'done'  Running setup.py install for python-stdnum: started  Running setup.py install for python-stdnum: finished with status 'done'  Running setup.py install for pytz: started  Running setup.py install for pytz: finished with status 'done'  Running setup.py install for rdflib: started  Running setup.py install for rdflib: finished with status 'done'  Running setup.py install for requests-foauth: started  Running setup.py install for requests-foauth: finished with status 'done'  Running setup.py install for selenium: started  Running setup.py install for selenium: finished with status 'done'  Running setup.py install for suds: started  Running setup.py install for suds: finished with status 'done'  Running setup.py install for tweepy: started  Running setup.py install for tweepy: finished with status 'done'  Running setup.py install for tweetstream: started  Running setup.py install for tweetstream: finished with status 'done'  Running setup.py install for w3lib: started  Running setup.py install for w3lib: finished with status 'done'  Running setup.py install for xlwt: started  Running setup.py install for xlwt: finished with status 'done'  Running setup.py install for xlutils: started  Running setup.py install for xlutils: finished with status 'done'  Running setup.py install for xmltodict: started  Running setup.py install for xmltodict: finished with status 'done'  Running setup.py install for lxml: started  Running setup.py install for lxml: finished with status 'done'  Running setup.py install for chromium-compact-language-detector: started  Running setup.py install for chromium-compact-language-detector: finished with status 'done'  Running setup.py install for icalendar: started  Running setup.py install for icalendar: finished with status 'done'  Running setup.py install for pyquery: started  Running setup.py install for pyquery: finished with status 'done'  Running setup.py install for scrapely: started  Running setup.py install for scrapely: finished with status 'done'  Running setup.py install for Fom: started  Running setup.py install for Fom: finished with status 'done'  Running setup.py install for Scrapy: started  Running setup.py install for Scrapy: finished with status 'done'  Running setup.py install for adspygoogle.adwords: started  Running setup.py install for adspygoogle.adwords: finished with status 'done'  Running setup.py install for nltk: started  Running setup.py install for nltk: finished with status 'done'  Running setup.py install for pydot: started  Running setup.py install for pydot: finished with status 'done'  Running setup.py install for M2Crypto: started  Running setup.py install for M2Crypto: finished with status 'done'  Successfully installed BeautifulSoup-3.2.0 Creoleparser-0.7.4 Fom-0.9.8 Genshi-0.6 Jinja2-2.6 M2Crypto-0.22.3 Mako-1.0.7 Markdown-2.2.0 MarkupSafe-1.0 PyYAML-3.10 Pygments-1.4 SQLAlchemy-0.6.6 Scrapy-0.14.1 Twisted-11.1.0 Unidecode-0.4.9 adspygoogle.adwords-15.6.2 alembic-1.0.0 anyjson-0.3.3 argparse-1.2.1 beautifulsoup4-4.1.3 bitlyapi-0.1.1 blinker-1.2 cartodb-0.6 certifi-0.0.8 chardet-2.1.1 chromium-compact-language-detector-0.31415 ckanclient-0.10 colormath-1.0.8 csvkit-0.3.0 dataset-0.5.2 demjson-1.6 distribute-0.7.3 dropbox-1.4 dumptruck-0.1.6 errorhandler-1.1.1 feedparser-5.0.1 fluidinfo.py-1.1.2 gdata-2.0.15 geopy-0.94.1 gevent-0.13.6 google-api-python-client-1.0b8 googlemaps-1.0.2 greenlet-0.3.2 html5lib-0.90 httplib2-0.7.4 icalendar-3.0.1b1 imposm.parser-1.0.3 jellyfish-0.2.0 lxml-2.3.3 mechanize-0.2.5 mock-0.7.2 networkx-1.6 ngram-3.3.0 nltk-3.0.2 nose-1.1.2 numpy-1.14.5 oauth-1.0.1 oauth2-1.5.170 oauthlib-0.1.2 openpyxl-1.5.7 ordereddict-1.1 pbkdf2-1.3 pdfminer-20110515 pexpect-2.4 pipe2py-0.9.2 pyOpenSSL-0.13 pycrypto-2.5 pycurl-7.19.0 pydot-1.0.2 pyephem-3.7.5.1 pyparsing-1.5.6 pyquery-1.0 pyth-0.5.6 python-Levenshtein-0.10.2 python-dateutil-1.5 python-editor-1.0.3 python-gflags-2.0 python-modargs-1.2 python-slugify-1.2.5 python-stdnum-0.7 pytz-2011k rdflib-3.1.0 requests-1.0.4 requests-foauth-0.1.1 scrapely-0.9 scraperwiki selenium-2.5.0 simplejson-2.2.1 suds-0.4 tweepy-1.7.1 tweetstream-1.1.1 w3lib-1.0 xlrd-0.7.1 xlutils-1.4.1 xlwt-0.7.2 xmltodict-0.4 zope.interface-3.8.0   ! Hello! It looks like your application is using an outdated version of Python.  ! This caused the security warning you saw above during the 'pip install' step.  ! We recommend 'python-3.6.2', which you can specify in a 'runtime.txt' file.  ! -- Much Love, Heroku.   ! Hello! Your requirements.txt file contains the distribute package.  ! This library is automatically installed by Heroku and shouldn't be in  ! Your requirements.txt file. This can cause unexpected behavior.  ! -- Much Love, Heroku.  -----> Downloading NLTK corpora…  ! 'nltk.txt' not found, not downloading any corpora  ! Learn more: https://devcenter.heroku.com/articles/python-nltk  -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running...

Data

Downloaded 543 times by SimKennedy MikeRalphson

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (20 KB) Use the API

rows 10 / 48

d l f
2015-12-17 12:24:24.506931
DEC066_DOEACC_gov_2015_09
2016-11-16 05:51:25.936884
DEC066_DOEACC_gov_2016_04
2016-11-16 05:51:26.668421
DEC066_DOEACC_gov_2016_03
2016-11-16 05:51:27.281945
DEC066_DOEACC_gov_2016_02
2016-11-16 05:51:27.782779
DEC066_DOEACC_gov_2016_01
2016-11-16 05:51:28.751865
DEC066_DOEACC_gov_2015_12
2016-11-16 05:51:29.290100
DEC066_DOEACC_gov_2015_11
2016-11-16 05:51:30.330444
DEC066_DOEACC_gov_2015_10
2016-11-16 05:51:30.881571
DEC066_DOEACC_gov_2015_09
2016-11-16 05:51:31.447303
DEC066_DOEACC_gov_2015_08

Statistics

Average successful run time: 1 minute

Total run time: 1 day

Total cpu time used: about 1 hour

Total disk space used: 49.1 KB

History

  • Auto ran revision fd0f8051 and completed successfully .
    nothing changed in the database
  • Auto ran revision fd0f8051 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision fd0f8051 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision fd0f8051 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision fd0f8051 and completed successfully .
    nothing changed in the database
    1 page scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

sp_DEC066_DOEACC_gov / scraper.py