This is a scraper that runs on Morph. To get started see the documentation

Contributors woodbine blablupcom

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Python app detected -----> Installing python-2.7.14 -----> Installing pip -----> Installing requirements with pip  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 2))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to revision morph_defaults) to /app/.heroku/src/scraperwiki  Collecting BeautifulSoup==3.2.0 (from -r /tmp/build/requirements.txt (line 9))  Downloading https://files.pythonhosted.org/packages/33/fe/15326560884f20d792d3ffc7fe8f639aab88647c9d46509a240d9bfbb6b1/BeautifulSoup-3.2.0.tar.gz  Collecting Creoleparser==0.7.4 (from -r /tmp/build/requirements.txt (line 10))  Downloading https://files.pythonhosted.org/packages/2a/0c/f442415eae7f0bb077fed28f6f8b1ddb07b32f216b2268dca9c267655f0d/Creoleparser-0.7.4.zip  Collecting Genshi==0.6 (from -r /tmp/build/requirements.txt (line 11))  Downloading https://files.pythonhosted.org/packages/3e/ef/64bd33c8acc94839db8e1b1df6110e0efcb495997c8578b68144886cdc82/Genshi-0.6.tar.gz (433kB)  Collecting Jinja2==2.6 (from -r /tmp/build/requirements.txt (line 12))  Downloading https://files.pythonhosted.org/packages/25/c8/212b1c2fd6df9eaf536384b6c6619c4e70a3afd2dffdd00e5296ffbae940/Jinja2-2.6.tar.gz (389kB)  Collecting Markdown==2.2.0 (from -r /tmp/build/requirements.txt (line 13))  Downloading https://files.pythonhosted.org/packages/ac/99/288a81a38526a42c98b5b9832c6e339ca8d5dd38b19a53abfac7c8037c7f/Markdown-2.2.0.tar.gz (236kB)  Collecting Pygments==1.4 (from -r /tmp/build/requirements.txt (line 14))  Downloading https://files.pythonhosted.org/packages/6c/0a/2174e016cf4c799fb30b37d0ab4329c99bc1bf5f949e1c0ec3aa0e5cf2ed/Pygments-1.4.tar.gz (3.5MB)  Collecting SQLAlchemy==0.6.6 (from -r /tmp/build/requirements.txt (line 15))  Downloading https://files.pythonhosted.org/packages/0e/a6/732b93ca53774a7a572477b5debc11ecd556ae685ca5bd2a073560765d18/SQLAlchemy-0.6.6.tar.gz (2.1MB)  Collecting Twisted==11.1.0 (from -r /tmp/build/requirements.txt (line 16))  Downloading https://files.pythonhosted.org/packages/c7/82/c71021c15625960e11b32cdba7c93bf9cdf79b9fe4f0a2dcde3a97ffcad3/Twisted-11.1.0.tar.bz2 (2.8MB)  Collecting Unidecode==0.04.9 (from -r /tmp/build/requirements.txt (line 17))  Downloading https://files.pythonhosted.org/packages/69/8a/de936836c087769b23d5937da47bc141fb302013fc7ca89599c3dd2f2f9f/Unidecode-0.04.9.tar.gz (196kB)  Collecting anyjson==0.3.3 (from -r /tmp/build/requirements.txt (line 18))  Downloading https://files.pythonhosted.org/packages/c3/4d/d4089e1a3dd25b46bebdb55a992b0797cff657b4477bc32ce28038fdecbc/anyjson-0.3.3.tar.gz  Collecting argparse==1.2.1 (from -r /tmp/build/requirements.txt (line 19))  Downloading https://files.pythonhosted.org/packages/6f/ad/86448942ad49c5fe05bfdf7ebc874807f521dfcca5ee543afaca2974ad5a/argparse-1.2.1.tar.gz (69kB)  Collecting beautifulsoup4==4.1.3 (from -r /tmp/build/requirements.txt (line 20))  Downloading https://files.pythonhosted.org/packages/8c/f3/e0e62c314c6f93f306415eec3fb1b665a710dc492879709c027db051db9d/beautifulsoup4-4.1.3.tar.gz (58kB)  Collecting bitlyapi==0.1.1 (from -r /tmp/build/requirements.txt (line 21))  Downloading https://files.pythonhosted.org/packages/2b/71/ae97330c791cb3c08ad479c64ad65fcb749bde5236dbe8ab86f3176d1d30/bitlyapi-0.1.1.tar.gz  Collecting blinker==1.2 (from -r /tmp/build/requirements.txt (line 22))  Downloading https://files.pythonhosted.org/packages/bf/92/b8c23de91e995d0f0245c5ebbae0e8a803bc1811be15921258a15efa1df5/blinker-1.2.tar.gz (66kB)  Collecting cartodb==0.6 (from -r /tmp/build/requirements.txt (line 23))  Downloading https://files.pythonhosted.org/packages/1c/a8/4d142bab9ab56142fbb3ad0e8268fd68213e8214219a649c0a19ad711211/cartodb-0.6.tar.gz  Collecting certifi==0.0.8 (from -r /tmp/build/requirements.txt (line 24))  Downloading https://files.pythonhosted.org/packages/38/70/d777da670969367780cb0cb66f43799e17e050dcdeb0fa4e26189519f9f2/certifi-0.0.8.tar.gz (118kB)  Collecting chardet==2.1.1 (from -r /tmp/build/requirements.txt (line 25))  Downloading https://files.pythonhosted.org/packages/f2/f1/2b5ab854299fe1ea312a9c10dda58421ea24af98a128ad1bff6b87c0c927/chardet-2.1.1.tar.gz (178kB)  Collecting ckanclient==0.10 (from -r /tmp/build/requirements.txt (line 26))  Downloading https://files.pythonhosted.org/packages/52/c3/8c9e69709811039d9e707ceaf9132dd7542eeb41c524ae41e17f89d8ec51/ckanclient-0.10.tar.gz  Collecting colormath==1.0.8 (from -r /tmp/build/requirements.txt (line 27))  Downloading https://files.pythonhosted.org/packages/5d/49/af6e9f9ef10b94be0d12da80895f2e5ae2b83d095c7f41343029d488e6b5/colormath-1.0.8.tar.gz  Collecting csvkit==0.3.0 (from -r /tmp/build/requirements.txt (line 28))  Downloading https://files.pythonhosted.org/packages/b4/2b/0c79ea25083f5400cf059b17d4ff7ff785bffb71a4a50435fa6a0ef46397/csvkit-0.3.0.tar.gz  Collecting dataset==0.5.2 (from -r /tmp/build/requirements.txt (line 29))  Downloading https://files.pythonhosted.org/packages/7a/41/0c27c5053131563d29ba03565edcb3e50f34fb3bee9261a4ccbfe6845d21/dataset-0.5.2.tar.gz  Collecting demjson==1.6 (from -r /tmp/build/requirements.txt (line 30))  Downloading https://files.pythonhosted.org/packages/2a/65/97c43d134641af8fed5d8d3dc3c9d87d445a3693829351a02d2e6cdbf35d/demjson-1.6.tar.gz (64kB)  Collecting dropbox==1.4 (from -r /tmp/build/requirements.txt (line 31))  Downloading https://files.pythonhosted.org/packages/66/b5/2a6255e63ea930c29c3d4f10fff49e601dbc05a9e520426c32f452647eb8/dropbox-1.4.tar.gz  Collecting errorhandler==1.1.1 (from -r /tmp/build/requirements.txt (line 32))  Downloading https://files.pythonhosted.org/packages/62/3a/f80955c4741a3b7fed9c7b621adb6d4997a28c5a1ddbb5a367601c95d1b2/errorhandler-1.1.1.tar.gz  Collecting feedparser==5.0.1 (from -r /tmp/build/requirements.txt (line 33))  Downloading https://files.pythonhosted.org/packages/90/8d/7818ed122854a8b338d6a52dfde1900c248c3c1fba0bf8f09f03bdff40cf/feedparser-5.0.1.tar.bz2 (204kB)  Collecting fluidinfo.py==1.1.2 (from -r /tmp/build/requirements.txt (line 34))  Downloading https://files.pythonhosted.org/packages/29/cb/79b44ba372ffc5be337ba99befd1f26fe20fa2a61e2dbd69a8dd7b0e1176/fluidinfo.py-1.1.2.tar.gz  Collecting gdata==2.0.15 (from -r /tmp/build/requirements.txt (line 35))  Downloading https://files.pythonhosted.org/packages/e3/80/87d95ec4729f46bf320d8b3936fbd4118935eb025ccb993b557a29f860fa/gdata-2.0.15.tar.gz (2.0MB)  Collecting geopy==0.94.1 (from -r /tmp/build/requirements.txt (line 36))  Downloading https://files.pythonhosted.org/packages/94/0d/07852a0560047f5a7aa12bc5a7b0b8c95feb5ab21aa571232ab228267cc3/geopy-0.94.1.tar.gz  Collecting gevent==0.13.6 (from -r /tmp/build/requirements.txt (line 37))  Downloading https://files.pythonhosted.org/packages/14/83/37f998c61406cb765264db8b68a24296e1f40d05a57b18dbfafa0883b5bd/gevent-0.13.6.tar.gz (289kB)  Collecting google-api-python-client==1.0beta8 (from -r /tmp/build/requirements.txt (line 38))  Downloading https://files.pythonhosted.org/packages/80/e4/95916a88cf92211949cea70c56c246afdd0885ad19e2cbf3bc005115b329/google-api-python-client-1.0beta8.tar.gz (348kB)  Collecting googlemaps==1.0.2 (from -r /tmp/build/requirements.txt (line 39))  Downloading https://files.pythonhosted.org/packages/a0/de/b8d19fac34a080e9c4db877db33fd5bbc6a98e19c2d9e70bf01c346a8655/googlemaps-1.0.2.tar.gz (60kB)  Collecting greenlet==0.3.2 (from -r /tmp/build/requirements.txt (line 40))  Downloading https://files.pythonhosted.org/packages/4a/a3/df6960827911eb281a9b86b12785e22c88e7d7df55e68ff4eeef0904a449/greenlet-0.3.2.zip (50kB)  Collecting html5lib==0.90 (from -r /tmp/build/requirements.txt (line 41))  Downloading https://files.pythonhosted.org/packages/5a/a8/2e264a1fc01e6e32c1d7583f22c83b4ac1582b5d7f35214a01960f0bc979/html5lib-0.90.tar.gz (86kB)  Collecting httplib2==0.7.4 (from -r /tmp/build/requirements.txt (line 42))  Downloading https://files.pythonhosted.org/packages/62/89/95df81f893da90744f0086d6841086953c41d25da4950e034dcfdf8cf334/httplib2-0.7.4.tar.gz (106kB)  Collecting imposm.parser==1.0.3 (from -r /tmp/build/requirements.txt (line 43))  Downloading https://files.pythonhosted.org/packages/1e/aa/20c79986749e15bdd6709e54db3945d3e20e135657fec4787e23a77d2c32/imposm.parser-1.0.3.tar.gz  Collecting jellyfish==0.2.0 (from -r /tmp/build/requirements.txt (line 44))  Downloading https://files.pythonhosted.org/packages/f0/d3/c78b33c6dac5d27f2e5aed7a0c25a80a9d615b635e1547d2d2b4a88e83fe/jellyfish-0.2.0.tar.gz  Collecting mechanize==0.2.5 (from -r /tmp/build/requirements.txt (line 45))  Downloading https://files.pythonhosted.org/packages/32/bc/d5b44fe4a3b5079f035240a7c76bd0c71a60c6082f4bfcb1c7585604aa35/mechanize-0.2.5.tar.gz (383kB)  Collecting mock==0.7.2 (from -r /tmp/build/requirements.txt (line 46))  Downloading https://files.pythonhosted.org/packages/6d/7f/3dff8eb00b040fd25235c5aec76d24d17553b36b817662140c50ca63e94f/mock-0.7.2.tar.gz (896kB)  Collecting networkx==1.6 (from -r /tmp/build/requirements.txt (line 47))  Downloading https://files.pythonhosted.org/packages/97/46/9014afb2ef7a450b32269805b736720324c398ae55edbc1824b49073beee/networkx-1.6.tar.gz (707kB)  Collecting ngram==3.3.0 (from -r /tmp/build/requirements.txt (line 48))  Downloading https://files.pythonhosted.org/packages/78/ff/4a7a88047fe50ab9806446488730ab3f74fc277be2357ac46f6f0c9b0227/ngram-3.3.0.tar.gz  Collecting nose==1.1.2 (from -r /tmp/build/requirements.txt (line 49))  Downloading https://files.pythonhosted.org/packages/38/96/7aa1c2583ddec558a230175d6aeddba796cde7191852bf3e6eb3cfb873e1/nose-1.1.2.tar.gz (729kB)  Collecting oauth2==1.5.170 (from -r /tmp/build/requirements.txt (line 50))  Downloading https://files.pythonhosted.org/packages/8b/d2/d9613db75252cee85ff2a5064436931c7f8b751ee46044ba54638ea7de52/oauth2-1.5.170.tar.gz  Collecting oauth==1.0.1 (from -r /tmp/build/requirements.txt (line 51))  Downloading https://files.pythonhosted.org/packages/e2/10/d7d6ae26ef7686109a10b3e88d345c4ec6686d07850f4ef7baefb7eb61e1/oauth-1.0.1.tar.gz  Collecting oauthlib==0.1.2 (from -r /tmp/build/requirements.txt (line 52))  Downloading https://files.pythonhosted.org/packages/86/15/acc8e6170fcaaadf0e2679c69f1acc108cd6a4dbb1f44ed42b173d0fa3cf/oauthlib-0.1.2.tar.gz  Collecting openpyxl==1.5.7 (from -r /tmp/build/requirements.txt (line 53))  Downloading https://files.pythonhosted.org/packages/cf/56/7a1414fafc30066ac1c3018fd5b89e2d1ab969b73890ec7c5ca0471a3c1d/openpyxl-1.5.7.tar.gz (67kB)  Collecting ordereddict==1.1 (from -r /tmp/build/requirements.txt (line 54))  Downloading https://files.pythonhosted.org/packages/53/25/ef88e8e45db141faa9598fbf7ad0062df8f50f881a36ed6a0073e1572126/ordereddict-1.1.tar.gz  Collecting pbkdf2==1.3 (from -r /tmp/build/requirements.txt (line 55))  Downloading https://files.pythonhosted.org/packages/02/c0/6a2376ae81beb82eda645a091684c0b0becb86b972def7849ea9066e3d5e/pbkdf2-1.3.tar.gz  Collecting pdfminer==20110515 (from -r /tmp/build/requirements.txt (line 56))  Downloading https://files.pythonhosted.org/packages/ce/f8/512bcd1a116d0332ab9fab84c3771d4699216db1086e120d581535665c31/pdfminer-20110515.tar.gz (4.1MB)  Collecting pexpect==2.4 (from -r /tmp/build/requirements.txt (line 57))  Downloading https://files.pythonhosted.org/packages/fa/e1/c1f8fce7e7d578ae69aff616cabd5e61b6cb734aade2486b2140853d0f26/pexpect-2.4.tar.gz (113kB)  Collecting pipe2py==0.9.2 (from -r /tmp/build/requirements.txt (line 58))  Downloading https://files.pythonhosted.org/packages/14/d7/f6d94e55e2c267dc0d39ba0e3d3829a20a469ec20a00afa38b91947fb967/pipe2py-0.9.2.tar.gz (57kB)  Collecting pyOpenSSL==0.13 (from -r /tmp/build/requirements.txt (line 59))  Downloading https://files.pythonhosted.org/packages/8b/20/8f4230b281a2a9d0ee9e24fd89aeded0b25d40c84b3d61100a96438e1626/pyOpenSSL-0.13.tar.gz (250kB)  Collecting pycrypto==2.5 (from -r /tmp/build/requirements.txt (line 60))  Downloading https://files.pythonhosted.org/packages/eb/0d/80b7706fa181128f55b34b2ed49bca24e1fecf25101c0364b602cfdd3f6c/pycrypto-2.5.tar.gz (426kB)  Collecting pycurl==7.19.0 (from -r /tmp/build/requirements.txt (line 61))  Downloading https://files.pythonhosted.org/packages/11/73/abcfbbb6e1dd7087fa53042c301c056c11264e8a737a4688f834162d731e/pycurl-7.19.0.tar.gz (70kB)  Collecting pyephem==3.7.5.1 (from -r /tmp/build/requirements.txt (line 62))  Downloading https://files.pythonhosted.org/packages/f9/62/4b486cec967357add6df1f24ef56e5bf0da5bc2110e4b0b3ce7264ce2ad7/pyephem-3.7.5.1.tar.gz (703kB)  Collecting pyparsing==1.5.6 (from -r /tmp/build/requirements.txt (line 63))  Downloading https://files.pythonhosted.org/packages/fa/fa/e063a194dd48b8e76c1ef77bda6be80e8f988dc111b29e5029127d324b72/pyparsing-1.5.6.tar.gz (1.4MB)  Collecting pyth==0.5.6 (from -r /tmp/build/requirements.txt (line 64))  Downloading https://files.pythonhosted.org/packages/9c/fb/489f35bd27074d02333e2e1c3a7ad511c63c56aa00c555ac9399f6637df4/pyth-0.5.6.tar.gz  Collecting python-Levenshtein==0.10.2 (from -r /tmp/build/requirements.txt (line 65))  Downloading https://files.pythonhosted.org/packages/32/3c/46cd4e5b41d46ad309372b9b5de70776aa66d5db02bafb3444782b86a23c/python-Levenshtein-0.10.2.tar.gz (45kB)  Collecting python-dateutil==1.5 (from -r /tmp/build/requirements.txt (line 66))  Downloading https://files.pythonhosted.org/packages/b4/7c/df59c89a753eb33c7c44e1dd42de0e9bc2ccdd5a4d576e0bfad97cc280cb/python-dateutil-1.5.tar.gz (233kB)  Collecting python-gflags==2.0 (from -r /tmp/build/requirements.txt (line 67))  Downloading https://files.pythonhosted.org/packages/46/47/12c17c3216c04a85e5ffd9163ad09f0c1661c2cc2ccc0faf70e39cb8dc96/python-gflags-2.0.tar.gz (65kB)  Collecting python-modargs==1.2 (from -r /tmp/build/requirements.txt (line 68))  Downloading https://files.pythonhosted.org/packages/a1/61/24d8587b069364de03dee98b20f90e0ad2e025ccb1db2ee16b3caf639b0e/python-modargs-1.2.tar.gz  Collecting python-stdnum==0.7 (from -r /tmp/build/requirements.txt (line 69))  Downloading https://files.pythonhosted.org/packages/40/01/c495a308c6fac2ab9419fb1be21165ec18b0ea68b9ac26e099c73ec57b83/python-stdnum-0.7.tar.gz (113kB)  Collecting pytz==2011k (from -r /tmp/build/requirements.txt (line 70))  Downloading https://files.pythonhosted.org/packages/9c/56/3813cd4d4ec4cd8d93388b8934e421122d8a89f19cf1f143a3c7ebc8827c/pytz-2011k.tar.bz2 (166kB)  Collecting rdflib==3.1.0 (from -r /tmp/build/requirements.txt (line 71))  Downloading https://files.pythonhosted.org/packages/30/f0/6c07b9639ed34fb0b5dea1d225864fc1b339d19fb5b06b2836508648db01/rdflib-3.1.0.tar.gz (249kB)  Collecting requests-foauth==0.1.1 (from -r /tmp/build/requirements.txt (line 72))  Downloading https://files.pythonhosted.org/packages/b7/6c/7291fa76577d0eb4530829a041f86d294b45aee6b36bae2b191c5bfd4994/requests-foauth-0.1.1.tar.gz  Collecting requests==1.0.4 (from -r /tmp/build/requirements.txt (line 73))  Downloading https://files.pythonhosted.org/packages/5d/e8/f27e0868b9a49946b3f800722e02b19efebde22ae534276df3e5f6cca41d/requests-1.0.4.tar.gz (336kB)  Collecting selenium==2.5.0 (from -r /tmp/build/requirements.txt (line 74))  Downloading https://files.pythonhosted.org/packages/21/0f/dbc8580df0eb4b2ea451f1901573ae09629e3135dacb70e504b950ec0cad/selenium-2.5.0.tar.gz (2.4MB)  Collecting simplejson==2.2.1 (from -r /tmp/build/requirements.txt (line 75))  Downloading https://files.pythonhosted.org/packages/08/aa/49ce621718cb55f27cc9bc85e38cc552bfb90e281889c155b0a59d2b01ec/simplejson-2.2.1.tar.gz (49kB)  Collecting suds==0.4 (from -r /tmp/build/requirements.txt (line 76))  Downloading https://files.pythonhosted.org/packages/bc/d6/960acce47ee6f096345fe5a7d9be7708135fd1d0713571836f073efc7393/suds-0.4.tar.gz (104kB)  Collecting tweepy==1.7.1 (from -r /tmp/build/requirements.txt (line 77))  Downloading https://files.pythonhosted.org/packages/09/21/2e87597c60fff537ecfff0533b634e1fdb09d5585990308354952a9370a9/tweepy-1.7.1.tar.gz  Collecting tweetstream==1.1.1 (from -r /tmp/build/requirements.txt (line 78))  Downloading https://files.pythonhosted.org/packages/60/a4/30b6d372e6bb0b3290b1012f5f84ee9d5183880d09ebb865e87200a55142/tweetstream-1.1.1.tar.gz  Collecting w3lib==1.0 (from -r /tmp/build/requirements.txt (line 79))  Downloading https://files.pythonhosted.org/packages/a2/07/b2c6767a26d473a7812ba7daa2b334ffbf2caed363b872cda0a5b0651fdd/w3lib-1.0.tar.gz  Collecting xlrd==0.7.1 (from -r /tmp/build/requirements.txt (line 81))  Downloading https://files.pythonhosted.org/packages/a8/b1/bf7e936d9ea1e68c5e3247fcafaa45fe3967282bb419a8c54312be0f64af/xlrd-0.7.1.tar.gz (118kB)  Collecting xlutils==1.4.1 (from -r /tmp/build/requirements.txt (line 82))  Downloading https://files.pythonhosted.org/packages/c2/fa/e264c0a2fdf3db9152b7b37551d4f8a8a432dbf9afef54f68c91da7a0233/xlutils-1.4.1.tar.gz (40kB)  Collecting xlwt==0.7.2 (from -r /tmp/build/requirements.txt (line 83))  Downloading https://files.pythonhosted.org/packages/98/e9/e9c551e993bde8eccfaf4fd3f991990850a3b1d2ddf9ba9e30f495462497/xlwt-0.7.2.tar.gz (114kB)  Collecting xmltodict==0.4 (from -r /tmp/build/requirements.txt (line 84))  Downloading https://files.pythonhosted.org/packages/dd/3b/14f5583a5128d4dde7a14738b554e7449d2f08d710af3fe1a3c49ca4b636/xmltodict-0.4.tar.gz  Collecting zope.interface==3.8.0 (from -r /tmp/build/requirements.txt (line 88))  Downloading https://files.pythonhosted.org/packages/a9/8d/cea179e663f9656f07d09b0b181299a2d8949fb6491ce3c5bc923ca9dd9f/zope.interface-3.8.0.tar.gz (111kB)  Collecting lxml==2.3.3 (from -r /tmp/build/requirements.txt (line 89))  Downloading https://files.pythonhosted.org/packages/09/70/9176a425aa436677dcc4ddd39b78ba598ca55683aefb47c6b7f617fa17cc/lxml-2.3.3.tar.gz (3.1MB)  Collecting chromium-compact-language-detector==0.031415 (from -r /tmp/build/requirements.txt (line 90))  Downloading https://files.pythonhosted.org/packages/90/9b/6754a53f6b622420e511e95d600ac7dbc90e459796009e2502f384666210/chromium_compact_language_detector-0.031415.tar.gz (2.2MB)  Collecting icalendar==3.0.1b1 (from -r /tmp/build/requirements.txt (line 93))  Downloading https://files.pythonhosted.org/packages/b3/1b/2dbc75d7b5b77dcc5e9c36eeae522a859ac3569bd22081bc02fce0e698e5/icalendar-3.0.1b1.tar.gz  Collecting pyquery==1.0 (from -r /tmp/build/requirements.txt (line 96))  Downloading https://files.pythonhosted.org/packages/92/43/4435ff3612477cbabf72d3688c510b5d031358ec5e9a6cef10f8babc4d33/pyquery-1.0.tar.gz  Collecting scrapely==0.9 (from -r /tmp/build/requirements.txt (line 98))  Downloading https://files.pythonhosted.org/packages/1d/16/892d94655015dc5e0dd1f0013e78b5c510a6da6b1f1d92b7b4ec7d43cdbd/scrapely-0.9.tar.gz  Collecting Fom==0.9.8 (from -r /tmp/build/requirements.txt (line 103))  Downloading https://files.pythonhosted.org/packages/52/cf/179866c56774cea643e9a8a046e612222eb6b3ed3f824de220d211c7b3e4/Fom-0.9.8.tar.gz (70kB)  Collecting PyYAML==3.10 (from -r /tmp/build/requirements.txt (line 105))  Downloading https://files.pythonhosted.org/packages/00/17/3b822893a1789a025d3f676a381338516a8f65e686d915b0834ecc9b4979/PyYAML-3.10.tar.gz (241kB)  Collecting Scrapy==0.14.1 (from -r /tmp/build/requirements.txt (line 107))  Downloading https://files.pythonhosted.org/packages/c1/7f/d898f6f3b19a3556c31224d137ec6864144d64d8b6a26a20f4096c3bee67/Scrapy-0.14.1.tar.gz (719kB)  Collecting adspygoogle.adwords==15.6.2 (from -r /tmp/build/requirements.txt (line 109))  Downloading https://files.pythonhosted.org/packages/88/f5/be287bdc6df013c571e2a0d0a4046d174d862cfd31751459efce5457dba3/adspygoogle.adwords-15.6.2.tar.gz (166kB)  Collecting nltk==3.0.2 (from -r /tmp/build/requirements.txt (line 111))  Downloading https://files.pythonhosted.org/packages/06/85/4ac5762ba85980b4250931d80d1d1ea3917de2f13c56d2c270a4b902ecd0/nltk-3.0.2.tar.gz (991kB)  Collecting pydot==1.0.2 (from -r /tmp/build/requirements.txt (line 113))  Downloading https://files.pythonhosted.org/packages/02/ff/cbd177256cfed9d0e6578a40ee74e1609d0532350f3cc8c66912831221dd/pydot-1.0.2.tar.gz  Collecting M2Crypto==0.22.3 (from -r /tmp/build/requirements.txt (line 115))  Downloading https://files.pythonhosted.org/packages/80/d4/09524cdccd88cb9a6ef99a1cf6a4996e2bb48dceb16a23530ca04f59f390/M2Crypto-0.22.3.tar.gz (74kB)  Collecting dumptruck>=0.1.2 (from scraperwiki->-r /tmp/build/requirements.txt (line 2))  Downloading https://files.pythonhosted.org/packages/15/27/3330a343de80d6849545b6c7723f8c9a08b4b104de964ac366e7e6b318df/dumptruck-0.1.6.tar.gz  Collecting alembic>=0.6.2 (from dataset==0.5.2->-r /tmp/build/requirements.txt (line 29))  Downloading https://files.pythonhosted.org/packages/89/03/756d5b8e1c90bf283c3f435766aa3f20208d1c3887579dd8f2122e01d5f4/alembic-0.9.9.tar.gz (1.0MB)  Collecting python-slugify>=0.0.6 (from dataset==0.5.2->-r /tmp/build/requirements.txt (line 29))  Downloading https://files.pythonhosted.org/packages/70/c1/98bfb2c981787dcec4613c5da2c17d6f54613935b0e3a877e87a9fa974e4/python-slugify-1.2.5.tar.gz  Collecting distribute (from python-stdnum==0.7->-r /tmp/build/requirements.txt (line 69))  Downloading https://files.pythonhosted.org/packages/5f/ad/1fde06877a8d7d5c9b60eff7de2d452f639916ae1d48f0b8f97bf97e570a/distribute-0.7.3.zip (145kB)  Collecting numpy (from scrapely==0.9->-r /tmp/build/requirements.txt (line 98))  Downloading https://files.pythonhosted.org/packages/c0/e7/08f059a00367fd613e4f2875a16c70b6237268a1d6d166c6d36acada8301/numpy-1.14.3-cp27-cp27mu-manylinux1_x86_64.whl (12.1MB)  Collecting Mako (from alembic>=0.6.2->dataset==0.5.2->-r /tmp/build/requirements.txt (line 29))  Downloading https://files.pythonhosted.org/packages/eb/f3/67579bb486517c0d49547f9697e36582cd19dafb5df9e687ed8e22de57fa/Mako-1.0.7.tar.gz (564kB)  Collecting python-editor>=0.3 (from alembic>=0.6.2->dataset==0.5.2->-r /tmp/build/requirements.txt (line 29))  Downloading https://files.pythonhosted.org/packages/65/1e/adf6e000ea5dc909aa420352d6ba37f16434c8a3c2fa030445411a1ed545/python-editor-1.0.3.tar.gz  Collecting MarkupSafe>=0.9.2 (from Mako->alembic>=0.6.2->dataset==0.5.2->-r /tmp/build/requirements.txt (line 29))  Downloading https://files.pythonhosted.org/packages/4d/de/32d741db316d8fdb7680822dd37001ef7a448255de9699ab4bfcbdf4172b/MarkupSafe-1.0.tar.gz  python-slugify 1.2.5 has requirement Unidecode>=0.04.16, but you'll have unidecode 0.4.9 which is incompatible.  alembic 0.9.9 has requirement SQLAlchemy>=0.7.6, but you'll have sqlalchemy 0.6.6 which is incompatible.  dataset 0.5.2 has requirement sqlalchemy>=0.9.1, but you'll have sqlalchemy 0.6.6 which is incompatible.  Installing collected packages: dumptruck, requests, scraperwiki, BeautifulSoup, Genshi, Creoleparser, Jinja2, Markdown, Pygments, SQLAlchemy, zope.interface, Twisted, Unidecode, anyjson, argparse, beautifulsoup4, bitlyapi, blinker, httplib2, oauth2, cartodb, certifi, chardet, ckanclient, colormath, xlrd, python-dateutil, csvkit, MarkupSafe, Mako, python-editor, alembic, python-slugify, PyYAML, dataset, demjson, oauth, simplejson, dropbox, errorhandler, feedparser, fluidinfo.py, gdata, geopy, greenlet, gevent, python-gflags, google-api-python-client, googlemaps, html5lib, imposm.parser, jellyfish, mechanize, mock, networkx, ngram, nose, pycrypto, oauthlib, openpyxl, ordereddict, pbkdf2, pdfminer, pexpect, pipe2py, pyOpenSSL, pycurl, pyephem, pyparsing, pyth, python-Levenshtein, python-modargs, distribute, python-stdnum, pytz, rdflib, requests-foauth, selenium, suds, tweepy, tweetstream, w3lib, xlwt, xlutils, xmltodict, lxml, chromium-compact-language-detector, icalendar, pyquery, numpy, scrapely, Fom, Scrapy, adspygoogle.adwords, nltk, pydot, M2Crypto  Running setup.py install for dumptruck: started  Running setup.py install for dumptruck: finished with status 'done'  Running setup.py install for requests: started  Running setup.py install for requests: finished with status 'done'  Running setup.py develop for scraperwiki  Running setup.py install for BeautifulSoup: started  Running setup.py install for BeautifulSoup: finished with status 'done'  Running setup.py install for Genshi: started  Running setup.py install for Genshi: finished with status 'done'  Running setup.py install for Creoleparser: started  Running setup.py install for Creoleparser: finished with status 'done'  Running setup.py install for Jinja2: started  Running setup.py install for Jinja2: finished with status 'done'  Running setup.py install for Markdown: started  Running setup.py install for Markdown: finished with status 'done'  Running setup.py install for Pygments: started  Running setup.py install for Pygments: finished with status 'done'  Running setup.py install for SQLAlchemy: started  Running setup.py install for SQLAlchemy: finished with status 'done'  Running setup.py install for zope.interface: started  Running setup.py install for zope.interface: finished with status 'done'  Running setup.py install for Twisted: started  Running setup.py install for Twisted: finished with status 'done'  Running setup.py install for Unidecode: started  Running setup.py install for Unidecode: finished with status 'done'  Running setup.py install for anyjson: started  Running setup.py install for anyjson: finished with status 'done'  Running setup.py install for argparse: started  Running setup.py install for argparse: finished with status 'done'  Running setup.py install for beautifulsoup4: started  Running setup.py install for beautifulsoup4: finished with status 'done'  Running setup.py install for bitlyapi: started  Running setup.py install for bitlyapi: finished with status 'done'  Running setup.py install for blinker: started  Running setup.py install for blinker: finished with status 'done'  Running setup.py install for httplib2: started  Running setup.py install for httplib2: finished with status 'done'  Running setup.py install for oauth2: started  Running setup.py install for oauth2: finished with status 'done'  Running setup.py install for cartodb: started  Running setup.py install for cartodb: finished with status 'done'  Running setup.py install for certifi: started  Running setup.py install for certifi: finished with status 'done'  Running setup.py install for chardet: started  Running setup.py install for chardet: finished with status 'done'  Running setup.py install for ckanclient: started  Running setup.py install for ckanclient: finished with status 'done'  Running setup.py install for colormath: started  Running setup.py install for colormath: finished with status 'done'  Running setup.py install for xlrd: started  Running setup.py install for xlrd: finished with status 'done'  Running setup.py install for python-dateutil: started  Running setup.py install for python-dateutil: finished with status 'done'  Running setup.py install for csvkit: started  Running setup.py install for csvkit: finished with status 'done'  Running setup.py install for MarkupSafe: started  Running setup.py install for MarkupSafe: finished with status 'done'  Running setup.py install for Mako: started  Running setup.py install for Mako: finished with status 'done'  Running setup.py install for python-editor: started  Running setup.py install for python-editor: finished with status 'done'  Running setup.py install for alembic: started  Running setup.py install for alembic: finished with status 'done'  Running setup.py install for python-slugify: started  Running setup.py install for python-slugify: finished with status 'done'  Running setup.py install for PyYAML: started  Running setup.py install for PyYAML: finished with status 'done'  Running setup.py install for dataset: started  Running setup.py install for dataset: finished with status 'done'  Running setup.py install for demjson: started  Running setup.py install for demjson: finished with status 'done'  Running setup.py install for oauth: started  Running setup.py install for oauth: finished with status 'done'  Running setup.py install for simplejson: started  Running setup.py install for simplejson: finished with status 'done'  Running setup.py install for dropbox: started  Running setup.py install for dropbox: finished with status 'done'  Running setup.py install for errorhandler: started  Running setup.py install for errorhandler: finished with status 'done'  Running setup.py install for feedparser: started  Running setup.py install for feedparser: finished with status 'done'  Running setup.py install for fluidinfo.py: started  Running setup.py install for fluidinfo.py: finished with status 'done'  Running setup.py install for gdata: started  Running setup.py install for gdata: finished with status 'done'  Running setup.py install for geopy: started  Running setup.py install for geopy: finished with status 'done'  Running setup.py install for greenlet: started  Running setup.py install for greenlet: finished with status 'done'  Running setup.py install for gevent: started  Running setup.py install for gevent: finished with status 'done'  Running setup.py install for python-gflags: started  Running setup.py install for python-gflags: finished with status 'done'  Running setup.py install for google-api-python-client: started  Running setup.py install for google-api-python-client: finished with status 'done'  Running setup.py install for googlemaps: started  Running setup.py install for googlemaps: finished with status 'done'  Running setup.py install for html5lib: started  Running setup.py install for html5lib: finished with status 'done'  Running setup.py install for imposm.parser: started  Running setup.py install for imposm.parser: finished with status 'done'  Running setup.py install for jellyfish: started  Running setup.py install for jellyfish: finished with status 'done'  Running setup.py install for mechanize: started  Running setup.py install for mechanize: finished with status 'done'  Running setup.py install for mock: started  Running setup.py install for mock: finished with status 'done'  Running setup.py install for networkx: started  Running setup.py install for networkx: finished with status 'done'  Running setup.py install for ngram: started  Running setup.py install for ngram: finished with status 'done'  Running setup.py install for nose: started  Running setup.py install for nose: finished with status 'done'  Running setup.py install for pycrypto: started  Running setup.py install for pycrypto: finished with status 'done'  Running setup.py install for oauthlib: started  Running setup.py install for oauthlib: finished with status 'done'  Running setup.py install for openpyxl: started  Running setup.py install for openpyxl: finished with status 'done'  Running setup.py install for ordereddict: started  Running setup.py install for ordereddict: finished with status 'done'  Running setup.py install for pbkdf2: started  Running setup.py install for pbkdf2: finished with status 'done'  Running setup.py install for pdfminer: started  Running setup.py install for pdfminer: finished with status 'done'  Running setup.py install for pexpect: started  Running setup.py install for pexpect: finished with status 'done'  Running setup.py install for pipe2py: started  Running setup.py install for pipe2py: finished with status 'done'  Running setup.py install for pyOpenSSL: started  Running setup.py install for pyOpenSSL: finished with status 'done'  Running setup.py install for pycurl: started  Running setup.py install for pycurl: finished with status 'done'  Running setup.py install for pyephem: started  Running setup.py install for pyephem: finished with status 'done'  Running setup.py install for pyparsing: started  Running setup.py install for pyparsing: finished with status 'done'  Running setup.py install for pyth: started  Running setup.py install for pyth: finished with status 'done'  Running setup.py install for python-Levenshtein: started  Running setup.py install for python-Levenshtein: finished with status 'done'  Running setup.py install for python-modargs: started  Running setup.py install for python-modargs: finished with status 'done'  Running setup.py install for distribute: started  Running setup.py install for distribute: finished with status 'done'  Running setup.py install for python-stdnum: started  Running setup.py install for python-stdnum: finished with status 'done'  Running setup.py install for pytz: started  Running setup.py install for pytz: finished with status 'done'  Running setup.py install for rdflib: started  Running setup.py install for rdflib: finished with status 'done'  Running setup.py install for requests-foauth: started  Running setup.py install for requests-foauth: finished with status 'done'  Running setup.py install for selenium: started  Running setup.py install for selenium: finished with status 'done'  Running setup.py install for suds: started  Running setup.py install for suds: finished with status 'done'  Running setup.py install for tweepy: started  Running setup.py install for tweepy: finished with status 'done'  Running setup.py install for tweetstream: started  Running setup.py install for tweetstream: finished with status 'done'  Running setup.py install for w3lib: started  Running setup.py install for w3lib: finished with status 'done'  Running setup.py install for xlwt: started  Running setup.py install for xlwt: finished with status 'done'  Running setup.py install for xlutils: started  Running setup.py install for xlutils: finished with status 'done'  Running setup.py install for xmltodict: started  Running setup.py install for xmltodict: finished with status 'done'  Running setup.py install for lxml: started  Running setup.py install for lxml: finished with status 'done'  Running setup.py install for chromium-compact-language-detector: started  Running setup.py install for chromium-compact-language-detector: finished with status 'done'  Running setup.py install for icalendar: started  Running setup.py install for icalendar: finished with status 'done'  Running setup.py install for pyquery: started  Running setup.py install for pyquery: finished with status 'done'  Running setup.py install for scrapely: started  Running setup.py install for scrapely: finished with status 'done'  Running setup.py install for Fom: started  Running setup.py install for Fom: finished with status 'done'  Running setup.py install for Scrapy: started  Running setup.py install for Scrapy: finished with status 'done'  Running setup.py install for adspygoogle.adwords: started  Running setup.py install for adspygoogle.adwords: finished with status 'done'  Running setup.py install for nltk: started  Running setup.py install for nltk: finished with status 'done'  Running setup.py install for pydot: started  Running setup.py install for pydot: finished with status 'done'  Running setup.py install for M2Crypto: started  Running setup.py install for M2Crypto: finished with status 'done'  Successfully installed BeautifulSoup-3.2.0 Creoleparser-0.7.4 Fom-0.9.8 Genshi-0.6 Jinja2-2.6 M2Crypto-0.22.3 Mako-1.0.7 Markdown-2.2.0 MarkupSafe-1.0 PyYAML-3.10 Pygments-1.4 SQLAlchemy-0.6.6 Scrapy-0.14.1 Twisted-11.1.0 Unidecode-0.4.9 adspygoogle.adwords-15.6.2 alembic-0.9.9 anyjson-0.3.3 argparse-1.2.1 beautifulsoup4-4.1.3 bitlyapi-0.1.1 blinker-1.2 cartodb-0.6 certifi-0.0.8 chardet-2.1.1 chromium-compact-language-detector-0.31415 ckanclient-0.10 colormath-1.0.8 csvkit-0.3.0 dataset-0.5.2 demjson-1.6 distribute-0.7.3 dropbox-1.4 dumptruck-0.1.6 errorhandler-1.1.1 feedparser-5.0.1 fluidinfo.py-1.1.2 gdata-2.0.15 geopy-0.94.1 gevent-0.13.6 google-api-python-client-1.0b8 googlemaps-1.0.2 greenlet-0.3.2 html5lib-0.90 httplib2-0.7.4 icalendar-3.0.1b1 imposm.parser-1.0.3 jellyfish-0.2.0 lxml-2.3.3 mechanize-0.2.5 mock-0.7.2 networkx-1.6 ngram-3.3.0 nltk-3.0.2 nose-1.1.2 numpy-1.14.3 oauth-1.0.1 oauth2-1.5.170 oauthlib-0.1.2 openpyxl-1.5.7 ordereddict-1.1 pbkdf2-1.3 pdfminer-20110515 pexpect-2.4 pipe2py-0.9.2 pyOpenSSL-0.13 pycrypto-2.5 pycurl-7.19.0 pydot-1.0.2 pyephem-3.7.5.1 pyparsing-1.5.6 pyquery-1.0 pyth-0.5.6 python-Levenshtein-0.10.2 python-dateutil-1.5 python-editor-1.0.3 python-gflags-2.0 python-modargs-1.2 python-slugify-1.2.5 python-stdnum-0.7 pytz-2011k rdflib-3.1.0 requests-1.0.4 requests-foauth-0.1.1 scrapely-0.9 scraperwiki selenium-2.5.0 simplejson-2.2.1 suds-0.4 tweepy-1.7.1 tweetstream-1.1.1 w3lib-1.0 xlrd-0.7.1 xlutils-1.4.1 xlwt-0.7.2 xmltodict-0.4 zope.interface-3.8.0   ! Hello! Your requirements.txt file contains the distribute package.  ! This library is automatically installed by Heroku and shouldn't be in  ! Your requirements.txt file. This can cause unexpected behavior.  ! -- Much Love, Heroku.  -----> Downloading NLTK corpora…  ! 'nltk.txt' not found, not downloading any corpora  ! Learn more: https://devcenter.heroku.com/articles/python-nltk  -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... E5042_HLBC_gov_2018_03 E5042_HLBC_gov_2018_02 E5042_HLBC_gov_2018_01 E5042_HLBC_gov_2017_12 E5042_HLBC_gov_2017_11 E5042_HLBC_gov_2017_10 E5042_HLBC_gov_2017_09 E5042_HLBC_gov_2017_08 E5042_HLBC_gov_2017_07 E5042_HLBC_gov_2017_06 E5042_HLBC_gov_2017_05 E5042_HLBC_gov_2017_04 E5042_HLBC_gov_2017_03 E5042_HLBC_gov_2017_02 E5042_HLBC_gov_2017_01 E5042_HLBC_gov_2016_12 E5042_HLBC_gov_2016_11 E5042_HLBC_gov_2016_10 E5042_HLBC_gov_2016_09 E5042_HLBC_gov_2016_08 E5042_HLBC_gov_2016_07 E5042_HLBC_gov_2016_06 E5042_HLBC_gov_2016_05 E5042_HLBC_gov_2016_04 E5042_HLBC_gov_2016_03 E5042_HLBC_gov_2015_02 E5042_HLBC_gov_2016_01 E5042_HLBC_gov_2015_12 E5042_HLBC_gov_2015_Q0 E5042_HLBC_gov_2010_Q0 E5042_HLBC_gov_2015_09 E5042_HLBC_gov_2015_08 E5042_HLBC_gov_2015_07 E5042_HLBC_gov_2015_06 E5042_HLBC_gov_2011_Y1 E5042_HLBC_gov_2012_Y1 E5042_HLBC_gov_2013_Y1 E5042_HLBC_gov_2015_05 E5042_HLBC_gov_2014_Y1 E5042_HLBC_gov_2015_04

Data

Downloaded 683 times by SimKennedy MikeRalphson woodbine rootkit1989

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (22 KB) Use the API

rows 10 / 40

f d l
E5042_HLBC_gov_2018_03
2018-05-28 02:31:45.003400
E5042_HLBC_gov_2018_02
2018-05-28 02:31:47.100128
E5042_HLBC_gov_2018_01
2018-05-28 02:31:49.221211
E5042_HLBC_gov_2017_12
2018-05-28 02:31:51.228520
E5042_HLBC_gov_2017_11
2018-05-28 02:31:53.074646
E5042_HLBC_gov_2017_10
2018-05-28 02:31:55.057837
E5042_HLBC_gov_2017_09
2018-05-28 02:31:56.995477
E5042_HLBC_gov_2017_08
2018-05-28 02:31:58.960369
E5042_HLBC_gov_2017_07
2018-05-28 02:32:00.898875
E5042_HLBC_gov_2017_06
2018-05-28 02:32:02.684481

Statistics

Average successful run time: 12 minutes

Total run time: 9 months

Total cpu time used: 2 days

Total disk space used: 57.4 KB

History

  • Auto ran revision 0b760c42 and completed successfully .
    40 records added, 40 records removed in the database
  • Auto ran revision 0b760c42 and completed successfully .
    40 records added, 40 records removed in the database
  • Auto ran revision 0b760c42 and completed successfully .
    40 records added, 40 records removed in the database
    45 pages scraped
  • Auto ran revision 0b760c42 and completed successfully .
    40 records added, 40 records removed in the database
  • Auto ran revision 0b760c42 and completed successfully .
    40 records added, 40 records removed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

sp_E5042_HLBC_gov / scraper.py