mobeets / intriguing-things-scraper

scraper for Alexis Madrigal's 5 Intriguing Things

Scrapes tinyletter.com

TinyLetter


Alexis Madrigal's 5 Intriguing Things is a usually-daily newsletter containing links to things. The file scraper.py is run daily by Morph to update a (good enough) archive of those things.

View the archive here, or better yet, browse the results.

Contributors mobeets

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... Loading previous entries... Found 1863 urls Starting at http://tinyletter.com/realfuture/letters/from-fuld-hall-to-olden-farm-witnessed-enviously Currently have 1863 entries Starting at http://tinyletter.com/realfuture/letters/from-fuld-hall-to-olden-farm-witnessed-enviously http://tinyletter.com/realfuture/letters/from-fuld-hall-to-olden-farm-witnessed-enviously Traceback (most recent call last): File "scraper.py", line 144, in <module> main() File "scraper.py", line 141, in main io(starturl, urls, inds) File "scraper.py", line 106, in io dt, ts, new_url = load(next_url) File "scraper.py", line 76, in load dt, contents, next_url = parse(read(url)) File "scraper.py", line 31, in read response = urllib2.urlopen(url) File "/app/.heroku/python/lib/python2.7/urllib2.py", line 127, in urlopen return _opener.open(url, data, timeout) File "/app/.heroku/python/lib/python2.7/urllib2.py", line 410, in open response = meth(req, response) File "/app/.heroku/python/lib/python2.7/urllib2.py", line 523, in http_response 'http', request, response, code, msg, hdrs) File "/app/.heroku/python/lib/python2.7/urllib2.py", line 442, in error result = self._call_chain(*args) File "/app/.heroku/python/lib/python2.7/urllib2.py", line 382, in _call_chain result = func(*args) File "/app/.heroku/python/lib/python2.7/urllib2.py", line 629, in http_error_302 return self.parent.open(new, timeout=req.timeout) File "/app/.heroku/python/lib/python2.7/urllib2.py", line 410, in open response = meth(req, response) File "/app/.heroku/python/lib/python2.7/urllib2.py", line 523, in http_response 'http', request, response, code, msg, hdrs) File "/app/.heroku/python/lib/python2.7/urllib2.py", line 448, in error return self._call_chain(*args) File "/app/.heroku/python/lib/python2.7/urllib2.py", line 382, in _call_chain result = func(*args) File "/app/.heroku/python/lib/python2.7/urllib2.py", line 531, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 404: Not found

Data

Downloaded 1775 times by mobeets MikeRalphson

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (2.25 MB) Use the API

rows 10 / 1863

ps title url index number src_url dt
<p>"But [Emory University's Gregory] Berns hopes to respond with future fMRI work, which will compare brain activity in dogs being fed by automated mechanisms with that of dogs being fed by humans."</p><p> </p>
People are putting dogs in MRI machines to determine if they love us like we love them. But will they love robots, too?
2013-11-5.1
1
2013-11-5
<p>"If you want a picture of the future, imagine a robot hand playing rock paper scissors with a human hand -- forever" -- <a href="https://twitter.com/flaneur/status/397503991707598848">@MatthewOgle</a></p><p> </p>
You can't beat this robot at rock-paper-scissors because it detects your initial hand movement and forms its own fingers into a winning configuration before you can finish.
2013-11-5.2
2
2013-11-5
<p>"...<span>more than offsetting losses in other divisions."</span></p><p> </p>
AOL's dial-up Internet business generated almost $150 million in income in the last quarter
2013-11-5.3
3
2013-11-5
<p>"<span>So he took my computer and sort of typed a few things, and right in front of me he downloaded, like, 4,000 words from the pool of the internet."</span></p><p> </p>
Julian Assange helped MIA find words with T-E-N-T in them for a song about the plight of refugees
2013-11-5.4
4
2013-11-5
<p>"<span>The camp was centered around a beautiful wild hot spring. 70 miles to the nearest phone. They erected a dome in the desert and then battled the winds while trying to erect an inflatable structure. It was Burning Man 40 years ago."</span></p><p><span><br /></span></p><p><span>As always, send feedback and ideas for inclusion to amadrigal@theatlantic.com.</span></p>
Whole Earth Catalog founder Steward Brand shot this footage in the desert in 1971.
2013-11-5.5
5
2013-11-5
<p>"Mobile is eating the world."</p><p> </p>
Analyst Benedict Evans lays out the 73-slide case for the end of the Internet, media, and technology industries as we've known them
2013-11-6.1
1
2013-11-6
<p>"<span>The institution with the most to gain is the Internal Revenue Service."</span></p><p> </p>
Steven Levy's 1994 Wired article on digital currency, including a swath of defunct BitCoin wannabes.
2013-11-6.2
2
2013-11-6
<p>"<span>David Milarch of the <a href="http://www.ancienttreearchive.org/">Archangel Ancient Tree Archive</a>, the group cloning the trees, says the clones are living links to Muir's life."</span></p><p> </p>
A Michigan company successfully cloned a 130-year old sequoia that Atlantic contributor John Muir planted in his yard in the 19th century
2013-11-6.3
3
2013-11-6
<p>"<span>The Jawbone Canyon siphon, pictured above in </span><a href="http://digitallibrary.usc.edu/cdm/singleitem/collection/p15799coll65/id/17789/rec/1" target="_blank">a photograph from the California Historical Society Collection at the USC Libraries</a><span>, is among the aqueduct's most impressive features. Workers assembled the massive steel pipe (measuring 8,095 feet in length and up to ten feet in diameter) in 36-foot, 25-ton segments, each hauled to the work site by </span><a href="http://content.cdlib.org/ark:/13030/hb2q2nb1jr/" target="_blank">a team of 52 mules</a><span>. Water falls through the tube, 850 feet to the canyon floor, generating hydraulic pressure that then forces it up and over the opposite ridge without the aid of a pump."</span></p><p> </p>
A reflection on the Rube Goldbergian engineering of Los Angeles' Owens Valley aqueduct.
2013-11-6.4
4
2013-11-6
<p>* Thanks to <a href="https://twitter.com/smc90/status/397971728153862145">Sonal Chokshi</a>, <a href="https://twitter.com/NathanUnbound/status/397790865193578496">Nathan Masters</a>, <a href="https://twitter.com/MattPRD/status/397933227584655360">Matt Schlicht</a>.</p>
UX Archive, a site that has collected 241 "user flows," which show how people accomplish anything with their phones
2013-11-6.5
5
2013-11-6

Statistics

Average successful run time: 2 minutes

Total run time: 14 days

Total cpu time used: about 1 hour

Total disk space used: 2.3 MB

History

  • Auto ran revision c1dbd724 and failed .
    nothing changed in the database
    2 pages scraped
  • Auto ran revision c1dbd724 and failed .
    nothing changed in the database
    2 pages scraped
  • Auto ran revision c1dbd724 and failed .
    nothing changed in the database
  • Auto ran revision c1dbd724 and failed .
    nothing changed in the database
    2 pages scraped
  • Auto ran revision c1dbd724 and failed .
    nothing changed in the database
    2 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

intriguing-things-scraper / scraper.py