Stanpol / ted_com_scraper

ted.com talks

Scrapes www.ted.com

TED Talks are influential videos from expert speakers on education, business, science, tech and creativity, with subtitles in 100+ languages. Ideas free to stream and download.


This is a scraper that runs on Morph.

It scrapes the latest talks from TED.com.

Contributors Stanpol

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Python app detected  ! The latest version of Python 2 is python-2.7.14 (you are using python-2.7.9, which is unsupported).  ! We recommend upgrading by specifying the latest version (python-2.7.14).  Learn More: https://devcenter.heroku.com/articles/python-runtimes -----> Installing python-2.7.9 -----> Installing pip -----> Installing requirements with pip  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 6))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to revision morph_defaults) to /app/.heroku/src/scraperwiki  Collecting lxml==3.4.4 (from -r /tmp/build/requirements.txt (line 8))  Downloading https://files.pythonhosted.org/packages/63/c7/4f2a2a4ad6c6fa99b14be6b3c1cece9142e2d915aa7c43c908677afc8fa4/lxml-3.4.4.tar.gz (3.5MB)  Collecting bs4==0.0.1 (from -r /tmp/build/requirements.txt (line 9))  Downloading https://files.pythonhosted.org/packages/10/ed/7e8b97591f6f456174139ec089c769f89a94a1a4025fe967691de971f314/bs4-0.0.1.tar.gz  Collecting cssselect==0.9.1 (from -r /tmp/build/requirements.txt (line 10))  Downloading https://files.pythonhosted.org/packages/aa/e5/9ee1460d485b94a6d55732eb7ad5b6c084caf73dd6f9cb0bb7d2a78fafe8/cssselect-0.9.1.tar.gz  Collecting dumptruck>=0.1.2 (from scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading https://files.pythonhosted.org/packages/15/27/3330a343de80d6849545b6c7723f8c9a08b4b104de964ac366e7e6b318df/dumptruck-0.1.6.tar.gz  Collecting requests (from scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading https://files.pythonhosted.org/packages/49/df/50aa1999ab9bde74656c2919d9c0c085fd2b3775fd3eca826012bef76d8c/requests-2.18.4-py2.py3-none-any.whl (88kB)  Collecting beautifulsoup4 (from bs4==0.0.1->-r /tmp/build/requirements.txt (line 9))  Downloading https://files.pythonhosted.org/packages/a6/29/bcbd41a916ad3faf517780a0af7d0254e8d6722ff6414723eedba4334531/beautifulsoup4-4.6.0-py2-none-any.whl (86kB)  Collecting idna<2.7,>=2.5 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl (56kB)  Collecting urllib3<1.23,>=1.21.1 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading https://files.pythonhosted.org/packages/63/cb/6965947c13a94236f6d4b8223e21beb4d576dc72e8130bd7880f600839b8/urllib3-1.22-py2.py3-none-any.whl (132kB)  Collecting certifi>=2017.4.17 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl (150kB)  Collecting chardet<3.1.0,>=3.0.2 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)  Installing collected packages: dumptruck, idna, urllib3, certifi, chardet, requests, scraperwiki, lxml, beautifulsoup4, bs4, cssselect  Running setup.py install for dumptruck: started  Running setup.py install for dumptruck: finished with status 'done'  Running setup.py develop for scraperwiki  Running setup.py install for lxml: started  Running setup.py install for lxml: still running...  Running setup.py install for lxml: finished with status 'done'  Running setup.py install for bs4: started  Running setup.py install for bs4: finished with status 'done'  Running setup.py install for cssselect: started  Running setup.py install for cssselect: finished with status 'done'  Successfully installed beautifulsoup4-4.6.0 bs4-0.0.1 certifi-2018.4.16 chardet-3.0.4 cssselect-0.9.1 dumptruck-0.1.6 idna-2.6 lxml-3.4.4 requests-2.18.4 scraperwiki urllib3-1.22   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Scraping the latest page Nancy Rabalais: The "dead zone" of the Gulf of Mexico Mark Tyndall: The harm reduction model of drug addiction treatment Hannah Bürckstümmer: A printable, flexible, organic solar cell Yasin Kakande: What's missing in the global debate over refugees Robin Steinberg: What if we ended the injustice of bail? Malika Whitley: How the arts help homeless youth heal and build Lera Boroditsky: How language shapes the way we think José Andrés: How a team of chefs fed Puerto Rico after Hurricane Maria Tara Houska: The Standing Rock resistance and our fight for indigenous rights Kasiva Mutua: How I use the drum to tell my story Danny Hillis: Should we create a solar shade to cool the earth? Andrew Dent: To eliminate waste, we need to rediscover thrift Drew Philp: My $500 house in Detroit -- and the neighbors who helped me rebuild it Irina Kareva: Math can help uncover cancer's secrets Raphael Arar: How we can teach computers to make sense of our emotions Christian Picciolini: My descent into America's neo-Nazi movement -- and how I got out Judith Heumann: Our fight for disability rights -- and why we're not done yet Leo Igwe: Why I choose humanism over faith Ndidi Nwuneli: The role of faith and belief in modern Africa Erica Stone: Academic research is publicly funded -- why isn't it publicly available? Mennat El Ghalid: How fungi recognize (and infect) plants Vikram Sharma: How quantum physics can make encryption stronger Matthias Müllenbeck: What if we paid doctors to keep people healthy? Amishi Jha: How to tame your wandering mind Sauti Sol: The rhythm of Afrobeat Eve Abrams: The human stories behind mass incarceration Vittorio Loreto: Need a new idea? Start at the edge of what is known Soka Moses: For survivors of Ebola, the crisis isn't over Bob Stein: A rite of passage for late life Liz Ogbu: What if gentrification was about healing communities instead of displacing them? Adong Judith: How I use art to bridge misunderstanding Chris Nowinski: Can I have your brain? The quest for truth on concussions and CTE Sally Kohn: What we can do about the culture of hate Rei: "my mama" / "BLACK BANANA" Hadi Eldebek: Why must artists be poor? Isabel Wilkerson: The Great Migration and the power of a single decision

Data

Downloaded 1 time by Stanpol

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (1.63 MB) Use the API

rows 10 / 2795

link_high link_medium link_low event_url talk_url event_name time date talk_name
/talks?q=TEDMED+2015
/talks/sarah_gray_how_my_son_s_short_life_made_a_lasting_difference
TEDMED 2015
10:17
May 2016
Sarah Gray: How my son's short life made a lasting difference
/talks?q=TED2016
/talks/riccardo_sabatini_how_to_read_the_genome_and_build_a_human_being
TED2016
15:28
Apr 2016
Riccardo Sabatini: How to read the genome and build a human being
/talks?q=TED%40State+Street+Boston
/talks/michael_metcalfe_a_provocative_way_to_finance_the_fight_against_climate_change
TED@State Street Boston
12:52
Apr 2016
Michael Metcalfe: A provocative way to finance the fight against climate change
/talks?q=TED2016
/talks/ameera_harouda_why_i_put_myself_in_danger_to_tell_the_stories_of_gaza
TED2016
8:38
Apr 2016
Ameera Harouda: Why I put myself in danger to tell the stories of Gaza
/talks?q=TED2016
/talks/r_luke_dubois_insightful_human_portraits_made_from_data
TED2016
12:43
Apr 2016
R. Luke DuBois: Insightful human portraits made from data
/talks?q=TED2016
/talks/shivani_siroya_a_smart_loan_for_people_with_no_credit_history_yet
TED2016
8:11
Apr 2016
Shivani Siroya: A smart loan for people with no credit history (yet)
/talks?q=TED2016
/talks/kenneth_lacovara_hunting_for_dinosaurs_showed_me_our_place_in_the_universe
TED2016
15:49
Apr 2016
Kenneth Lacovara: Hunting for dinosaurs showed me our place in the universe
/talks?q=TEDxGatewayWomen
/talks/aditi_gupta_a_taboo_free_way_to_talk_about_periods
TEDxGatewayWomen
11:10
Apr 2016
Aditi Gupta: A taboo-free way to talk about periods
/talks?q=TED+Talks+Live
/talks/juan_enriquez_we_can_reprogram_life_how_to_do_it_wisely
TED Talks Live
14:49
Apr 2016
Juan Enriquez: We can reprogram life. How to do it wisely
/talks?q=TEDActive+2015
/talks/joshua_prager_wisdom_from_great_writers_on_every_year_of_life
TEDActive 2015
6:01
Apr 2016
Joshua Prager: Wisdom from great writers on every year of life

Statistics

Average successful run time: 2 minutes

Total run time: about 1 month

Total cpu time used: 6 minutes

Total disk space used: 3.33 MB

History

  • Auto ran revision e7e4a317 and completed successfully .
    36 records added, 34 records removed in the database
    2 pages scraped
  • Auto ran revision e7e4a317 and completed successfully .
    36 records added, 35 records removed in the database
    2 pages scraped
  • Auto ran revision e7e4a317 and completed successfully .
    36 records added, 35 records removed in the database
    2 pages scraped
  • Auto ran revision e7e4a317 and completed successfully .
    36 records added, 36 records removed in the database
    2 pages scraped
  • Auto ran revision e7e4a317 and completed successfully .
    36 records added, 36 records removed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

ted_com_scraper / scraper.py