Stanpol / ted_com_scraper

ted.com talks


This is a scraper that runs on Morph.

It scrapes the latest talks from TED.com.

Contributors Stanpol

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Python app detected  ! The latest version of Python 2 is python-2.7.14 (you are using python-2.7.9, which is unsupported).  ! We recommend upgrading by specifying the latest version (python-2.7.14).  Learn More: https://devcenter.heroku.com/articles/python-runtimes -----> Installing python-2.7.9 -----> Installing pip -----> Installing requirements with pip  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 6))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to morph_defaults) to /app/.heroku/src/scraperwiki  Collecting lxml==3.4.4 (from -r /tmp/build/requirements.txt (line 8))  Downloading lxml-3.4.4.tar.gz (3.5MB)  Collecting bs4==0.0.1 (from -r /tmp/build/requirements.txt (line 9))  Downloading bs4-0.0.1.tar.gz  Collecting cssselect==0.9.1 (from -r /tmp/build/requirements.txt (line 10))  Downloading cssselect-0.9.1.tar.gz  Collecting dumptruck>=0.1.2 (from scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading dumptruck-0.1.6.tar.gz  Collecting requests (from scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading requests-2.18.4-py2.py3-none-any.whl (88kB)  Collecting beautifulsoup4 (from bs4==0.0.1->-r /tmp/build/requirements.txt (line 9))  Downloading beautifulsoup4-4.6.0-py2-none-any.whl (86kB)  Collecting idna<2.7,>=2.5 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading idna-2.6-py2.py3-none-any.whl (56kB)  Collecting urllib3<1.23,>=1.21.1 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading urllib3-1.22-py2.py3-none-any.whl (132kB)  Collecting certifi>=2017.4.17 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading certifi-2018.1.18-py2.py3-none-any.whl (151kB)  Collecting chardet<3.1.0,>=3.0.2 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB)  Installing collected packages: dumptruck, idna, urllib3, certifi, chardet, requests, scraperwiki, lxml, beautifulsoup4, bs4, cssselect  Running setup.py install for dumptruck: started  Running setup.py install for dumptruck: finished with status 'done'  Running setup.py develop for scraperwiki  Running setup.py install for lxml: started  Running setup.py install for lxml: still running...  Running setup.py install for lxml: finished with status 'done'  Running setup.py install for bs4: started  Running setup.py install for bs4: finished with status 'done'  Running setup.py install for cssselect: started  Running setup.py install for cssselect: finished with status 'done'  Successfully installed beautifulsoup4-4.6.0 bs4-0.0.1 certifi-2018.1.18 chardet-3.0.4 cssselect-0.9.1 dumptruck-0.1.6 idna-2.6 lxml-3.4.4 requests-2.18.4 scraperwiki urllib3-1.22   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Scraping the latest page Azim Khamisa and Ples Felix: What comes after tragedy? Forgiveness Cleo Wade: Want to change the world? Start by being brave enough to care Anna Rosling Rönnlund: See how the rest of the world lives, organized by income Bob Inglis: American bipartisan politics can be saved -- here's how Scott Williams: The hidden role informal caregivers play in health care Michelle Knox: Talk about your death while you're still healthy Matt Goldman: The search for "aha!" moments Vivek Maru: How to put the power of law in people's hands Jacob Collier: A one-man musical phenomenon Marily Oppezzo: Want to be more creative? Go for a walk Alexis Charpentier: How record collectors find lost music and preserve our cultural heritage Kevin Njabo: How we can stop Africa's scientific brain drain Soyapi Mumba: Medical tech designed to meet Africa's needs Mindy Scheier: How adaptive clothing empowers people with disabilities Touria El Glaoui: Inside Africa's thriving art scene Stewart Brand and Chris Anderson: Mammoths resurrected, geoengineering and other thoughts from a futurist Sue Jaye Johnson: What we don't teach kids about sex Arik Hartmann: Our treatment of HIV has advanced. Why hasn't the stigma changed? Lana Mazahreh: 3 thoughtful ways to conserve water Lisa Feldman Barrett: You aren't at the mercy of your emotions -- your brain creates them Christopher Ategeka: How adoption worked for me Heather Lanier: "Good" and "bad" are incomplete stories we tell ourselves Christian Benimana: The next generation of African architects and designers Deborah Willis and Hank Willis Thomas: A mother and son united by love and art Yvette Alberdingk Thijm: The power of citizen video to create undeniable truths Joel Jackson: A vehicle built in Africa, for Africa Tiffany Watt Smith: The history of human emotions Javed Akhtar: The gift of words Atul Gawande: Want to get great at something? Get a coach Angela Wang: How China is changing the future of shopping David Brenner: A new weapon in the fight against superbugs Kamau Gachigi: Success stories from Kenya's first makerspace Miho Janvier: Lessons from a solar storm chaser Joan Blades and John Gable: Free yourself from your filter bubbles Xavier De Kestelier: Adventures of an interplanetary architect Gautam Bhan: A bold plan to house 100 million people

Data

Downloaded 0 times

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (1.55 MB) Use the API

rows 10 / 2689

link_high link_medium link_low event_url talk_url event_name time date talk_name
/talks?q=TEDMED+2015
/talks/sarah_gray_how_my_son_s_short_life_made_a_lasting_difference
TEDMED 2015
10:17
May 2016
Sarah Gray: How my son's short life made a lasting difference
/talks?q=TED2016
/talks/riccardo_sabatini_how_to_read_the_genome_and_build_a_human_being
TED2016
15:28
Apr 2016
Riccardo Sabatini: How to read the genome and build a human being
/talks?q=TED%40State+Street+Boston
/talks/michael_metcalfe_a_provocative_way_to_finance_the_fight_against_climate_change
TED@State Street Boston
12:52
Apr 2016
Michael Metcalfe: A provocative way to finance the fight against climate change
/talks?q=TED2016
/talks/ameera_harouda_why_i_put_myself_in_danger_to_tell_the_stories_of_gaza
TED2016
8:38
Apr 2016
Ameera Harouda: Why I put myself in danger to tell the stories of Gaza
/talks?q=TED2016
/talks/r_luke_dubois_insightful_human_portraits_made_from_data
TED2016
12:43
Apr 2016
R. Luke DuBois: Insightful human portraits made from data
/talks?q=TED2016
/talks/shivani_siroya_a_smart_loan_for_people_with_no_credit_history_yet
TED2016
8:11
Apr 2016
Shivani Siroya: A smart loan for people with no credit history (yet)
/talks?q=TED2016
/talks/kenneth_lacovara_hunting_for_dinosaurs_showed_me_our_place_in_the_universe
TED2016
15:49
Apr 2016
Kenneth Lacovara: Hunting for dinosaurs showed me our place in the universe
/talks?q=TEDxGatewayWomen
/talks/aditi_gupta_a_taboo_free_way_to_talk_about_periods
TEDxGatewayWomen
11:10
Apr 2016
Aditi Gupta: A taboo-free way to talk about periods
/talks?q=TED+Talks+Live
/talks/juan_enriquez_we_can_reprogram_life_how_to_do_it_wisely
TED Talks Live
14:49
Apr 2016
Juan Enriquez: We can reprogram life. How to do it wisely
/talks?q=TEDActive+2015
/talks/joshua_prager_wisdom_from_great_writers_on_every_year_of_life
TEDActive 2015
6:01
Apr 2016
Joshua Prager: Wisdom from great writers on every year of life

Statistics

Average successful run time: 2 minutes

Total run time: about 1 month

Total cpu time used: 5 minutes

Total disk space used: 3.25 MB

History

  • Auto ran revision e7e4a317 and completed successfully .
    36 records added, 36 records removed in the database
  • Auto ran revision e7e4a317 and completed successfully .
    36 records added, 35 records removed in the database
  • Auto ran revision e7e4a317 and completed successfully .
    36 records added, 29 records removed in the database
    2 pages scraped
  • Auto ran revision e7e4a317 and failed .
    nothing changed in the database
  • Auto ran revision e7e4a317 and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

ted_com_scraper / scraper.py