Stanpol / ted_com_scraper

ted.com talks


This is a scraper that runs on Morph.

It scrapes the latest talks from TED.com.

Contributors Stanpol

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Python app detected  ! The latest version of Python 2 is python-2.7.14 (you are using python-2.7.9, which is unsupported).  ! We recommend upgrading by specifying the latest version (python-2.7.14).  Learn More: https://devcenter.heroku.com/articles/python-runtimes -----> Installing python-2.7.9 -----> Installing pip -----> Installing requirements with pip  DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 6))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to revision morph_defaults) to /app/.heroku/src/scraperwiki  Running command git clone -q http://github.com/openaustralia/scraperwiki-python.git /app/.heroku/src/scraperwiki  Running command git checkout -b morph_defaults --track origin/morph_defaults  Switched to a new branch 'morph_defaults'  Branch morph_defaults set up to track remote branch morph_defaults from origin.  Collecting lxml==3.4.4  Downloading lxml-3.4.4.tar.gz (3.5 MB)  Collecting bs4==0.0.1  Downloading bs4-0.0.1.tar.gz (1.1 kB)  Collecting cssselect==0.9.1  Downloading cssselect-0.9.1.tar.gz (32 kB)  Collecting dumptruck>=0.1.2  Downloading dumptruck-0.1.6.tar.gz (15 kB)  Collecting requests  Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)  Collecting beautifulsoup4  Downloading beautifulsoup4-4.9.3-py2-none-any.whl (115 kB)  Collecting idna<3,>=2.5; python_version < "3"  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)  Collecting certifi>=2017.4.17  Downloading certifi-2021.10.8-py2.py3-none-any.whl (149 kB)  Collecting chardet<5,>=3.0.2; python_version < "3"  Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)  Collecting urllib3<1.27,>=1.21.1  Downloading urllib3-1.26.9-py2.py3-none-any.whl (138 kB)  Collecting soupsieve<2.0,>1.2; python_version < "3.0"  Downloading soupsieve-1.9.6-py2.py3-none-any.whl (33 kB)  Collecting backports.functools-lru-cache; python_version < "3"  Downloading backports.functools_lru_cache-1.6.4-py2.py3-none-any.whl (5.9 kB)  Building wheels for collected packages: lxml, bs4, cssselect, dumptruck  Building wheel for lxml (setup.py): started  Building wheel for lxml (setup.py): still running...  Building wheel for lxml (setup.py): finished with status 'done'  Created wheel for lxml: filename=lxml-3.4.4-cp27-cp27m-linux_x86_64.whl size=2989850 sha256=5b3aef6801436f66a630b965919b4730f81e2f0ac161cf9dea03d5e86a6e02a8  Stored in directory: /tmp/pip-ephem-wheel-cache-fJnFi1/wheels/d6/de/81/11ae6edd05c75aac677e67dd154c85da758ba6f3e8e80e962e  Building wheel for bs4 (setup.py): started  Building wheel for bs4 (setup.py): finished with status 'done'  Created wheel for bs4: filename=bs4-0.0.1-py2-none-any.whl size=1273 sha256=abf8e7066b42dbd401e4e91936d9f4475838cbf09decdb8a130ce0b3131aef1c  Stored in directory: /tmp/pip-ephem-wheel-cache-fJnFi1/wheels/98/b9/dc/90f1e36fc6bf9564491a69c9c3d7ae38b8f72986256e416be6  Building wheel for cssselect (setup.py): started  Building wheel for cssselect (setup.py): finished with status 'done'  Created wheel for cssselect: filename=cssselect-0.9.1-py2-none-any.whl size=26992 sha256=3f89f18913a55a5e29dfaf3ecf7c53facbac034aa862eef5b1d2e39689672b3a  Stored in directory: /tmp/pip-ephem-wheel-cache-fJnFi1/wheels/85/fe/00/b94036d8583cec9791d8cda24c184f2d2ac1397822f7f0e8d4  Building wheel for dumptruck (setup.py): started  Building wheel for dumptruck (setup.py): finished with status 'done'  Created wheel for dumptruck: filename=dumptruck-0.1.6-py2-none-any.whl size=11844 sha256=8fc58eccf7a2a5bbf6f649124d63d677b3fc604d4bac4db0138447ec3ed14aec  Stored in directory: /tmp/pip-ephem-wheel-cache-fJnFi1/wheels/dc/75/e9/1e61c4080c73e7bda99614549591f83b53bcc2d682f26fce62  Successfully built lxml bs4 cssselect dumptruck  Installing collected packages: dumptruck, idna, certifi, chardet, urllib3, requests, scraperwiki, lxml, backports.functools-lru-cache, soupsieve, beautifulsoup4, bs4, cssselect  Running setup.py develop for scraperwiki  Successfully installed backports.functools-lru-cache-1.6.4 beautifulsoup4-4.9.3 bs4-0.0.1 certifi-2021.10.8 chardet-4.0.0 cssselect-0.9.1 dumptruck-0.1.6 idna-2.10 lxml-3.4.4 requests-2.27.1 scraperwiki soupsieve-1.9.6 urllib3-1.26.9 DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.    -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Scraping the latest page Deepa Purushothaman: The workplace, redefined by women of color Dorie Clark: The real reason you feel so busy (and what to do about it) Diana Sierra Becerra: The rebel radio that brought down a war criminal Adam Mosseri: A creator-led internet, built on blockchain Resson Kantai Duff: Why Africa needs community-led conservation Bektour Iskender: The crime-fighting power of cross-border investigative journalism Jonathan Smith: Does trickle-down economics actually work? Lucie Pinson: How to stop banks from investing in dirty energy Juliet Schor: The case for a 4-day work week Kayla Wolf: The biggest mistakes in mapmaking history Srishti Bakshi: My long walk across India for women's freedom Ndidi Okonkwo Nwuneli: The future of the food ecosystem -- and the power of your plate Edith Elliott and Shahed Alam: The most powerful untapped resource in health care Amanda Renteria: A bold plan to transform access to the US social safety net Anushka Ratnayake: A transparent, easy way for smallholder farmers to save Tiana Epps-Johnson: An election redesign to restore trust in US democracy ! No links available Tima Kurdi: How one photo captured a humanitarian crisis Iseult Gillespie: Love vs. Honor: The Irish myth of Diarmuid's betrayal Celina de Sola: Mental health care that disrupts cycles of violence Sue Natali: How ancient Arctic carbon threatens everyone on the planet Lucia Carminati: How the Suez Canal changed the world Nonette Royo: Why Indigenous forest guardianship is crucial to climate action Becca Heller: A safe pathway to resettlement for migrants and refugees Michèle Lamont: How to heal a divided world Becky Barker: The restorative power of medical tattoos Shannon Zhao: The myth of the original star-crossed lovers Melissa J. Moore: How mRNA medicine will change the world Henri Picciotto: Can you solve the feeding frenzy riddle? Temie Giwa-Tubosun: The funding gap in start-up investing John C. Mather: How the James Webb Space Telescope will unfold the universe Bill Gates: We can make COVID-19 the last pandemic Shervin Khodabandeh: Why people and AI make good business partners Christopher M. Moreman: The dark history of zombies Nat Keohane: A new economic model for protecting tropical forests Allyson Felix: An Olympic champion's mindset for overcoming fear Ameya Gondhalekar: Why are cockroaches so hard to kill?

Data

Downloaded 1 time by Stanpol

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (3.23 MB) Use the API

rows 10 / 5062

link_high link_medium link_low event_url talk_url event_name time date talk_name
/talks?q=TEDMED+2015
/talks/sarah_gray_how_my_son_s_short_life_made_a_lasting_difference
TEDMED 2015
10:17
May 2016
Sarah Gray: How my son's short life made a lasting difference
/talks?q=TED2016
/talks/riccardo_sabatini_how_to_read_the_genome_and_build_a_human_being
TED2016
15:28
Apr 2016
Riccardo Sabatini: How to read the genome and build a human being
/talks?q=TED%40State+Street+Boston
/talks/michael_metcalfe_a_provocative_way_to_finance_the_fight_against_climate_change
TED@State Street Boston
12:52
Apr 2016
Michael Metcalfe: A provocative way to finance the fight against climate change
/talks?q=TED2016
/talks/ameera_harouda_why_i_put_myself_in_danger_to_tell_the_stories_of_gaza
TED2016
8:38
Apr 2016
Ameera Harouda: Why I put myself in danger to tell the stories of Gaza
/talks?q=TED2016
/talks/r_luke_dubois_insightful_human_portraits_made_from_data
TED2016
12:43
Apr 2016
R. Luke DuBois: Insightful human portraits made from data
/talks?q=TED2016
/talks/shivani_siroya_a_smart_loan_for_people_with_no_credit_history_yet
TED2016
8:11
Apr 2016
Shivani Siroya: A smart loan for people with no credit history (yet)
/talks?q=TED2016
/talks/kenneth_lacovara_hunting_for_dinosaurs_showed_me_our_place_in_the_universe
TED2016
15:49
Apr 2016
Kenneth Lacovara: Hunting for dinosaurs showed me our place in the universe
/talks?q=TEDxGatewayWomen
/talks/aditi_gupta_a_taboo_free_way_to_talk_about_periods
TEDxGatewayWomen
11:10
Apr 2016
Aditi Gupta: A taboo-free way to talk about periods
/talks?q=TED+Talks+Live
/talks/juan_enriquez_we_can_reprogram_life_how_to_do_it_wisely
TED Talks Live
14:49
Apr 2016
Juan Enriquez: We can reprogram life. How to do it wisely
/talks?q=TEDActive+2015
/talks/joshua_prager_wisdom_from_great_writers_on_every_year_of_life
TEDActive 2015
6:01
Apr 2016
Joshua Prager: Wisdom from great writers on every year of life

Statistics

Average successful run time: 2 minutes

Total run time: about 1 month

Total cpu time used: 23 minutes

Total disk space used: 5.14 MB

History

  • Auto ran revision e7e4a317 and completed successfully .
    36 records added, 30 records removed in the database
  • Auto ran revision e7e4a317 and failed .
    nothing changed in the database
  • Auto ran revision e7e4a317 and completed successfully .
    36 records added, 36 records removed in the database
  • Auto ran revision e7e4a317 and completed successfully .
    36 records added, 36 records removed in the database
  • Auto ran revision e7e4a317 and completed successfully .
    36 records added, 35 records removed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

ted_com_scraper / scraper.py