openaustralia / example_ruby_chrome_headless_scraper

Example scraper showing how to use Chrome headless from a ruby scraper

Scrapes morph.io, faye.morph.io, www.gravatar.com, and 6 other domains

Get structured data out of the web. Code collaboration through GitHub. Run your scrapers in the cloud.

README.md

This is a simple scraper showing you how to use Chrome headless with Ruby. Here's what it does:

  1. Visits the morph.io home page
  2. Because the headless browser is a small window it now has to click the button to open the navigation menu so we can see the search box
  3. It enters a search for "planningalerts" into the search box and clicks the submit button
  4. After waiting for the results to appear (see the important gotcha in the code comments!) it outputs the full names of all the scrapers on the search results page

Any questions? Hit up the help forum.

Contributors mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... henare/planningalerts_scraperwiki_scrapers PuZZleDucK/rockingham-node planningalerts-scrapers/kiama openaustralia/PlanningAlerts_vanity_stats CloCkWeRX/sorell planningalerts-scrapers/sorell openaustralia/planningalerts_coverage openaustralia/jacaranda JasonThomasData/Hume_City_Council planningalerts-scrapers/logan

Statistics

Average successful run time: half a minute

Total run time: 3 minutes

Total cpu time used: half a minute

Total disk space used: 35.7 KB

History

  • Manually ran revision f65c09f0 and completed successfully .
    nothing changed in the database
    22 pages scraped
  • Manually ran revision 9e2d2474 and completed successfully .
    nothing changed in the database
    35 pages scraped
  • Manually ran revision 9e2d2474 and failed .
    1 page scraped
  • Manually ran revision 61331b7a and failed .
    2 pages scraped
  • Manually ran revision e28dc29b and failed .
  • ...
  • Created on morph.io

Show complete history