You can do this by using a headless browser. On morph.io Google Chrome is pre-installed ready for you to use.
The Google Chrome binary is installed at
/usr/bin/google-chrome on every morph.io container
as part of the build process.
You can use Google Chrome directly by running it with
google-chrome --headless --disable-gpu or
you can control it via WebDriver with ChromeDriver
which is installed on morph.io at
With Capybara and Selenium Webdriver you can control Chrome. To install them add them to your scraper Gemfile:
gem 'capybara' gem 'selenium-webdriver'
Then in your scraper start a Capybara session using
require "capybara" require "selenium-webdriver" capybara = Capybara::Session.new(:selenium_chrome_headless) # Start scraping capybara.visit("https://morph.io/") puts capybara.find("#banner h2").text
Sometimes there's nothing like seeing a real-life example. If you have a scraper you would like to add to this list, please let us know.
morph.io, faye.morph.io, www.gravatar.com, and 6 others