blimpage / sash_windows

Sash windows for sale on renovaterestorerecycle.com.au

README.md

Oh hi! This is the code for a scraper that runs daily on Morph.io. It finds all of the sash windows available for sale at https://renovaterestorerecycle.com.au/ and https://www.renovatorsparadise.com.au/, and sends an email (via Sendgrid) to a nominated email address if any new windows are found.

If that sounds incredibly niche and not very interesting, that's because it is!

Contributors blimpage

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... [1G [1G-----> Ruby app detected [1G-----> Compiling Ruby/Rack [1G-----> Using Ruby version: ruby-2.6.2 [1G-----> Installing dependencies using bundler version 1.17.2 [1G Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment [1G Fetching gem metadata from https://rubygems.org/........... [1G Fetching https://github.com/openaustralia/scraperwiki-ruby.git [1G Using bundler 1.17.2 [1G Fetching httpclient 2.6.0.1 [1G Fetching mime-types 2.5 [1G Fetching unf_ext 0.0.7.1 [1G Installing unf_ext 0.0.7.1 with native extensions [1G Installing httpclient 2.6.0.1 [1G Fetching net-http-digest_auth 1.4 [1G Installing net-http-digest_auth 1.4 [1G Fetching net-http-persistent 2.9.4 [1G Installing mime-types 2.5 [1G Installing net-http-persistent 2.9.4 [1G Fetching mini_portile2 2.4.0 [1G Fetching ntlm-http 0.1.1 [1G Installing mini_portile2 2.4.0 [1G Installing ntlm-http 0.1.1 [1G Fetching webrobots 0.1.1 [1G Fetching mustermann 1.0.3 [1G Installing webrobots 0.1.1 [1G Installing mustermann 1.0.3 [1G Fetching rack 2.0.7 [1G Installing rack 2.0.7 [1G Fetching ruby_http_client 3.3.0 [1G Installing ruby_http_client 3.3.0 [1G Fetching sqlite3 1.3.10 [1G Installing sqlite3 1.3.10 with native extensions [1G Fetching tilt 2.0.9 [1G Installing tilt 2.0.9 [1G Fetching nokogiri 1.10.8 [1G Installing nokogiri 1.10.8 with native extensions [1G Fetching rack-protection 2.0.5 [1G Installing rack-protection 2.0.5 [1G Fetching unf 0.1.4 [1G Installing unf 0.1.4 [1G Fetching sinatra 2.0.5 [1G Installing sinatra 2.0.5 [1G Fetching domain_name 0.5.24 [1G Installing domain_name 0.5.24 [1G Fetching sendgrid-ruby 5.3.0 [1G Installing sendgrid-ruby 5.3.0 [1G Fetching http-cookie 1.0.2 [1G Installing http-cookie 1.0.2 [1G Fetching sqlite_magic 0.0.3 [1G Installing sqlite_magic 0.0.3 [1G Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176) [1G Fetching mechanize 2.7.3 [1G Installing mechanize 2.7.3 [1G Bundle complete! 3 Gemfile dependencies, 24 gems now installed. [1G Gems in the groups development and test were not installed. [1G Bundled gems are installed into `./vendor/bundle` [1G Removing bundler (1.15.2) [1G Bundle completed (13.98s) [1G Cleaning up the bundler cache. [1G-----> Detecting rake tasks [1G [1G [1G-----> Discovering process types [1G Procfile declares types -> scraper Injecting scraper and running... Scraping renovaterestorerecycle.com.au: Scraping page: https://www.renovaterestorerecycle.com.au/category/sash-windows/?post_type=rrr_stock 12 products found. Scraping page: https://www.renovaterestorerecycle.com.au/category/sash-windows/page/2/?post_type=rrr_stock 12 products found. Scraping page: https://www.renovaterestorerecycle.com.au/category/sash-windows/page/3/?post_type=rrr_stock 7 products found. 3 pages scraped. 31 total products found. 0 available product(s) found. Scraping renovatorsparadise.com.au: Finding pages to scrape. 7 page(s) of interest found. Scraping page: https://www.renovatorsparadise.com.au/product-category/windows/single-sash-window/interwar-sash/interwar-0-1395-mm-high/ 24 product(s) found. Scraping page: https://www.renovatorsparadise.com.au/product-category/windows/single-sash-window/interwar-sash/interwar-1400-1595-high/ 17 product(s) found. Scraping page: https://www.renovatorsparadise.com.au/product-category/windows/single-sash-window/interwar-sash/interwar-1600-1895-high/ 15 product(s) found. Scraping page: https://www.renovatorsparadise.com.au/product-category/windows/single-sash-window/interwar-sash/interwar-1900-mm-high/ 13 product(s) found. Scraping page: https://www.renovatorsparadise.com.au/product-category/windows/single-sash-window/leadlight-sash/ 14 product(s) found. Scraping page: https://www.renovatorsparadise.com.au/product-category/windows/sash-window/ 29 product(s) found. Scraping page: https://www.renovatorsparadise.com.au/product-category/windows/triple-quadruple-sash/ 45 product(s) found. 7 pages scraped. 157 total products found. 78 new product(s) found! Sending email notification... Something went wrong while sending email. Response: #<SendGrid::Response:0x00005575c32c5e00 @status_code="401", @body="{\"errors\":[{\"message\":\"Authenticated user is not authorized to send mail\",\"field\":null,\"help\":null}]}", @headers={"server"=>["nginx"], "date"=>["Thu, 11 Apr 2024 23:15:31 GMT"], "content-type"=>["application/json"], "content-length"=>["101"], "connection"=>["close"], "access-control-allow-origin"=>["https://sendgrid.api-docs.io"], "access-control-allow-methods"=>["POST"], "access-control-allow-headers"=>["Authorization, Content-Type, On-behalf-of, x-sg-elas-acl"], "access-control-max-age"=>["600"], "x-no-cors-reason"=>["https://sendgrid.com/docs/Classroom/Basics/API/cors.html"], "strict-transport-security"=>["max-age=600; includeSubDomains"]}> New products saved to the database. All done! Bye!

Data

Downloaded 2 times by blimpage

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (144 KB) Use the API

rows 10 / 591

name description price url
Pair of Victorian Single Sash Windows.
1185 wide x 2060 high.
$1090
https://www.renovaterestorerecycle.com.au/rrr-stock/pair-of-victorian-single-sash-windows/
Pair of Victorian Sidelight Sash Windows.
2 @ 2220 wide x 2145 high. Both centre sashes + smaller sidelight sashes open + close on each window. All the architrave ( internal + external sides ) is also included.
$3500
https://www.renovaterestorerecycle.com.au/rrr-stock/pair-of-victorian-sidelight-sash-windows/
Obscure Etched Glass Single Sash Window.
1110 wide x 1935 high.
$490
https://www.renovaterestorerecycle.com.au/rrr-stock/obscure-etched-glass-single-sash-window/
House Lot of 3 Single Sash Windows.
3 @ 930 wide x 2145 high.
$690 each
https://www.renovaterestorerecycle.com.au/rrr-stock/house-lot-of-12-single-sash-windows/
Victorian Federation Single Sash Window.
900 wide x 1340 high.
$290
https://www.renovaterestorerecycle.com.au/rrr-stock/victorian-federation-single-sash-window/
Sandblasted Obscure Glass Single Sash Window.
1025 wide x 1510 high.
$490
https://www.renovaterestorerecycle.com.au/rrr-stock/sandblasted-obscure-glass-single-sash-window/
8 Light Colonial Single Sash Window.
1040 wide x 1550 high.
$240
https://www.renovaterestorerecycle.com.au/rrr-stock/8-light-colonial-single-sash-window/
Colonial Single Sash Window with Opening Fanlight.
995 wide x 2440 high.
$1250
https://www.renovaterestorerecycle.com.au/rrr-stock/colonial-single-sash-window-with-opening-fanlight/
Cricket Bat Victorian Single Sash Window.
1110 wide x 1995 high.
$550
https://www.renovaterestorerecycle.com.au/rrr-stock/cricket-bat-victorian-single-sash-window/
4 Light Colonial Single Sash Window.
1100 wide x 1400 high.
$240
https://www.renovaterestorerecycle.com.au/rrr-stock/4-light-colonial-single-sash-window/

Statistics

Average successful run time: 2 minutes

Total run time: 1 day

Total cpu time used: 30 minutes

Total disk space used: 221 KB

History

  • Manually ran revision 476d08bb and completed successfully .
    77 records added in the database
  • Auto ran revision 476d08bb and failed .
    nothing changed in the database
  • Auto ran revision 476d08bb and failed .
    nothing changed in the database
  • Auto ran revision 476d08bb and failed .
    nothing changed in the database
  • Auto ran revision 476d08bb and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

sash_windows / scraper.rb