benedict-chan / scraper-astro-databank-pageid

A scraper to scrape the page ids of astro databank, for https://morph.io

Complete history

• Manually ran revision 17f4c029 and failed 2015-05-25.
  run time 3 days
  26447 pages scraped
• Manually ran revision e6e4baa9 and failed 2015-05-21.
  run time about 2 hours
  11704 records added in the database
  11729 pages scraped
• Manually ran revision c5ec91f9 and completed successfully 2015-05-21.
  run time 18 s
  10 records added in the database
  11 pages scraped
• Manually ran revision c5ec91f9 and completed successfully 2015-05-21.
  run time 21 s
  10 records added, 10 records removed, 37976 records updated in the database
  11 pages scraped
• Manually ran revision 45994947 and completed successfully 2015-05-19.
  run time 4 minutes
  37986 records added, 37500 records removed in the database
  76 pages scraped
• Manually ran revision 1af262a0 and completed successfully 2015-05-19.
  run time 3 minutes
  37500 records added in the database
  76 pages scraped
• Manually ran revision 94a589fe and failed 2015-05-19.
  run time 3 minutes
  37500 records added in the database
  76 pages scraped
• Created on morph.io 2015-05-19