ninurtalabs / biz_agrix_news_scraper

Agriculture and business news from different RSS feeds


This scraper scrapes multiple RSS feeds for business and agriculture news. It's hosted on morph.io

Working

  1. Every morning, the scraper runs and stores the day's news in a SQLLite table.
  2. When the scraping is complete, the scraper notifies an AWS Lambda defined here via an AWS API Gateway endpoint
  3. The Lambda gets the day's news by hitting the scraper's data extraction endpoint
  4. The Lambda then appends the data to a DynamoDB table

Contributors ninurtalabs kelvingakuo

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Python app detected -----> Installing python-3.6.2 -----> Installing pip -----> Installing requirements with pip  Collecting apsw  Downloading apsw-3.9.2-r1.tar.gz (2.6 MB)  Collecting feedparser==5.2.1  Downloading feedparser-5.2.1.tar.bz2 (192 kB)  Collecting webpreview==1.6.0  Downloading webpreview-1.6.0-py3-none-any.whl (9.2 kB)  Collecting beautifulsoup4>=4.4.0  Downloading beautifulsoup4-4.8.2-py3-none-any.whl (106 kB)  Collecting requests>=2.10.0  Downloading requests-2.23.0-py2.py3-none-any.whl (58 kB)  Collecting soupsieve>=1.2  Downloading soupsieve-2.0-py2.py3-none-any.whl (32 kB)  Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1  Downloading urllib3-1.25.8-py2.py3-none-any.whl (125 kB)  Collecting idna<3,>=2.5  Downloading idna-2.9-py2.py3-none-any.whl (58 kB)  Collecting chardet<4,>=3.0.2  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)  Collecting certifi>=2017.4.17  Downloading certifi-2019.11.28-py2.py3-none-any.whl (156 kB)  Building wheels for collected packages: apsw, feedparser  Building wheel for apsw (setup.py): started  Building wheel for apsw (setup.py): finished with status 'done'  Created wheel for apsw: filename=apsw-3.9.2.post1-cp36-cp36m-linux_x86_64.whl size=1922564 sha256=6e061c50674d3b211871805c2fd12691e70198082b6dbac2b45c55777ceeaa17  Stored in directory: /tmp/pip-ephem-wheel-cache-uio6ezyd/wheels/6c/9e/d9/c34e5ed7b56a012d023fb66b1f615ceefbf9fd813ba01cf21a  Building wheel for feedparser (setup.py): started  Building wheel for feedparser (setup.py): finished with status 'done'  Created wheel for feedparser: filename=feedparser-5.2.1-py3-none-any.whl size=44939 sha256=49d5a1229a9998bcb97abd7ffe3c0d536f32174edbb63825604b15f125348222  Stored in directory: /tmp/pip-ephem-wheel-cache-uio6ezyd/wheels/2f/70/50/16138238f6447854a1300f7d457766789dc7122cfaf46bcef2  Successfully built apsw feedparser  Installing collected packages: apsw, feedparser, soupsieve, beautifulsoup4, urllib3, idna, chardet, certifi, requests, webpreview  Successfully installed apsw-3.9.2.post1 beautifulsoup4-4.8.2 certifi-2019.11.28 chardet-3.0.4 feedparser-5.2.1 idna-2.9 requests-2.23.0 soupsieve-2.0 urllib3-1.25.8 webpreview-1.6.0   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running...

Data

Downloaded 5 times by ninurtalabs

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (43 KB) Use the API

rows 10 / 85

image link source summary title year month day
standard_agrix
Though his farm is modest in size, he has ensured that all the birds have enough room to forage, rest and live without feeling cramped for space.
How I rear chicken, turkey in one space
2020
3
28
standard_agrix
As the impact of Covid-19 begins to hit home, farmers are also feeling the heat and to stay afloat, they are taking drastic but smart measures
Lessons on farming during a health crisis
2020
3
28
standard_agrix
Flower exports wither as demand drops
2020
3
28
standard_agrix
Animal welfare is also important in this health pandemic season. This is not the time to focus on our health and forget the animal.
Virus crisis: As you plan, don't forget the animals
2020
3
28
standard_agrix
At Chepseon Dairies, to maximise production and minimise contamination, everything has been automated from feeding to milking
Lessons from 300-acre ultra-modern dairy unit
2020
3
28
standard_agrix
Having been at it for more than 20 years they know everything from pest control to harvest and storage.
2020
3
28
standard_agrix
In Kenya at least 40 to 50 per cent of the fruits and vegetables produced are usually lost or wasted along the supply chain.
Ensuring your perishables last longer in the time of coronavirus
2020
3
28
standard_agrix
Despite being a lucrative venture, farmers always face low production and shrunk profits, often due to disease infestations
Increasing yields: How to control common diseases in potatoes
2020
3
28
standard_agrix
The Laikipia County land bank initiative aims at facilitating leasing of idle land for commercial use
Laikipia to establish land bank
2020
3
28
standard_agrix
A Sh400 million milk processing facility in Nandi County could help restore hope in farmers who have struggled with low prices for long.
Sh400 million factory boost for farmers
2020
3
28

Statistics

Average successful run time: 7 minutes

Total run time: about 1 hour

Total cpu time used: 3 minutes

Total disk space used: 96.3 KB

History

  • Auto ran revision c822f4df and completed successfully .
    85 records updated in the database
  • Auto ran revision c822f4df and completed successfully .
    85 records updated in the database
  • Auto ran revision c822f4df and completed successfully .
    85 records updated in the database
  • Auto ran revision c822f4df and completed successfully .
    85 records updated in the database
  • Auto ran revision c822f4df and completed successfully .
    85 records updated in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

biz_agrix_news_scraper / scraper.py