Automated collection and handling of data

Abstract

This independent project involves web scraping, which is the process of extracting data from websites using automated software programs. The project specifically focuses on scraping Excel files and uploading index data to a database, with the aim of providing financial analysts and fund managers at Kvika Asset Management with valuable information on a variety of topics, including purchasing patterns and inflation. The project is being undertaken due to the lack of manpower at Kvika Asset Management.Web scraping is a useful technique that enables automated data collection from websites, and it is fair to assume that competitors in the sectors already have similar programs that are being updated regularly. These systems are typically built on following the main pillars of web-scraping systems, 1. A scraping program or module 2. a data patterns and recognition program and finally, 3. a SQL database. All these systems connect to create a seamless scraping system.