About Product
The product offers the ability to collect news from various news websites, perform data analysis and processing, and then display the results on the website in the desired format.
Business challenges
The tasks set by the client within the scope of the project were:
- To create a platform for parsing and aggregating a large number of online print publications.
- To gather all mentions of events or subjects that appeared in the media over a specific period of time.
- To identify when these mentions began and how media coverage of them developed over time.
- To investigate the emotional tone of these events over time and across different media outlets, quantified numerically.
- To automatically generate analytical reports based on this data, which can be manually supplemented from the editor’s dashboard
Key results
Within the scope of the task, the following methodologies and technologies were employed:
- Language recognition
- Machine translation
- Named entity recognition
- Construction of word vector representations
- Content categorization
- Design of a specialized database for the construction, vector representation, and search of graphs
- The system processes tens of thousands of new materials daily, building and handling hundreds of thousands of relations.
Our Solutions
- The client defines a list of media sources that are then fully parsed for all or newly created news items.
- Each news item is analyzed for named entities — individuals, administrative entities, geographical mentions, etc., as well as for its emotional tone.
- The data is vectorized in chunks and compared to all previously processed content.
- As a result, relationships between the vector representations of the materials are built in a graph database, allowing for the construction of a temporal and informational path for each event.
Technology stack
Programming and Frameworks:
Image Processing and Scientific Computing Libraries:
Machine Learning and Deep Learning:
Database Management Systems:
Messaging and Task Management:
Data Search:
Team
- 1 PM
- 1 Data Scientist
- 2 Python devs
- 1 DevOps
- 1 QA
