Downloads · 30 days
0
datasci-final-project-2024
when-my-cat-learn-datasci/datasci-final-project-2024
datasci-final-project-2024 is a machine learning model from when-my-cat-learn-datasci. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Presentations: https://youtu.be/z3PZ1i0JpA
Access
Public
Updated May 10, 2024
Repo size
2.3 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
Other3 GB · 68%
At a glance
- Access
- Public
- Created
- May 3, 2024
- Updated
- May 10, 2024
- SHA
- e4017f97
Model card
From the Hugging Face model README
This repository is for 2110446 Data Science and Data Engineering project.
Presentations: https://youtu.be/_z3PZ1i0JpA
How to run the project
Video: https://youtu.be/vnNhSXqAvZc
Prerequisites
- git
- docker
How to run
- Clone the repository
git clone https://huggingface.co/when-my-cat-learn-datasci/datasci-final-project-2024
- Change directory to the project
cd datasci-final-project-2024
- Start with command
chmod +x start.sh
./start.sh
OR
3 Build the docker image
docker compose build
- Start with docker-compose
docker-compose up
Project Structure
DataGathering
This folder is mainly for collecting data from other sources.
1. GoogleGeocoding
Gather geolocation of affiliation name using google geocoding API.
| Directory/File | Description |
|---|---|
| GoogleGeocoding.ipynb | Python jupiter notebook for gathering latitude and longitude of country |
| geocode_aff_country.csv | contains geolocation of each affiliation (aff_country, lat, lon) |
2. ScopusAPI
Query abstract scopus data using scopus API
| Directory/File | Description |
|---|---|
| ScopusAPI.ipynb | Python jupiter notebook for fetching abstract data from scopus API |
| example_abs_data.json | contains example data fetched from abstract API from scopus |
| example_abs_data.json | contains example data fetched from abstracts API from scopus |
3. SubjectCode
Webscraping scopus subject areas
| Directory/File | Description |
|---|---|
| SubjectCode.ipynb | Python jupiter notebook for scraping subject code from web |
| scopus_subject_areas.csv | contains subject code and it corresponds name (code,name) |
Pipeline
This folder is for data processing and data cleaning. It contains the following files:
| Directory/File | Description |
|---|---|
| Pipeline.ipynb | Python jupiter notebook for extract data from raw data |
| spark-3.5.1-bin-hadoop3.tgz | It will use in docker for connecting with spark |
Raw and Raw_Extra
This folder contains raw data and extra raw data that will be used in the project.
Visualization
| Directory/File | Description |
|---|---|
| dockerfile | Dockerfile with python image |
| main.py | Python script used by streamlit to visualize the data |
- Created
- May 3, 2024
- Updated
- May 10, 2024