mahjong_vision

Mahjong Vision Assistant

This project uses computer vision and machine learning to provide real-time discard suggestions for the game Mahjong Soul.

Features

• Tile Recognition: Identifies Mahjong tiles from the Mahjong Soul game window using a fine-tuned Vision Transformer model (pjura/mahjong_soul_vision).
• Game State Analysis: Parses the recognized tiles to understand the current game state (player's hand, melds, discard pools).
• Discard Suggestion: Employs a neural network (ImprovedNN), based on the architecture from the pjura/mahjong_ai repository, to predict the optimal discard based on the analyzed game state.
• Live Overlay: Captures the game window and overlays suggestions directly onto the screen, highlighting the recommended discard tile.

Project Structure

• live_feed.py: The main script to run the live assistant. It captures the screen, performs tile recognition, predicts discards, and displays the overlay.
• hf_vision_model.ipynb: Jupyter notebook detailing the training process for the Hugging Face Vision Transformer used for tile recognition.
• tools.py: Contains utility functions for data processing, model prediction, loss calculation, MLflow interaction, and tile representation translation used by live_feed.py. Many cross repo functions.
• model.safetensors: Saved weights for the discard prediction neural network (ImprovedNN).

Setup

1. Environment: Ensure you have Python installed along with necessary libraries. Key libraries include:

   • torch (with CUDA support if available)
   • transformers
   • datasets
   • evaluate
   • opencv-python (cv2)
   • Pillow (PIL)
   • pygetwindow
   • numpy
   • pyautogui
   • keyboard
   • safetensors
   • mlflow (Optional, used in tools.py, you can use whatever you like to serve the model)
   • scipy
   • matplotlib

   (A requirements.txt file would be beneficial here, but didn't made one at the time)

2. Models:

   • The tile recognition model (pjura/mahjong_soul_vision) will be downloaded automatically by the transformers library.
   • The discard prediction model (model.safetensors) should be present in the root directory.

Usage

1. Ensure the Mahjong Soul game window is open and titled "MahjongSoul".
2. Run the main script:

   python live_feed.py
   

3. The script will capture the game window, analyze the tiles, and highlight the suggested discard tile in the player's hand region. The color of the highlight indicates the model's confidence (Green=High, Red=Low).
4. Press 'q' to quit the application.
5. Auto-Click: When it is your turn (14 tiles in hand/melds) and a suggestion is highlighted, hold the Spacebar to automatically move the mouse and click the suggested tile. If Spacebar is not held, only the highlight will be shown.

Notes

• The script relies on specific window coordinates and aspect ratios which might need adjustment depending on screen resolution and game layout.
• The discard prediction model architecture (ImprovedNN) originates from the pjura/mahjong_ai repository. The included model.safetensors file is an example set of weights for this model, also from that repository, but potentially not the latest version. It was trained on the pjura/mahjong_board_states dataset, primarily using the tenhou_prediction_deepLearning_basic.ipynb notebook as detailed on the model card. You can add your own logic to load different weights or the latest version from the Hub.

This model is a fine-tuned version of google/vit-base-patch16-224-in21k on a local imagefolder dataset consisting of pictures of Mahjong tiles. It achieves the following results on the evaluation set:

• Loss: 0.0466
• Accuracy: 0.9967
• F1: 0.9966
• Recall: 0.9967

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 250

Training results

Framework versions

• Transformers 4.34.1
• Pytorch 2.1.0+cu121
• Datasets 2.14.6
• Tokenizers 0.14.1