rlbs

Authors

Alexandre Caspers, Grace Jiyoung Yun

Introduction

This project aims to utilize the brainpy library to imitate realistic brain behavior, in order to choose the order in which to perform a given list of tasks. Since the brainpy components need to be heavily adapted to the available tasks, we provide a predefined list of 6 possible tasks.

The user provides the program with wich of the 6 tasks he/she needs to do (possible all), and also which type of work style he/she prefers. For the latter we have 3 predefined choices:

• low effort first (lowest - lower - low - high - higher - highest)
• high effort first (highest - higher - high - low - lower - lowest)
• alternating effort (highest - lowest - higher - lower - high - low)

With this information, the program will then return the most optimal order.

In order to do these predictions we train 3 different models (1 for each work style) that get called according to the user input.

Components

• DRL - Deep Reinforcement Learning:

  • N = amount of total possible tasks (10)

  • States: complexities of last performed task, actions that need to be performed (1 or 0), and each their physical and mental complexity (size: 2 + 3 * N)

  • Output: softmax of which task to perform (size: N)

  • Reward Function: Utilizes BrainPy module to compute the brain activity. Taking this information in relation to the work style of the user, generate a list of rewards for each possible task

  • Other Features: decaying epsilon, bootstrapping

• BrainPy:

  • Decision Making Model: two components E and I

  • E is partitioned into N + 1 components (N and the remainder)

  • Each of those components (N + 2) are linked with one another (synapses), so everything has an impact on everything

  • The N sub-parts of E each receive a unique input signal, generated in relation to the physical and mental complexity of its associated task

  • The model computes and records the brain activity of these N sub-parts of this complex network

  • These brain activities are signals, from which we take the average. This result is passed to the reward function of the DRL model

  • Input signal construction: shaped like a bar graph, with up- and down-time. Physical complexity defines the amplitude, the mental complexity defines the duration of the up-time

• Other

  • A frontend app designed with Streamlit, allowing the user to input the data in a cleaner fashion
  • Code Generation components that create the needed brain model, synapses, and more in relation to how many tasks we have (to allow for easy flexibility)
  • Section 1 of frontend: uses DRL and pre-computed BrainPy results to perform choices of predefined tasks and complexities
  • Section 2 of frontend: user-defined tasks and complexities, skips DRL, as tradeoff directly calls BrainPy (user-defined but much much slower) (for complexity reasons the allowed number of tasks is restricted to 18)

Reasons for DRL & BrainPy

Research in the domain of brain and personality is not advanced, so we wanted to give our contribution, as we believe that the more knowledge we have on this topic, the more facinating techniques and tools could be built in the future.

BrainPy is currently one of the most powerful brain modeling libraries there are. It allows for very close definition of the exact brain components we want, the synapses, the resting / reset states of neurons, number of neurons and other intrinsic behaviors of the brain.

However, BrainPy is very slow to use for multiple iterations (1 iteration with 5 task takes about 10 seconds, while 1 iteration with 17 can take up more than 1 minute).

With DRL we want to develop a model that can mimic the decision of the brain model but in significant quicker complexity (since running a NN is just a few matrix computations).

Problems & Solutions

• Data and Knowledge on the domain of brain and personality is beyond scarce

  • using self-build datasets (built from research and common sense, e.g. complexity of certain tasks)
  • DRL allows us to avoid using existing datasets (which do not exist in the first place)

• BrainPy requires a vast amount of computation

  • a DRL needs many iterations, so calling the brain model every iteration would take forever to just train 1 model
  • what we do instead is to run the brain model with every possible combination in advance, and save the resulting brain activities
  • the DRL then uses this cached data to compute the rewards
  • this trick saves an unfathomable amount of computation time

• Brainpy documentation exists but only shows a fraction of the capabilities of BrainPy & examples (outside of the documentation) are close to non-existant

  • Trial-and-Error

• Running BrainPy for many iterations slows down PC and eventually crashes (BrainPy model can only be called a certain amount of times before it crashes)

  • Calls to BrainPy have to be limited
  • Cashed Outputs (But you have to restrict how many tasks, because with 5 tasks there are only 2^5=32 possibilities, but with 2^17=131k it is too much to test everything)
  • The maximum amount of tasks that avoid crashing is 6
  • Outputs have to be cashed (2^6 calls), because during DRL we would vastly surpass that number
  • Cashed Outputs Allow us to precompute the results once, and then for any following training session one iteration (where we would call BrainPy before) would yield results almost instantly, instead of waiting several minutes

• This idea restricts the choices from the user, but we gain computation speed. In order to have the other side too, we implement another section where the user can specify tasks with physical and mental complexity him/herself up to 18 tasks (more than that and BrainPy crashes). This allows choice, but computation time takes much longer too. 3 Tasks is only around 17 seconds whereas 18 tasks may take up to 35 minutes.

Trained Models Steps

Model low-effort first: Model high-effort first: Model alternating-effort first:

Example BrainPy Iteration

Lower Graph: Input Signals (based on difficulty of the tasks, i.e. physical difficulty determines amplitude, mental difficulty determines duration of a high). Given input activities were: ['workout' 'videogame' 'studying']
Upper Graph: Resulting Brain Activity

Architecture

.
├── AdaptiveBrain                       # same Structure as Brain/, used for section 2 of frontend
├── Brain
│   ├── caller.py                       # Call the brain code
│   ├── mini_brain.py                   # Core brain model with all the components and synapses
│   ├── plotter.py                      # Plots a Brain model iteration
│   ├── pre_run_brain.py                # Creates the cahed brain activity
│   └── stim.py                         # Input Signal Generation
├── Code_Generation                     # Code to automatically write other python files in relation to tasks.csv (for easy scalability)
│   ├── gen_caller.py
│   ├── gen_minibrain.py
│   ├── gen_plotter.py
│   ├── gen_setup.py
│   └── main_generator.py               # calles all the code generation functions above
├── Data
│   ├── prerun_brain_activities.csv     # Cached brain activities
│   └── tasks.csv                       # Data about tasks to use, and their complexities
├── DRL
│   ├── agent.py                        # Agent, States, Calling training functions
│   ├── model.py                        # The NN part of DRL
│   ├── plotter.py                      # Plots the performance of the trained agent
│   └── rewards.py                      # The different evaluation types of brain activity (3 work styles)
├── drl-model/                          # Saved models
│   ├── model-high_effort_first.h5
│   ├── model-low_effort_first.h5
│   └── model-alternating_effort.h5
├── Pictures/                           # Pictures used in this README
├── ResGraphs/                          # Folder in which we saw the steps chosen in section 2 (and the related matplotlib images)
├── Utils/
│   └── utils.py                        # Auxilary functions to save and access the ResGraphs Folder more easily
├── app.py                              # Frontend
├── main_inference.py                   # Main Code for section 2 of frontend (user defined inputs)
├── main_inference.ipynb                # Main Code during Prediction Phase
├── main_training.ipynb                 # Main Code during Training Phase
├── README.md
├── requirements.txt
└── setup.py                            # Global variables used by some files

References

Links to the BrainPy documentation & research paper:

• https://brainpy.readthedocs.io/en/latest/quickstart/simulation.html
• https://doi.org/10.1101/2022.10.28.514024

@article {Wang2022brainpy,
    author = {Wang, Chaoming and Chen, Xiaoyu and Zhang, Tianqiu and Wu, Si},
    title = {BrainPy: a flexible, integrative, efficient, and extensible framework towards general-purpose brain dynamics programming},
    elocation-id = {2022.10.28.514024},
    year = {2022},
    doi = {10.1101/2022.10.28.514024},
    publisher = {Cold Spring Harbor Laboratory},
    URL = {https://www.biorxiv.org/content/early/2022/10/28/2022.10.28.514024},
    eprint = {https://www.biorxiv.org/content/early/2022/10/28/2022.10.28.514024.full.pdf},
    journal = {bioRxiv}
}