AI With Reinforcement Learning: Fall 2020 Assignment 1 Tasks ✓ Solved

AI with reinforcement Learning, Fall 2020 Assignment 1 Tasks

Your task is to design a reinforcement learning algorithm to teach the mouse how to find the food. The fundamental task is as follows:

• There is a 100x100 matrix representing a grid where any space can either be occupied by a mouse or a piece of food. There is only one mouse, and an arbitrary amount of food.
• The mouse is able to sense the food with a 3x3 matrix representing its sense of smell, the range of which is the full grid, and stacks (Two foods next to each other will generate twice as much smell). This will be the input for your algorithm. Note that the center of this matrix will always be 0, as that space represents the mouse itself.
• The mouse has a limited amount of energy, which is fully replenished when it finds food. If it runs out of energy, it dies, and the game is over.
• The mouse is able to move in any cardinal direction (North, South, East, and West). The goal is for them to eat all of the food in the grid as quickly as possible.
• This simulation is visually represented using PyGame. This task can also be solved by a trivial algorithm using nothing but simple arithmetic, as the 'scent' of food is a function of distance from the mouse. You can try to find a non-RL solution and compare it to the best version of the RL algorithm's results.

For each frame, a forward pass is run through your model. The input for each frame is an array (or tuple) with four numbers between 0 and 1. These are the probabilities generated from your model. These will determine how the mouse moves.

Questions:

• Does the order matter for the reinforcement learning model? (Ex: Inputting (N,S,E,W) vs (N,E,S,W))
• Does the order matter for a closed form solution?
• What would be better for reinforcement learning; taking the highest value from the array as the movement choice, or choosing a random direction weighted by the given probabilities? Why?

Tasks for the students:

• Write a reward function. You have access to the current game state, as well as the number of previous frames of input matrices, food level, and number of food tiles found. This number of frames will be the same number of frames it takes to starve from a full energy level. This can be very simple, like just using the number of food tiles found, or use reward shaping using multiple frames of previous input matrices to create more frequent positive/negative rewards (Ex: when the mouse moves closer or farther away from food).

Questions:

• What would be a sparse reward function for this model?
• How can the reward function be improved?

Write a model: As described above, write a model that takes in 8 inputs (The mouse's sensory matrix minus the center), and outputs four probabilities for each cardinal direction. In addition, you will be handling when to back-propagate a reward, or when to keep running.

Additional task (Optional worth extra credit): There are many variables that you can mess around with to complicate the problem to make it more suitable to Reinforcement learning. One of them is decreasing the range of the mouse's scent with the variable SCENT_RANGE. Another is the variable VARIABLE_TERRAIN which gives the mouse a second sense (sight) and adds a value to each terrain section which indicates how much energy is spend by stepping over that tile.

Q: How does the reward function change if you add the new variables?

Paper For Above Instructions

The design of a reinforcement learning (RL) algorithm to teach a mouse to find food in a grid-like environment is a compelling computational problem that illustrates the capacities of AI in making autonomous decisions based on sensory input. This approach is deeply rooted in principles of RL, where agents learn to make choices that maximize cumulative rewards through interactions with their environments.

The fundamental aspect of the assigned task is to create an algorithm capable of navigating a 100x100 grid-level environment populated by food items and the mouse itself. In this scenario, the mouse must use its sensing capabilities, represented as a 3x3 sensory matrix, to detect food and maneuver accordingly. The center of this matrix always represents the mouse’s current location, which isolates the neighboring food inputs around it.

The mouse's movement decision is based on probabilities generated by the RL model. Given an input consisting of the sensory matrix excluding its center, the model outputs probabilities corresponding to movement in four cardinal directions: north, south, east, and west. As a crucial part of the algorithm, the mouse possesses a limited energy resource, which is restored upon finding food, creating tension in energy management throughout the navigation process.

In crafting the reward function, students must navigate the nuances of reward shaping to ensure efficient learning. Students can utilize various strategies, including direct rewards for every food item consumed or triggering a reward upon the mouse moving closer to food, thus guiding learning not only from immediate rewards but also from exploring prior states. Sparse reward functions could entail minimal rewards, primarily when food is acquired, whereas more complex formulations could factor in distances to food sources and the number of moves taken.

The design choices when implementing the reinforcement learning model raise pertinent questions about the influence of input order on agent behavior. For instance, establishing whether the model prioritizes the highest probability direction for movement or selects randomly from movements weighted by calculated probabilities presents an interesting governance of exploration versus exploitation. This decision could significantly impact learning speed and efficiency.

Regarding model construction, taking advantage of the mouse's sensory matrix to derive the output probabilities for the four directions is a key component. For each frame of interaction, the model should process the incoming data dynamically, continuously learning from experiences while adjusting its decision-making policy accordingly. This iterative process is essential to achieving an agent adept at efficiently collecting food.

Further complexity can be introduced into the model through optional variables like SCENT_RANGE and VARIABLE_TERRAIN. By limiting the radius of the mouse's scent detection range, one can observe how this influences the reward function. Likewise, incorporating terrain variances could simulate energy costs for the mouse’s movements, compelling the model to assess trade-offs and strategize movement based on both smell and obstacle considerations.

To ensure the effectiveness of the designed algorithm, testing it against a baseline model that employs simple arithmetic to determine food proximity can furnish students with contrasting performance metrics. Observational comparisons between RL-driven behaviors and those guided by simpler heuristics will reveal insights about the relative merits of RL methods in dynamic environments.

The challenges and intricacies of this design task serve to enrich students' understanding of reinforcement learning principles, enabling them to construct robust AI models that can adapt to complex scenarios. Through iterative learning and reward function calibration, students can pursue the objective of formulating a proficient algorithm capable of navigating the grid while optimizing food collection efficiency.

References

• Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction. MIT Press.
• Arulkumaran, K., Deisenroth, M. P., Neuneier, R., & Silver, D. (2017). A Brief Survey of Deep Reinforcement Learning. arXiv preprint arXiv:1708.05866.
• Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996). Reinforcement Learning: A Survey. Journal of Artificial Intelligence Research, 4, 237-285.
• Haarnoja, E., Zhou, A., Abbeel, P., & Schulman, J. (2018). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with A Stochastic Actor. arXiv preprint arXiv:1812.05905.
• Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347.
• Silver, D., Huang, A., Maddison, C. J., Guez, A., Lanctot, M., & Sifre, L. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587), 484-489.
• DeepMind Technologies. (2013). Playing Atari with Deep Reinforcement Learning. arXiv preprint arXiv:1312.5602.
• Zhang, S., & Sutton, R. S. (2017). A survey on policy gradient methods for reinforcement learning. arXiv preprint arXiv:1705.10501.