Merge branch 'main' into reflective-mouse-click

Author: klxu03

CONTRIBUTING.md

@@ -1,8 +1,25 @@
-Contributing
-============
+# Contributing
+We appreciate your contributions!
 
+## Process
 1. Fork it
 2. Create your feature branch (`git checkout -b my-new-feature`)
 3. Commit your changes (`git commit -am 'Add some feature'`)
 4. Push to the branch (`git push origin my-new-feature`)
 5. Create new Pull Request
+
+## Contribution Ideas
+- **Remove necessity for `pip install .`**: I think by uploading packages to PyPi we can reduce the installation code steps by consolidating `pip install -r requirements.txt` and `pip install .`. If that's possible that'd be great. 
+- **Improve performance by finding optimal screenshot grid**: A primary element of the framework is that it overlays a percentage grid on the screenshot which GPT-4v uses to estimate click locations. If someone is able to find the optimal grid and some evaluation metrics to confirm it is an improvement on the current method then we will merge that PR. 
+- **Improve the `SUMMARY_PROMPT`**
+- **Create an evaluation system**
+- **Improve Linux and Windows compatibility**: There are still some issues with Linux and Windows compatibility. PRs to fix the issues are encouraged. 
+- **Enabling New Mouse Capabilities**: (drag, hover, etc.)
+- **Adding New Multimodal Models**: Integration of new multimodal models is welcomed. If you have a specific model in mind that you believe would be a valuable addition, please feel free to integrate it and submit a PR.
+- **Framework Architecture Improvements**: Think you can enhance the framework architecture described in the intro? We welcome suggestions and PRs.
+
+## Guidelines
+This will primarily be a [Software 2.0](https://karpathy.medium.com/software-2-0-a64152b37c35) project. For this reason: 
+
+- Let's try to hold off refactors into separate files until `main.py` is more than 1000 lines
+

README.md

@@ -89,17 +89,13 @@ operate
   <img src="https://github.com/OthersideAI/self-operating-computer/blob/main/readme/terminal-access-2.png" width="300"  style="margin: 10px;"/>
 </div>
 
-### Contributions are Welcomed! Some Ideas: 
-- **Improve performance by finding optimal screenshot grid**: A primary element of the framework is that it overlays a percentage grid on the screenshot which GPT-4v uses to estimate click locations. If someone is able to find the optimal grid and some evaluation metrics to confirm it is an improvement on the current method then we will merge that PR. 
-- **Improve the `SUMMARY_PROMPT`**
-- **Create an evaluation system**
-- **Improve Linux and Windows compatibility**: There are still some issues with Linux and Windows compatibility. PRs to fix the issues are encouraged. 
-- **Enabling New Mouse Capabilities**: (drag, hover, etc.)
-- **Adding New Multimodal Models**: Integration of new multimodal models is welcomed. If you have a specific model in mind that you believe would be a valuable addition, please feel free to integrate it and submit a PR.
-- **Framework Architecture Improvements**: Think you can enhance the framework architecture described in the intro? We welcome suggestions and PRs.
-- **Implement a Reflective Mouse Click Mode**: Introduce a new mode that enhances click accuracy by adding a 'reflect and correct' step. In this mode, the system will 'move mouse, reflect on position, and click if accurate; otherwise, adjust position closer.' This approach, more akin to human interaction, could increase accuracy before the implementation of `Agent-1-vision` for precise clicking. The main challenge is the increased time due to current multimodal model latency. We propose an optional `-accurate` terminal flag to activate this mode. This feature has the potential to significantly boost performance and offers an interesting area for development.
-
-For any input on improving this project, feel free to reach out to [Josh](https://twitter.com/josh_bickett) on Twitter.If you want to contribute yourself, see [CONTRIBUTING.md](https://github.com/OthersideAI/self-operating-computer/blob/main/CONTRIBUTING.md).
+### Contributions are Welcomed!:
+
+If you want to contribute yourself, see [CONTRIBUTING.md](https://github.com/OthersideAI/self-operating-computer/blob/main/CONTRIBUTING.md).
+
+### Feedback
+
+For any input on improving this project, feel free to reach out to [Josh](https://twitter.com/josh_bickett) on Twitter. 
 
 ### Follow HyperWriteAI for More Updates
 
@@ -108,4 +104,8 @@ Stay updated with the latest developments:
 - Follow HyperWriteAI on [LinkedIn](https://www.linkedin.com/company/othersideai/).
 
 ### Compatibility
-- This project is compatible with Mac OS, Windows, and Linux (with X server installed).
+- This project is compatible with Mac OS, Windows, and Linux (with X server installed).
+
+### Star History
+
+[![Star History Chart](https://api.star-history.com/svg?repos=OthersideAI/self-operating-computer&type=Timeline)](https://star-history.com/#OthersideAI/self-operating-computer&Timeline)

operate/main.py

@@ -3,7 +3,6 @@
 """
 import os
 import time
-import random
 import base64
 import json
 import math