3309a47e10
- ADB client (connect, tap, swipe, screencap, app control) - Vision module (in-memory screenshot + OpenCV template matching) - Action loader (auto-discovers .py files in actions/) - Engine (screenshot → detect → act loop with YAML playbook) - 12 built-in actions: tap, click, wait, swipe, find, find_and_click, wait_until, click_seq, keyevent, app, log - Example custom action template - Debug screenshot tool for template creation - Detailed plan.md with 20 actions across 9 phases - Sample playbook with 10 steps
144 lines
4.8 KiB
Markdown
144 lines
4.8 KiB
Markdown
# State of Survival Bot
|
||
|
||
ADB + OpenCV Python bot for State of Survival running in BlueStacks.
|
||
|
||
## Quick Start
|
||
|
||
```bash
|
||
# 1. Activate venv
|
||
cd sos_bot
|
||
source .venv/bin/activate
|
||
|
||
# 2. Verify ADB connects to BlueStacks
|
||
adb connect 127.0.0.1:5555
|
||
adb devices
|
||
|
||
# 3. Capture the first screenshot
|
||
python debug_screenshot.py
|
||
|
||
# 4. Crop reference templates from the screenshot, save to templates/
|
||
# (use GIMP, Krita, MSPaint, or any image editor)
|
||
|
||
# 5. Edit playbook/default.yaml to add steps matching your templates
|
||
|
||
# 6. Run the bot
|
||
python main.py
|
||
```
|
||
|
||
## Architecture
|
||
|
||
```
|
||
sos_bot/
|
||
├── main.py # Entry point
|
||
├── debug_screenshot.py # Capture screenshots for template creation
|
||
├── requirements.txt
|
||
├── core/
|
||
│ ├── adb.py # ADB client (connect, tap, swipe, screencap)
|
||
│ ├── vision.py # In-memory screenshot + OpenCV template matching
|
||
│ ├── engine.py # Main loop: screenshot → detect → act
|
||
│ └── action_loader.py # Dynamic action discovery (.py files)
|
||
├── actions/
|
||
│ ├── tap.py # Tap at (x, y)
|
||
│ ├── click.py # Alias for tap
|
||
│ ├── wait.py # Sleep N seconds
|
||
│ ├── swipe.py # Swipe gesture
|
||
│ ├── find.py # Locate a template (no click)
|
||
│ ├── find_and_click.py # Locate template → tap it
|
||
│ ├── wait_until.py # Wait for template to appear
|
||
│ ├── click_seq.py # Tap multiple coordinates in sequence
|
||
│ ├── keyevent.py # Android key events (back, home, etc.)
|
||
│ ├── app.py # Launch/stop/restart apps
|
||
│ └── log.py # Write to log
|
||
├── templates/ # Reference PNG images for matching
|
||
│ └── README.txt
|
||
└── playbook/
|
||
└── default.yaml # Step-by-step behavior definition
|
||
```
|
||
|
||
## How It Works
|
||
|
||
1. **Main loop** takes a screenshot every `loop_delay` seconds
|
||
2. Each playbook **step** checks its condition against the screenshot
|
||
3. The **first matching step** runs its action sequence
|
||
4. Screenshots are **never saved to disk** — they live in memory as numpy arrays
|
||
|
||
## Playbook Conditions
|
||
|
||
Each step can use one of these triggers:
|
||
|
||
| Condition | Description |
|
||
|-----------|-------------|
|
||
| `if_template` | Template matching — looks for a reference PNG |
|
||
| `if_pixel` | Pixel color check at specific coordinates |
|
||
| `if_multiple` | All specified templates must match (AND) |
|
||
| `if_counter` | Triggers every N iterations |
|
||
| `always: true` | Always matches (fallback / idle) |
|
||
|
||
## Playbook Actions
|
||
|
||
| Action | Params | What it does |
|
||
|--------|--------|-------------|
|
||
| `tap` | `x`, `y` | Tap at coordinates |
|
||
| `click` | `x`, `y` | Same as tap |
|
||
| `wait` | `seconds` | Pause N seconds |
|
||
| `find` | `template`, `confidence`, `region` | Find template (returns position) |
|
||
| `find_and_click` | `template`, `confidence`, `region`, `offset_x/y` | Find + tap |
|
||
| `wait_until` | `template`, `confidence`, `timeout`, `poll_interval` | Wait for template |
|
||
| `swipe` | `x1`, `y1`, `x2`, `y2`, `duration_ms` | Swipe gesture |
|
||
| `click_seq` | `coords` (list of [x,y]), `delay` | Tap sequence |
|
||
| `keyevent` | `key` (back/home/menu/enter) | Android key event |
|
||
| `app` | `action` (launch/stop/restart), `package` | App control |
|
||
| `log` | `message` | Write to log |
|
||
|
||
## Extending: Custom Actions
|
||
|
||
Create a new `.py` file in `actions/` with this structure:
|
||
|
||
```python
|
||
"""Action: my_action — description of what it does."""
|
||
|
||
|
||
class Action:
|
||
name = "my_action"
|
||
|
||
def run(self, ctx, params):
|
||
# ctx = {adb, vision, engine, screenshot, log, _templates_dir}
|
||
# params = dict from playbook YAML
|
||
|
||
x = params["x"]
|
||
y = params["y"]
|
||
ctx["adb"].tap(x, y)
|
||
ctx["log"].info("my_action @ (%d, %d)", x, y)
|
||
```
|
||
|
||
The file is auto-discovered on next startup. No registration needed.
|
||
|
||
## Creating Templates
|
||
|
||
1. Run `python debug_screenshot.py` → saves current screen
|
||
2. Open the PNG in an image editor
|
||
3. Crop EXACTLY the button/icon you want to detect
|
||
4. Save as `templates/my_button.png`
|
||
5. Reference it in the playbook: `if_template: "my_button.png"`
|
||
|
||
**Template tips:**
|
||
- Crop tightly around the target (no excess background)
|
||
- Use a unique, static UI element
|
||
- Start with `confidence: 0.8`, lower to 0.7 if it's not found
|
||
- Use `region: [x1, y1, x2, y2]` to constrain search area (faster, fewer false positives)
|
||
|
||
## Common ADB Ports
|
||
|
||
| BlueStacks Instance | Port |
|
||
|-------------------|------|
|
||
| First instance | 5555 |
|
||
| Second instance | 5557 |
|
||
| Third instance | 5559 |
|
||
| (Nth) | 5555 + (N-1)*2 |
|
||
|
||
## BlueStacks Settings Required
|
||
|
||
- Settings → Advanced → Android Debug Bridge (ADB): **ON**
|
||
- Settings → Display → Resolution: 1600×900 (recommended)
|
||
- Settings → Display → DPI: 240
|