sos-bot/plan.md

State of Survival Bot — Implementation Plan

Overview

Automated bot for State of Survival running in BlueStacks. Uses ADB + OpenCV template matching. All screenshots stay in memory. Behavior driven by YAML playbook. New actions added as standalone .py files in actions/.

---

Phase 1 — Foundation (DONE)

1.1 Framework

Module	File	Status
ADB client	core/adb.py	✅ Done
Vision (template matching)	core/vision.py	✅ Done
Action loader (plugin system)	core/action_loader.py	✅ Done
Engine (main loop)	core/engine.py	✅ Done
Entry point	main.py	✅ Done
Debug screenshot tool	debug_screenshot.py	✅ Done
Default playbook	playbook/default.yaml	✅ Done
12 built-in actions	actions/*.py	✅ Done

1.2 BlueStacks Setup (manual, one-time)

1. BlueStacks → Settings → Advanced → Android Debug Bridge: ON
2. BlueStacks → Settings → Display → Resolution: 1600×900, DPI: 240
3. Install State of Survival, log in
4. adb connect 127.0.0.1:5555
5. adb devices — confirm "device" state

---

Phase 2 — Screen Detection System

2.1 Main Screen Detection

Goal: Reliably know when we're on the settlement main screen vs. a sub-menu, dialog, or loading screen.

Strategy: Template matching on 2-3 unique static UI elements.

Template	What to crop	Suggested file
Bottom nav bar icon (settlement)	The settlement/globe icon at bottom-left	main_screen_marker.png
Resource bar (any corner)	A section of the resource bar at the top	resource_bar.png
Left menu button	The menu button (~22, 478)	left_menu_btn.png

Playbook step:

- name: "Verify main screen"
  if_multiple: ["main_screen_marker.png", "resource_bar.png"]
  confidence: 0.8
  then:
    - action: log
      params: { message: "On main screen" }

Negative check (not on main screen):

- name: "Not on main screen — force restart"
  if_not_template: "main_screen_marker.png"    # <-- needs engine support
  then:
    - action: app
      params: { action: "restart", package: "com.kingsgroup.sos", wait: 15 }

2.2 Popup / Ad Detection

Goal: Detect and dismiss all overlay dialogs, ads, and offers.

Template	Location	Confidence
close_ad.png	X button in top-right of ad banners	0.75
dialog_close.png	X button for dialogs	0.75
offer_close.png	Small X on special offers	0.7
back_btn.png	Back arrow top-left	0.75

Region constraint for top-right X buttons: [1440, 0, 1600, 300]

Strategy: Check for known close buttons every cycle, before any other action. Multiple close attempts in sequence for layered dialogs (some offers have 2-3 popups stacked).

steps:
  - name: "Close ads (layer 1)"
    if_template: "close_ad.png"
    region: [1440, 0, 1600, 300]
    then:
      - action: find_and_click
        params: { template: "close_ad.png", region: [1440, 0, 1600, 300] }
      - action: wait
        params: { seconds: 2 }

  - name: "Close ads (layer 2)"
    if_template: "dialog_close.png"
    region: [1480, 0, 1600, 100]
    then:
      - action: find_and_click
        params: { template: "dialog_close.png", region: [1480, 0, 1600, 100] }

---

Phase 3 — Resource Collection

3.1 Collect Finished Gathers

Goal: Detect completed gathering nodes on the world map and collect resources.

Template	What to crop	Notes
collect_btn.png	"Collect" button that appears on finished nodes	Crop tight, green text on dark bg
gather_complete_icon.png	The checkmark/ribbon on gathering buildings	Visible from settlement

Flow:

1. Scan settlement for gather-complete markers
2. Click marker → opens node dialog
3. Click "Collect" button
4. Close result dialog

Playbook:

- name: "Collect resources"
  if_template: "gather_complete_icon.png"
  confidence: 0.7
  then:
    - action: find_and_click
      params: { template: "gather_complete_icon.png" }
    - action: wait
      params: { seconds: 2 }
    - action: find_and_click
      params: { template: "collect_btn.png" }
    - action: wait
      params: { seconds: 2 }
    - action: tap
      params: { x: 1553, y: 32 }    # close

---

Phase 4 — Military Actions

4.1 Train Troops

Goal: Keep training queues active for all troop types.

Flow:

1. Open left menu (tap 22, 478)
2. Tap troop type 1 (550, 182)
3. Tap building center (806, 432) twice
4. Check if training icon exists (932, 645)
5. If training icon exists → check if button is "Train" or "Already Training"
6. If Train → click it, else close
7. Repeat for troop types 2 (550, 261) and 3 (550, 348)

Template	What to crop	Notes
troops_building.png	The barracks/training building on screen	For find-and-click
train_btn.png	The "Train" button text/icon	Green button
training_in_progress.png	The indicator that training is active	Grayed out or timer
troop_type_1.png	Right-menu troop icon line 1	Region-crop
troop_type_2.png	Right-menu troop icon line 2
troop_type_3.png	Right-menu troop icon line 3

Strategy choice: We have two approaches:

A) Coordinate-based (simpler, brittle if UI shifts):

- name: "Train troops type 1"
  if_counter: 5
  then:
    - action: tap
      params: { x: 22, y: 478 }      # left menu
    - action: wait
      params: { seconds: 1 }
    - action: tap
      params: { x: 550, y: 182 }     # troop type 1
    - action: wait
      params: { seconds: 2 }
    - action: tap
      params: { x: 806, y: 432 }     # building center
    - action: wait
      params: { seconds: 1 }
    - action: find_and_click
      params: { template: "train_btn.png", confidence: 0.75 }
    - action: tap
      params: { x: 1553, y: 32 }     # close

B) Template-based (more robust):

- name: "Find barracks and train"
  if_template: "troops_building.png"
  confidence: 0.7
  then:
    - action: find_and_click
      params: { template: "troops_building.png" }
    - action: wait
      params: { seconds: 2 }
    - action: find_and_click
      params: { template: "train_btn.png", confidence: 0.75 }
    - action: wait
      params: { seconds: 1 }
    - action: tap
      params: { x: 1553, y: 32 }

Recommendation: Start with approach A, add templates as fallbacks.

4.2 Heal Troops

Goal: Heal wounded troops from the hospital.

Template	What to crop
heal_btn.png	Hospital/heal button
heal_all.png	"Heal All" button
hospital_icon.png	Hospital building icon on settlement map

Flow:

1. Find hospital on screen → tap
2. Wait for dialog
3. Click "Heal All"
4. Close

4.3 Attack Warning → Shield

Goal: When an attack is incoming, activate a shield.

Template	What to crop
attack_banner.png	Red attack warning banner at top of screen
shield_2h.png	2-hour shield icon/button
shield_8h.png	8-hour shield icon/button
confirm_shield.png	Confirm / activate shield button

Flow (from original code):

1. Check pixel (1525, 468) for red color (195, 69, 69) — attack warning indicator
2. Tap (1401, 850) — open rally response panel
3. Tap (90, 600) — switch to world chat to see the warning
4. Search region (182-861, 80-263) for green shield buttons
5. Tap shield → tap confirm (1338, 825)
6. Close dialogs

Template version:

- name: "Shield on attack"
  if_template: "attack_banner.png"
  confidence: 0.7
  then:
    - action: find_and_click
      params: { template: "shield_2h.png", confidence: 0.7 }
    - action: wait
      params: { seconds: 1 }
    - action: find_and_click
      params: { template: "confirm_shield.png", confidence: 0.7 }
    - action: wait
      params: { seconds: 2 }
    - action: tap
      params: { x: 1553, y: 32 }
    - action: log
      params: { message: "SHIELD ACTIVATED" }

4.4 Join Rally Groups

Goal: Join alliance rallies against infected/marauders when available.

Flow (from original code):

1. Look for group invitation banner (amber color ~174,114,40) at top-right (1540-1584, 305-348)
2. Click to open group screen
3. Search for available rally in region (1227-1527, 102-882) — olive color ~95,120,46
4. Check stamina — if ended, close and return
5. Check march limit — if limited, close and return
6. Check empty troops — if no troops selected, tap clear (773, 789) then add 1 (1289, 279)
7. March (1269, 801)
8. Handle "not enough troops" message if it appears

Template	What to crop
rally_invite_banner.png	Amber banner at top
march_btn.png	"March" button
stamina_empty.png	Indicator that stamina is depleted
rally_available.png	A rally you can join in the list

---

Phase 5 — Progression Actions

5.1 Upgrade Buildings

Goal: Upgrade buildings when upgrade arrows are visible.

Flow:

1. Tap building slot 1 (63, 240) — first column, first row
2. Wait for building dialog
3. Scan buildings area (576-1059, 431-768) for green upgrade arrow (4 shades of green)
4. If upgrade arrow found → tap it → opens upgrade dialog
5. Check if "Upgrade" button is active (not gray ~85,85,85)
6. If active → tap Upgrade
7. If gray → resources missing → close
8. Close dialog
9. Repeat for building slot 2 (63, 350)

Template	What to crop
upgrade_arrow_green.png	Green arrow pointing up on a building
upgrade_btn.png	"Upgrade" button in upgrade dialog (active, not gray)
upgrade_gray.png	Grayed out upgrade button (missing resources)
building_slot_1.png	First building position on left column
building_slot_2.png	Second building position on left column

5.2 Research

Goal: Progress research trees (Fight and Economy).

Flow:

1. Open left menu (22, 478)
2. Tap research tab (550, 493)
3. Open research screen (987, 605)
4. Check if research is already in progress (pixel 489, 844 == brown ~123,82,57) → if yes, close
5. Toggle between Fight branch (652, 835) and Economy branch (1437, 838)
6. Tap "Go" (1158, 823)
7. Confirm (831, 301)
8. Close

Template	What to crop
research_tab.png	Research icon in left menu
research_in_progress.png	"In Progress" indicator
research_go_btn.png	"Go" / "Start Research" button
research_fight_tab.png	Fight research branch
research_economy_tab.png	Economy research branch

5.3 Intelligence Collection

Goal: Collect completed intelligence mission rewards.

Flow:

1. Open left menu (22, 478)
2. Check pixel (35, 568) for orange-brown color (~157, 94, 77) — indicates rewards available
3. If reward available → tap intelligence screen (551, 624)
4. Tap Collect (1115, 607)
5. Close (1188, 305)
6. If no reward → close left menu (668, 446)

Template	What to crop
intel_tab.png	Intelligence tab in left menu
intel_notification.png	Orange dot/indicator on intel tab
intel_collect_btn.png	"Collect" button in intel screen

---

Phase 6 — Alliance Actions

6.1 Alliance Tech Donations

Goal: Contribute to alliance technology research.

Flow:

1. Open clan (1140, 848)
2. Tap tech tab (1482, 542)
3. Wait 3s for loading
4. Tap recommendations filter (1005, 42)
5. Scan region (257-1500, 174-804) for red recommendation indicators (~114,27,27)
6. If found → tap it 7× (opens the donation cart)
7. Click 11 rounds of donation slots (3 per round: 614, 675, 749 at x=1149)
8. Close cart (1333, 140)
9. Return (46, 33)
10. Send alliance help handshake (890, 730) → (848, 802)
11. Open gift box (1495, 745)
12. Collect Ali's Gift (1386, 347) → Big Gift (363, 411)
13. Close ads, collect all green checkmarks in gift panel

Template	What to crop
clan_btn.png	Clan/alliance button on main screen
tech_tab.png	Technology tab
donate_btn.png	Donate button
alliance_help_btn.png	Handshake/help button
gift_box_btn.png	Gift box button
red_dot_recommendation.png	Red dot on recommended tech

6.2 Alliance Gift Collection

Goal: Collect daily gifts from alliance members.

Flow:

1. Open gift box → Ali's Gift → Big Gift
2. Close ad popups that appear
3. Collect all gifts (scan for green checkmark color ~112,145,54)
4. Collect the final big reward button

---

Phase 7 — World Map / Intel

7.1 Intel Radar Missions

Goal: Auto-run radar intel missions for rewards.

Flow:

1. Open wilderness (77, 827)
2. Tap Search (71, 704)
3. Tap Intel tab (126, 645)
4. Go Search (1362, 822)
5. Click center of map (797, 365)
6. Check for available target (brown ~137,88,32) or raid button (olive ~100,124,42)
7. If target → check march count, level, send march
8. Wait for result (10s intervals):
   • Win: yellow text (255,221,162) at (597, 202) → click center 4×
   • Lose: olive text (76,101,35) at (1186, 637) → close
9. Also collect already-completed intel markers (red ~157,44,31)

Template	What to crop
wilderness_btn.png	Wilderness toggle button bottom-left
search_btn.png	Search button on world map
intel_tab_world.png	Intel tab in search panel
go_search_btn.png	"Search" / "Go" button
march_btn_intel.png	March button in intel target dialog
intel_win_text.png	Victory text indicator
intel_lose_text.png	Defeat text indicator
intel_collect_marker.png	Red marker for completed intel

7.2 Hero Raid Stages

Goal: Auto-run hero raid stages.

Flow (from original code, currently commented out):

1. Open wilderness → settlement
2. Tap Build Hero (1139, 518)
3. Open Raid (1401, 507)
4. Open Stage (1112, 476)
5. Scan for red enemy markers (~243,51,68) up to 6 scans
6. If found → tap marker → Go Fight (1028, 801)
7. Wait for win/lose (same as intel)
8. Repeat until no markers found

Template	What to crop
hero_building.png	Hero building icon
raid_btn.png	Raid button inside hero screen
enemy_marker_red.png	Red enemy icon on raid map
fight_btn.png	"Fight" / "Attack" button
stage_select.png	Stage selector

---

Phase 8 — Utility Actions

8.1 App Restart on Hang

Goal: Force-restart the game when stuck on loading screen or frozen.

Strategy:

• if_counter: 180 (every ~9 minutes at 3s loop) — proactive restart
• Or after 3 consecutive "not main screen" detections

Implementation:

- name: "Periodic restart"
  if_counter: 180
  then:
    - action: app
      params: { action: "restart", package: "com.kingsgroup.sos", wait: 15 }

8.2 Swipe / Navigate

Gesture helpers for custom scenarios:

Action	Implementation
Swipe up	swipe: { x1: 250, y1: 700, x2: 250, y2: 600, duration_ms: 500 }
Swipe left	swipe: { x1: 767, y1: 449, x2: 255, y2: 449, duration_ms: 500 }
Swipe right	swipe: { x1: 255, y1: 449, x2: 767, y2: 449, duration_ms: 500 }
Open left menu	tap: { x: 22, y: 478 }
Close dialog X	tap: { x: 1553, y: 32 }
Back button	keyevent: { key: "back" }
Toggle world/settlement	tap: { x: 77, y: 827 }

---

Phase 9 — Advanced Features (Future)

9.1 OCR for Reading Values

Add Tesseract or EasyOCR to read:

• Resource counts (wood, food, metal, oil)
• March/troop counts
• Timers (how long until building/training finishes)
• VIP level, power score

New actions needed:

• actions/ocr_read.py — reads text from a screen region → returns string
• actions/ocr_click.py — finds text via OCR and taps it

9.2 State Machine / Decision Tree

Replace flat playbook step evaluation with a proper FSM:

• States: MAIN_SCREEN, TRAINING, HEALING, UPGRADING, INTEL, OFF_SCREEN, POPUP
• Transitions based on template detection
• Timeouts per state (don't stay in INTEL for more than 60s)
• Counter for consecutive "off screen" detections → trigger restart

9.3 Multi-Instance Support

• Loop over multiple ADB ports (5555, 5557, 5559…)
• Each instance runs its own engine thread
• Shared template directory
• Per-instance playbook or config override

9.4 Statistics & Logging

• Log kills, resources gained, shield activations
• Export stats to CSV
• Discord/Telegram webhook notifications on events (attack, shield, level-up)
• Screenshot on error for debugging

---

Action Checklist — Master Table

#	Action	Phase	Priority	Detection Method	Templates Needed
1	Main screen detection	2	🔴 Critical	Template × 2	main_screen_marker.png, resource_bar.png
2	Close popup ads	2	🔴 Critical	Template (region)	close_ad.png, dialog_close.png
3	Attack → shield	4	🔴 Critical	Template	attack_banner.png, shield_2h.png, confirm_shield.png
4	Restart on hang	8	🟡 High	Counter	None
5	Collect resources	3	🟡 High	Template	gather_complete_icon.png, collect_btn.png
6	Train troops	4	🟡 High	Coord + Template	train_btn.png, troops_building.png
7	Heal troops	4	🟡 High	Template	heal_btn.png, heal_all.png
8	Upgrade buildings	5	🟡 High	Pixel + Template	upgrade_arrow_green.png, upgrade_btn.png
9	Research	5	🟡 High	Coord + Pixel	research_in_progress.png, research_go_btn.png
10	Intelligence collect	5	🟡 High	Coord + Pixel	intel_notification.png, intel_collect_btn.png
11	Alliance tech donate	6	🟢 Medium	Coord + Pixel	clan_btn.png, tech_tab.png, red_dot_recommendation.png
12	Alliance gifts	6	🟢 Medium	Coord	None (coordinate-based)
13	Alliance help handshake	6	🟢 Medium	Coord + Template	alliance_help_btn.png
14	Join rallies	4.4	🟢 Medium	Template	rally_invite_banner.png, march_btn.png
15	Intel radar missions	7	🔵 Low	Coord + Pixel	intel_tab_world.png, march_btn_intel.png
16	Hero raid stages	7.2	🔵 Low	Pixel	enemy_marker_red.png, fight_btn.png
17	OCR reading	9	🔵 Low	Tesseract	None (text ↔ image)
18	FSM state machine	9	🔵 Low	Engine upgrade	None
19	Multi-instance	9	⚪ Later	Threading	None
20	Webhook notifications	9	⚪ Later	HTTP	None

---

Recommended Build Order

Phase 2: Screens → Popups → Main detection       [Day 1 — you're here]
    ↓
Phase 3: Collect resources                        [Day 1]
    ↓
Phase 4: Shield → Train → Heal → Join rallies     [Day 2]
    ↓
Phase 5: Upgrade → Research → Intel               [Day 2-3]
    ↓
Phase 6: Alliance actions                          [Day 3]
    ↓
Phase 7: Intel missions → Hero raids              [Day 4]
    ↓
Phase 9: OCR → FSM → Multi-instance → Webhooks   [Ongoing]

---

How to Create Templates (Step by Step)

Method 1: Using debug_screenshot.py

cd ~/sos_bot
source .venv/bin/activate

# 1. Capture current screen
python debug_screenshot.py -o screen.png

# 2. Open screen.png in GIMP/Krita
# 3. Crop around a button (e.g., the Train button)
# 4. Save as templates/train_btn.png

Method 2: In-game pixel coordinate finder

If you're on the main screen and want to verify coordinates match the bot's expectations, use:

source .venv/bin/activate
python3 -c "
from PIL import Image
im = Image.open('screen.png')
# Get color at any (x, y)
print(im.getpixel((1533, 615)))  # main screen check pixel
print(im.getpixel((1525, 468)))  # attack banner check
"

Template Design Rules

Rule	Why
Crop tightly around the target	Less background = fewer false positives
Include the whole button (border to border)	Partial buttons match poorly
Use PNG format (lossless)	JPG artifacts reduce match confidence
Start with confidence: 0.8	Lower to 0.7 if not found, never below 0.6
Use region: to limit search area	Speeds up matching, reduces false positives
Keep templates under 50×50 px where possible	Smaller templates = faster matching
For dynamic elements (timers, counts)	Use OCR instead (Phase 9)
For text buttons	Template works if font is consistent
For icons (shield, heal, x button)	Template works perfectly — icons are static

---

Playbook File Organization

As we add features, keep the playbook clean:

Option A: Single file with sections (recommended for now)

playbook/default.yaml
    ↓
    Steps organized by function:
    1. Ad/Popup closing (high priority)
    2. Attack/Shield (urgent)
    3. Resource collection
    4. Training/Healing
    5. Building/Research
    6. Alliance
    7. Intel/Raid
    8. Idle/Restart

Option B: Multiple playbook files (future)

playbook/
├── default.yaml          # main config + includes
├── core.yaml             # detection, popups, restart
├── military.yaml         # train, heal, shield, rallies
├── progression.yaml      # build, research, intel
├── alliance.yaml         # donate, gifts, help
└── world.yaml            # search, intel missions, raids

Requires !include YAML extension or pre-processor.

---

Custom Actions to Build

We'll need these new actions beyond the 12 already built:

Action	File	Purpose
pixel_check	actions/pixel_check.py	Check pixel color, return True/False for branching
loop_until	actions/loop_until.py	Repeat sub-actions until a condition is met
click_while	actions/click_while.py	Keep clicking a position until screen changes
scroll_find	actions/scroll_find.py	Scroll a list until a template appears
random_delay	actions/random_delay.py	Human-like variable delay (anti-detection)
screenshot_debug	actions/screenshot_debug.py	Save screenshot on demand (for debugging)
notify	actions/notify.py	Discord/Telegram webhook notification

Pixel Check Action (needed for branching logic)

# actions/pixel_check.py
class Action:
    name = "pixel_check"

    def run(self, ctx, params):
        x = params["x"]
        y = params["y"]
        target = tuple(params["color"])
        tolerance = params.get("tolerance", 10)
        match = ctx["vision"].color_match(x, y, target, tolerance)
        ctx["_last_pixel_match"] = match
        ctx["log"].info("pixel_check(%d,%d) = %s", x, y, match)
        return match

---

Engine Improvements Needed

#	Feature	File	What to change
1	if_not_template condition	core/engine.py	Negation of template match
2	Step cooldown / rate limit	core/engine.py	Don't run same step more than once per N seconds
3	Action result → context	core/engine.py	Pass action return values into ctx for subsequent actions
4	Step timeout guard	core/engine.py	If a step's actions take > 60s, abort and re-screenshot
5	Consecutive failure counter	core/engine.py	After 3 failures, restart app

---

Next Steps — Right Now

1. Start BlueStacks with SoS running
2. adb connect 127.0.0.1:5555
3. python debug_screenshot.py — capture your first screen
4. Open the screenshot in GIMP, crop these first templates:
   • The close/X button in top-right corner → close_ad.png
   • The settlement/globe icon bottom-left → main_screen_marker.png
   • The attack banner (if visible) → attack_banner.png
5. Put them in templates/
6. Uncomment/simplify playbook/default.yaml to test close_ad + main_screen_marker
7. Run python main.py — verify detection works
8. Capture more screenshots in each game state → crop more templates
9. Add steps to playbook one at a time
10. After each new step, run for 5+ minutes to verify it doesn't get stuck