From 2e1c80b7c2e86407ac52bd48428d9a26eeb29d26 Mon Sep 17 00:00:00 2001 From: Matt Hills Date: Sat, 11 Apr 2026 19:35:31 -0400 Subject: [PATCH] Keep screen on during gameplay + ESP32 bridge design notes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit GameScreen: add FLAG_KEEP_SCREEN_ON while a game is active. Keeps the system out of the deeper sleep states that aggravate OTG selective-suspend on Samsung, where the 8BitDo adapter otherwise drops every 60-120 seconds. Cleared on dispose so other screens are unaffected. docs: drop a design sketch for a DIY Joy-Con bridge built on an original ESP32 or Raspberry Pi Pico W. Covers the two architectures (fully wireless via BT Classic host + BLE peripheral, or wired hybrid via BT Classic host + USB-CDC serial), the Joy-Con init subcommand requirement, a proposed wire protocol for the serial path, and the changes we'd make app-side to consume it. Also captures the dongle rabbit hole we went through so future-us doesn't repeat the same experiments. Notes that Bluepad32 already implements BT Classic HID host with first-class Joy-Con support on original ESP32 and Pico W, which shrinks the firmware side considerably. Also warns explicitly against ESP32-S3 for this project — despite being the obvious "newer, better" ESP32, it is BLE-only and cannot pair with Joy-Cons (BR/EDR-only). --- .../emulate/ui/screens/game/GameScreen.kt | 13 + docs/joycon-esp32-bridge.md | 356 ++++++++++++++++++ 2 files changed, 369 insertions(+) create mode 100644 docs/joycon-esp32-bridge.md diff --git a/app/src/main/java/com/lazy/emulate/ui/screens/game/GameScreen.kt b/app/src/main/java/com/lazy/emulate/ui/screens/game/GameScreen.kt index 59bbd97..6fa4ba4 100644 --- a/app/src/main/java/com/lazy/emulate/ui/screens/game/GameScreen.kt +++ b/app/src/main/java/com/lazy/emulate/ui/screens/game/GameScreen.kt @@ -1,5 +1,6 @@ package com.lazy.emulate.ui.screens.game +import android.app.Activity import android.graphics.Bitmap import android.graphics.BitmapFactory import android.os.Handler @@ -7,6 +8,7 @@ import android.os.Looper import android.view.PixelCopy import android.view.SurfaceHolder import android.view.SurfaceView +import android.view.WindowManager import androidx.compose.foundation.Image import androidx.compose.foundation.background import androidx.compose.foundation.border @@ -109,6 +111,17 @@ fun GameScreen( ControllerLayout.defaultForConsole(game.consoleType) } + // Keep the screen on while a game is on-screen. This also keeps the system out of the + // deeper sleep / power-saving states that aggravate USB OTG selective-suspend on + // Samsung devices, where the adapter drops every ~60-120 seconds otherwise. + DisposableEffect(Unit) { + val window = (context as? Activity)?.window + window?.addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON) + onDispose { + window?.clearFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON) + } + } + DisposableEffect(game.id) { val loaded = EmulationEngine.startGame(context, game.consoleType, game.romPath) if (loaded != null) { diff --git a/docs/joycon-esp32-bridge.md b/docs/joycon-esp32-bridge.md new file mode 100644 index 0000000..19e9b21 --- /dev/null +++ b/docs/joycon-esp32-bridge.md @@ -0,0 +1,356 @@ +# Joy-Con ESP32 Bridge — Design Notes + +Design sketch for replacing the 8BitDo USB Wireless Adapter 2 with a DIY +ESP32-based Joy-Con bridge. Not yet implemented — this doc captures the +plan so it's ready to pick up later. + +## Why + +The current "working" path uses an 8BitDo USB Wireless Adapter 2 in PS-Classic +mode, plugged into the Samsung Fold 6 via an OTG cable. It works, but has three +annoying problems: + +1. **It wedges periodically.** The adapter stops sending HID reports every + 60–120 seconds on this phone. Unplug + replug clears it. Likely root cause + is Samsung's aggressive USB selective-suspend / OTG power management + kicking the idle device off the bus. See the "dongle rabbit hole" notes + below for everything we tried. +2. **PS-Classic mode is fragile.** We picked it because it's the only mode + where Samsung's kernel `usbhid` actually binds the device as a real + gamepad. Direct Joy-Con Bluetooth, XInput mode, DInput mode, and macOS/DS4 + mode all fail for various reasons specific to Samsung's kernel + this + adapter's firmware. +3. **It's a cable dangling off a foldable phone.** Not great ergonomically. + +Doing our own Joy-Con-to-gamepad translation on an ESP32 lets us control both +ends of the conversation: we pair the Joy-Con to hardware we own, we do the +init handshake correctly, and we present the result to the phone as either a +standard BLE HID gamepad (no cable) or a USB-CDC serial stream (cable, but we +control the firmware so the selective-suspend problem goes away). + +## Why the Joy-Con is non-trivial + +Joy-Cons are Bluetooth **Classic** HID devices. They are NOT BLE. This means +any bridge MCU must have BT Classic on its radio. + +Once paired, they default to input report mode `0x3F`, which only emits a +small subset of buttons as a dumb HID "joystick". To get the full +button-mask + analog sticks + IMU + battery, you have to send a sequence of +subcommands: + +1. Read SPI flash calibration at `0x6020` (stick factory calibration) and + `0x8010` (stick user calibration) +2. `set_player_lights` (subcommand `0x30`) +3. `enable_imu` (subcommand `0x40`) if you want gyro/accel +4. **`set_input_report_mode` to `0x30`** (subcommand `0x03`) — this is the + important one; switches to the 12-byte input report that contains + everything you actually want + +Once the Joy-Con is in `0x30` mode, every input report is a fixed layout: + +``` +byte purpose + 0 0x30 (report id) + 1 timer (rolling counter) + 2 battery + connection info + 3 buttons right (Y, X, B, A, SR, SL, R, ZR) + 4 buttons shared (-, +, R-stick, L-stick, home, capture) + 5 buttons left (down, up, right, left, SR, SL, L, ZL) + 6-8 left stick (12-bit X + 12-bit Y, packed) + 9-11 right stick (same) + 12 vibrator ack + 13+ IMU samples (3 frames x 12 bytes) if IMU enabled +``` + +The full protocol is documented here: +- https://github.com/dekuNukem/Nintendo_Switch_Reverse_Engineering +- https://github.com/DanielOgorchock/joycond (Linux userspace init daemon) +- Linux mainline `drivers/hid/hid-nintendo.c` (kernel driver that does the + same init sequence in C) + +Either joycond or hid-nintendo is a fine starting point for porting to C on +ESP32. + +## Hardware + +Joy-Cons use **Bluetooth Classic (BR/EDR)**, not BLE. This is the single +most important constraint on the hardware. Most modern MCU boards that +have "Bluetooth" actually only have BLE, which won't work. + +Only these chips have BT Classic: + +| Chip | BT Classic | BLE | Native USB | Fits? | +|----------------------------|:----------:|:---:|:----------:|:-----:| +| **ESP32** (D0WDQ6 / WROOM-32 / WROVER-32) | ✅ | ✅ | ❌ | ✅ **best default** | +| **Raspberry Pi Pico W** (CYW43439) | ✅ | ✅ | ✅ | ✅ also works | +| ESP32-S3 | ❌ | ✅ | ✅ | ❌ BLE only (common mistake) | +| ESP32-S2 | ❌ | ❌ | ✅ | ❌ no BT at all | +| ESP32-C3 | ❌ | ✅ | ❌ | ❌ BLE only | +| ESP32-C6 | ❌ | ✅ | ❌ | ❌ BLE + 802.15.4 only | +| ESP32-H2 | ❌ | ✅ | ❌ | ❌ BLE + 802.15.4 only | + +**⚠️ Do not buy ESP32-S3 for this project** even though it looks like the +obvious upgrade over the original ESP32. Espressif confirmed in their own +datasheet that ESP32-S3 only supports Bluetooth 5 LE — no BR/EDR — and +you cannot connect a Joy-Con to it over Bluetooth. The Bluepad32 FAQ +makes this explicit: *"controllers like Switch, Wii, DualSense, DualShock, +etc. only talk BR/EDR... you cannot use an ESP32-S3 as a Bluetooth HID +host to connect to Joy-Con controllers."* + +**Recommended: original ESP32-WROOM-32 dev board** — any of the Amazon +"DOIT DEVKIT V1", "HiLetgo", "ELEGOO", "DIYmall" variants. They're all +the same chip, usually with a CP2102 or CH340 USB-to-serial chip on +board so you get programming + serial communication over a single USB +port. $8–12 Canadian. BT Classic is built into the ESP32-D0WDQ6 chip. +No native USB-OTG, so in Path B the chip talks to the phone over the +on-board USB-serial bridge. + +**Also good: Raspberry Pi Pico W** — the CYW43439 WiFi/BT combo chip +on the Pico W supports BT Classic, and the RP2040 has native USB. If +you prefer C SDK + CMake to Arduino, Pico W is a nicer dev experience, +and the Bluepad32 library supports it as a first-class target alongside +the original ESP32. + +**ESP32-WROVER-32** is the same silicon as WROOM-32 plus an extra PSRAM +chip. Works identically for this project but costs a bit more for PSRAM +we don't need. Fine if that's what you already have. + +## Two architectures + +### Path A — fully wireless (BT Classic host + BLE peripheral) + +``` +[Joy-Con] --BT Classic HID--> [ESP32-S3] --BLE HID--> [Phone] +``` + +The ESP32 runs two radio profiles at once on its dual-mode stack: + +- **BT Classic HID host** (ESP-IDF `esp_hidh` API) — pairs with the Joy-Con, + runs the init subcommand sequence, subscribes to `0x30` input reports +- **BLE HID device** (ESP-IDF `esp_hids` API, or the `ESP32-BLE-Gamepad` + Arduino library) — advertises itself to the phone as a standard BLE + gamepad with a generic HID descriptor + +The Android side needs **no code changes**. Samsung's kernel binds BLE HID +gamepads natively (BLE HID is a completely different code path from the +USB HID mess we've been fighting), the existing `InputDevice`/ +`dispatchKeyEvent` path picks up button events, and our +`ControllerManager` routes them into the emulator the same way it does +for the 8BitDo today. + +**Pros** +- No cable. Charge the ESP32 off a tiny LiPo or a power brick, use it as a + wireless dongle sitting next to the phone. +- Nothing to change in the app. +- Once it works, it works the same for every game on every emulator we + already support. + +**Cons** +- Running BT Classic host + BLE peripheral simultaneously on the same radio + is non-trivial. ESP-IDF supports dual mode but the profiles have to share + a single controller — expect to spend time in `menuconfig` and the + Bluetooth controller's coexistence settings. +- Bigger firmware, more things that can go wrong during bring-up. + +### Path B — wired hybrid (BT Classic host + USB-CDC serial) + +``` +[Joy-Con] --BT Classic HID--> [ESP32-S3] --USB CDC serial--> [Phone] +``` + +Same Joy-Con side. Different phone side: instead of re-broadcasting over +BLE, the ESP32 streams button state as short binary frames over USB CDC. +The phone reads them via `usb-serial-for-android` or directly via the +`UsbManager` + `bulkTransfer` APIs we already use. + +**Pros** +- Simpler firmware — no dual-mode radio, no BLE HID descriptor, no BLE + pairing dance. Get it working in a weekend. +- We own the wire protocol, so it's trivially debuggable. Dump the serial + bytes in Serial Monitor and you see exactly what the Joy-Con is sending. +- **Selective-suspend goes away** because we control the firmware on + both sides. Send a 60-Hz heartbeat frame even when no buttons are held + and the USB bus stays active. The ESP32 can also be externally + powered, so the phone's bus-power policy becomes irrelevant. +- Direct fit for the existing `ControllerManager.dispatchButton` plumbing. + +**Cons** +- Still a cable between the ESP32 and the phone (we're swapping the 8BitDo + OTG cable for an ESP32 OTG cable — ergonomically the same). +- Requires a small addition to the app to read USB serial and parse the + wire protocol. + +Recommended order of attack: **build Path B first**. The hard part (Joy-Con +BT Classic host + init sequence + `0x30` report parsing) is identical in +both paths, so Path B gets you to a working gamepad fastest and validates +the Joy-Con side. Once Path B is rock-solid, swap the output stage for BLE +HID and you have Path A for free (modulo the dual-mode coexistence tuning). + +## Wire protocol sketch (Path B) + +Keep it tiny, keep it fixed-size, keep it self-synchronizing. + +``` +byte 0 0xA5 sync / frame header +byte 1 seq rolling sequence counter, wraps at 255 +byte 2 btns_lo (A, B, X, Y, L1, R1, L2, R2) +byte 3 btns_hi (Start, Select, L3, R3, Home, Capture, reserved, reserved) +byte 4 dpad packed: 4 bits for hat direction (0-7 clockwise from N, 8=none), + 4 bits for Joy-Con-specific buttons (SR/SL etc) +byte 5 lx signed int8, -127..127 (already deadzoned + calibrated on ESP32) +byte 6 ly signed int8 +byte 7 rx signed int8 +byte 8 ry signed int8 +byte 9 flags bit 0: battery low, bit 1: charging, bit 2: imu_valid, bits 3-7: reserved +byte 10 crc8 CRC-8 over bytes 0..9 (or just XOR checksum if we're lazy) +``` + +11 bytes per frame, 60 Hz = 660 bytes/sec. USB CDC at 115200+ baud handles +this with orders of magnitude to spare. Even ESP32's default 921600 baud is +fine for 240 Hz polling. + +If later we want motion / gyro for games that use it, we append another ~12 +bytes and bump a version byte in the header. Forward-compatible. + +## App-side changes (Path B) + +All contained in `ControllerManager.kt` and a new helper file: + +1. **New dependency** on [usb-serial-for-android](https://github.com/mik3y/usb-serial-for-android) + (tiny, ~50 KB; supports CH340, CP210x, FTDI, and CDC-ACM, which covers + every ESP32 dev board). +2. **Add ESP32-S3 native USB VID/PID** (`0x303A / 0x1001`) and CP2102 + (`0x10C4 / 0xEA60`) to `targetVidPids` in `ControllerManager`. +3. **In `onUsbDeviceAttached`**, after the existing HID-class short-circuit, + check whether the device matches one of the ESP32 VID/PIDs. If so, open + it as a USB serial port instead of going through the HID path, start a + reader thread, and parse incoming `[0xA5, seq, btns_lo, btns_hi, dpad, + lx, ly, rx, ry, flags, crc]` frames. +4. **For each frame**, diff against the previous frame and call + `dispatchButton(...)` for any changed button bits, plus `onAnalogEvent(...)` + for stick updates. Existing plumbing carries it into the emulator core. +5. **Heartbeat / liveness**: track the frame sequence counter, and if we + don't see a frame for 500 ms, log a warning and mark the controller as + disconnected so the test screen reflects it. + +Approximate LOC: ~150 lines in ControllerManager + 50 lines for the wire +protocol parser. No changes needed anywhere else in the app. + +## Firmware-side notes + +### Bluepad32 changes the math + +There's a project called **[Bluepad32](https://bluepad32.readthedocs.io/)** +that already implements BT Classic HID host for original ESP32 and Pico W, +with **explicit first-class Joy-Con support** alongside DualShock 3/4/5, +Switch Pro Controller, Wii Remote, Xbox, and generic HID gamepads. It does +the init subcommand dance, SPI-flash calibration reads, and 0x30 input +report parsing for you. MIT licensed. + +This means the Joy-Con side of this project essentially does not need to +be written — it's library-level `#include`. Our firmware job shrinks to: + +1. Initialize Bluepad32 and register a gamepad callback. +2. Format the callback's button/stick state into our wire protocol. +3. Send it out the output stage (USB serial for Path B, BLE HID for Path A). + +The Joy-Con init quirks, reconnect handling, stick calibration, and BT +pairing UX are already solved. + +### Starting points + +- **[Bluepad32](https://github.com/ricardoquesada/bluepad32)** — the main + library. Has ESP-IDF and Arduino examples. The `controllers/` demo example + pairs gamepads and prints button state to serial; adapting it to our + wire format is ~50 lines. +- For **Path A** (BLE output): glue the Bluepad32 input side to + [`ESP32-BLE-Gamepad`](https://github.com/lemmingDev/ESP32-BLE-Gamepad) — + working BLE HID gamepad profile in ~30 lines of Arduino. Caveat: BLE + peripheral coexisting with BT Classic host on one chip is tricky. +- For **Path B** (USB serial): just `Serial.write()` the frame from the + Bluepad32 callback. Trivial. +- If you'd rather not use Bluepad32 and write the protocol yourself (for + learning or licensing reasons), reference joycond (C++, Apache 2.0) or + the Linux kernel `drivers/hid/hid-nintendo.c` (C, GPL). + +### Prior art + +- https://github.com/dekuNukem/Nintendo_Switch_Reverse_Engineering — full + protocol docs +- https://github.com/DanielOgorchock/joycond — Linux userspace init daemon +- https://github.com/pipe01/joycon-rs — Rust Joy-Con protocol library +- Various ESP32-based Nintendo Switch Pro Controller *emulators* also + exist — they go the other direction (pretend to be a Pro Controller + talking to a Switch console) but share the same protocol knowledge. + +### Estimated effort (with Bluepad32) + +- **Path B (serial out)**: a weekend to get working, another weekend to + harden against reconnect/pair-loss edge cases. Total ~10–20 hobby hours. +- **Path A (BLE out)**: same Path B starting point plus 1–2 weeks of + wrestling with BT Classic + BLE dual-mode coexistence on one chip. + Bluepad32 itself only targets single-mode BT Classic, so you'd be + extending it or running the BLE side in parallel via raw ESP-IDF calls. + Harder to estimate. + +## The dongle rabbit hole (what we already tried and why it didn't work) + +For anyone picking this up later, here's the short history so you don't +re-run the same experiments: + +- **Direct Joy-Con over Bluetooth to Samsung Fold 6.** Pairs fine, kernel + creates two evdev nodes (main + IMU), IMU streams MSC_TIMESTAMP so the + link is alive, but the main node has only `BTN_TL`/`BTN_TL2` in its key + capability set (no face buttons, no dpad, no stick) and even those fire + zero events. Samsung's `hid-nintendo` driver parks the node waiting for a + userspace init handshake that nothing on the system provides, and shell + can't `chmod` `/sys/bus/hid/devices/.../` to kick it. +- **8BitDo adapter, XInput mode.** Enumerates as an Xbox 360 wired + controller (vendor-spec class 0xFF / subclass 0x5D). Android has no + XInput driver in its kernel. Adapter endlessly cycles modes because it + can't complete the Xbox 360 handshake with the phone. +- **8BitDo adapter, DInput mode.** Enumerates as a clean HID-class device, + but Samsung's kernel `usbhid` refuses to bind it for unknown reasons + (verified by uninstalling the app entirely and replugging — no + `/dev/input/event*` is ever created). Not fixable from userspace. +- **8BitDo adapter, macOS mode.** Enumerates as a Sony DualShock 4 + (`054C:05C4`). Same failure as DInput — Samsung's kernel doesn't bind + it. Despite Android having native DS4 support. +- **8BitDo adapter, PS-Classic mode.** Enumerates as a "Sony Interactive + Entertainment Controller" (`054C:0CDA`). Samsung's kernel actually binds + this one as a real gamepad. **This is what we ship today.** Downside: the + adapter wedges every 60–120 seconds and needs an unplug+replug to + recover, presumably from OTG selective-suspend. + +Things that partially work or haven't been fully explored: +- Plugging the adapter into a powered USB hub between the phone and the + adapter helps somewhat with the wedging but does not fully fix it. +- Disabling Samsung battery optimization for the app does not fix the + wedging. +- Holding `FLAG_KEEP_SCREEN_ON` on the game window helps (the system + stays out of the deep-sleep regime that aggravates selective-suspend) + but again doesn't fully fix it. + +All three workarounds stack with the ESP32 bridge idea if we go Path B — +but Path A (BLE) makes them all irrelevant because there's no USB bus to +suspend in the first place. + +## Open questions to resolve during bring-up + +1. Does the Samsung Fold 6 kernel actually bind ESP32-S3 native USB-CDC + as a `/dev/bus/usb/` device visible to our app, or does it do something + weird to USB-CDC too? Should be fine — CDC-ACM is boring and + well-supported — but verify before writing firmware. +2. What's the latency floor from Joy-Con button press to emulator core? + Joy-Con → BT Classic (~4-8 ms) + ESP32 parse (<1 ms) + USB serial + (<1 ms) + app dispatch (<1 ms) should come in under 15 ms, comfortably + below the ~16 ms/frame budget at 60 Hz. Worth measuring once hardware + exists. +3. Left+Right Joy-Con combined as one "Pro Controller" style pad, or + single-Joy-Con sideways mode? Single-Joy-Con is simpler for NES/SNES + and matches the NES Joy-Con we have. Pairing two Joy-Cons to the same + ESP32 is a separate BT Classic multi-device problem that we can punt + on. +4. Rumble — do we care? Joy-Con rumble is HD Rumble, which is a nightmare + even by Joy-Con standards. Probably punt; the emulated consoles we + support didn't have rumble anyway.