Merge branch 'master' into cubecell

README.md

@@ -39,7 +39,7 @@ We currently support three models of radios.
 
 - US/JP/AU/NZ - 915MHz
 - CN - 470MHz
-- EU - 870MHz
+- EU - 868MHz, 433MHz
 
 Getting a version that includes a screen is optional, but highly recommended.
 

bin/version.sh

@@ -1,3 +1,3 @@
 
 
-export VERSION=0.6.7
+export VERSION=0.7.6

docs/software/TODO.md

@@ -1,205 +1,77 @@
 # High priority
 
-Items to complete soon (next couple of alpha releases).
+- why is the net so chatty now?
+- modem sleep should work if we lower serial rate to 115kb?
+- device wakes, turns BLE on and phone doesn't notice (while phone was sitting in auto-connect)
+- E22 bringup
+- encryption review findings writeup
 
-- lower wait_bluetooth_secs to 30 seconds once we have the GPS power on (but GPS in sleep mode) across light sleep. For the time
-  being I have it set at 2 minutes to ensure enough time for a GPS lock from scratch.
+- turn on modem-sleep mode - https://github.com/espressif/arduino-esp32/issues/1142#issuecomment-512428852
+last EDF release in arduino is: https://github.com/espressif/arduino-esp32/commit/1977370e6fc069e93ffd8818798fbfda27ae7d99
+IDF release/v3.3 46b12a560
+IDF release/v3.3 367c3c09c
+https://docs.espressif.com/projects/esp-idf/en/release-v3.3/get-started/linux-setup.html
+kevinh@kevin-server:~/development/meshtastic/esp32-arduino-lib-builder$ python /home/kevinh/development/meshtastic/esp32-arduino-lib-builder/esp-idf/components/esptool_py/esptool/esptool.py --chip esp32 --port /dev/ttyUSB0 --baud 921600 --before default_reset --after hard_reset write_flash -z --flash_mode dout --flash_freq 40m --flash_size detect 0x1000 /home/kevinh/development/meshtastic/esp32-arduino-lib-builder/build/bootloader/bootloader.bin
+cp -a out/tools/sdk/* components/arduino/tools/sdk
+cp -ar components/arduino/* ~/.platformio/packages/framework-arduinoespressif32@src-fba9d33740f719f712e9f8b07da6ea13/
 
 # Medium priority
 
 Items to complete before the first beta release.
 
-- Use 32 bits for message IDs
-- Use fixed32 for node IDs
-- Remove the "want node" node number arbitration process
-- Don't store position packets in the to phone fifo if we are disconnected. The phone will get that info for 'free' when it
-  fetches the fresh nodedb.
-- Use the RFM95 sequencer to stay in idle mode most of the time, then automatically go to receive mode and automatically go from transmit to receive mode. See 4.2.8.2 of manual.
-- possibly switch to https://github.com/SlashDevin/NeoGPS for gps comms
-- good source of battery/signal/gps icons https://materialdesignicons.com/
-- research and implement better mesh algorithm - investigate changing routing to https://github.com/sudomesh/LoRaLayer2 ?
-- check fcc rules on duty cycle. we might not need to freq hop. https://www.sunfiretesting.com/LoRa-FCC-Certification-Guide/
-- use fuse bits to store the board type and region. So one load can be used on all boards
-- the BLE stack is leaking about 200 bytes each time we go to light sleep
+- turn on watchdog timer (because lib code seems buggy)
+- show battery level as % full
 - rx signal measurements -3 marginal, -9 bad, 10 great, -10 means almost unusable. So scale this into % signal strength. preferably as a graph, with an X indicating loss of comms.
-- assign every "channel" a random shared 8 bit sync word (per 4.2.13.6 of datasheet) - use that word to filter packets before even checking CRC. This will ensure our CPU will only wake for packets on our "channel"
-- Note: we do not do address filtering at the chip level, because we might need to route for the mesh
-  is in cleartext (so that nodes will route for other radios that are cryptoed with a key we don't know)
-- add frequency hopping, dependent on the gps time, make the switch moment far from the time anyone is going to be transmitting
-- share channel settings over Signal (or qr code) by embedding an an URL which is handled by the MeshUtil app.
-- publish update articles on the web
 
 # Pre-beta priority
 
-During the beta timeframe the following improvements 'would be nice' (and yeah - I guess some of these items count as features, but it is a hobby project ;-) )
+During the beta timeframe the following improvements 'would be nice'
 
-- If the phone doesn't read fromradio mailbox within X seconds, assume the phone is gone and we can stop queing location msgs
-  for it (because it will redownload the nodedb when it comes back)
-- Figure out why the RF95 ISR is never seeing RH_RF95_VALID_HEADER, so it is not protecting our rx packets from getting stomped on by sends
-- fix the frequency error reading in the RF95 RX code (can't do floating point math in an ISR ;-)
-- See CustomRF95::send and fix the problem of dropping partially received packets if we want to start sending
-- make sure main cpu is not woken for packets with bad crc or not addressed to this node - do that in the radio hw
-- triple check fcc compliance
+- finish DSR for unicast
+- check fcc rules on duty cycle. we might not need to freq hop. https://www.sunfiretesting.com/LoRa-FCC-Certification-Guide/ . Might need to add enforcement for europe though.
 - pick channel center frequency based on channel name? "dolphin" would hash to 900Mhz, "cat" to 905MHz etc? allows us to hide the concept of channel # from hte user.
-- scan to find channels with low background noise? (Use CAD mode of the RF95 to automatically find low noise channels)
 - make a no bluetooth configured yet screen - include this screen in the loop if the user hasn't yet paired
 - if radio params change fundamentally, discard the nodedb
-- reneable the bluetooth battery level service on the T-BEAM, because we can read battery level there
-
-# Spinoff project ideas
-
-- an open source version of https://www.burnair.ch/skynet/
-- a paragliding app like http://airwhere.co.uk/
-- a version with a solar cell for power, just mounted high to permanently provide routing for nodes in a valley. Someone just pointed me at disaster.radio
-- How do avalanche beacons work? Could this do that as well? possibly by using beacon mode feature of the RF95?
-- provide generalized (but slow) internet message forwarding servie if one of our nodes has internet connectivity
+- re-enable the bluetooth battery level service on the T-BEAM
+- implement first cut of router mode: preferentially handle flooding, and change sleep and GPS behaviors
+- provide generalized (but slow) internet message forwarding service if one of our nodes has internet connectivity (MQTT) [ Not a requirement but a personal interest ]
 
 # Low priority
 
 Items after the first final candidate release.
 
-- use variable length arduino Strings in protobufs (instead of current fixed buffers)
+- Change back to using a fixed sized MemoryPool rather than MemoryDynamic (see bug #149)
+- scan to find channels with low background noise? (Use CAD mode of the RF95 to automatically find low noise channels)
+- If the phone doesn't read fromradio mailbox within X seconds, assume the phone is gone and we can stop queing location msgs
+  for it (because it will redownload the nodedb when it comes back)
+- add frequency hopping, dependent on the gps time, make the switch moment far from the time anyone is going to be transmitting
+- assign every "channel" a random shared 8 bit sync word (per 4.2.13.6 of datasheet) - use that word to filter packets before even checking CRC. This will ensure our CPU will only wake for packets on our "channel"
+- the BLE stack is leaking about 200 bytes each time we go to light sleep
+- use fuse bits to store the board type and region. So one load can be used on all boards
+- Don't store position packets in the to phone fifo if we are disconnected. The phone will get that info for 'free' when it
+  fetches the fresh nodedb.
+- Use the RFM95 sequencer to stay in idle mode most of the time, then automatically go to receive mode and automatically go from transmit to receive mode. See 4.2.8.2 of manual.
+- Use fixed32 for node IDs, packetIDs, successid, failid, and lat/lon - will require all nodes to be updated, but make messages slightly smaller.
+- add "store and forward" support for messages, or move to the DB sync model. This would allow messages to be eventually delivered even if nodes are out of contact at the moment.
+- use variable length Strings in protobufs (instead of current fixed buffers). This would save lots of RAM
 - use BLEDevice::setPower to lower our BLE transmit power - extra range doesn't help us, it costs amps and it increases snoopability
-- make an install script to let novices install software on their boards
-- use std::map<NodeInfo\*, std::string> in node db
-- make a HAM build: yep - that's a great idea. I'll add it to the TODO. should be pretty painless - just a new frequency list, a bool to say 'never do encryption' and use hte callsign as that node's unique id. -from Girts
+- make a HAM build: just a new frequency list, a bool to say 'never do encryption' and use hte callsign as that node's unique id. -from Girts
 - don't forward redundant pings or ping responses to the phone, it just wastes phone battery
-- use https://platformio.org/lib/show/1260/OneButton if necessary
-- don't send location packets if we haven't moved
+- don't send location packets if we haven't moved significantly
 - scrub default radio config settings for bandwidth/range/speed
-- answer to pings (because some other user is looking at our nodeinfo) with our latest location (not a stale location)
 - show radio and gps signal strength as an image
 - only BLE advertise for a short time after the screen is on and button pressed - to save power and prevent people for sniffing for our BT app.
-- make mesh aware network timing state machine (sync wake windows to gps time)
+- make mesh aware network timing state machine (sync wake windows to gps time) - this can save LOTS of battery
 - split out the software update utility so other projects can use it. Have the appload specify the URL for downloads.
 - read the PMU battery fault indicators and blink/led/warn user on screen
-- the AXP debug output says it is trying to charge at 700mA, but the max I've seen is 180mA, so AXP registers probably need to be set to tell them the circuit can only provide 300mAish max. So that the low charge rate kicks in faster and we don't wear out batteries.
-- increase the max charging rate a bit for 18650s, currently it limits to 180mA (at 4V). Work backwards from the 500mA USB limit (at 5V) and let the AXP charge at that rate.
 - discard very old nodedb records (> 1wk)
-- using the genpartitions based table doesn't work on TTGO so for now I stay with my old memory map
-- We let anyone BLE scan for us (FIXME, perhaps only allow that until we are paired with a phone and configured)
-- use two different buildenv flags for ttgo vs lora32. https://docs.platformio.org/en/latest/ide/vscode.html#key-bindings
-- sim gps data for testing nodes that don't have hardware
-- do debug serial logging to android over bluetooth
-- break out my bluetooth OTA software as a seperate library so others can use it
-- Heltec LoRa32 has 8MB flash, use a bigger partition table if needed - TTGO is 4MB but has PSRAM
 - add a watchdog timer
 - handle millis() rollover in GPS.getTime - otherwise we will break after 50 days
 - report esp32 device code bugs back to the mothership via android
+- change BLE bonding to something more secure. see comment by pSecurity->setAuthenticationMode(ESP_LE_AUTH_BOND)
 
-# Done
+# Spinoff project ideas
 
-- change the partition table to take advantage of the 4MB flash on the wroom: http://docs.platformio.org/en/latest/platforms/espressif32.html#partition-tables
-- wrap in nice MeshRadio class
-- add mesh send & rx
-- make message send from android go to service, then to mesh radio
-- make message receive from radio go through to android
-- test loopback tx/rx path code without using radio
-- notify phone when rx packets arrive, currently the phone polls at startup only
-- figure out if we can use PA_BOOST - yes, it seems to be on both boards
-- implement new ble characteristics
-- have MeshService keep a node DB by sniffing user messages
-- have a state machine return the correct FromRadio packet to the phone, it isn't always going to be a MeshPacket. Do a notify on fromnum to force the radio to read our state machine generated packets
-- send my_node_num when phone sends WantsNodes
-- have meshservice periodically send location data on mesh (if device has a GPS)
-- implement getCurrentTime() - set based off gps but then updated locally
-- make default owner record have valid usernames
-- message loop between node 0x28 and 0x7c
-- check in my radiolib fixes
-- figure out what is busted with rx
-- send our owner info at boot, reply if we see anyone send theirs
-- add manager layers
-- confirm second device receives that gps message and updates device db
-- send correct hw vendor in the bluetooth info - needed so the android app can update different radio models
-- correctly map nodeids to nodenums, currently we just do a proof of concept by always doing a broadcast
-- add interrupt detach/sleep mode config to lora radio so we can enable deepsleep without panicing
-- make jtag work on second board
-- implement regen owner and radio prefs
-- use a better font
-- make nice screens (boot, about to sleep, debug info (gps signal, #people), latest text, person info - one frame per person on network)
-- turn framerate from ui->state.frameState to 1 fps (or less) unless in transition
-- switch to my gui layout manager
-- make basic gui. different screens: debug, one page for each user in the user db, last received text message
-- make button press cycle between screens
-- save our node db on entry to sleep
-- fix the logo
-- sent/received packets (especially if a node was just reset) have variant of zero sometimes - I think there is a bug (race-condtion?) in the radio send/rx path.
-- DONE dynamic nodenum assignment tasks
-- make jtag debugger id stable: https://askubuntu.com/questions/49910/how-to-distinguish-between-identical-usb-to-serial-adapters
-- reported altitude is crap
-- good tips on which bands might be more free https://github.com/TheThingsNetwork/ttn/issues/119
-- finish power measurements (GPS on during sleep vs LCD on during sleep vs LORA on during sleep) and est battery life
-- make screen sleep behavior work
-- make screen advance only when a new node update arrives, a new text arrives or the user presses a button, turn off screen after a while
-- after reboot, channel number is getting reset to zero! fix!
-- send user and location events much less often
-- send location (or if not available user) when the user wakes the device from display sleep (both for testing and to improve user experience)
-- make real implementation of getNumOnlineNodes
-- very occasionally send our position and user packet based on the schedule in the radio info (if for nothing else so that other nodes update last_seen)
-- show real text info on the text screen
-- apply radio settings from android land
-- cope with nodes that have 0xff or 0x00 as the last byte of their mac
-- allow setting full radio params from android
-- add receive timestamps to messages, inserted by esp32 when message is received but then shown on the phone
-- update build to generate both board types
-- have node info screen show real info (including distance and heading)
-- blink the power led less often
-- have radiohead ISR send messages to RX queue directly, to allow that thread to block until we have something to send
-- move lora rx/tx to own thread and block on IO
-- keep our pseudo time moving forward even if we enter deep sleep (use esp32 rtc)
-- for non GPS equipped devices, set time from phone
-- GUI on oled hangs for a few seconds occasionally, but comes back
-- update local GPS position (but do not broadcast) at whatever rate the GPS is giving it
-- don't send our times to other nodes
-- don't trust times from other nodes
-- draw compass rose based off local walking track
-- add requestResponse optional bool - use for location broadcasts when sending tests
-- post sample video to signal forum
-- support non US frequencies
-- send pr https://github.com/ThingPulse/esp8266-oled-ssd1306 to tell them about this project
-- document rules for sleep wrt lora/bluetooth/screen/gps. also: if I have text messages (only) for the phone, then give a few seconds in the hopes BLE can get it across before we have to go back to sleep.
-- wake from light sleep as needed for our next scheduled periodic task (needed for gps position broadcasts etc)
-- turn bluetooth off based on our sleep policy
-- blink LED while in LS sleep mode
-- scrolling between screens based on press is busted
-- Use Neo-M8M API to put it in sleep mode (on hold until my new boards arrive)
-- update the prebuilt bins for different regulatory regions
-- don't enter NB state if we've recently talked to the phone (to prevent breaking syncing or bluetooth sw update)
-- have sw update prevent BLE sleep
-- manually delete characteristics/descs
-- leave lora receiver always on
-- protobufs are sometimes corrupted after sleep!
-- stay awake while charging
-- check gps battery voltage
-- if a position report includes ground truth time and we don't have time yet, set our clock from that. It is better than nothing.
-- retest BLE software update for both board types
-- report on wikifactory
-- send note to the guy who designed the cases
-- turn light sleep on aggressively (while lora is on but BLE off)
-- Use the Periodic class for both position and user periodic broadcasts
-- don't treat north as up, instead adjust shown bearings for our guess at the users heading (i.e. subtract one from the other)
-- sendToMesh can currently block for a long time, instead have it just queue a packet for a radio freertos thread
-- don't even power on bluetooth until we have some data to send to the android phone. Most of the time we should be sleeping in a lowpower "listening for lora" only mode. Once we have some packets for the phone, then power on bluetooth
-  until the phone pulls those packets. Ever so often power on bluetooth just so we can see if the phone wants to send some packets. Possibly might need ULP processor to help with this wake process.
-- do hibernation mode to get power draw down to 2.5uA https://lastminuteengineers.com/esp32-sleep-modes-power-consumption/
-- fix GPS.zeroOffset calculation it is wrong
-- (needs testing) fixed the following during a plane flight:
-  Have state machine properly enter deep sleep based on loss of mesh and phone comms.
-  Default to enter deep sleep if no LORA received for two hours (indicates user has probably left the mesh).
-- (fixed I think) text messages are not showing on local screen if screen was on
-- add links to todos
-- link to the kanban page
-- add a getting started page
-- finish mesh alg reeval
-- ublox gps parsing seems a little buggy (we shouldn't be sending out read solution commands, the device is already broadcasting them)
-- turn on gps https://github.com/sparkfun/SparkFun_Ublox_Arduino_Library/blob/master/examples/Example18_PowerSaveMode/Example18_PowerSaveMode.ino
-- switch gps to 38400 baud https://github.com/sparkfun/SparkFun_Ublox_Arduino_Library/blob/master/examples/Example11_ResetModule/Example2_FactoryDefaultsviaSerial/Example2_FactoryDefaultsviaSerial.ino
-- Use Neo-M8M API to put it in sleep mode
-- use gps sleep mode instead of killing its power (to allow fast position when we wake)
-- enable fast lock and low power inside the gps chip
-- Make a FAQ
-- add a SF12 transmit option for _super_ long range
-- figure out why this fixme is needed: "FIXME, disable wake due to PMU because it seems to fire all the time?"
-- "AXP192 interrupt is not firing, remove this temporary polling of battery state"
-- make debug info screen show real data (including battery level & charging) - close corresponding github issue
-- remeasure wake time power draws now that we run CPU down at 80MHz
+- an open source version of https://www.burnair.ch/skynet/
+- a paragliding app like http://airwhere.co.uk/
+- How do avalanche beacons work? Could this do that as well? possibly by using beacon mode feature of the RF95?

docs/software/bluetooth-api.md

@@ -10,19 +10,28 @@ This device will work with any MTU size, but it is highly recommended that you c
 
 This is the main bluetooth service for the device and provides the API your app should use to get information about the mesh, send packets or provision the radio.
 
-For a reference implementation of a client that uses this service see [RadioInterfaceService](https://github.com/meshtastic/Meshtastic-Android/blob/master/app/src/main/java/com/geeksville/mesh/service/RadioInterfaceService.kt). Typical flow when
-a phone connects to the device should be the following:
+For a reference implementation of a client that uses this service see [RadioInterfaceService](https://github.com/meshtastic/Meshtastic-Android/blob/master/app/src/main/java/com/geeksville/mesh/service/RadioInterfaceService.kt).
 
+Typical flow when a phone connects to the device should be the following (if you want to watch this flow from the python app just run "meshtastic --debug --info" - the flow over BLE is identical):
+
+- There are only three relevant endpoints (and they have built in BLE documentation - so use a BLE tool of your choice to watch them): FromRadio, FromNum (sends notifies when new data is available in FromRadio) and ToRadio
 - SetMTU size to 512
+- Write a ToRadio.startConfig protobuf to the "ToRadio" endpoint" - this tells the radio you are a new connection and you need the entire NodeDB sent down.
+- Read repeatedly from the "FromRadio" endpoint. Each time you read you will get back a FromRadio protobuf (see Meshtatastic-protobuf). Keep reading from this endpoint until you get back and empty buffer.
+- See below for the expected sequence for your initial download.
+- After the initial download, you should subscribe for BLE "notify" on the "FromNum" endpoint. If a notification arrives, that means there are now one or more FromRadio packets waiting inside FromRadio. Read from FromRadio until you get back an empty packet.
+- Any time you want to send packets to the radio, you should write a ToRadio packet into ToRadio.
+
+Expected sequence for initial download:
+
+- After your send startConfig, you will receive a series of FromRadio packets. The sequence of these packets will be as follows (but you are best not counting on this, instead just update your model for whatever packet you receive - based on looking at the type)
 - Read a RadioConfig from "radio" - used to get the channel and radio settings
-- Read (and write if incorrect) a User from "user" - to get the username for this node
+- Read a User from "user" - to get the username for this node
 - Read a MyNodeInfo from "mynode" to get information about this local device
 - Write an empty record to "nodeinfo" to restart the nodeinfo reading state machine
-- Read from "nodeinfo" until it returns empty to build the phone's copy of the current NodeDB for the mesh
-- Read from "fromradio" until it returns empty to get any messages that arrived for this node while the phone was away
-- Subscribe to notify on "fromnum" to get notified whenever the device has a new received packet
-- Read that new packet from "fromradio"
-- Whenever the phone has a packet to send write to "toradio"
+- Read a series of NodeInfo packets to build the phone's copy of the current NodeDB for the mesh
+- Read a endConfig packet that indicates that the entire state you need has been sent.
+- Read a series of MeshPackets until it returns empty to get any messages that arrived for this node while the phone was away
 
 For definitions (and documentation) on FromRadio, ToRadio, MyNodeInfo, NodeInfo and User protocol buffers see [mesh.proto](https://github.com/meshtastic/Meshtastic-protobufs/blob/master/mesh.proto)
 

docs/software/build-instructions.md

@@ -6,10 +6,11 @@ in these instructions I describe use of their command line tool.
 1. Purchase a suitable radio (see above)
 2. Install [PlatformIO](https://platformio.org/platformio-ide)
 3. Download this git repo and cd into it
-4. If you are outside the USA, edit [platformio.ini](/platformio.ini) to set the correct frequency range for your country. The line you need to change starts with "hw_version" and instructions are provided above that line. Options are provided for EU433, EU835, CN, JP and US. Pull-requests eagerly accepted for other countries.
-5. Plug the radio into your USB port
-6. Type "pio run --environment XXX -t upload" (This command will fetch dependencies, build the project and install it on the board via USB). For XXX, use the board type you have (either tbeam, heltec, ttgo-lora32-v1, ttgo-lora32-v2).
-7. Platform IO also installs a very nice VisualStudio Code based IDE, see their [tutorial](https://docs.platformio.org/en/latest/tutorials/espressif32/arduino_debugging_unit_testing.html) if you'd like to use it.
+4. Run `git submodule update --init --recursive` to pull in dependencies this project needs.
+5. If you are outside the USA, edit [platformio.ini](/platformio.ini) to set the correct frequency range for your country. The line you need to change starts with `hw_version` and instructions are provided above that line. Options are provided for `EU433`, `EU835`, `CN`, `JP` and `US` (default). Pull-requests eagerly accepted for other countries.
+6. Plug the radio into your USB port
+7. Type `pio run --environment XXX -t upload` (This command will fetch dependencies, build the project and install it on the board via USB). For XXX, use the board type you have (either `tbeam`, `heltec`, `ttgo-lora32-v1`, `ttgo-lora32-v2`).
+8. Platform IO also installs a very nice VisualStudio Code based IDE, see their [tutorial](https://docs.platformio.org/en/latest/tutorials/espressif32/arduino_debugging_unit_testing.html) if you'd like to use it.
 
 ## Decoding stack traces
 

docs/software/crypto.md

@@ -34,7 +34,4 @@ Note that for both stategies, sizes are measured in blocks and that an AES block
 
 ## Remaining todo
 
-- Make the packet numbers 32 bit
-- Confirm the packet #s are stored in flash across deep sleep (and otherwise in in RAM)
 - Have the app change the crypto key when the user generates a new channel
-- Implement for NRF52 [NRF52](https://infocenter.nordicsemi.com/topic/com.nordic.infocenter.sdk5.v15.0.0/lib_crypto_aes.html#sub_aes_ctr)

docs/software/nrf52-TODO.md

@@ -40,7 +40,6 @@ Needed to be fully functional at least at the same level of the ESP32 boards. At
 
 ## Items to be 'feature complete'
 
-- change packet numbers to be 32 bits
 - check datasheet about sx1262 temperature compensation
 - enable brownout detection and watchdog
 - stop polling for GPS characters, instead stay blocked on read in a thread
@@ -60,8 +59,6 @@ Needed to be fully functional at least at the same level of the ESP32 boards. At
 
 Nice ideas worth considering someday...
 
-- Use flego to me an iOS/linux app? https://felgo.com/doc/qt/qtbluetooth-index/ or
-- Use flutter to make an iOS/linux app? https://github.com/Polidea/FlutterBleLib
 - enable monitor mode debugging (need to use real jlink): https://devzone.nordicsemi.com/nordic/nordic-blog/b/blog/posts/monitor-mode-debugging-with-j-link-and-gdbeclipse
 - Improve efficiency of PeriodicTimer by only checking the next queued timer event, and carefully sorting based on schedule
 - make a Mfg Controller and device under test classes as examples of custom app code for third party devs. Make a post about this. Use a custom payload type code. Have device under test send a broadcast with max hopcount of 0 for the 'mfgcontroller' payload type. mfg controller will read SNR and reply. DOT will declare failure/success and switch to the regular app screen.
@@ -128,6 +125,7 @@ Nice ideas worth considering someday...
 - scheduleOSCallback doesn't work yet - it is way too fast (causes rapid polling of busyTx, high power draw etc...)
 - find out why we reboot while debugging - it was bluetooth/softdevice
 - make a file system implementation (preferably one that can see the files the bootloader also sees) - preferably https://github.com/adafruit/Adafruit_nRF52_Arduino/blob/master/libraries/InternalFileSytem/examples/Internal_ReadWrite/Internal_ReadWrite.ino else use https://infocenter.nordicsemi.com/topic/com.nordic.infocenter.sdk5.v15.3.0/lib_fds_usage.html?cp=7_5_0_3_55_3
+- change packet numbers to be 32 bits
 
 ```
 

docs/software/power.md

@@ -36,6 +36,10 @@ From lower to higher power consumption.
   onEntry: setBluetoothOn(true), screen.setOn(true)
   onExit: screen.setOn(false)
 
+- serial API usage (SERIAL) - Screen is on, device doesn't sleep, bluetooth off
+  onEntry: setBluetooth off, screen on
+  onExit:
+
 ## Behavior
 
 ### events that increase CPU activity
@@ -51,9 +55,11 @@ From lower to higher power consumption.
 - While in DARK/ON: If we receive EVENT_BLUETOOTH_PAIR we transition to ON and start our screen_on_secs timeout
 - While in NB/DARK/ON: If we receive EVENT_NODEDB_UPDATED we transition to ON (so the new screen can be shown)
 - While in DARK: While the phone talks to us over BLE (EVENT_CONTACT_FROM_PHONE) reset any sleep timers and stay in DARK (needed for bluetooth sw update and nice user experience if the user is reading/replying to texts)
+- while in LS/NB/DARK: if SERIAL_CONNECTED, go to serial
 
 ### events that decrease cpu activity
 
+- While in SERIAL: if SERIAL_DISCONNECTED, go to NB
 - While in ON: If PRESS event occurs, reset screen_on_secs timer and tell the screen to handle the pess
 - While in ON: If it has been more than screen_on_secs since a press, lower to DARK
 - While in DARK: If time since last contact by our phone exceeds phone_timeout_secs (15 minutes), we transition down into NB mode

platformio.ini

@@ -22,8 +22,6 @@ default_envs = cubecellplus ; Note: the github actions CI test build can't yet b
 ; HW_VERSION (default emptystring)
 
 [env]
-platform = espressif32
-framework = arduino
 
 ; customize the partition table
 ; http://docs.platformio.org/en/latest/platforms/espressif32.html#partition-tables
@@ -79,6 +77,8 @@ lib_deps =
   
 ; Common settings for ESP targes, mixin with extends = esp32_base
 [esp32_base]
+platform = espressif32
+framework = arduino
 src_filter = 
   ${env.src_filter} -<nrf52/>
 upload_speed = 921600
@@ -86,6 +86,8 @@ debug_init_break = tbreak setup
 build_flags =
   ${env.build_flags} -Wall -Wextra -Isrc/esp32 
 lib_ignore = segger_rtt
+platform_packages =
+    framework-arduinoespressif32 @ https://github.com/meshtastic/arduino-esp32.git
 
 ; The 1.0 release of the TBEAM board
 [env:tbeam]

proto

@@ -1 +1 @@
-Subproject commit 9d083d5d4ff4ef095135b18468004eaba77cb691
+Subproject commit e7f181ef6fd4e38c40e0d0be552149f8bbe75a66

src/PowerFSM.cpp

@@ -26,46 +26,49 @@ static void sdsEnter()
 
 #include "error.h"
 
+static uint32_t secsSlept;
+
 static void lsEnter()
 {
     DEBUG_MSG("lsEnter begin, ls_secs=%u\n", radioConfig.preferences.ls_secs);
     screen.setOn(false);
+    secsSlept = 0; // How long have we been sleeping this time
 
     DEBUG_MSG("lsEnter end\n");
 }
 
 static void lsIdle()
 {
-    DEBUG_MSG("lsIdle begin ls_secs=%u\n", radioConfig.preferences.ls_secs);
+    // DEBUG_MSG("lsIdle begin ls_secs=%u\n", radioConfig.preferences.ls_secs);
 
 #ifndef NO_ESP32
-    uint32_t secsSlept = 0;
     esp_sleep_source_t wakeCause = ESP_SLEEP_WAKEUP_UNDEFINED;
-    bool reached_ls_secs = false;
 
-    while (!reached_ls_secs) {
+    // Do we have more sleeping to do?
+    if (secsSlept < radioConfig.preferences.ls_secs) {
         // Briefly come out of sleep long enough to blink the led once every few seconds
-        uint32_t sleepTime = 5;
+        uint32_t sleepTime = 30;
 
+        // If some other service would stall sleep, don't let sleep happen yet
+        if (doPreflightSleep()) {
             setLed(false); // Never leave led on while in light sleep
             wakeCause = doLightSleep(sleepTime * 1000LL);
-        if (wakeCause != ESP_SLEEP_WAKEUP_TIMER)
-            break;
+
+            if (wakeCause == ESP_SLEEP_WAKEUP_TIMER) {
+                // Normal case: timer expired, we should just go back to sleep ASAP
 
                 setLed(true);                // briefly turn on led
-        doLightSleep(1);
-        if (wakeCause != ESP_SLEEP_WAKEUP_TIMER)
-            break;
+                wakeCause = doLightSleep(1); // leave led on for 1ms
 
                 secsSlept += sleepTime;
-        reached_ls_secs = secsSlept >= radioConfig.preferences.ls_secs;
+                // DEBUG_MSG("sleeping, flash led!\n");
             }
-    setLed(false);
-
-    if (reached_ls_secs) {
-        // stay in LS mode but let loop check whatever it wants
-        DEBUG_MSG("reached ls_secs, servicing loop()\n");
+            if (wakeCause == ESP_SLEEP_WAKEUP_UART) {
+                // Not currently used (because uart triggers in hw have problems)
+                powerFSM.trigger(EVENT_SERIAL_CONNECTED);
             } else {
+                // We woke for some other reason (button press, uart, device interrupt)
+                // uint64_t status = esp_sleep_get_ext1_wakeup_status();
                 DEBUG_MSG("wakeCause %d\n", wakeCause);
 
 #ifdef BUTTON_PIN
@@ -81,6 +84,16 @@ static void lsIdle()
                     powerFSM.trigger(EVENT_WAKE_TIMER);
                 }
             }
+        } else {
+            // Someone says we can't sleep now, so just save some power by sleeping the CPU for 100ms or so
+            delay(100);
+        }
+    } else {
+        // Time to stop sleeping!
+        setLed(false);
+        DEBUG_MSG("reached ls_secs, servicing loop()\n");
+        powerFSM.trigger(EVENT_WAKE_TIMER);
+    }
 #endif
 }
 
@@ -104,6 +117,12 @@ static void darkEnter()
     screen.setOn(false);
 }
 
+static void serialEnter()
+{
+    setBluetoothEnable(false);
+    screen.setOn(true);
+}
+
 static void onEnter()
 {
     screen.setOn(true);
@@ -133,6 +152,7 @@ State stateSDS(sdsEnter, NULL, NULL, "SDS");
 State stateLS(lsEnter, lsIdle, lsExit, "LS");
 State stateNB(nbEnter, NULL, NULL, "NB");
 State stateDARK(darkEnter, NULL, NULL, "DARK");
+State stateSERIAL(serialEnter, NULL, NULL, "SERIAL");
 State stateBOOT(bootEnter, NULL, NULL, "BOOT");
 State stateON(onEnter, NULL, NULL, "ON");
 Fsm powerFSM(&stateBOOT);
@@ -148,7 +168,7 @@ void PowerFSM_setup()
 
     powerFSM.add_transition(&stateNB, &stateNB, EVENT_RECEIVED_PACKET, NULL, "Received packet, resetting win wake");
 
-    // Handle press events
+    // Handle press events - note: we ignore button presses when in API mode
     powerFSM.add_transition(&stateLS, &stateON, EVENT_PRESS, NULL, "Press");
     powerFSM.add_transition(&stateNB, &stateON, EVENT_PRESS, NULL, "Press");
     powerFSM.add_transition(&stateDARK, &stateON, EVENT_PRESS, NULL, "Press");
@@ -160,6 +180,7 @@ void PowerFSM_setup()
     powerFSM.add_transition(&stateNB, &stateSDS, EVENT_LOW_BATTERY, NULL, "LowBat");
     powerFSM.add_transition(&stateDARK, &stateSDS, EVENT_LOW_BATTERY, NULL, "LowBat");
     powerFSM.add_transition(&stateON, &stateSDS, EVENT_LOW_BATTERY, NULL, "LowBat");
+    powerFSM.add_transition(&stateSERIAL, &stateSDS, EVENT_LOW_BATTERY, NULL, "LowBat");
 
     powerFSM.add_transition(&stateDARK, &stateON, EVENT_BLUETOOTH_PAIR, NULL, "Bluetooth pairing");
     powerFSM.add_transition(&stateON, &stateON, EVENT_BLUETOOTH_PAIR, NULL, "Bluetooth pairing");
@@ -173,6 +194,13 @@ void PowerFSM_setup()
     powerFSM.add_transition(&stateDARK, &stateON, EVENT_RECEIVED_TEXT_MSG, NULL, "Received text");
     powerFSM.add_transition(&stateON, &stateON, EVENT_RECEIVED_TEXT_MSG, NULL, "Received text"); // restarts the sleep timer
 
+    powerFSM.add_transition(&stateLS, &stateSERIAL, EVENT_SERIAL_CONNECTED, NULL, "serial API");
+    powerFSM.add_transition(&stateNB, &stateSERIAL, EVENT_SERIAL_CONNECTED, NULL, "serial API");
+    powerFSM.add_transition(&stateDARK, &stateSERIAL, EVENT_SERIAL_CONNECTED, NULL, "serial API");
+    powerFSM.add_transition(&stateON, &stateSERIAL, EVENT_SERIAL_CONNECTED, NULL, "serial API");
+
+    powerFSM.add_transition(&stateSERIAL, &stateNB, EVENT_SERIAL_DISCONNECTED, NULL, "serial disconnect");
+
     powerFSM.add_transition(&stateDARK, &stateDARK, EVENT_CONTACT_FROM_PHONE, NULL, "Contact from phone");
 
     powerFSM.add_transition(&stateNB, &stateDARK, EVENT_PACKET_FOR_PHONE, NULL, "Packet for phone");

src/PowerFSM.h

@@ -14,6 +14,8 @@
 #define EVENT_NODEDB_UPDATED 8     // NodeDB has a big enough change that we think you should turn on the screen
 #define EVENT_CONTACT_FROM_PHONE 9 // the phone just talked to us over bluetooth
 #define EVENT_LOW_BATTERY 10 // Battery is critically low, go to sleep
+#define EVENT_SERIAL_CONNECTED 11
+#define EVENT_SERIAL_DISCONNECTED 12
 
 extern Fsm powerFSM;
 

src/SerialConsole.cpp

@@ -1,4 +1,5 @@
 #include "SerialConsole.h"
+#include "PowerFSM.h"
 #include "configuration.h"
 #include <Arduino.h>
 
@@ -26,12 +27,19 @@ void SerialConsole::init()
  */
 void SerialConsole::handleToRadio(const uint8_t *buf, size_t len)
 {
-    // Note: for the time being we could _allow_ debug printing to keep going out the console
-    // I _think_ this is okay because we currently only print debug msgs from loop() and we are only
-    // dispatching serial protobuf msgs from loop() as well.  When things are more threaded in the future this
-    // will need to change.
-    // setDestination(&noopPrint);
+    // Turn off debug serial printing once the API is activated, because other threads could print and corrupt packets
+    setDestination(&noopPrint);
     canWrite = true;
 
     StreamAPI::handleToRadio(buf, len);
 }
+
+/// Hookable to find out when connection changes
+void SerialConsole::onConnectionChanged(bool connected)
+{
+    if (connected) { // To prevent user confusion, turn off bluetooth while using the serial port api
+        powerFSM.trigger(EVENT_SERIAL_CONNECTED);
+    } else {
+        powerFSM.trigger(EVENT_SERIAL_DISCONNECTED);
+    }
+}

src/SerialConsole.h

@@ -26,6 +26,10 @@ class SerialConsole : public StreamAPI, public RedirectablePrint
             RedirectablePrint::write('\r');
         return RedirectablePrint::write(c);
     }
+
+  protected:
+    /// Hookable to find out when connection changes
+    virtual void onConnectionChanged(bool connected);
 };
 
 extern SerialConsole console;

src/WorkerThread.cpp

@@ -1,4 +1,5 @@
 #include "WorkerThread.h"
+#include "debug.h"
 #include <assert.h>
 
 #ifdef configUSE_PREEMPTION
@@ -18,6 +19,15 @@ void WorkerThread::doRun()
 {
     while (!wantExit) {
         block();
+
+#ifdef DEBUG_STACK
+        static uint32_t lastPrint = 0;
+        if (millis() - lastPrint > 10 * 1000L) {
+            lastPrint = millis();
+            meshtastic::printThreadInfo("net");
+        }
+#endif
+
         loop();
     }
 }
@@ -30,8 +40,6 @@ void NotifiedWorkerThread::notify(uint32_t v, eNotifyAction action)
     xTaskNotify(taskHandle, v, action);
 }
 
-
-
 void NotifiedWorkerThread::block()
 {
     xTaskNotifyWait(0,                                          // don't clear notification on entry

src/WorkerThread.h

@@ -18,6 +18,8 @@ class Thread
 
     virtual ~Thread() { vTaskDelete(taskHandle); }
 
+    uint32_t getStackHighwaterMark() { return uxTaskGetStackHighWaterMark(taskHandle); }
+
   protected:
     /**
      * The method that will be called when start is called.

src/configuration.h

@@ -120,6 +120,10 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 #define SSD1306_ADDRESS 0x3C
 
+// The SH1106 controller is almost, but not quite, the same as SSD1306
+// Define this if you know you have that controller or your "SSD1306" misbehaves.
+//#define USE_SH1106
+
 // Flip the screen upside down by default as it makes more sense on T-BEAM
 // devices. Comment this out to not rotate screen 180 degrees.
 #define FLIP_SCREEN_VERTICALLY
@@ -184,6 +188,13 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 // This string must exactly match the case used in release file names or the android updater won't work
 #define HW_VENDOR "heltec"
 
+// the default ESP32 Pin of 15 is the Oled SCL, set to 36 and 37 and works fine.
+//Tested on Neo6m module. 
+#undef GPS_RX_PIN
+#undef GPS_TX_PIN
+#define GPS_RX_PIN 36
+#define GPS_TX_PIN 37
+
 #ifndef USE_JTAG  // gpio15 is TDO for JTAG, so no I2C on this board while doing jtag
 #define I2C_SDA 4 // I2C pins for this board
 #define I2C_SCL 15
@@ -275,6 +286,8 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 // What platforms should use SEGGER?
 #ifdef NRF52_SERIES
 #define USE_SEGGER
+#else
+#define SERIAL0_RX_GPIO 3 // Always GPIO3 on ESP32
 #endif
 
 #ifdef USE_SEGGER

src/esp32/BluetoothSoftwareUpdate.cpp

@@ -30,8 +30,6 @@ class TotalSizeCharacteristic : public CallbackCharacteristic
 
     void onWrite(BLECharacteristic *c)
     {
-        BLEKeepAliveCallbacks::onWrite(c);
-
         LockGuard g(updateLock);
         // Check if there is enough to OTA Update
         uint32_t len = getValue32(c, 0);
@@ -67,8 +65,6 @@ class DataCharacteristic : public CallbackCharacteristic
 
     void onWrite(BLECharacteristic *c)
     {
-        BLEKeepAliveCallbacks::onWrite(c);
-
         LockGuard g(updateLock);
         std::string value = c->getValue();
         uint32_t len = value.length();
@@ -92,8 +88,6 @@ class CRC32Characteristic : public CallbackCharacteristic
 
     void onWrite(BLECharacteristic *c)
     {
-        BLEKeepAliveCallbacks::onWrite(c);
-
         LockGuard g(updateLock);
         uint32_t expectedCRC = getValue32(c, 0);
         uint32_t actualCRC = crc.finalize();

src/esp32/BluetoothUtil.cpp

@@ -8,54 +8,6 @@
 
 SimpleAllocator btPool;
 
-/**
- * Create standard device info service
- **/
-BLEService *createDeviceInfomationService(BLEServer *server, std::string hwVendor, std::string swVersion,
-                                          std::string hwVersion = "")
-{
-    BLEService *deviceInfoService = server->createService(BLEUUID((uint16_t)ESP_GATT_UUID_DEVICE_INFO_SVC));
-
-    BLECharacteristic *swC =
-        new BLECharacteristic(BLEUUID((uint16_t)ESP_GATT_UUID_SW_VERSION_STR), BLECharacteristic::PROPERTY_READ);
-    BLECharacteristic *mfC = new BLECharacteristic(BLEUUID((uint16_t)ESP_GATT_UUID_MANU_NAME), BLECharacteristic::PROPERTY_READ);
-    // BLECharacteristic SerialNumberCharacteristic(BLEUUID((uint16_t) ESP_GATT_UUID_SERIAL_NUMBER_STR),
-    // BLECharacteristic::PROPERTY_READ);
-
-    /*
-           * Mandatory characteristic for device info service?
-
-          BLECharacteristic *m_pnpCharacteristic = m_deviceInfoService->createCharacteristic(ESP_GATT_UUID_PNP_ID,
-      BLECharacteristic::PROPERTY_READ);
-
-      uint8_t sig, uint16_t vid, uint16_t pid, uint16_t version;
-          uint8_t pnp[] = { sig, (uint8_t) (vid >> 8), (uint8_t) vid, (uint8_t) (pid >> 8), (uint8_t) pid, (uint8_t) (version >>
-      8), (uint8_t) version }; m_pnpCharacteristic->setValue(pnp, sizeof(pnp));
-      */
-    swC->setValue(swVersion);
-    deviceInfoService->addCharacteristic(addBLECharacteristic(swC));
-    mfC->setValue(hwVendor);
-    deviceInfoService->addCharacteristic(addBLECharacteristic(mfC));
-    if (!hwVersion.empty()) {
-        BLECharacteristic *hwvC =
-            new BLECharacteristic(BLEUUID((uint16_t)ESP_GATT_UUID_HW_VERSION_STR), BLECharacteristic::PROPERTY_READ);
-        hwvC->setValue(hwVersion);
-        deviceInfoService->addCharacteristic(addBLECharacteristic(hwvC));
-    }
-    // SerialNumberCharacteristic.setValue("FIXME");
-    // deviceInfoService->addCharacteristic(&SerialNumberCharacteristic);
-
-    // m_manufacturerCharacteristic = m_deviceInfoService->createCharacteristic((uint16_t) 0x2a29,
-    // BLECharacteristic::PROPERTY_READ); m_manufacturerCharacteristic->setValue(name);
-
-    /* add these later?
-      ESP_GATT_UUID_SYSTEM_ID
-      */
-
-    // caller must call service->start();
-    return deviceInfoService;
-}
-
 bool _BLEClientConnected = false;
 
 class MyServerCallbacks : public BLEServerCallbacks
@@ -106,6 +58,54 @@ void addWithDesc(BLEService *service, BLECharacteristic *c, const char *descript
     addBLEDescriptor(desc);
 }
 
+/**
+ * Create standard device info service
+ **/
+BLEService *createDeviceInfomationService(BLEServer *server, std::string hwVendor, std::string swVersion,
+                                          std::string hwVersion = "")
+{
+    BLEService *deviceInfoService = server->createService(BLEUUID((uint16_t)ESP_GATT_UUID_DEVICE_INFO_SVC));
+
+    BLECharacteristic *swC =
+        new BLECharacteristic(BLEUUID((uint16_t)ESP_GATT_UUID_SW_VERSION_STR), BLECharacteristic::PROPERTY_READ);
+    BLECharacteristic *mfC = new BLECharacteristic(BLEUUID((uint16_t)ESP_GATT_UUID_MANU_NAME), BLECharacteristic::PROPERTY_READ);
+    // BLECharacteristic SerialNumberCharacteristic(BLEUUID((uint16_t) ESP_GATT_UUID_SERIAL_NUMBER_STR),
+    // BLECharacteristic::PROPERTY_READ);
+
+    /*
+           * Mandatory characteristic for device info service?
+
+          BLECharacteristic *m_pnpCharacteristic = m_deviceInfoService->createCharacteristic(ESP_GATT_UUID_PNP_ID,
+      BLECharacteristic::PROPERTY_READ);
+
+      uint8_t sig, uint16_t vid, uint16_t pid, uint16_t version;
+          uint8_t pnp[] = { sig, (uint8_t) (vid >> 8), (uint8_t) vid, (uint8_t) (pid >> 8), (uint8_t) pid, (uint8_t) (version >>
+      8), (uint8_t) version }; m_pnpCharacteristic->setValue(pnp, sizeof(pnp));
+      */
+    swC->setValue(swVersion);
+    deviceInfoService->addCharacteristic(addBLECharacteristic(swC));
+    mfC->setValue(hwVendor);
+    deviceInfoService->addCharacteristic(addBLECharacteristic(mfC));
+    if (!hwVersion.empty()) {
+        BLECharacteristic *hwvC =
+            new BLECharacteristic(BLEUUID((uint16_t)ESP_GATT_UUID_HW_VERSION_STR), BLECharacteristic::PROPERTY_READ);
+        hwvC->setValue(hwVersion);
+        deviceInfoService->addCharacteristic(addBLECharacteristic(hwvC));
+    }
+    // SerialNumberCharacteristic.setValue("FIXME");
+    // deviceInfoService->addCharacteristic(&SerialNumberCharacteristic);
+
+    // m_manufacturerCharacteristic = m_deviceInfoService->createCharacteristic((uint16_t) 0x2a29,
+    // BLECharacteristic::PROPERTY_READ); m_manufacturerCharacteristic->setValue(name);
+
+    /* add these later?
+      ESP_GATT_UUID_SYSTEM_ID
+      */
+
+    // caller must call service->start();
+    return deviceInfoService;
+}
+
 static BLECharacteristic *batteryLevelC;
 
 /**
@@ -223,11 +223,15 @@ void deinitBLE()
 
     pServer->getAdvertising()->stop();
 
+    if (pUpdate != NULL) {
         destroyUpdateService();
 
-    pUpdate->stop();
-    pDevInfo->stop();
         pUpdate->stop(); // we delete them below
+        pUpdate->executeDelete();
+    }
+
+    pDevInfo->stop();
+    pDevInfo->executeDelete();
 
     // First shutdown bluetooth
     BLEDevice::deinit(false);
@@ -235,14 +239,16 @@ void deinitBLE()
     // do not delete this - it is dynamically allocated, but only once - statically in BLEDevice
     // delete pServer->getAdvertising();
 
+    if (pUpdate != NULL)
         delete pUpdate;
     delete pDevInfo;
     delete pServer;
 
     batteryLevelC = NULL; // Don't let anyone generate bogus notifies
 
-    for (int i = 0; i < numChars; i++)
+    for (int i = 0; i < numChars; i++) {
         delete chars[i];
+    }
     numChars = 0;
 
     for (int i = 0; i < numDescs; i++)
@@ -276,15 +282,19 @@ BLEServer *initBLE(StartBluetoothPinScreenCallback startBtPinScreen, StopBluetoo
     // We now let users create the battery service only if they really want (not all devices have a battery)
     // BLEService *pBattery = createBatteryService(pServer);
 
+// #define BLE_SOFTWARE_UPDATE
+#ifdef BLE_SOFTWARE_UPDATE
     pUpdate = createUpdateService(pServer, hwVendor, swVersion,
                                   hwVersion); // We need to advertise this so our android ble scan operation can see it
 
+    pUpdate->start();
+#endif
+
     // It seems only one service can be advertised - so for now don't advertise our updater
     // pServer->getAdvertising()->addServiceUUID(pUpdate->getUUID());
 
     // start all our services (do this after creating all of them)
     pDevInfo->start();
-    pUpdate->start();
 
     // FIXME turn on this restriction only after the device is paired with a phone
     // advert->setScanFilter(false, true); // We let anyone scan for us (FIXME, perhaps only allow that until we are paired with a
@@ -293,7 +303,11 @@ BLEServer *initBLE(StartBluetoothPinScreenCallback startBtPinScreen, StopBluetoo
     static BLESecurity security; // static to avoid allocs
     BLESecurity *pSecurity = &security;
     pSecurity->setCapability(ESP_IO_CAP_OUT);
+
+    // FIXME - really should be ESP_LE_AUTH_REQ_SC_BOND but it seems there is a bug right now causing that bonding info to be lost
+    // occasionally?
     pSecurity->setAuthenticationMode(ESP_LE_AUTH_REQ_SC_BOND);
+
     pSecurity->setInitEncryptionKey(ESP_BLE_ENC_KEY_MASK | ESP_BLE_ID_KEY_MASK);
 
     return pServer;

src/esp32/CallbackCharacteristic.h

@@ -1,33 +1,12 @@
 #pragma once
-#include "PowerFSM.h" // FIXME - someday I want to make this OTA thing a separate lb at at that point it can't touch this
 #include "BLECharacteristic.h"
-
-/**
- * This mixin just lets the power management state machine know the phone is still talking to us
- */
-class BLEKeepAliveCallbacks : public BLECharacteristicCallbacks
-{
-public:
-    void onRead(BLECharacteristic *c)
-    {
-        powerFSM.trigger(EVENT_CONTACT_FROM_PHONE);
-    }
-
-    void onWrite(BLECharacteristic *c)
-    {
-        powerFSM.trigger(EVENT_CONTACT_FROM_PHONE);
-    }
-};
+#include "PowerFSM.h" // FIXME - someday I want to make this OTA thing a separate lb at at that point it can't touch this
 
 /**
  * A characterstic with a set of overridable callbacks
  */
-class CallbackCharacteristic : public BLECharacteristic, public BLEKeepAliveCallbacks
+class CallbackCharacteristic : public BLECharacteristic, public BLECharacteristicCallbacks
 {
   public:
-    CallbackCharacteristic(const char *uuid, uint32_t btprops)
-        : BLECharacteristic(uuid, btprops)
-    {
-        setCallbacks(this);
-    }
+    CallbackCharacteristic(const char *uuid, uint32_t btprops) : BLECharacteristic(uuid, btprops) { setCallbacks(this); }
 };

src/esp32/MeshBluetoothService.cpp

@@ -23,9 +23,6 @@ static CallbackCharacteristic *meshFromNumCharacteristic;
 
 BLEService *meshService;
 
-// If defined we will also support the old API
-#define SUPPORT_OLD_BLE_API
-
 class BluetoothPhoneAPI : public PhoneAPI
 {
     /**
@@ -44,150 +41,6 @@ class BluetoothPhoneAPI : public PhoneAPI
 
 BluetoothPhoneAPI *bluetoothPhoneAPI;
 
-class ProtobufCharacteristic : public CallbackCharacteristic
-{
-    const pb_msgdesc_t *fields;
-    void *my_struct;
-
-  public:
-    ProtobufCharacteristic(const char *uuid, uint32_t btprops, const pb_msgdesc_t *_fields, void *_my_struct)
-        : CallbackCharacteristic(uuid, btprops), fields(_fields), my_struct(_my_struct)
-    {
-        setCallbacks(this);
-    }
-
-    void onRead(BLECharacteristic *c)
-    {
-        BLEKeepAliveCallbacks::onRead(c);
-        size_t numbytes = pb_encode_to_bytes(trBytes, sizeof(trBytes), fields, my_struct);
-        DEBUG_MSG("pbread from %s returns %d bytes\n", c->getUUID().toString().c_str(), numbytes);
-        c->setValue(trBytes, numbytes);
-    }
-
-    void onWrite(BLECharacteristic *c)
-    {
-        BLEKeepAliveCallbacks::onWrite(c);
-        writeToDest(c, my_struct);
-    }
-
-  protected:
-    /// like onWrite, but we provide an different destination to write to, for use by subclasses that
-    /// want to optionally ignore parts of writes.
-    /// returns true for success
-    bool writeToDest(BLECharacteristic *c, void *dest)
-    {
-        // dumpCharacteristic(pCharacteristic);
-        std::string src = c->getValue();
-        DEBUG_MSG("pbwrite to %s of %d bytes\n", c->getUUID().toString().c_str(), src.length());
-        return pb_decode_from_bytes((const uint8_t *)src.c_str(), src.length(), fields, dest);
-    }
-};
-
-#ifdef SUPPORT_OLD_BLE_API
-class NodeInfoCharacteristic : public BLECharacteristic, public BLEKeepAliveCallbacks
-{
-  public:
-    NodeInfoCharacteristic()
-        : BLECharacteristic("d31e02e0-c8ab-4d3f-9cc9-0b8466bdabe8",
-                            BLECharacteristic::PROPERTY_WRITE | BLECharacteristic::PROPERTY_READ)
-    {
-        setCallbacks(this);
-    }
-
-    void onRead(BLECharacteristic *c)
-    {
-        BLEKeepAliveCallbacks::onRead(c);
-
-        const NodeInfo *info = nodeDB.readNextInfo();
-
-        if (info) {
-            DEBUG_MSG("Sending nodeinfo: num=0x%x, lastseen=%u, id=%s, name=%s\n", info->num, info->position.time, info->user.id,
-                      info->user.long_name);
-            size_t numbytes = pb_encode_to_bytes(trBytes, sizeof(trBytes), NodeInfo_fields, info);
-            c->setValue(trBytes, numbytes);
-        } else {
-            c->setValue(trBytes, 0); // Send an empty response
-            DEBUG_MSG("Done sending nodeinfos\n");
-        }
-    }
-
-    void onWrite(BLECharacteristic *c)
-    {
-        BLEKeepAliveCallbacks::onWrite(c);
-        DEBUG_MSG("Reset nodeinfo read pointer\n");
-        nodeDB.resetReadPointer();
-    }
-};
-
-// wrap our protobuf version with something that forces the service to reload the config
-class RadioCharacteristic : public ProtobufCharacteristic
-{
-  public:
-    RadioCharacteristic()
-        : ProtobufCharacteristic("b56786c8-839a-44a1-b98e-a1724c4a0262",
-                                 BLECharacteristic::PROPERTY_WRITE | BLECharacteristic::PROPERTY_READ, RadioConfig_fields,
-                                 &radioConfig)
-    {
-    }
-
-    void onRead(BLECharacteristic *c)
-    {
-        DEBUG_MSG("Reading radio config, sdsecs %u\n", radioConfig.preferences.sds_secs);
-        ProtobufCharacteristic::onRead(c);
-    }
-
-    void onWrite(BLECharacteristic *c)
-    {
-        DEBUG_MSG("Writing radio config\n");
-        ProtobufCharacteristic::onWrite(c);
-        bluetoothPhoneAPI->handleSetRadio(radioConfig);
-    }
-};
-
-// wrap our protobuf version with something that forces the service to reload the owner
-class OwnerCharacteristic : public ProtobufCharacteristic
-{
-  public:
-    OwnerCharacteristic()
-        : ProtobufCharacteristic("6ff1d8b6-e2de-41e3-8c0b-8fa384f64eb6",
-                                 BLECharacteristic::PROPERTY_WRITE | BLECharacteristic::PROPERTY_READ, User_fields, &owner)
-    {
-    }
-
-    void onWrite(BLECharacteristic *c)
-    {
-        BLEKeepAliveCallbacks::onWrite(
-            c); // NOTE: We do not call the standard ProtobufCharacteristic superclass, because we want custom write behavior
-
-        static User o; // if the phone doesn't set ID we are careful to keep ours, we also always keep our macaddr
-        if (writeToDest(c, &o)) {
-            bluetoothPhoneAPI->handleSetOwner(o);
-        }
-    }
-};
-
-class MyNodeInfoCharacteristic : public ProtobufCharacteristic
-{
-  public:
-    MyNodeInfoCharacteristic()
-        : ProtobufCharacteristic("ea9f3f82-8dc4-4733-9452-1f6da28892a2", BLECharacteristic::PROPERTY_READ, MyNodeInfo_fields,
-                                 &myNodeInfo)
-    {
-    }
-
-    void onRead(BLECharacteristic *c)
-    {
-        // update gps connection state
-        myNodeInfo.has_gps = gps->isConnected;
-
-        ProtobufCharacteristic::onRead(c);
-
-        myNodeInfo.error_code = 0; // The phone just read us, so throw it away
-        myNodeInfo.error_address = 0;
-    }
-};
-
-#endif
 
 class ToRadioCharacteristic : public CallbackCharacteristic
 {
@@ -196,9 +49,6 @@ class ToRadioCharacteristic : public CallbackCharacteristic
 
     void onWrite(BLECharacteristic *c)
     {
-        BLEKeepAliveCallbacks::onWrite(c);
-        DEBUG_MSG("Got on write\n");
-
         bluetoothPhoneAPI->handleToRadio(c->getData(), c->getValue().length());
     }
 };
@@ -212,7 +62,6 @@ class FromRadioCharacteristic : public CallbackCharacteristic
 
     void onRead(BLECharacteristic *c)
     {
-        BLEKeepAliveCallbacks::onRead(c);
         size_t numBytes = bluetoothPhoneAPI->getFromRadio(trBytes);
 
         // Someone is going to read our value as soon as this callback returns.  So fill it with the next message in the queue
@@ -236,11 +85,7 @@ class FromNumCharacteristic : public CallbackCharacteristic
         // observe(&service.fromNumChanged);
     }
 
-    void onRead(BLECharacteristic *c)
-    {
-        BLEKeepAliveCallbacks::onRead(c);
-        DEBUG_MSG("FIXME implement fromnum read\n");
-    }
+    void onRead(BLECharacteristic *c) { DEBUG_MSG("FIXME implement fromnum read\n"); }
 };
 
 /*
@@ -263,12 +108,6 @@ BLEService *createMeshBluetoothService(BLEServer *server)
     addWithDesc(service, meshFromNumCharacteristic, "fromRadio");
     addWithDesc(service, new ToRadioCharacteristic, "toRadio");
     addWithDesc(service, new FromRadioCharacteristic, "fromNum");
-#ifdef SUPPORT_OLD_BLE_API
-    addWithDesc(service, new MyNodeInfoCharacteristic, "myNode");
-    addWithDesc(service, new RadioCharacteristic, "radio");
-    addWithDesc(service, new OwnerCharacteristic, "owner");
-    addWithDesc(service, new NodeInfoCharacteristic, "nodeinfo");
-#endif
 
     meshFromNumCharacteristic->addDescriptor(addBLEDescriptor(new BLE2902())); // Needed so clients can request notification
 
@@ -292,6 +131,7 @@ void stopMeshBluetoothService()
 {
     assert(meshService);
     meshService->stop();
+    meshService->executeDelete();
 }
 
 void destroyMeshBluetoothService()

src/esp32/main-esp32.cpp

@@ -4,6 +4,7 @@
 #include "configuration.h"
 #include "main.h"
 #include "power.h"
+#include "sleep.h"
 #include "target_specific.h"
 
 bool bluetoothOn;
@@ -154,12 +155,31 @@ void axp192Init()
 }
 #endif
 
+/*
+static void printBLEinfo() {
+        int dev_num = esp_ble_get_bond_device_num();
+
+    esp_ble_bond_dev_t *dev_list = (esp_ble_bond_dev_t *)malloc(sizeof(esp_ble_bond_dev_t) * dev_num);
+    esp_ble_get_bond_device_list(&dev_num, dev_list);
+    for (int i = 0; i < dev_num; i++) {
+        // esp_ble_remove_bond_device(dev_list[i].bd_addr);
+    }
+
+} */
+
 void esp32Setup()
 {
     uint32_t seed = esp_random();
     DEBUG_MSG("Setting random seed %u\n", seed);
     randomSeed(seed); // ESP docs say this is fairly random
 
+    DEBUG_MSG("Total heap: %d\n", ESP.getHeapSize());
+    DEBUG_MSG("Free heap: %d\n", ESP.getFreeHeap());
+    DEBUG_MSG("Total PSRAM: %d\n", ESP.getPsramSize());
+    DEBUG_MSG("Free PSRAM: %d\n", ESP.getFreePsram());
+
+    // enableModemSleep();
+
 #ifdef AXP192_SLAVE_ADDRESS
     axp192Init();
 #endif

src/main.cpp

@@ -33,6 +33,7 @@
 #include "power.h"
 // #include "rom/rtc.h"
 #include "DSRRouter.h"
+#include "debug.h"
 #include "main.h"
 #include "screen.h"
 #include "sleep.h"
@@ -282,6 +283,8 @@ void loop()
     DEBUG_PORT.loop(); // Send/receive protobufs over the serial port
 #endif
 
+    // heap_caps_check_integrity_all(true); // FIXME - disable this expensive check
+
 #ifndef NO_ESP32
     esp32Loop();
 #endif
@@ -314,6 +317,14 @@ void loop()
         showingBootScreen = false;
     }
 
+#ifdef DEBUG_STACK
+    static uint32_t lastPrint = 0;
+    if (millis() - lastPrint > 10 * 1000L) {
+        lastPrint = millis();
+        meshtastic::printThreadInfo("main");
+    }
+#endif
+
     // Update the screen last, after we've figured out what to show.
     screen.debug()->setNodeNumbersStatus(nodeDB.getNumOnlineNodes(), nodeDB.getNumNodes());
     screen.debug()->setChannelNameStatus(channelSettings.name);

src/mesh/MemoryPool.h

@@ -5,12 +5,80 @@
 
 #include "PointerQueue.h"
 
+template <class T> class Allocator
+{
+
+  public:
+    virtual ~Allocator() {}
+
+    /// Return a queable object which has been prefilled with zeros.  Panic if no buffer is available
+    /// Note: this method is safe to call from regular OR ISR code
+    T *allocZeroed()
+    {
+        T *p = allocZeroed(0);
+
+        assert(p); // FIXME panic instead
+        return p;
+    }
+
+    /// Return a queable object which has been prefilled with zeros - allow timeout to wait for available buffers (you probably
+    /// don't want this version).
+    T *allocZeroed(TickType_t maxWait)
+    {
+        T *p = alloc(maxWait);
+
+        if (p)
+            memset(p, 0, sizeof(T));
+        return p;
+    }
+
+    /// Return a queable object which is a copy of some other object
+    T *allocCopy(const T &src, TickType_t maxWait = portMAX_DELAY)
+    {
+        T *p = alloc(maxWait);
+        assert(p);
+
+        if (p)
+            *p = src;
+        return p;
+    }
+
+    /// Return a buffer for use by others
+    virtual void release(T *p) = 0;
+
+  protected:
+    // Alloc some storage
+    virtual T *alloc(TickType_t maxWait) = 0;
+};
+
+/**
+ * An allocator that just uses regular free/malloc
+ */
+template <class T> class MemoryDynamic : public Allocator<T>
+{
+  public:
+    /// Return a buffer for use by others
+    virtual void release(T *p)
+    {
+        assert(p);
+        free(p);
+    }
+
+  protected:
+    // Alloc some storage
+    virtual T *alloc(TickType_t maxWait)
+    {
+        T *p = (T *)malloc(sizeof(T));
+        assert(p);
+        return p;
+    }
+};
+
 /**
  * A pool based allocator
  *
- * Eventually this routine will even be safe for ISR use...
  */
-template <class T> class MemoryPool
+template <class T> class MemoryPool : public Allocator<T>
 {
     PointerQueue<T> dead;
 
@@ -30,39 +98,8 @@ template <class T> class MemoryPool
 
     ~MemoryPool() { delete[] buf; }
 
-    /// Return a queable object which has been prefilled with zeros.  Panic if no buffer is available
-    /// Note: this method is safe to call from regular OR ISR code
-    T *allocZeroed()
-    {
-        T *p = allocZeroed(0);
-
-        assert(p); // FIXME panic instead
-        return p;
-    }
-
-    /// Return a queable object which has been prefilled with zeros - allow timeout to wait for available buffers (you probably
-    /// don't want this version).
-    T *allocZeroed(TickType_t maxWait)
-    {
-        T *p = dead.dequeuePtr(maxWait);
-
-        if (p)
-            memset(p, 0, sizeof(T));
-        return p;
-    }
-
-    /// Return a queable object which is a copy of some other object
-    T *allocCopy(const T &src, TickType_t maxWait = portMAX_DELAY)
-    {
-        T *p = dead.dequeuePtr(maxWait);
-
-        if (p)
-            *p = src;
-        return p;
-    }
-
     /// Return a buffer for use by others
-    void release(T *p)
+    virtual void release(T *p)
     {
         assert(dead.enqueue(p, 0));
         assert(p >= buf &&
@@ -78,4 +115,9 @@ template <class T> class MemoryPool
                (size_t)(p - buf) <
                    maxElements); // sanity check to make sure a programmer didn't free something that didn't come from this pool
     }
+
+  protected:
+    /// Return a queable object which has been prefilled with zeros - allow timeout to wait for available buffers (you
+    /// probably don't want this version).
+    virtual T *alloc(TickType_t maxWait) { return dead.dequeuePtr(maxWait); }
 };

src/mesh/MeshService.cpp

@@ -86,12 +86,14 @@ void MeshService::sendOurOwner(NodeNum dest, bool wantReplies)
 const MeshPacket *MeshService::handleFromRadioUser(const MeshPacket *mp)
 {
     bool wasBroadcast = mp->to == NODENUM_BROADCAST;
-    bool isCollision = mp->from == myNodeInfo.my_node_num;
 
+    // Disable this collision testing if we use 32 bit nodenums
+    bool isCollision = (sizeof(NodeNum) == 1) && (mp->from == myNodeInfo.my_node_num);
+
+    if (isCollision) {
         // we win if we have a lower macaddr
         bool weWin = memcmp(&owner.macaddr, &mp->decoded.user.macaddr, sizeof(owner.macaddr)) < 0;
 
-    if (isCollision) {
         if (weWin) {
             DEBUG_MSG("NOTE! Received a nodenum collision and we are vetoing\n");
 
@@ -158,7 +160,7 @@ int MeshService::handleFromRadio(const MeshPacket *mp)
 
     // If we veto a received User packet, we don't put it into the DB or forward it to the phone (to prevent confusing it)
     if (mp) {
-        DEBUG_MSG("Forwarding to phone, from=0x%x, rx_time=%u\n", mp->from, mp->rx_time);
+        printPacket("Forwarding to phone", mp);
         nodeDB.updateFrom(*mp); // update our DB state based off sniffing every RX packet from the radio
 
         fromNum++;

src/mesh/MeshTypes.h

@@ -6,8 +6,8 @@
 #include "mesh.pb.h"
 #include <Arduino.h>
 
-typedef uint8_t NodeNum;
-typedef uint8_t PacketId; // A packet sequence number
+typedef uint32_t NodeNum;
+typedef uint32_t PacketId; // A packet sequence number
 
 #define NODENUM_BROADCAST (sizeof(NodeNum) == 4 ? UINT32_MAX : UINT8_MAX)
 #define ERRNO_OK 0
@@ -29,4 +29,4 @@ typedef uint8_t PacketId; // A packet sequence number
 typedef int ErrorCode;
 
 /// Alloc and free packets to our global, ISR safe pool
-extern MemoryPool<MeshPacket> packetPool;
+extern Allocator<MeshPacket> &packetPool;

src/mesh/NodeDB.cpp

@@ -30,7 +30,7 @@ DeviceState versions used to be defined in the .proto file but really only this
 #define here.
 */
 
-#define DEVICESTATE_CUR_VER 8
+#define DEVICESTATE_CUR_VER 10
 #define DEVICESTATE_MIN_VER DEVICESTATE_CUR_VER
 
 #ifndef NO_ESP32
@@ -101,10 +101,12 @@ void NodeDB::resetRadioConfig()
     crypto->setKey(channelSettings.psk.size, channelSettings.psk.bytes);
 
     // temp hack for quicker testing
+
     /*
     radioConfig.preferences.screen_on_secs = 30;
     radioConfig.preferences.wait_bluetooth_secs = 30;
-    radioConfig.preferences.position_broadcast_secs = 15;
+    radioConfig.preferences.position_broadcast_secs = 6 * 60;
+    radioConfig.preferences.ls_secs = 60;
     */
 }
 
@@ -114,7 +116,6 @@ void NodeDB::init()
     devicestate.has_my_node = true;
     devicestate.has_radio = true;
     devicestate.has_owner = true;
-    devicestate.has_radio = false;
     devicestate.radio.has_channel_settings = true;
     devicestate.radio.has_preferences = true;
     devicestate.node_db_count = 0;
@@ -124,10 +125,8 @@ void NodeDB::init()
 
     // default to no GPS, until one has been found by probing
     myNodeInfo.has_gps = false;
-    myNodeInfo.node_num_bits = sizeof(NodeNum) * 8;
-    myNodeInfo.packet_id_bits = sizeof(PacketId) * 8;
     myNodeInfo.message_timeout_msec = FLOOD_EXPIRE_TIME;
-    myNodeInfo.min_app_version = 167;
+    myNodeInfo.min_app_version = 172;
     generatePacketId(); // FIXME - ugly way to init current_packet_id;
 
     // Init our blank owner info to reasonable defaults
@@ -135,18 +134,13 @@ void NodeDB::init()
     sprintf(owner.id, "!%02x%02x%02x%02x%02x%02x", ourMacAddr[0], ourMacAddr[1], ourMacAddr[2], ourMacAddr[3], ourMacAddr[4],
             ourMacAddr[5]);
     memcpy(owner.macaddr, ourMacAddr, sizeof(owner.macaddr));
+
+    // Set default owner name
+    pickNewNodeNum(); // Note: we will repick later, just in case the settings are corrupted, but we need a valid
+    // owner.short_name now
     sprintf(owner.long_name, "Unknown %02x%02x", ourMacAddr[4], ourMacAddr[5]);
-
-    // Crummy guess at our nodenum
-    pickNewNodeNum();
-
     sprintf(owner.short_name, "?%02X", myNodeInfo.my_node_num & 0xff);
 
-    // Include our owner in the node db under our nodenum
-    NodeInfo *info = getOrCreateNode(getNodeNum());
-    info->user = owner;
-    info->has_user = true;
-
     if (!FSBegin()) // FIXME - do this in main?
     {
         DEBUG_MSG("ERROR filesystem mount Failed\n");
@@ -157,6 +151,20 @@ void NodeDB::init()
     loadFromDisk();
     // saveToDisk();
 
+    // We set node_num and packet_id _after_ loading from disk, because we always want to use the values this
+    // rom was compiled for, not what happens to be in the save file.
+    myNodeInfo.node_num_bits = sizeof(NodeNum) * 8;
+    myNodeInfo.packet_id_bits = sizeof(PacketId) * 8;
+
+    // Note! We do this after loading saved settings, so that if somehow an invalid nodenum was stored in preferences we won't
+    // keep using that nodenum forever. Crummy guess at our nodenum (but we will check against the nodedb to avoid conflicts)
+    pickNewNodeNum();
+
+    // Include our owner in the node db under our nodenum
+    NodeInfo *info = getOrCreateNode(getNodeNum());
+    info->user = owner;
+    info->has_user = true;
+
     // We set these _after_ loading from disk - because they come from the build and are more trusted than
     // what is stored in flash
     strncpy(myNodeInfo.region, optstr(HW_VERSION), sizeof(myNodeInfo.region));
@@ -176,8 +184,11 @@ void NodeDB::init()
  */
 void NodeDB::pickNewNodeNum()
 {
-    // Pick an initial nodenum based on the macaddr
-    NodeNum r = sizeof(NodeNum) == 1 ? ourMacAddr[5]
+    NodeNum r = myNodeInfo.my_node_num;
+
+    // If we don't have a nodenum at app - pick an initial nodenum based on the macaddr
+    if (r == 0)
+        r = sizeof(NodeNum) == 1 ? ourMacAddr[5]
                                  : ((ourMacAddr[2] << 24) | (ourMacAddr[3] << 16) | (ourMacAddr[4] << 8) | ourMacAddr[5]);
 
     if (r == NODENUM_BROADCAST || r < NUM_RESERVED)
@@ -247,15 +258,18 @@ void NodeDB::saveToDisk()
         if (!pb_encode(&stream, DeviceState_fields, &devicestate)) {
             DEBUG_MSG("Error: can't write protobuf %s\n", PB_GET_ERROR(&stream));
             // FIXME - report failure to phone
-        }
 
             f.close();
+        } else {
+            // Success - replace the old file
+            f.close();
 
             // brief window of risk here ;-)
             if (!FS.remove(preffile))
                 DEBUG_MSG("Warning: Can't remove old pref file\n");
             if (!FS.rename(preftmp, preffile))
                 DEBUG_MSG("Error: can't rename new pref file\n");
+        }
     } else {
         DEBUG_MSG("ERROR: can't write prefs\n"); // FIXME report to app
     }

src/mesh/PacketHistory.cpp

@@ -2,8 +2,6 @@
 #include "configuration.h"
 #include "mesh-pb-constants.h"
 
-
-
 PacketHistory::PacketHistory()
 {
     recentPackets.reserve(MAX_NUM_NODES); // Prealloc the worst case # of records - to prevent heap fragmentation
@@ -48,7 +46,7 @@ bool PacketHistory::wasSeenRecently(const MeshPacket *p, bool withUpdate)
         r.sender = p->from;
         r.rxTimeMsec = now;
         recentPackets.push_back(r);
-        DEBUG_MSG("Adding packet record for fr=0x%x,to=0x%x,id=%d\n", p->from, p->to, p->id);
+        printPacket("Adding packet record", p);
     }
 
     return false;

src/mesh/PhoneAPI.cpp

@@ -1,6 +1,8 @@
 #include "PhoneAPI.h"
 #include "MeshService.h"
 #include "NodeDB.h"
+#include "PowerFSM.h"
+#include "RadioInterface.h"
 #include <assert.h>
 
 PhoneAPI::PhoneAPI()
@@ -14,16 +16,35 @@ void PhoneAPI::init()
     observe(&service.fromNumChanged);
 }
 
+void PhoneAPI::checkConnectionTimeout()
+{
+    if (isConnected) {
+        bool newConnected = (millis() - lastContactMsec < radioConfig.preferences.phone_timeout_secs * 1000L);
+        if (!newConnected) {
+            isConnected = false;
+            onConnectionChanged(isConnected);
+        }
+    }
+}
+
 /**
  * Handle a ToRadio protobuf
  */
 void PhoneAPI::handleToRadio(const uint8_t *buf, size_t bufLength)
 {
+    powerFSM.trigger(EVENT_CONTACT_FROM_PHONE); // As long as the phone keeps talking to us, don't let the radio go to sleep
+    lastContactMsec = millis();
+    if (!isConnected) {
+        isConnected = true;
+        onConnectionChanged(isConnected);
+    }
+    // return (lastContactMsec != 0) &&
+
     if (pb_decode_from_bytes(buf, bufLength, ToRadio_fields, &toRadioScratch)) {
         switch (toRadioScratch.which_variant) {
         case ToRadio_packet_tag: {
-            // If our phone is sending a position, see if we can use it to set our RTC
             MeshPacket &p = toRadioScratch.variant.packet;
+            printPacket("PACKET FROM PHONE", &p);
             service.handleToRadio(p);
             break;
         }
@@ -71,8 +92,12 @@ void PhoneAPI::handleToRadio(const uint8_t *buf, size_t bufLength)
  */
 size_t PhoneAPI::getFromRadio(uint8_t *buf)
 {
-    if (!available())
+    if (!available()) {
+        DEBUG_MSG("getFromRadio, !available\n");
         return false;
+    } else {
+        DEBUG_MSG("getFromRadio, state=%d\n", state);
+    }
 
     // In case we send a FromRadio packet
     memset(&fromRadioScratch, 0, sizeof(fromRadioScratch));
@@ -227,6 +252,9 @@ void PhoneAPI::handleToRadioPacket(MeshPacket *p) {}
 /// If the mesh service tells us fromNum has changed, tell the phone
 int PhoneAPI::onNotify(uint32_t newValue)
 {
+    checkConnectionTimeout(); // a handy place to check if we've heard from the phone (since the BLE version doesn't call this
+                              // from idle)
+
     if (state == STATE_SEND_PACKETS || state == STATE_LEGACY) {
         DEBUG_MSG("Telling client we have new packets %u\n", newValue);
         onNowHasData(newValue);

src/mesh/PhoneAPI.h

@@ -50,6 +50,11 @@ class PhoneAPI
     /// Use to ensure that clients don't get confused about old messages from the radio
     uint32_t config_nonce = 0;
 
+    /** the last msec we heard from the client on the other side of this link */
+    uint32_t lastContactMsec = 0;
+
+    bool isConnected = false;
+
   public:
     PhoneAPI();
 
@@ -85,6 +90,12 @@ class PhoneAPI
     /// Our fromradio packet while it is being assembled
     FromRadio fromRadioScratch;
 
+    /// Hookable to find out when connection changes
+    virtual void onConnectionChanged(bool connected) {}
+
+  /// If we haven't heard from the other side in a while then say not connected
+    void checkConnectionTimeout();
+
     /**
      * Subclasses can use this as a hook to provide custom notifications for their transport (i.e. bluetooth notifies)
      */

src/mesh/RadioInterface.cpp

@@ -24,7 +24,53 @@ separated by 2.16 MHz with respect to the adjacent channels. Channel zero starts
 // 1kb was too small
 #define RADIO_STACK_SIZE 4096
 
-RadioInterface::RadioInterface() : txQueue(MAX_TX_QUEUE)
+void printPacket(const char *prefix, const MeshPacket *p)
+{
+    DEBUG_MSG("%s (id=0x%08x Fr0x%02x To0x%02x, WantAck%d, HopLim%d", prefix, p->id, p->from & 0xff, p->to & 0xff, p->want_ack,
+              p->hop_limit);
+    if (p->which_payload == MeshPacket_decoded_tag) {
+        auto &s = p->decoded;
+        switch (s.which_payload) {
+        case SubPacket_data_tag:
+            DEBUG_MSG(" Payload:Data");
+            break;
+        case SubPacket_position_tag:
+            DEBUG_MSG(" Payload:Position");
+            break;
+        case SubPacket_user_tag:
+            DEBUG_MSG(" Payload:User");
+            break;
+        case 0:
+            DEBUG_MSG(" Payload:None");
+            break;
+        default:
+            DEBUG_MSG(" Payload:%d", s.which_payload);
+            break;
+        }
+        if (s.want_response)
+            DEBUG_MSG(" WANTRESP");
+
+        if (s.source != 0)
+            DEBUG_MSG(" source=%08x", s.source);
+
+        if (s.dest != 0)
+            DEBUG_MSG(" dest=%08x", s.dest);
+
+        if (s.which_ack == SubPacket_success_id_tag)
+            DEBUG_MSG(" successId=%08x", s.ack.success_id);
+        else if (s.which_ack == SubPacket_fail_id_tag)
+            DEBUG_MSG(" failId=%08x", s.ack.fail_id);
+    } else {
+        DEBUG_MSG(" encrypted");
+    }
+
+    if (p->rx_time != 0) {
+        DEBUG_MSG(" rxtime=%u", p->rx_time);
+    }
+    DEBUG_MSG(")\n");
+}
+
+RadioInterface::RadioInterface()
 {
     assert(sizeof(PacketHeader) == 4 || sizeof(PacketHeader) == 16); // make sure the compiler did what we expected
 

src/mesh/RadioInterface.h

@@ -59,7 +59,6 @@ class RadioInterface : protected NotifiedWorkerThread
 
   protected:
     MeshPacket *sendingPacket = NULL; // The packet we are currently sending
-    PointerQueue<MeshPacket> txQueue;
     uint32_t lastTxStart = 0L;
 
     /**
@@ -163,3 +162,6 @@ class SimRadio : public RadioInterface
     /// \return true if initialisation succeeded.
     virtual bool init() { return true; }
 };
+
+/// Debug printing for packets
+void printPacket(const char *prefix, const MeshPacket *p);

src/mesh/RadioLibInterface.cpp

@@ -114,8 +114,8 @@ bool RadioLibInterface::canSendImmediately()
 /// bluetooth comms code.  If the txmit queue is empty it might return an error
 ErrorCode RadioLibInterface::send(MeshPacket *p)
 {
-    DEBUG_MSG("enqueuing for send on mesh fr=0x%x,to=0x%x,id=%d (txGood=%d,rxGood=%d,rxBad=%d)\n", p->from, p->to, p->id, txGood,
-              rxGood, rxBad);
+    printPacket("enqueuing for send", p);
+    DEBUG_MSG("txGood=%d,rxGood=%d,rxBad=%d\n", txGood, rxGood, rxBad);
     ErrorCode res = txQueue.enqueue(p, 0) ? ERRNO_OK : ERRNO_UNKNOWN;
 
     if (res != ERRNO_OK) { // we weren't able to queue it, so we must drop it to prevent leaks
@@ -134,7 +134,7 @@ bool RadioLibInterface::canSleep()
 {
     bool res = txQueue.isEmpty();
     if (!res) // only print debug messages if we are vetoing sleep
-        DEBUG_MSG("radio wait to sleep, txEmpty=%d\n", txQueue.isEmpty());
+        DEBUG_MSG("radio wait to sleep, txEmpty=%d\n", res);
 
     return res;
 }
@@ -173,11 +173,13 @@ void RadioLibInterface::loop()
     case ISR_TX:
         handleTransmitInterrupt();
         startReceive();
+        // DEBUG_MSG("tx complete - starting timer\n");
         startTransmitTimer();
         break;
     case ISR_RX:
         handleReceiveInterrupt();
         startReceive();
+        // DEBUG_MSG("rx complete - starting timer\n");
         startTransmitTimer();
         break;
     case TRANSMIT_DELAY_COMPLETED:
@@ -192,6 +194,8 @@ void RadioLibInterface::loop()
                 assert(txp);
                 startSend(txp);
             }
+        } else {
+            // DEBUG_MSG("done with txqueue\n");
         }
         break;
     default:
@@ -216,7 +220,7 @@ void RadioLibInterface::startTransmitTimer(bool withDelay)
         uint32_t delay =
             !withDelay ? 1 : random(MIN_TX_WAIT_MSEC, MAX_TX_WAIT_MSEC); // See documentation for loop() wrt these values
                                                                          // DEBUG_MSG("xmit timer %d\n", delay);
-
+        // DEBUG_MSG("delaying %u\n", delay);
         setPeriod(delay);
     }
 }
@@ -233,7 +237,7 @@ void RadioLibInterface::completeSending()
 {
     if (sendingPacket) {
         txGood++;
-        DEBUG_MSG("Completed sending to=0x%x, id=%u\n", sendingPacket->to, sendingPacket->id);
+        printPacket("Completed sending", sendingPacket);
 
         // We are done sending that packet, release it
         packetPool.release(sendingPacket);
@@ -287,7 +291,7 @@ void RadioLibInterface::handleReceiveInterrupt()
             memcpy(mp->encrypted.bytes, payload, payloadLen);
             mp->encrypted.size = payloadLen;
 
-            DEBUG_MSG("Lora RX interrupt from=0x%x, id=%u\n", mp->from, mp->id);
+            printPacket("Lora RX", mp);
 
             deliverToReceiver(mp);
         }
@@ -297,7 +301,7 @@ void RadioLibInterface::handleReceiveInterrupt()
 /** start an immediate transmit */
 void RadioLibInterface::startSend(MeshPacket *txp)
 {
-    DEBUG_MSG("Starting low level send from=0x%x, to=0x%x, id=%u, want_ack=%d\n", txp->from, txp->to, txp->id, txp->want_ack);
+    printPacket("Starting low level send", txp);
     setStandby(); // Cancel any already in process receives
 
     size_t numbytes = beginSending(txp);

src/mesh/RadioLibInterface.h

@@ -29,6 +29,8 @@ class RadioLibInterface : public RadioInterface, private PeriodicTask
      */
     uint32_t rxBad = 0, rxGood = 0, txGood = 0;
 
+    PointerQueue<MeshPacket> txQueue = PointerQueue<MeshPacket>(MAX_TX_QUEUE);
+
   protected:
     float bw = 125;
     uint8_t sf = 9;

src/mesh/ReliableRouter.cpp

@@ -27,7 +27,7 @@ ErrorCode ReliableRouter::send(MeshPacket *p)
 bool ReliableRouter::shouldFilterReceived(const MeshPacket *p)
 {
     if (p->to == NODENUM_BROADCAST && p->from == getNodeNum()) {
-        DEBUG_MSG("Received someone rebroadcasting for us fr=0x%x,to=0x%x,id=%d\n", p->from, p->to, p->id);
+        printPacket("Rx someone rebroadcasting for us", p);
 
         // We are seeing someone rebroadcast one of our broadcast attempts.
         // If this is the first time we saw this, cancel any retransmissions we have queued up and generate an internal ack for

src/mesh/Router.cpp

@@ -19,11 +19,15 @@
     4 // max number of packets destined to our queue, we dispatch packets quickly so it doesn't need to be big
 
 // I think this is right, one packet for each of the three fifos + one packet being currently assembled for TX or RX
+// And every TX packet might have a retransmission packet or an ack alive at any moment
 #define MAX_PACKETS                                                                                                              \
-    (MAX_RX_TOPHONE + MAX_RX_FROMRADIO + MAX_TX_QUEUE +                                                                          \
+    (MAX_RX_TOPHONE + MAX_RX_FROMRADIO + 2 * MAX_TX_QUEUE +                                                                      \
      2) // max number of packets which can be in flight (either queued from reception or queued for sending)
 
-MemoryPool<MeshPacket> packetPool(MAX_PACKETS);
+static MemoryPool<MeshPacket> staticPool(MAX_PACKETS);
+// static MemoryDynamic<MeshPacket> staticPool;
+
+Allocator<MeshPacket> &packetPool = staticPool;
 
 /**
  * Constructor
@@ -56,8 +60,11 @@ PacketId generatePacketId()
 
     if (!didInit) {
         didInit = true;
-        i = random(0, numPacketId +
-                          1); // pick a random initial sequence number at boot (to prevent repeated reboots always starting at 0)
+
+        // pick a random initial sequence number at boot (to prevent repeated reboots always starting at 0)
+        // Note: we mask the high order bit to ensure that we never pass a 'negative' number to random
+        i = random(numPacketId & 0x7fffffff);
+        DEBUG_MSG("Initial packet id %u, numPacketId %u\n", i, numPacketId);
     }
 
     i++;
@@ -143,8 +150,8 @@ ErrorCode Router::send(MeshPacket *p)
  */
 void Router::sniffReceived(const MeshPacket *p)
 {
-    DEBUG_MSG("FIXME-update-db Sniffing packet fr=0x%x,to=0x%x,id=%d\n", p->from, p->to, p->id);
-    // FIXME, update nodedb
+    DEBUG_MSG("FIXME-update-db Sniffing packet\n");
+    // FIXME, update nodedb here for any packet that passes through us
 }
 
 bool Router::perhapsDecode(MeshPacket *p)
@@ -195,7 +202,7 @@ void Router::handleReceived(MeshPacket *p)
         sniffReceived(p);
 
         if (p->to == NODENUM_BROADCAST || p->to == getNodeNum()) {
-            DEBUG_MSG("Notifying observers of received packet fr=0x%x,to=0x%x,id=%d\n", p->from, p->to, p->id);
+            printPacket("Delivering rx packet", p);
             notifyPacketReceived.notifyObservers(p);
         }
     }

src/mesh/StreamAPI.cpp

@@ -9,6 +9,7 @@ void StreamAPI::loop()
 {
     writeStream();
     readStream();
+    checkConnectionTimeout();
 }
 
 /**

src/mesh/mesh.pb.c

@@ -9,7 +9,7 @@
 PB_BIND(Position, Position, AUTO)
 
 
-PB_BIND(Data, Data, 2)
+PB_BIND(Data, Data, AUTO)
 
 
 PB_BIND(User, User, AUTO)

src/mesh/mesh.pb.h

@@ -47,7 +47,7 @@ typedef struct _ChannelSettings {
     char name[12];
 } ChannelSettings;
 
-typedef PB_BYTES_ARRAY_T(251) Data_payload_t;
+typedef PB_BYTES_ARRAY_T(240) Data_payload_t;
 typedef struct _Data {
     Data_Type typ;
     Data_payload_t payload;
@@ -586,20 +586,20 @@ extern const pb_msgdesc_t ManufacturingData_msg;
 
 /* Maximum encoded size of messages (where known) */
 #define Position_size                            39
-#define Data_size                                256
+#define Data_size                                245
 #define User_size                                72
 #define RouteDiscovery_size                      88
-#define SubPacket_size                           285
-#define MeshPacket_size                          324
+#define SubPacket_size                           274
+#define MeshPacket_size                          313
 #define ChannelSettings_size                     60
 #define RadioConfig_size                         157
 #define RadioConfig_UserPreferences_size         93
 #define NodeInfo_size                            132
 #define MyNodeInfo_size                          110
-#define DeviceState_size                         15463
+#define DeviceState_size                         15100
 #define DebugString_size                         258
-#define FromRadio_size                           333
-#define ToRadio_size                             327
+#define FromRadio_size                           322
+#define ToRadio_size                             316
 /* ManufacturingData_size depends on runtime parameters */
 
 #ifdef __cplusplus

src/screen.cpp

@@ -34,7 +34,11 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 #define FONT_HEIGHT 14 // actually 13 for "ariel 10" but want a little extra space
 #define FONT_HEIGHT_16 (ArialMT_Plain_16[1] + 1)
+#ifdef USE_SH1106
+#define SCREEN_WIDTH 132
+#else
 #define SCREEN_WIDTH 128
+#endif
 #define SCREEN_HEIGHT 64
 #define TRANSITION_FRAMERATE 30 // fps
 #define IDLE_FRAMERATE 10       // in fps

src/screen.h

@@ -3,7 +3,12 @@
 #include <cstring>
 
 #include <OLEDDisplayUi.h>
+
+#ifdef USE_SH1106
+#include <SH1106Wire.h>
+#else
 #include <SSD1306Wire.h>
+#endif
 
 #include "PeriodicTask.h"
 #include "TypedQueue.h"
@@ -211,7 +216,11 @@ class Screen : public PeriodicTask
     /// Holds state for debug information
     DebugInfo debugInfo;
     /// Display device
+#ifdef USE_SH1106
+    SH1106Wire dispdev;
+#else
     SSD1306Wire dispdev;
+#endif
     /// UI helper for rendering to frames and switching between them
     OLEDDisplayUi ui;
 };

src/sleep.cpp

@@ -14,6 +14,7 @@
 #include "esp_pm.h"
 #include "rom/rtc.h"
 #include <driver/rtc_io.h>
+#include <driver/uart.h>
 
 #include "BluetoothUtil.h"
 
@@ -111,8 +112,7 @@ void initDeepSleep()
 #endif
 }
 
-/// return true if sleep is allowed
-static bool doPreflightSleep()
+bool doPreflightSleep()
 {
     if (preflightSleep.notifyObservers(NULL) != 0)
         return false; // vetoed
@@ -129,6 +129,7 @@ static void waitEnterSleep()
 
         if (millis() - now > 30 * 1000) { // If we wait too long just report an error and go to sleep
             recordCriticalError(ErrSleepEnterWait);
+            ESP.restart(); // FIXME - for now we just restart, need to fix bug #167
             break;
         }
     }
@@ -256,6 +257,17 @@ esp_sleep_wakeup_cause_t doLightSleep(uint64_t sleepMsec) // FIXME, use a more r
     gpio_pullup_en((gpio_num_t)BUTTON_PIN);
 #endif
 
+#ifdef SERIAL0_RX_GPIO
+    // We treat the serial port as a GPIO for a fast/low power way of waking, if we see a rising edge that means
+    // someone started to send something
+
+    // Alas - doesn't work reliably, instead need to use the uart specific version (which burns a little power)
+    // FIXME: gpio 3 is RXD for serialport 0 on ESP32
+    // Send a few Z characters to wake the port
+    gpio_wakeup_enable((gpio_num_t)SERIAL0_RX_GPIO, GPIO_INTR_LOW_LEVEL);
+    // uart_set_wakeup_threshold(UART_NUM_0, 3);
+    // esp_sleep_enable_uart_wakeup(0);
+#endif
 #ifdef BUTTON_PIN
     gpio_wakeup_enable((gpio_num_t)BUTTON_PIN, GPIO_INTR_LOW_LEVEL); // when user presses, this button goes low
 #endif
@@ -279,7 +291,7 @@ esp_sleep_wakeup_cause_t doLightSleep(uint64_t sleepMsec) // FIXME, use a more r
 }
 #endif
 
-#if 0
+
 // not legal on the stock android ESP build
 
 /**
@@ -294,8 +306,8 @@ void enableModemSleep()
   static esp_pm_config_esp32_t config; // filled with zeros because bss
 
   config.max_freq_mhz = CONFIG_ESP32_DEFAULT_CPU_FREQ_MHZ;
-  config.min_freq_mhz = 10; // 10Mhz is minimum recommended
+  config.min_freq_mhz = 20; // 10Mhz is minimum recommended
   config.light_sleep_enable = false;
   DEBUG_MSG("Sleep request result %x\n", esp_pm_configure(&config));
 }
-#endif
+

src/sleep.h

@@ -19,6 +19,9 @@ void initDeepSleep();
 void setCPUFast(bool on);
 void setLed(bool ledOn);
 
+/** return true if sleep is allowed right now */
+bool doPreflightSleep();
+
 extern int bootCount;
 
 // is bluetooth sw currently running?
@@ -32,3 +35,5 @@ extern Observable<void *> notifySleep;
 
 /// Called to tell observers we are now entering (deep) sleep and you should prepare.  Must return 0
 extern Observable<void *> notifyDeepSleep;
+
+void enableModemSleep();