Completed / Frozen V1 prototype

AMSAN — Building an Affordable AI Companion Robot From Scratch

A smartphone-powered autonomous companion robot combining embedded electronics, Android software, physical safety, expressive interaction, and Realtime AI.

AMSAN V1.0 viewed from the front at an angle with its smartphone face, blue body, front sensors and wheels visible.
Case study updated with final project media — 4 September 2026

AMSAN is a companion robot I designed and built around an Android smartphone and an ESP32. The phone handles the face, voice, memory, and high-level behaviour. The ESP32 controls movement, sensors, arms, RGB, and physical safety.

I wanted to learn robotics by making one complete physical system, not only testing individual sensors or pieces of code. Commercial companion robots can be expensive and hard to customise or repair, so I tried combining accessible electronics with a phone I could use as the display, microphone, speaker, processor, storage, and internet interface.

The result is a stable functional companion-robot prototype. It is not commercial hardware or a certified consumer product.

What V1 had to do

Movement and safety came first.

  • Move in four directions and stop reliably
  • Detect front and rear obstacles and recover directionally
  • Detect floor edges before a fall
  • Keep a hardware motor-safety layer
  • React through its face, touch, arms, RGB, and voice
  • Run local games, memory, and autonomous behaviour
  • Stay repairable and reasonably affordable

System architecture

Personality on the phone. Safety on the ESP32.

The division is intentional. Android can decide what AMSAN wants to do, but the ESP32 can refuse an unsafe movement request.

AMSAN control pathThe Android phone sends high-level commands over Bluetooth Classic SPP to an ESP32. The ESP32 checks sensors and safety rules before the TB6612FNG controls two motors.
High-level interaction

Android phone

FaceRig · microphone · speaker · Realtime voice · Life Engine · games · memory · settings · Developer Hub

Physical safety authority

ESP32 · SHR-01

Motors · front/rear distance · edge sensors · touch · arms · RGB · telemetry · watchdog · STOP rules

TB6612FNG

Motor driver

Two TT motors

Differential drive

Sensors + servos

Physical feedback

Hardware

A phone head on a repairable robot body.

The blue rectangular body carries a horizontally mounted Android phone, two front ultrasonic modules, an RGB ring, padded side arms, two powered wheels, a caster, and the internal controller and power wiring.

Side profile of AMSAN showing the body, phone mount, arm and drive wheel.
Side profile and drive layout.
Rear view of AMSAN showing the smartphone mount and rear-mounted power system.
Rear view with the mounted power system.
ControllerESP32 Dev Module
DriveTwo TT geared motors, about 7 cm wheels, caster wheel
Motor driverTB6612FNG
Obstacle sensingTwo HC-SR04 sensors, front and rear
Edge sensingFive TCRT5000-type sensors
TouchTwo TTP223 capacitive sensors
ArmsTwo servo-controlled arms
Lighting12-pixel addressable RGB ring
PowerRONIN R-125 power bank, 10,000 mAh / 37 Wh
Approximate chassis22.0 cm × 14.7 cm

The INA219 voltage/current sensor is provisioned at address 0x40 with firmware retry logic. The robot’s main movement and safety functions do not depend on it starting successfully.

GPIO reference

Motors AIN1 16 · AIN2 17 · PWMA 18 · BIN1 19 · BIN2 23 · PWMB 25

Ultrasonic Front TRIG 26 / ECHO 34 · Rear TRIG 32 / ECHO 35

Edge Front 27 · Rear 4

Servos Left 14 · Right 13

RGB 33 · 12 pixels

Touch Left 36 · Right 39

INA219 SDA 21 · SCL 22 · 0x40

Autonomy

Sense → Decide → Act → Recheck

AMSAN has one local character: curious, playful, mischievous, expressive, and sometimes a little stubborn. The behaviour engine coordinates movement, gaze, expressions, arms, RGB, and pauses so it feels coherent rather than random.

Front obstacle

  1. Front sensor detects an obstacle
  2. Immediate STOP
  3. Read fresh rear distance
  4. Reverse briefly if the rear is safe
  5. STOP and recheck
  6. Turn, check the front, then resume

Rear obstacle

  1. Rear sensor detects an obstacle
  2. Immediate STOP
  3. Read fresh front distance
  4. Move forward briefly if the front is safe
  5. Change heading
  6. Continue after another check
A front obstacle makes forward movement unsafe, not automatically backward movement. A rear obstacle blocks reverse, not automatically forward movement.

Obstacle distance

The two HC-SR04 sensors use an approximate 18 cm obstacle threshold.

Touch and gestures

Two touch sensors can trigger local face, RGB, and arm reactions without an API call.

RGB timing

The RGB engine is non-blocking so it does not delay sensors, Bluetooth, watchdog refreshes, or STOP commands.

Safety hierarchy

Safety always has priority over personality or autonomous behaviour.

  1. Hardware roller / STBY protectionAn extra physical layer can disable the motor driver.
  2. Edge and fall protectionFive TCRT5000-type sensors stop the robot before a drop.
  3. Bluetooth-loss STOPDisconnected control means the motors stop.
  4. Explicit or manual STOPA direct stop overrides lower-priority behaviour.
  5. Directional obstacle protectionUnsafe movement is refused while a safe recovery direction can remain available.
  6. Autonomous behaviourMovement runs only after every higher layer allows it.

Motor watchdog

If drive refreshes stop for about 1500 ms while the motors are active, the ESP32 stops them. Android refreshes active commands roughly every 400–500 ms.

Edge layout

Three front edge sensors are combined to GPIO27 and two rear sensors to GPIO4. Edge events outrank ordinary roaming and obstacle recovery.

Physical switches

Two roller/limit switches add a safety path through the TB6612 STBY arrangement, so motor safety does not depend entirely on Android.

AMSAN showing an ESP32 safety-stop state on its smartphone display while the RGB ring is red.
Safety-stop state shown on the phone.
Underside of AMSAN showing its wheels, motors, caster and physical sensor hardware.
Underside with the drive and physical sensor hardware.

Android, voice, and local behaviour

The cloud is for conversation, not basic robotics.

AMSAN used gpt-realtime-2.1-mini during development for intentional voice conversations. Obstacles, recovery, roaming, touch reactions, expressions, arms, RGB, games, and mood changes remain local.

Validated voice actions

Voice can request actions such as wave, dance, celebrate, arms home, happy, surprised, mischievous, sleepy, wake, or a game. The model cannot send unrestricted motor commands; requests are mapped into typed local actions and still pass through ESP32 safety.

Conversation

The app was tested in English, Urdu, and mixed Urdu/English. The phone microphone supports voice activity handling and barge-in where available, so a person can interrupt AMSAN while it is speaking.

Cost control

Autonomous events do not start Realtime turns. If nobody is intentionally talking to AMSAN, conversational API use should be zero. The Realtime session closes after about 25 seconds of inactivity.

Face and local reactions

The FaceRig uses large cyan eyes, pupils, highlights, eyebrows, eyelids, blinking, gaze, mouth movement, cheeks, and roughly 30 expressions and animations. Local reactions also include stylised animal sounds, birthday behaviour, and a short original tune or chant.

Front view of AMSAN with its animated smartphone face and illuminated RGB ring.
FaceRig and RGB ring during a real AMSAN demonstration.

Tic-Tac-Toe

A local 3×3 game with touch input, validation, winner and draw checks, minimax strategy, rematch, exit, sounds, and robot reactions.

Rock-Paper-Scissors

Local choice and scoring with rematch, exit, sounds, expressions, arms, and RGB reactions.

Memory

Profile memory on the phone can retain AMSAN’s identity, Hassan’s profile, useful facts, and preferences across app restarts. It is not unlimited human-like memory.

Development problems

The useful part was finding where the whole system failed.

Servo brownouts

Moving the arms made the ESP32 unstable. I revised the 5V distribution and common-ground arrangement instead of treating the resets as a software problem.

Watchdog timing

The ESP32 stops the motors after about 1500 ms without a valid drive refresh. Android was refreshing too slowly, so I changed it to send active commands about every 400–500 ms and kept the watchdog enabled.

Obstacle recovery

The robot could detect an obstacle and stop, but sometimes stayed stopped or tried the blocked direction again. Fresh telemetry was clearing a temporary Android stop marker before recovery ran. I kept the directional obstacle state until recovery finished.

Transport confusion

During development I had to separate simulator behaviour from the real Bluetooth transport. A screen that looked connected was not enough; commands and fresh sensor telemetry had to agree.

Audio routing

Realtime speech first used the phone's call-style earpiece path. I moved it to media audio and set a clear order: wired or USB output, intentional Bluetooth media, then the phone loudspeaker.

Local games

Tic-Tac-Toe and Rock-Paper-Scissors first existed only inside the Developer Hub. I connected validated voice actions to the local game engines without letting the remote model decide the rules.

Removing camera ML

CameraX, ML Kit, FaceNet, TensorFlow Lite, face enrollment, pose detection, and gestures were tested near the end. Automated tests worked, but the physical phone became less reliable. I removed the entire camera and ML runtime from V1.

Early AMSAN electronics prototype during hardware integration.
Early hardware integration before the final body was assembled.
I learned that a feature is not useful if it makes the complete system less reliable.

V1 result

A complete working prototype, with clear limits.

Verified V1 features

  • Autonomous forward, reverse, and turning behaviour
  • Front and rear obstacle detection with directional recovery
  • Edge protection, hardware STBY protection, Bluetooth-loss stop, manual stop, and motor watchdog
  • Animated face with roughly 30 core expressions and animations
  • Touch reactions, two servo arms, wave, dance, and celebrate actions
  • Non-blocking RGB moods and status patterns
  • English, Urdu, and mixed Urdu/English voice conversation
  • External audio support, phone-speaker fallback, microphone input, and barge-in
  • Offline Tic-Tac-Toe and Rock-Paper-Scissors
  • Persistent local profile memory and local animal and birthday reactions

Not included in V1

  • No camera vision, face recognition, hand-gesture recognition, or visual ML
  • No SLAM, mapping, room localisation, human following, or advanced object recognition
  • No side ultrasonic sensors
  • Realtime conversation needs internet access, API credentials, and available API budget
  • This is an engineering prototype, not a certified safety product

Possible V2 research

Camera-based social awareness, local face detection, recognition of Hassan, local face embeddings, gesture recognition, on-device ML, richer adaptation, and more sensors are possible future experiments. They are not V1 features.

Embedded

ESP32 · Arduino/C++ · Bluetooth Classic · Adafruit NeoPixel · INA219 support

Android

Kotlin · Jetpack Compose · Material 3 · Android audio APIs · DataStore

Hardware

TB6612FNG · TT motors · HC-SR04 · TCRT5000 · TTP223 · servos · RGB ring

Control and AI

Android Life Engine · validated actions · ESP32 safety · JSON protocol · OpenAI Realtime

Final demonstration

AMSAN V1.0 — Full Demonstration

A real demonstration of the completed AMSAN V1 prototype.

A real demonstration of the completed AMSAN V1 prototype.
← All projectsAsk about AMSAN