======================================
Making the LEGO Grand Piano play music
======================================
:CreationDate: 2020-08-28 10:39:28
:Id: modelli/lego-piano
:tags: - 3dPrint
- hardware
- software
- models
The `LEGO Grand Piano 21323
<https://www.lego.com/en-gb/product/grand-piano-21323>`_ is a great
model, full of clever details and impressive engineering.
It contains a bit of electronics, that allow it to talk with a phone
app via Bluetooth.
.. image:: lego-electronics.jpg
:width: 100%
:alt: a Lego optical sensor, motor, power&control brick, mounted on
a wide base of sideways pieces; the motor moves a long axle
with many pegs at right angles to each other
The app can play a few built-in tunes while running the motor inside
the piano, which pulls some keys down in a regular pattern, so you can
pretend to have a "player piano". The app can also wait for you to
press any key on the piano (they all move a little flag in front of
the optical sensor) before playing the next note, so you can pretend
to be playing.
That's fun, but not really enough: there are 25 independent keys,
surely we can make them play the right note!
.. image:: keys.jpg
:width: 100%
:alt: the full keyboard of the Lego piano, 2 octaves + 1, with
correctly-alternating white and black keys, each key has a
matching hammer
.. video:: keys-hammers.mp4
:width: 100%
:type: video/mp4
After some research, I decided to use optical sensors and a big
microcontroller. How hard can it be? ☺
As sensor I picked the `QRD1114
<https://learn.sparkfun.com/tutorials/qrd1114-optical-detector-hookup-guide/all>`_,
which is dead easy to use: infrared LED plus phototransistor, with a
very narrow and short area of sensitivity, so there's very little
chance of nearby keys setting off the wrong sensor.
The controller I used is a `Lilygo TTGO-T7 v1.3
<http://www.lilygo.cn/prod_view.aspx?TypeId=50033&Id=1258&FId=t3:50033:3>`_,
just because I had bought a bunch of them. It's a very small ESP32
board, with 40 pins broken out, and support for charging a lithium
battery from USB.
.. note::
The v1.3 is a ``d1_mini32`` board as far as the `esp32-arduino
<https://github.com/espressif/arduino-esp32>`_ libraries are
concerned, v1.4 is a ``esp32wrover``. There are probably other
variants, so you may need to adjust something if you want to
replicate my build
First step was figuring out how to connect the sensors to the
controller. The LED in the QRD1114 can easily work with about 20mA,
so I can drive one directly from the ESP32, whose GPIO pins can drive
up to about 30mA. 25 sensors fit nicely in a 5×5 matrix, so something
like this should work:
.. image:: lego-piano-scanner-schematic.svg
:width: 100%
:alt: schematic drawing of an electronic circuit; four QRD1114 are
arranged in a 2×2 grid; a pin labelled "row1" is connected to
2 LED anodes via a 330Ω resistor and to 2 phototransistor
collectors via a 10kΩ resistor; a pin labelled "row2" is
similarly connected to the other pair; a pin labelled "col1"
is connected to the LED cathode and phototransistor emitter
of two sensors, one per "row"; a pin labelled "col2" is
similarly connected to the other pair; between each 10kΩ
resistor and the phototransistor collectors are pins labelled
"sense1" and "sense2"
Pulling a "row" pin high (with the others low) and a "col" pin low
(with the others in high-impedance / tristate), we can turn on one
sensor at a time, so we can scan the matrix. When the phototransistor
sense light reflecting from an object in front of it, it will pull
down the "sense" pin, which we can read via the ADC in the controller.
Testing that design on a breadboard proved that it can work!
.. image:: sensor-matrix-test.jpg
:width: 100%
:alt: breadboard with a ESP32 dev board, 4 QRD1114 sensors, four
resistors, and a bunch of wires. One QRD1114's LED is shining
a weak pink, the others are off
How are we going to keep the sensors in the right place inside the
piano, though? We're going to print a support that's compatible with
Lego pieces!
.. image:: sensor-mount-test.jpg
:width: 100%
:alt: white plastic 3D-printed rectangle, 6×2 Lego studs in size,
with standard-sized studs only along one long side; 5 QRD1114
sensors occupy one long side, and 2 Lego 1×1 plates with C
clip are on the other side
My printer (a `Prusa i3 MK3S
<https://shop.prusa3d.com/en/3d-printers/180-original-prusa-i3-mk3s-kit.html>`_)
can print holes of the right size for standard vias, and can print the
whole sensor support in one go (29 studs long). The sensors need to be
aligned with the hammers (there's no space behind the stems of the
keys, also the hammers are white so more visible to the
phototransistor, and move more, so it's less probable we'll have false
positives). I used `OpenSCAD <http://www.openscad.org/>`_ to `model
the support
<https://www.thenautilus.net/cgit/lego-piano/tree/3d-print/qrd-holder.scad>`_.
.. image:: sensor-mount-size-check.jpg
:width: 100%
:alt: white plastic 3D-printer rectangle, 29×2 Lego studs in size,
with 25 groups of 4 small holes to hold the sensors, placed
on top of the "strings" on the piano, showing how the holes
for the sensors align with the hammers
We also must check positioning on the other two axes, of course.
.. image:: sensor-position-test-1.jpg
:width: 100%
:alt: one QRD1114 mounted on the 3D-printer plastic holder,
suspended below one of the "strings" with a Lego clip piece;
the sensor sits just above and behind the rightmost hammer on
the keyboard; the hammer is in its resting position
.. image:: sensor-position-test-2.jpg
:width: 100%
:alt: same pieces as before; the hammer is in its raised position,
precisely in front of the sensor
At this point someone must be asking: how are we going to solder those
sensors on a *plastic* board? And the answer is, we're not going to! I
decided to go with `wire-wrapping
<https://en.wikipedia.org/wiki/Wire_wrap>`_!
.. image:: sensors-mount-wiring.jpg
:width: 100%
:alt: reverse side of the 29×2 rectangle, with 10 sensors inserted
into their holes from the front, and many thin electrical
wires, insulated in yellow plastic, connecting some of their
pins
The sensors have 4 pins, numbered counter-clockwise looking at the LED
/ phototransistor faces:
1. phototransistor collector, to connect to pull-up resistor and to
"row" pin
2. phototransistor emitter, to connect to "column" pin
3. LED anode, to connect to current-limiter resistor and to "row" pin
4. LED cathode, to connect to "column" pin
so the whole wiring looks like this::
aM aN aO aP aQ cM cN cO cP cQ
14 14 14 14 14 14 14 14 14 14 …
23 23 23 23 23 23 23 23 23 23 …
Mb Nb Ob Pb Qb Md Nd Od Pd Qd
where ``a``, ``c`` go to the pull-up resistors; ``b``, ``d`` go to the
limiter resistors, and ``M``, ``N``, ``O``, ``P``, ``Q`` go to the
column pins.
The controller board is built in a similar way: 3D-printed and
wire-wrapped (see `the model
<https://www.thenautilus.net/cgit/lego-piano/tree/3d-print/controller.scad>`_).
.. image:: board.jpg
:width: 100%
:alt: white plastic 3D-printed rectangle, with two groups of 10x2
holes around a label "esp", two groups of 5×2 holes around
labels "220Ω" and "10kΩ", three groups of 5 holes beside
labels "led", "gnd", "pht", and one group of 9 holes beside a
label "amp"; there is a notch in the rectangle near the "esp"
label, and a small raised block near the "amp" label
The notch in the board corresponds to the battery connector, and the
raised block is to hold up the small `AdaFruit audio amplifier
<https://learn.adafruit.com/stereo-3-7w-class-d-audio-amplifier/inputs-and-outputs>`_
.. image:: board-populated.jpg
:width: 100%
:alt: same board, with all the components placed on it. The "led",
"gnd" and "pht" holes hold connectors
.. image:: board-wired.jpg
:width: 100%
:alt: reverse of the populated board, many electrical wires
insulated in yellow plastic connect various pins
.. image:: board-full.jpg
:width: 100%
:alt: same populated board, with a small naked speaker connected to
the amplifier
I had some problems with the row / column connections, because not all the
GPIO pins can actually be used. After some trial and error, I settled
on:
- row pins: 05 23 19 18 26
- colums pins: 17 33 16 21 22
- ADC pins: 02 04 12 27 14
- DAC pin: 25
- amp enable: 32
The program was a bit fiddly to get right, but not particularly
complicated, `you can see it in my Git repository
<https://www.thenautilus.net/cgit/lego-piano/tree/esp32/lego-piano.ino>`_.
.. note::
The TTGO board gave me some problems with uploading the compiled
image, with errors like ``A fatal error occurred: Timed out waiting
for packet content`` or ``Invalid head of packet (0xE0)``. To fix
those, I had to `set the upload speed by hand
<https://www.thenautilus.net/cgit/lego-piano/tree/esp32/Makefile>`_
in the ``Makefile``.
First test with a few sensors on a breadboard:
.. video:: sound-test-1.mp4
:width: 100%
:type: video/mp4
and with all the sensors, mounted inside the piano:
.. video:: sound-test-2.mp4
:width: 100%
:type: video/mp4
I removed the Lego electronics to make space for the wires and the
controller board.
And, finally, the whole assembled set:
.. video:: sound-test-3.mp4
:width: 100%
:type: video/mp4
At the moment the program can't really deal with more than one key
pressed at a time, as you may have noticed at the end of that last
video. The next step is to use a soundfont library, probably
`TinySoundFount <https://github.com/schellingb/TinySoundFont>`_ or `the
ESP-optimised version
<https://github.com/earlephilhower/ESP8266Audio/tree/master/src/libtinysoundfont>`_,
which should have no problems mixing multiple notes.
