Making the LEGO Grand Piano play music

Other languages

The LEGO Grand Piano 21323 is a great model, full of clever details and impressive engineering.

It contains a bit of electronics, that allow it to talk with a phone app via Bluetooth.

a Lego optical sensor, motor, power&control brick, mounted on
a wide base of sideways pieces; the motor moves a long axle
with many pegs at right angles to each other

The app can play a few built-in tunes while running the motor inside the piano, which pulls some keys down in a regular pattern, so you can pretend to have a "player piano". The app can also wait for you to press any key on the piano (they all move a little flag in front of the optical sensor) before playing the next note, so you can pretend to be playing.

That's fun, but not really enough: there are 25 independent keys, surely we can make them play the right note!

the full keyboard of the Lego piano, 2 octaves + 1, with
correctly-alternating white and black keys, each key has a
matching hammer

After some research, I decided to use optical sensors and a big microcontroller. How hard can it be? ☺

As sensor I picked the QRD1114, which is dead easy to use: infrared LED plus phototransistor, with a very narrow and short area of sensitivity, so there's very little chance of nearby keys setting off the wrong sensor.

The controller I used is a Lilygo TTGO-T7 v1.3, just because I had bought a bunch of them. It's a very small ESP32 board, with 40 pins broken out, and support for charging a lithium battery from USB.

Note

The v1.3 is a d1_mini32 board as far as the esp32-arduino libraries are concerned, v1.4 is a esp32wrover. There are probably other variants, so you may need to adjust something if you want to replicate my build

First step was figuring out how to connect the sensors to the controller. The LED in the QRD1114 can easily work with about 20mA, so I can drive one directly from the ESP32, whose GPIO pins can drive up to about 30mA. 25 sensors fit nicely in a 5×5 matrix, so something like this should work:

schematic drawing of an electronic circuit; four QRD1114 are
arranged in a 2×2 grid; a pin labelled "row1" is connected to
2 LED anodes via a 330Ω resistor and to 2 phototransistor
collectors via a 10kΩ resistor; a pin labelled "row2" is
similarly connected to the other pair; a pin labelled "col1"
is connected to the LED cathode and phototransistor emitter
of two sensors, one per "row"; a pin labelled "col2" is
similarly connected to the other pair; between each 10kΩ
resistor and the phototransistor collectors are pins labelled
"sense1" and "sense2"

Pulling a "row" pin high (with the others low) and a "col" pin low (with the others in high-impedance / tristate), we can turn on one sensor at a time, so we can scan the matrix. When the phototransistor sense light reflecting from an object in front of it, it will pull down the "sense" pin, which we can read via the ADC in the controller.

Testing that design on a breadboard proved that it can work!

breadboard with a ESP32 dev board, 4 QRD1114 sensors, four
resistors, and a bunch of wires. One QRD1114's LED is shining
a weak pink, the others are off

How are we going to keep the sensors in the right place inside the piano, though? We're going to print a support that's compatible with Lego pieces!

white plastic 3D-printed rectangle, 6×2 Lego studs in size,
with standard-sized studs only along one long side; 5 QRD1114
sensors occupy one long side, and 2 Lego 1×1 plates with C
clip are on the other side

My printer (a Prusa i3 MK3S) can print holes of the right size for standard vias, and can print the whole sensor support in one go (29 studs long). The sensors need to be aligned with the hammers (there's no space behind the stems of the keys, also the hammers are white so more visible to the phototransistor, and move more, so it's less probable we'll have false positives). I used OpenSCAD to model the support.

white plastic 3D-printer rectangle, 29×2 Lego studs in size,
with 25 groups of 4 small holes to hold the sensors, placed
on top of the "strings" on the piano, showing how the holes
for the sensors align with the hammers

We also must check positioning on the other two axes, of course.

one QRD1114 mounted on the 3D-printer plastic holder,
suspended below one of the "strings" with a Lego clip piece;
the sensor sits just above and behind the rightmost hammer on
the keyboard; the hammer is in its resting position same pieces as before; the hammer is in its raised position,
precisely in front of the sensor

At this point someone must be asking: how are we going to solder those sensors on a plastic board? And the answer is, we're not going to! I decided to go with wire-wrapping!

reverse side of the 29×2 rectangle, with 10 sensors inserted
into their holes from the front, and many thin electrical
wires, insulated in yellow plastic, connecting some of their
pins

The sensors have 4 pins, numbered counter-clockwise looking at the LED / phototransistor faces:

  1. phototransistor collector, to connect to pull-up resistor and to "row" pin
  2. phototransistor emitter, to connect to "column" pin
  3. LED anode, to connect to current-limiter resistor and to "row" pin
  4. LED cathode, to connect to "column" pin

so the whole wiring looks like this:

aM aN aO aP aQ cM cN cO cP cQ
14 14 14 14 14 14 14 14 14 14 …
23 23 23 23 23 23 23 23 23 23 …
Mb Nb Ob Pb Qb Md Nd Od Pd Qd

where a, c go to the pull-up resistors; b, d go to the limiter resistors, and M, N, O, P, Q go to the column pins.

The controller board is built in a similar way: 3D-printed and wire-wrapped (see the model).

white plastic 3D-printed rectangle, with two groups of 10x2
holes around a label "esp", two groups of 5×2 holes around
labels "220Ω" and "10kΩ", three groups of 5 holes beside
labels "led", "gnd", "pht", and one group of 9 holes beside a
label "amp"; there is a notch in the rectangle near the "esp"
label, and a small raised block near the "amp" label

The notch in the board corresponds to the battery connector, and the raised block is to hold up the small AdaFruit audio amplifier

same board, with all the components placed on it. The "led",
"gnd" and "pht" holes hold connectors reverse of the populated board, many electrical wires
insulated in yellow plastic connect various pins same populated board, with a small naked speaker connected to
the amplifier

I had some problems with the row / column connections, because not all the GPIO pins can actually be used. After some trial and error, I settled on:

  • row pins: 05 23 19 18 26
  • colums pins: 17 33 16 21 22
  • ADC pins: 02 04 12 27 14
  • DAC pin: 25
  • amp enable: 32

The program was a bit fiddly to get right, but not particularly complicated, you can see it in my Git repository.

Note

The TTGO board gave me some problems with uploading the compiled image, with errors like A fatal error occurred: Timed out waiting for packet content or Invalid head of packet (0xE0). To fix those, I had to set the upload speed by hand in the Makefile.

First test with a few sensors on a breadboard:

and with all the sensors, mounted inside the piano:

I removed the Lego electronics to make space for the wires and the controller board.

And, finally, the whole assembled set:

That's great, but it sounds nothing like a piano. It took about two days of experimentation, but I finally managed to get the ESP32 to use a soundfont, via the TinySoundFount library (actually the ESP-optimised version):

DatesCreated: 2020-08-28 10:39:28 Last modification: 2020-08-29 09:16:54