====================================== Making the LEGO Grand Piano play music ====================================== :CreationDate: 2020-08-28 10:39:28 :Id: modelli/lego-piano :tags: - 3dPrint - hardware - software - models The `LEGO Grand Piano 21323 `_ is a great model, full of clever details and impressive engineering. It contains a bit of electronics, that allow it to talk with a phone app via Bluetooth. .. image:: lego-electronics.jpg :width: 100% :alt: a Lego optical sensor, motor, power&control brick, mounted on a wide base of sideways pieces; the motor moves a long axle with many pegs at right angles to each other The app can play a few built-in tunes while running the motor inside the piano, which pulls some keys down in a regular pattern, so you can pretend to have a "player piano". The app can also wait for you to press any key on the piano (they all move a little flag in front of the optical sensor) before playing the next note, so you can pretend to be playing. That's fun, but not really enough: there are 25 independent keys, surely we can make them play the right note! .. image:: keys.jpg :width: 100% :alt: the full keyboard of the Lego piano, 2 octaves + 1, with correctly-alternating white and black keys, each key has a matching hammer .. video:: keys-hammers.mp4 :width: 100% :type: video/mp4 After some research, I decided to use optical sensors and a big microcontroller. How hard can it be? ☺ As sensor I picked the `QRD1114 `_, which is dead easy to use: infrared LED plus phototransistor, with a very narrow and short area of sensitivity, so there's very little chance of nearby keys setting off the wrong sensor. The controller I used is a `Lilygo TTGO-T7 v1.3 `_, just because I had bought a bunch of them. It's a very small ESP32 board, with 40 pins broken out, and support for charging a lithium battery from USB. .. note:: The v1.3 is a ``d1_mini32`` board as far as the `esp32-arduino `_ libraries are concerned, v1.4 is a ``esp32wrover``. There are probably other variants, so you may need to adjust something if you want to replicate my build First step was figuring out how to connect the sensors to the controller. The LED in the QRD1114 can easily work with about 20mA, so I can drive one directly from the ESP32, whose GPIO pins can drive up to about 30mA. 25 sensors fit nicely in a 5×5 matrix, so something like this should work: .. image:: lego-piano-scanner-schematic.svg :width: 100% :alt: schematic drawing of an electronic circuit; four QRD1114 are arranged in a 2×2 grid; a pin labelled "row1" is connected to 2 LED anodes via a 330Ω resistor and to 2 phototransistor collectors via a 10kΩ resistor; a pin labelled "row2" is similarly connected to the other pair; a pin labelled "col1" is connected to the LED cathode and phototransistor emitter of two sensors, one per "row"; a pin labelled "col2" is similarly connected to the other pair; between each 10kΩ resistor and the phototransistor collectors are pins labelled "sense1" and "sense2" Pulling a "row" pin high (with the others low) and a "col" pin low (with the others in high-impedance / tristate), we can turn on one sensor at a time, so we can scan the matrix. When the phototransistor sense light reflecting from an object in front of it, it will pull down the "sense" pin, which we can read via the ADC in the controller. Testing that design on a breadboard proved that it can work! .. image:: sensor-matrix-test.jpg :width: 100% :alt: breadboard with a ESP32 dev board, 4 QRD1114 sensors, four resistors, and a bunch of wires. One QRD1114's LED is shining a weak pink, the others are off How are we going to keep the sensors in the right place inside the piano, though? We're going to print a support that's compatible with Lego pieces! .. image:: sensor-mount-test.jpg :width: 100% :alt: white plastic 3D-printed rectangle, 6×2 Lego studs in size, with standard-sized studs only along one long side; 5 QRD1114 sensors occupy one long side, and 2 Lego 1×1 plates with C clip are on the other side My printer (a `Prusa i3 MK3S `_) can print holes of the right size for standard vias, and can print the whole sensor support in one go (29 studs long). The sensors need to be aligned with the hammers (there's no space behind the stems of the keys, also the hammers are white so more visible to the phototransistor, and move more, so it's less probable we'll have false positives). I used `OpenSCAD `_ to `model the support `_. .. image:: sensor-mount-size-check.jpg :width: 100% :alt: white plastic 3D-printer rectangle, 29×2 Lego studs in size, with 25 groups of 4 small holes to hold the sensors, placed on top of the "strings" on the piano, showing how the holes for the sensors align with the hammers We also must check positioning on the other two axes, of course. .. image:: sensor-position-test-1.jpg :width: 100% :alt: one QRD1114 mounted on the 3D-printer plastic holder, suspended below one of the "strings" with a Lego clip piece; the sensor sits just above and behind the rightmost hammer on the keyboard; the hammer is in its resting position .. image:: sensor-position-test-2.jpg :width: 100% :alt: same pieces as before; the hammer is in its raised position, precisely in front of the sensor At this point someone must be asking: how are we going to solder those sensors on a *plastic* board? And the answer is, we're not going to! I decided to go with `wire-wrapping `_! .. image:: sensors-mount-wiring.jpg :width: 100% :alt: reverse side of the 29×2 rectangle, with 10 sensors inserted into their holes from the front, and many thin electrical wires, insulated in yellow plastic, connecting some of their pins The sensors have 4 pins, numbered counter-clockwise looking at the LED / phototransistor faces: 1. phototransistor collector, to connect to pull-up resistor and to "row" pin 2. phototransistor emitter, to connect to "column" pin 3. LED anode, to connect to current-limiter resistor and to "row" pin 4. LED cathode, to connect to "column" pin so the whole wiring looks like this:: aM aN aO aP aQ cM cN cO cP cQ 14 14 14 14 14 14 14 14 14 14 … 23 23 23 23 23 23 23 23 23 23 … Mb Nb Ob Pb Qb Md Nd Od Pd Qd where ``a``, ``c`` go to the pull-up resistors; ``b``, ``d`` go to the limiter resistors, and ``M``, ``N``, ``O``, ``P``, ``Q`` go to the column pins. The controller board is built in a similar way: 3D-printed and wire-wrapped (see `the model `_). .. image:: board.jpg :width: 100% :alt: white plastic 3D-printed rectangle, with two groups of 10x2 holes around a label "esp", two groups of 5×2 holes around labels "220Ω" and "10kΩ", three groups of 5 holes beside labels "led", "gnd", "pht", and one group of 9 holes beside a label "amp"; there is a notch in the rectangle near the "esp" label, and a small raised block near the "amp" label The notch in the board corresponds to the battery connector, and the raised block is to hold up the small `AdaFruit audio amplifier `_ .. image:: board-populated.jpg :width: 100% :alt: same board, with all the components placed on it. The "led", "gnd" and "pht" holes hold connectors .. image:: board-wired.jpg :width: 100% :alt: reverse of the populated board, many electrical wires insulated in yellow plastic connect various pins .. image:: board-full.jpg :width: 100% :alt: same populated board, with a small naked speaker connected to the amplifier I had some problems with the row / column connections, because not all the GPIO pins can actually be used. After some trial and error, I settled on: - row pins: 05 23 19 18 26 - colums pins: 17 33 16 21 22 - ADC pins: 02 04 12 27 14 - DAC pin: 25 - amp enable: 32 The program was a bit fiddly to get right, but not particularly complicated, `you can see it in my Git repository `_. .. note:: The TTGO board gave me some problems with uploading the compiled image, with errors like ``A fatal error occurred: Timed out waiting for packet content`` or ``Invalid head of packet (0xE0)``. To fix those, I had to `set the upload speed by hand `_ in the ``Makefile``. First test with a few sensors on a breadboard: .. video:: sound-test-1.mp4 :width: 100% :type: video/mp4 and with all the sensors, mounted inside the piano: .. video:: sound-test-2.mp4 :width: 100% :type: video/mp4 I removed the Lego electronics to make space for the wires and the controller board. And, finally, the whole assembled set: .. video:: sound-test-3.mp4 :width: 100% :type: video/mp4 That's great, but it sounds nothing like a piano. It took about two days of experimentation, but I finally managed to get the ESP32 to use a soundfont, via the `TinySoundFount library `_ (actually `the ESP-optimised version `_): .. video:: soundfont.mp4 :width: 100% :type: video/mp4