How many DevOps Engineers does one need to hack a light bulb?
This question was answered at CLOUD&HEAT during our last in-house hackathon. This is an event that happens multiple times per year usually on a Friday.
At this day, we focus on learning new things or implementing tools that can help in our daily business. Every employee is invited to participate and to propose projects or things they are interested in. There is no limitation about the topics, even if Hackathon sounds a bit like hacking, which most people associate with IT stuff.
During our last hackathon in 2018, one of our employees started working on a so-called Opsi Light.
1. What we call the Opsi Light
An Opsi Light is meant to be a device that informs our colleagues from operations that there is some action required: For example a newly created ticket from a customer or an alarm informing that something in our data centers is not as it should be.
Like any good infrastructure operator, we have monitoring and alerting in place for our data centers. Normally, there is an acoustic alert to notify our on-duty folks when something is going really wrong. However, during office hours, the acoustic alert can be extremely distracting since we are working in an open plan office. So the idea was to have an optical notification device that does not disturb that much and could be placed in sight of the opsi on duty.
A lot of us are using a Raspberry Pi for their private hacking projects, so the obvious choice for controlling the opsi light was one of these small single-board computers. The advantages are that it runs linux, is very cheap, comes in a small form factor and has General Purpose Input/Output-Pins (GPIOs) that could be used to connect other devices like – as in our case – a LED over a matching resistor (more on that later).
For determining if there is something to display, we used a python script that queries our internal services for updates using a filter. After categorization of the events, a specific signal is generated and forwarded to a GPIO Pin of the Raspberry Pi that had a LED connected. Types of the signal could be:
- Flashing: Something went wrong during query of the services (network issue, malformed response)
- Blinking: Something is wrong in one of the data centers, immediate response required
- Solid light: At least one new customer ticket has been created, get back to customer asap
Whereas the more important signals override the less important ones (for example, if the light is blinking there can also be unprocessed customer tickets). The Opsi Light holds this state until the person on duty picks up to investigate on it.
It turns out that this mechanism is approximately 30s faster than our current “Opsi-phone” since alerts going through another specific filter to the phone. The downside is that you get some false positives using this method. So a task for the next hackathon will be to rework the application to use a push mechanism rather than pulling.
Anyway, 30s are already quite enough to have a first investigation and silence the phone to limit disturbance in our office. But a small LED doesn’t cut it to get your attention. And let’s be honest, it’s not as fun as a bright flashing light appliance.
2.1. The Hardware
So one of our engineers picked up an LED-based alarm light from a retailer which is driven using a 12V DC adapter. Unfortunately, the lamp had a circuit integrated which made it flash briefly at a fixed interval (probably on the order of 10ms flashes every second or so). From the outside, one can see that it consists of 15 LEDs covered by a red cap and two wires for power supply.
2.2. The Team
Quickly after the Hackathon started, a team of three DevOps Engineers found together to continue the development of the Opsi Light. Two of them already had electronics and hardware background, while the third was eager to learn more about those topics. So one goal of this exercise was to understand how the original light works and how to modify it, with minimal effort and reusing as much of the existing parts as possible, to interface with the Raspberry Pi; ideally as a drop-in replacement for the existing LED so that no code changes are required.
Luckily, we are in the fortunate position of having lots of electronics equipment at hand, since CLOUD&HEAT does in-house prototyping and development of some of the hardware used in our solutions. This gave us the hardware tools needed to work on this project.
The first step in this project was to analyse the existing hardware further to understand how it works and how it can be modified to suit our needs. Remember that the existing Opsi Light software essentially sets a pin on the Raspberry Pi to 3.3V when the lamp is supposed to be on and to 0V when the lamp is supposed to be off. We cannot source large amounts of current from those pins, nor can we have 12V on them (that’d fry the Pi quickly and painfully).
3.1. Reverse Engineering a Schematic
To analyse the lamp, we took it apart. We discovered a round printed circuit board (PCB) where the 15 LEDs are soldered, along with two electrolytic capacitors on the top side and a bunch of surface mount devices (SMD) on the other side. We are still not sure if the PCB is a single-layer or a dual-layer PCB; the top side is fully coated, so we can’t be fully sure if there are traces below it. There were vias (holes in the PCB which may connect top and bottom side) in it, however, we were able to reconstruct a plausible circuit from the bottom side alone.
Looking at the bottom side, the components we found were exclusively resistors and two identical things in SOT-23 package. Googling the label on the SOT-23 devices (“J3Y”) revealed that they are likely S8050 NPN bipolar transistors. By manually tracing the traces on the PCB, we eventually came up with the following schematic for the original device:
We took the liberty to organize the original schematic we came up with a litte and separate it in regions for instructional purposes. We’ll now interpret the regions, one at a time.
3.2.1. LED circuit
Let us go through it from top to bottom. The LEDs are wired in a series-parallel topology. Three of them are connected in series, protected by a resistor of 33 Ohm (we’ll get to that later) and those series are connected in parallel with their anode to 12V and the cathode to the collector of Q1 (which forms a low-side switch, essentially turning the LEDs off or on by (dis-)connecting them from GND – more on that later, too).
Now, LEDs are not classical light bulbs. In fact, they are non-linear devices like normal diodes or transistors, which can make them a bit unituitive to reason about. We’ll describe how they work in a simplified manner; for more details, we kindly redirect you to the wikipedia article about them.
Normal light bulbs can be modelled as resistors (in fact, they are mostly resistors which are just heated up by current enough to emit light). LEDs can not. We’ll not go into the details of their inner physical workings now (please see the wikipedia article for that), but what you need to know about them is that they have a so called Threshold Voltage. The threshold voltage is determined by the material chosen to build the actual light-emitting part, and is via that tightly coupled to the colour emitted by the LED. For example, red LEDs have a threshold voltage around 1.8V.
The threshold voltage is also the voltage at which the LED starts to conduct and emit light. As mentioned, LEDs work like normal diodes, which means that they only conduct if a voltage above a certain point is given. And even when they conduct, they drop that fixed amount of voltage across them (some people even use this property of diodes to build cheap voltage regulators…). If you try to apply more voltage than its threshold voltage to a LED without any resistor whatsoever, it will fail fatally very quickly, and here’s why.
LEDs are sensitive to the amount of current which passes through them. Standard bulk LEDs typically prefer currents around or below 5 mA, sometimes up to 20 mA. Higher current means more heat and reduced lifetime of the LED. Now the problem is, in contrast to resistors, the current is not a linear function of the voltage (you may recall Ohm’s Law for resistors, I = U / R, with U being the voltage, R being the resistance of the resistor and I the current). Instead, once you go beyond the threshold voltage, current increases roughly exponentially with the voltage.
Let’s take a concrete example of a 1.9V red LED which goes up to 7cd of light (datasheet). On the third page, you find the graph which plots the forward current over the forward voltage. Applying 2V thus would make the LED accept 30mA of current (which is its Absolute Maximum Rating, so don’t do that). Now let’s imagine you connect that LED to a 3.7V Li-Po cell. You can’t even find the current for that in the graph, because it is literally off-the-charts. A quick back-of-the-envelope approximation suggests that, assuming exponential increase of accepted current, the LED would be able to pass whopping 30 A. For a very, very short time. Expect smoke and possibly flames. You can think of it as if the “resistance” of LEDs became lower the more voltage you apply.
Note that there’s another peculiarity: The “resistance” of LEDs also falls with rising temperature. So even if you connect your LED to a 1.9V source (assuming you checked that the actual specific LED you’re using has 1.9V as reasonable forward/threshold voltage), continuous operation is still dangerous to the LED: it will heat up, conduct more current, heat up further (and faster, because more current), conduct even more current and so on until either your voltage source refuses to deliver more current or the LED disappears into a puff of smoke.
This is why typically, you have a current-limiting device in series with LEDs to prevent the current from getting out of hand. The rules of electronics say that for devices connected in series, the current passing through each of them must be the same. So if you have a resistor in series with an LED, the current passing through the LED will effectively be limited by the current passing through the resistor (as the relationship of voltage<->current is linear with resistors, it will limit the exponential growth of the current the LED would accept).
Knowing the threshold voltage of the LEDs (approx. 1.9V (measured)) and the input voltage (12V) and the fact that the LEDs are effectively connected to ground on the other side, we can now calculate the voltage across the resistor and, following that, the current passing through the LEDs. For this, we look at an LED series in isolation:
We have 12V distributed across 3 LEDs, a resistor with 33 Ohm and the transistor; we can assume that the transistor induces a voltage drop of approximately 0.7V. So there remain 11.3V for the LED-R chain. The LEDs have a threshold voltage of 1.9V each (and since the current will be limited, this will also be approximately the voltage we can expect them to drop), so in total 5.7V across the three LEDs. This leaves 5.6V for the resistor. Using Ohm’s law from above, we calculate 5.6V / 33Ohm = ~170 mA. 170 mA is quite a lot of current for an LED, but since the light is flashing in the default configuration, the average current over time is still low, which makes this an okay-ish thing to do.
Another interesting number to look at is the power dissipation of the resistor. There is Joule’s law which states P = U×I. Now that we know both U and I, we can calculate the heat dissipated by the resistor: 0.95W! This is quite a lot, of course, but again, the light is pulsed with a very short duty cycle so that there is not much to worry about. We could go on and calculate the actual effective power dissipated on the resistor, but we won’t go down this route.
There is an open question as to why the designers chose to put 3 LEDs in series instead of 5 (15 is out of question since the summed threshold voltage would’ve been 27V, way too much for the 12V supply). One possible reason is that the designers wanted to allow lower-voltage supplies (e.g. 9V) for lower brightness. A design with 5 LEDs in series would, however, have had a lower power dissipation across the resistor (which would’ve been dimensioned to around 10 Ohms, leading to 0.32W).
- Five chains of three LEDs are in parallel
- Each chain has a resistor which limits the current through the LEDs to ~170mA
- Without current limiting, the LEDs would burn out very quickly
3.2.2. Low-side switch
The LED circuit is connected to the collector of Q1. Just looking at Q1 with R5 and the LED circuit, we are looking at a so-called low-side switch.
Q1 is an NPN bipolar transistor. That means it allows current to flow from its collector (pin 3) to the emitter (pin 2) proportional to the current which is flowing from base (pin 1) to the emitter. Current can only flow from the collector to the emitter and from the base to the emitter, not in any other (or reverse) direction, as transistors are based on PN junctions just like diodes.
In this rather simple application, we can reduce the complexity of the bipolar transistor to a simple current-controlled switch. By sending a current down the base, we allow a current to flow into the collector.
This is really all that happens in this part of the circuit: when a current is on the base, the LEDs are allowed to emit light, because the circuit is now closed. When there’s no current on the base, the circuit is open and the LEDs are turned off.
One possibly non-intuitive thing about this part of the circuit might be that many have been taught in school to put the switch before the electronics which are switched (from the point of view of the positive voltage). However, in this circuit, the switch is “behind” the switched electronics.
There are two fundamental ways for switching: high-side (between the power supply positive rail and the load) and low-side (between the load and the power supply negative rail). Both have advantages and disadvantages. Low-side is easier to switch with logic-level circuits no matter what the input voltage is, because one only needs to go a slight bit above GND to turn the switch on (high-side needs to go close to the power supply to turn off). Low-side has the downside that when off, the switched circuit is essentially at the voltage of the positive rail, which may have adverse effects if it has outside connections, and also it pushes the low-side of the circuit slightly above ground (by the amount of the forward voltage of the transistor) even when turned on.
For LEDs, it does not matter much on which side one switches. In this case, we’re lucky that a low-side switch is deployed, because we can re-use it in our hack later on.
- In the off state, the NPN does not conduct; it is “open” (like an open switch). The LED circuit is not connected to GND and thus does not light up.
- In the on state, the NPN does conduct; it is “closed” (like a closed switch) and the LED circuit is connected to GND.
3.2.3. Astable multivibrator
This is the most complex part of the circuit. It is the part which is responsible for producing the perodic flashing the lamp does when connected to a power source. This is where simply looking at the steady state like we did above with the LEDs isn’t going to cut it and we’ll have to look at the transient evolution of the system to understand what is going on.
Again, we’ll refer you to Wikipedia for the in-depth details, since we can’t really go into the electronic depths of oscillating circuits and feedback networks here.
Essentially, it works like this. The capacitors C1 and C2 act, together with their associated resistors, as delay elements. They control the time rhythm in which the original lamp blinks. You can look up the details in the linked Wikipedia article.
So the circuit consists of a block of LEDs including pre-resistors to operate with 200 mA at 12 V. The low-side of those LEDs is connected to a NPN bipolar transistor acting as low-side switch which is part of an astable multivibrator responsible for the blinking effect.
In this section, we’ll determine the requirements and parameters for driving the LED lamp with completely customized blinking patterns. We will then compare the requirements with the hardware we analysed in the previous section and figure out and execute a way to make the hardware match the requirements.
4.1. Parameters and Requirements
The Raspberry Pi has fully controllable General Purpose Input/Output (GPIO) pins. We can set them to low (driving 0V and possibly sinking current), high (driving 3.3V and sourcing current), and high-impedance (essentially “disconnected”). The existing simple LED uses high and low states for on and off (since the LED is directly connected to the pin).
The GPIO pins of the Pi are limited to 16 mA of current per pin; trying to source more current than that from the pin may cause permanent damage to the SoC on the Raspberry Pi itself. In addition, we must not apply more than 3.3V to the GPIO pins in any fashion. This is especially important since we’re working with 12V on the LED lamp (and we can’t go much below 6V due to how the LEDs are put in series, as discussed above) and because the Raspberry Pi itself exposes a blank 5V You Must Not Connect To 3.3V.
To summarize, the signal we can send from the Pi is at 0V/3.3V and up to 16 mA.
4.2.1. LED resistors
As we discussed above, the LEDs are driven with 170 mA. 170 mA is most likely too much for the continuous operation of the lamp, so we had to reduce the current there. For that, we replaced the resistors R2, R3, R4, R8 and R9 with 330 Ohm resistors, effectively reducing the current by a factor of ten.
We first tried to do this ourselves, but quickly gave up trying to de-solder and re-solder SMD resistors. Luckily, it was easy enough to find an enthusiastic hardware team member who could fix that up within minutes for us.
4.2.2. Injection of the Control Signal
We have a control signal with 3.3V and 16mA max. We know that the PCB has a low-side switch behind the LED block, so essentially, all we have to do is to remove the other half of the astable multivibrator, attach a cable to the base of Q1 and connect it to the control output of the Pi. The modified schematic looks like this:
The resistor is required to limit the current which is sourced by the Pi into Q1. The current needs to be limited to protect the Raspberry which can source at most 16mA and to protect the transistor whose base-collector junction behaves similarly to an LED in that the current grows exponentially for applied voltages beyond the forward voltage. At the same time it needs to be large enough to saturate the transistor to drive the collector-emitter resistance low, such that only little power is dissipated in the transistor due to the comparatively large switched current. We decided to limit the current to approximately 5 mA which is safe enough for the Raspberry and large enough to efficiently switch the 100mA to drive the LEDs. The resistance required to limit the current can be calculated by taking the voltage between the base and the pin into account. The voltage between the pin and the base of Q1 is 3.3V minus the approximately 0.6V forward voltage drop of the base-emitter junction. So we end up with 2.6V / 5mA = 520 Ohm. Rounding up to the next available resistor we come up with 560 Ohm.
To implement these changes we simply stripped off R1, R6, R7 and Q2, as well as C1 and C2. Then we attached a wire to the base of Q1 and soldered a resistor at the end of it and added a connector compatible with the Raspberry Pi pins. After reassembling the case we ended up with a bright and shiny drop-in replacement for the single LED Opsi Light.
During the last hackathon, we managed to build on top of a project from the previous run where we created a visual alert mechansim for our operations team during office hours. The previous system used a single small LED which is simply not powerful (and fun) enough to alert folks.
In this run, we replaced the single LED with a store-bought LED-based lamp with 15 high-power red LEDs. We modified the lamp in such a way that it can be used as a drop-in replacement for the existing small single LED. The lamp uses a dedicated 12V DC supply whose ground is connected with the ground of the Raspberry Pi. Controlling happens via the transistor which was already found inside the lamp hardware.
The project was possible because we have our own hardware shop at CLOUD&HEAT, as well as skilled hardware people who were able to help the DevOps Engineers with the more tricky tasks.
Finally, we are able to answer the original question: With three DevOps Engineers, we were able to hack the light bulb to do what we want. 😉