PID

CoffeePID: optimizations

As we saw in the last post (CoffeePID: the build) the project integration was quite successful. Unfortunately the resulting temperature curve is not what I was hoping for. But to be able to fix this issue, we first need to understand the problem a bit better. The target temperature was set to 97°C, so the heater stopped at that limit. So lets start taking a closer look at that first measurement:

Initial heatup of the Gaggia Classic with CoffeePID

Initial heatup of the Gaggia Classic with CoffeePID

So what can we take away from this?

  • It seems that the heater is quite powerful, as it quickly manages reheating the boiler at falling temperatures. This means it should be possible to keep the temperature pretty stable, or at least keep the “underswing” of the temperature at a minimum.

  • The system is very inert. After only 2 seconds of heating the measured temperature is still increasing for more than 5 seconds. This means that we have to be more careful with (re-)heating phases and that it may be a good idea to add some delay after heating, before we take the next measurement and react on it.

  • Fun fact: We can clearly see that the frequency of the oscillation decreases over time, as the heat takes longer to dissipate and therefore increasing the cool down duration. So we basically know the state of the system (all components cold or heated up) by just taking a look at that frequency.

  • The last thing to state is, that the current heating strategy is crap. After reaching a “stable” system the temperature was still swinging between -2°C and +7°C around the target value.

Note: This curve is also exactly what we would expect from the static thermostat originally used. It just switches the heater at a “fixed” temperature. At this point I just digitized the original problem. Yay!

Let’s fix this

At this point I already tried at least 3 different strategies to optimize the temperature regulation and I am somehow sure that I’m not done. But this was the first real promising attempt:

Based on the measured temperature we need to calculate the duration to heat up to the target temperature. This will be a proportional factor, increasing the heating duration proportional to the temperature delta. For example heating double as long to heat up +20°C compared to +10°C. But this part is not enough, as it would never heat at or above the target temperature, but this may be a good idea on a quickly falling temperature. So we need a history of the temperature readings and calculate the incline. If the temperature is rapidly falling we need to heat longer to achieve the same heating effect as if the temperature was raising. We will later only have to find the constants needed to incorporate the formula to describe this system. So the formula simply looks like:

t = A * ΔT + B * a

A and B are the constants and a is the incline of the temperature curve, which is measured over the last 5 values in history. We can even simplify this by directly using the temperature delta as the incline. So the formula reads: the time to heat up to a specific temperature is equal to the sum of a constant times the temperature delta to overcome and a constant times the temperature delta of the last 5 values. It is obvious that B has to be a negative value here, as a temperature decrease (negative delta) should lead to a larger heating duration. I tested this a lot with my Gaggia Classic on my desk close to my computer and I came to these magic values:

A = 620, B = -8600

I’m sorry that those values most likely won’t work for you out of the box, as these are very system specific and timing critical. Even adding or removing Serial.println()-statements in the code will change the timing and therefore you have to optimize these values again. As I said, I spent some time doing this. But the values are actually not that magical, as we need a duration in milli seconds the heater should run. So A is just the time the heater runs to heat up the water 1°C. Whereas B is a more abstract factor which is important for the differential part. The factor can be substantial, e.g. when the current temperature is very close to the target temperature but rapidly falling. The first factor will then be close to zero, so the heating duration has to come from this part. So with my current values, if the temperature fell 0.25°C in the past, we would add 2150 ms to the heating duration. Reading these values the first time in written form, I have to say that it sounds really plausible. And the results were also pretty promising:

Optimized heatup of CoffeePID in my Gaggia Classic

Optimized heatup of CoffeePID in my Gaggia Classic

We can still see some minor oscillation in the curve, but if you check the y scaling, you will see that the temperature oscillation is less than 1°C in total amplitude.

One more thing

After I finished tweaking the values I was quite happy with the result. But that only lasted till the next day, when I made my first coffee. I realized that the values didn’t work as great in a cold-start scenario. The machine initially needed many heating phases to come close the target temperature although it was designed to “shoot for the target” with every heating phase. This is also plausible in retrospective, as the machine itself needs more “energy” on the first start to heat up all internal components and materials. The simple solution was to change the magic values a bit (A = 1000, B = -9000). This reduced the duration to reach a stable target temperature a lot and after around 10 minutes the system was swung in to less than ±0.3°C.

Conclusion

After using my Gaggia Classic with CoffeePID for some time now, I’m still satisfied. The machine obviously still takes about the same time to heat up completely, but it then manages to keep the temperature very stable. Even after pulling a shot of espresso it recovers pretty quickly. From this point of view I would declare this project successfully finished. Yay!

Outlook

It took me some time to write all these findings down and while I was working silently, a tweet and a github pull request reached me. A reader of this blog, built a CoffeePID himself and made quite some additions to the source code. This is so great and it promises us many features that I wanted to add in the future, namely:

  • Over-The-Air updating

  • a “real” PID-control

  • a PID-autotuner

I am currently in the state of checking this pull request and hope to come back to you pretty soon. Thanks for reading till the end, if you did ;) Stay tuned.

Other parts of this series:

CoffeePID: the build

Finally it’s time to put the CoffeePID to work. I have testet the software and hardware outside the Gaggia Classic and am very confident, that the integration will not be that big of a deal. Let’s see how this turns out… But first a little service and security advice:

Be aware, that the machine is internally working with 230V (or 110V, depending, where you are). So working on a powered machine can be very dangerous! This is why you should ALWAYS disconnect the power cable in the back BEFORE opening the machine and start fiddling with its internals.

Placement

The first time I took a closer look inside the machine to plan the component placement I was a bit disappointed, as the first plan to simply screw the SSR to the slots on the backside the machine didn’t work out. Besides that setback it turned out that the placement should be quite easy. I wanted to place the power supply, the WEMOS and the MAX31865-breakout on a mounting plate and put that on the backside of the machine, right behind the water filling funnel. That funnel makes it a bit tricky and is why I placed the components as close to the bottom as possible. Keeping those parts close together is generally a good idea, so we can use shorter cables and reduce the mess a bit. The SSR will be placed to the left side of the machine, this way it is close to the position of the original thermostat, simplifying wiring. There is also a lot of space below the front panel, but putting the relay there would be too much of a hassle, if you need to access the ports. I fixed the SSR using double sided thermal adhesive tape, this way it can dissipate the heat via the machine body.

The PT1000-element

Obviously I had to find a solution to the fact, that the thermal element is a small cylindrical element and the thermostat it replaces has a male M4 screw thread. But there is a simple fix, I just glued the PT1000-element inside a M4 spacer using thermal adhesive. That way the PT1000 is just a simple drop-in replacement to the current boiler thermostat. At least that is what I thought. Unfortunately if you only remove the top plate / water funnel, it is very difficult to screw in the PT1000. I only managed this with the help of my wife and a lot of patience. For the not so patient type it is a good idea to unscrew the boiler from the machine to put the new thermoelement in place.

Wiring

As I already had a look at wiring schematics of the machine, I was confident, that the integration should be not too hard. And as it turns out the modifications are really minimal. Starting at the thermostat, it is initially wired to the main power switch at the front and to the heater (and heating light). As this is now the job of the SSR (switching the heater), I simply connected the thermostat wires to the relay. Now there is only one thing missing: the power supply for the CoffeePID. As I want the CoffeePID to switch on with the machine, I need one wire from the just connected “input-wire” of the relay, which is switched by the main switch in the front and one wire that I can easily grab from the back.

LAST MINUTE CHANGES

During the integration planning I realized one possible issue. I want to be able to connect to the ESP (e.g. via USB serial monitor) after the CoffeePID has been integrated, also when the coffee machine is running. But it was planned to permanently connect the power supply to the power switch, so the ESP would be powered from the power supply and via USB, if connected. It is generally not advised to run a microcontroller board this way, as the power distribution on most boards is not designed for this mode and may damage the board permanently. To prevent this I simply added a connector to the AC side of the power supply, this way I can disconnect the power when connecting to the Wemos via USB. This is how the final build looks like and how I placed it in the Gaggia Classic.

POWER ON

As noted before, the temperature regulation is currently very simple, but I wanted to see the results first before tweaking the code. So I used the serial plotter of the Arduino IDE to create a chart of the initial heat up:

Initial temperature curve of CoffeePID in my Gaggia Classic

I am actually pretty ambivalent about this first graph. But what was I expecting? On the one hand it clearly shows that the build is working as planned. Below the target temperature the ESP switches the relay and therefore the heater on. Above the target temperature the relay is switched off, letting the water cool down. Unfortunately this is a pretty inert system leading to substantial overshoot and a quite large oscillation. Regarding temperature stability this is not what I was looking for, but I already indicated, that this might not be the final implementation. And with this new task of optimizing the temperature regulation, it seems that this will not be the last post in this series.

I hope you enjoyed the journey so far and stay with me on my way to a better coffee experience.

Other parts of this series:

CoffeePID: the web interface

This part was the most interesting one yet. I wanted to create a real simple user interface, so I can see the current temperature and if the machine is currently heating. This could be done writing all from scratch, but to make this more convenient I included Bootstrap 4 and jQuery to the page. As the libraries are also needed in access point mode, I couldn’t load them from a CDN and had to serve them from the ESP. 159 KB for BS4 and 88 KB for jQuery may not sound very much, but already have a very noticeable impact on loading time. More on that later.

The Frontend

As I wanted to keep this really simple, the main page only shows the current temperature of the boiler and an indicator if it is heating. And then of course there is the settings page to set the target temperature and change the wifi settings. I also added some convenience functions for setting up the wifi connection, this way I can see the signal strength of the received wifi networks.

After initial load of the site, the content is updated via ajax calls. Changing the settings or rebooting the ESP is also implemented via additional ajax endpoints.

Although the website is not complex at all, there was quite a some potential to optimize it for the ESP8266. This is mainly due to the poor data transfer rate you get in real life situations, especially compared to current computers and smartphones. But if the web server and -site are implemented correctly the result can be pretty snappy. Which brings me to the next point:

Optimize, Optimize, Optimize

I spent quite some time on this step, as everything should be as fast as possible, so I optimized the web server and website for minimal transfer size and maximum performance. To achieve this I

  • optimized html files using preload, async and defer

  • reduced overall file count (reducing the number of requests) - some people might think that combining files is a bad idea, because this way the files could not be loaded in parallel. But the ESP8266 doesn’t transfer the files over WiFi in parallel; in this case it’s a good idea to reduce the total number of requests to eliminate server processing and latencies from those extra calls.

  • purged all unused css from css-files (using purgecss) - in my case this reduces the size of the Bootstrap library from 159KB to 34KB 🎉

  • minified all js and css files

  • optimized all image files

  • compressed ALL files (assuring all files are transferred gzipped to the client)

  • optimized the critical code path on the server

After all optimizations I ended up at around 135KB transfer size to the client and less than 1 second of initial load time. As all files are cached on the client, all subsequent calls are much faster. Also I want to mention that my background image alone is 62 KB in (compressed) size, so the rest of the site is around 73 KB, which is pretty good.

To make the process of editing the website and prepare it for deployment more comfortable, I added a small shell script (included in the Github repo), which mainly executes all optimization steps on the website. After executing the script, the directory is ready to be copied to the LittleFS.

The ESP can obviously handle much more complex websites, even if the website itself doesn’t fit into the LittleFS area of the flash, you could for example use an SD card breakout to serve your site from a SD card. Also a complex site doesn’t have be slow, it depends a lot on the purpose and content. The optimization steps above can be applied to every website and are especially advised in such a resource constrained environment. But if you don’t need the site to work on access point mode and can assume that the client has an internet connection, you should offload as many resources to “the web” as you can. Fonts, CSS and JS (especially Bootstrap and jQuery) can easily be loaded via CDN and most other assets, like images, should also be loaded from a “real” web server on the internet. Just be sure to check the constraints of your final application before starting to optimize.

But I didn’t “just” optimize the website, I also tested different web servers on the ESP for their performance. That is why I also gave “ESPAsyncWebServer” a try, which works completely asynchronous and should be able to perform a lot better than “ESP8266WebServer” under certain conditions. Unfortunately in our special scenario, it didn’t show any performance improvements at all. This is most likely due to the fact, that there will be (at least most of the time) just one client accessing the web server at a time, so it seems the async implementation can’t really show its potential here. That is why I ended up using the package “ESP8266WebServer”.

The web server

Regarding the implementation of the “ESP8266WebServer” library, there are basically two ways to serve files from LittleFS. You can use sth. like:

This is cool, as it serves all files from the LittleFS-folder “/www/”, sets the cache header and automatically serves a “.gz” version of the requested file if available. So in many cases this is all you need to host a whole website.

But after a lot of testing I decided against the “serveStatic”-way, as it is slightly, but measurable, slower than handling the file serving yourself. This is only logical, as we can optimize the code executed on every request much better to our specific use-case. You can do this by defining handler functions (per route) or even easier, just only define a onNotFound-handler. It will then be called on every unregistered route, so for every file of the website. Our handler then loads the requested file from LittleFS, sets the correct file type and cache headers and serves the file using the “streamFile” functionality.

As we created the website ourself we can take an opportunistic approach in the handler function, therefore we assume all files exist as gzipped version and we can limit the support of file types to the ones used in the project. This way the critical path, meaning the code on the microcontroller that gets executed on every request, can be simplified. Even sorting the file types by frequency of occurrence will have a positive performance impact (although this most likely wont have a noticeable effect).

Finishing touches

Now we have a fast frontend to check our water temperature, but you still have to open a browser and navigate to “http://coffepid.local“. Even if you bookmarked the address, this is not really user friendly. To make this more convenient I added some meta tags to the head of the page and added an app icon, so you can use it as a web app. I don’t have any Android devices around, but on iOS you can easily create a link to a website on your home screen (this is most likely also true for Android, I just can test it). Now I can open the web interface using the app icon on my iPhone or iPad. Cool.

NOT FINISHED AT ALL

After playing around with this once called “sweet solution” of web applications, I reached one limitation. My website was generally based on two files (with additional css and js), a homepage and a settings page. That meant navigating between the sites changes the route in the browser and therefore leaving fullscreen mode of the web application. To overcome this limitation the page had to be served from one file, to implement navigation I used hash routing. It was very interesting to implement this, what is basically a very simple SPA-framework. This is mainly done just using one html file to load all the resources and act as a template and one javascript file managing routing and AJAX calls. I won’t go into the details here, as it would go beyond the scope of this post, but you can take a look into the code over at Github.

This solution works fine, but changing the content of the website is now pretty uncomfortable, as the html content of the site is now dynamically injected as a javascript string (from the file “content.js”). Also for such a minimal web interface using Bootstrap is definitely a bit heavy, but as my CSS skills are quite limited, this was the obvious choice. But if you are a designer / frontend developer and want to help out, feel free to reach out to me. A little help would be really appreciated.

Wrap up

As the hardware setup, “firmware” and frontend are somehow finished, the next step will be the integration into the Gaggia Classic. I will share the details on the build and of course the first results on how the heat up process looks like.

Other parts of this series:

CoffeePID: the software

After I had decided on all the hardware (see here), I was really motivated to get the software part running quickly. I love these kind of projects, as I always learn a lot of new stuff.

But first we need to know how the components are wired together and how they talk to each other. The PT1000 (2-wire) is simply connected via the terminal block to the MAX31865-board, which itself is connected to the ESP8266 using SPI. The SSR can then be controlled by almost any remaining free IO-pin on the ESP. Two little anecdotes regarding this:

  • Somebody taking a look at my final pinout may think: “Why didn’t he just use the SlaveSelect port of the WEMOS D1 mini pro as the ChipSelect port for SPI? This way you can’t use hardware SPI and the wiring is a bit odd.” - I would reply: “Thanks for the hint, but I already tried that. And while SS was connected to CS of the MAX31865, the ESP8266 couldn’t be programmed anymore. This may be related to the fact that the flash of the WEMOS D1 mini pro is itself connected to the ESP8266 via SPI. The funny part is, that this only effected programming, the code was actually running fine using hardware SPI. To fix the programming issue, I simply switched the CS port to another free IO-pin, as I don’t really care for hardware SPI anyway.”

  • Don’t use the port “D4” of the WEMOS to switch the relay. DON’T. Believe me. RealIy really! I actually initially chose that port randomly, but experienced some strange issues during uploading, when the SSR was connected to the ESP. This is simply because D4 is connected to the on-board led, which is switched on multiple times during programming. This meant the SSR was switched multiple times pretty quickly while uploading the firmware, what apparently can lead to problems. I also simply changed to another free port and finally everything is working as intended.

So the setup I ended up with for developing the software looked like this. I’ll redo the wiring later, when I know where and how I want to place the components inside the machine.

Components of CoffeePID (PT1000 bottom left, MAX31865 top left, WEMOS D1 mini pro top right and the Solid-State-Relay bottom right)

Components of CoffeePID (PT1000 bottom left, MAX31865 top left, WEMOS D1 mini pro top right and the Solid-State-Relay bottom right)

But now to the software, really really

First we have to decide on the platform we want to use. I usually use the Arduino environment in most of my ESP8266-projects, as it offers a great ecosystem regarding programming and libraries and has pretty good ESP-support. So there were also libraries for every needed component in this project, like web server, pid, MAX31865, mdns and persistent storage. This will make the coding of those components very lightweight and simple, as you will see later on. Any finally it’s very easy to program the WEMOS D1 using the Arduino IDE.

Before writing any code you need to setup the Arduino environment including all esp8266 stuff (see here). Then you should be able to select your board via the tools menu. And here (in the tools menu) is another quality I like about the ESP8266: You can easily overclock it. It runs on a base clock of 80 MHz, but you can set it to 160 MHz via Tools > CPU Frequency. I run all my ESPs overclocked, for years now, and I never ran into any thermal or power issues. So from my perspective it is totally safe to permanently run these chips on 160 MHz which greatly improves performance.

As it would go beyond the scope of this post, I will not explain every detail of the final code here, but I commented the code quite thoroughly to make it easier to understand what’s happening. The full source code is available at the Github repository, feel free to use and modify it, as you like. Also if you have any issues or questions regarding the code, just create an issue in the repository.

Using the PT1000

Let’s start with the code right at the core functionality: measuring the temperature. As I use a breakout board similar to the Adafruit MAX31865, I can simply use the Adafruit library. And that is quite nice, as Adafruit has always great examples and tutorials (here), so the implementation was a breeze. I just had to install the “Adafruit_MAX31865“ library via the Arduino IDE and the integration of the breakout and measuring the temperature were reduced to just these few lines of code:

Be aware, that there are two different “versions” of the MAX31865 breakout board. The PT100 and the PT1000 version, which only differ in the on-board reference resistor (PT100 -> 430 Ohm, PT1000 -> 4,3 kOhm). I actually had to switch this resistor on my board, which was a bit tricky, as it is a pretty small SMD component.

Integrating the Solid-State-Relay

Next up is switching the relay. As we use a relay that can be triggered with 3,3V TTL-level we just define an output pin and initialize it with “LOW” (meaning the heating being switched off). Then later, depending on the temperature reading, we can switch it on and off as we like.

I actually don’t know yet how I should implement this for best temperature stability / accuracy. Of course it is quite simple to implement a real software PID, but as our controlled variable (switching the SSR) is digital and not analog, I doubt that this will give us an ideal result. It is also a lot of work finding optimal PID parameters preventing a temperature overshoot and reducing the (damped) oscillation around the target value. At this point the firmware simply works as a binary switch, heating below the target temperature and switching off, when it’s reached. I’ll come back to this later, when I can test different heating strategies in real live.

WiFi

Let’s continue with the next part: make it connect to the wifi and start a web server. It was important to me, that the wifi credentials are not hardcoded in any way, so you can change your wifi password without recompiling the software. There is a great project called “WifiManager” (Github repository) that solves the problem for you by implementing a captive portal to enter the wifi credentials. But as the frontend is not very appealing and we want to optimize the process as much as possible, I created my own somehow similar implementation. This is the setup process I came up with:

  1. Check for saved credentials, if there are credentials stored, the ESP will try to connect to the network.

    • On success: the CoffeePID is now reachable in your wifi network.

    • On fail: If no credentials are found or a connect is not possible for more than 20 seconds, the controller starts in access point mode and you can connect to its network.

  2. Now the chip is reachable via any device connected to its network.

The code to connect to a wifi network or create an access point is also pretty straight forward:

The microcontroller is now reachable in the network, but I didn’t want to type the IP address into the browser, so addressing it by name would be great. This can simply be achieved by using mDNS responder (aka: Apple Bonjour) which makes devices announce their name on the network and are therefore discoverable by all other network clients. And again, there is a library for that (named “esp8266_mdns”), reducing the code to:

Great, we now can reach the web server of the ESP8266 at the address “http://coffeepid“ (Depending on the network it may be necessary to append “.local” to the url).

Webserver

As I will go into the details in a later post, let me just show you the basic integration. I used the library “ESP8266WebServer“ (installed via the library manager) to host the website on the microcontroller.

LittleFS

So finally we need to have some kind of internal logic which manages the wifi credentials, being able to persistently store and retrieve them from some kind of memory. The easiest way to store data permanently on a ESP8266 is to use its integrated file system. You can choose between two different filesystems, SPIFFS is the “classic” implementation and generally very lightweight, LittleFS is a pretty new implementation and focused on real directory support and performance. I did a performance comparison and in this scenario LittleFS was much faster for serving the files. And again, using the according Arduino library you can access files with very few code. But how do you write files to the LittleFS area using the Arduino IDE, because we want to write all files needed for the website in this area. There is a plugin for the Arduino IDE which allows you to write all files from a folder named “data” in the project directory (GitHub repository). For security reasons I separated the paths for the configuration file and the web server content, so the web server has no access to the configuration file.

Now we can check for a valid wifi configuration and switch the network mode correspondingly. The controller starts in wifi client mode if there are saved wifi credentials or in access point mode if not. So it’s up to the enduser if you want to use the coffeePID inside your existing wifi network, or if you want it to create its own.

RECAP

So, the ESP8266 is now connecting to the network (or creates its own), can handle local configuration files, measure the temperature and switch the relay. We now “just” need the functionality to set and show the temperature to the user. And as already mentioned, this will be done using a web frontend. But that is definitely a completely different story / post. I will report on my journey of creating and optimizing the frontend soon, so stay tuned…

Other parts of this series: