We’ve started to talk about how Data Visualization in the User Portal can help you develop applications that use SAMI’s diverse data sources with a user in mind. How do we display this data? What do we use to create the charts? Why did we do it that way?
Charts are created with at least 2 values: x and y. In our case, the charts are created against time, so our X value is time. The Y value is the value tracked by the device. Having an array with pairs of values, each sorted by timestamp, allows us to pull data in specific ranges of time. We can chart 10 seconds of data, or 30 seconds, or 1 minute, or more!
That’s what we thought—and it would’ve worked if we didn’t also want a continuous flow of incoming data, and a continuous update of the charts. We have to update the charts every 1 second. If you have incoming data from a WebSocket, you will see a continuous change of the charts every 1 second. We call this live mode.
In the beginning, our proof of concept was to chart data into one line. We had only one flow, one data storage, and one chart. That chart had only one line. Everything was so cool, and it worked like a charm. At this time we were using a charting library called Dygraphs. Our spec worked, and everything was happiness, until, well, someone noticed that Dygraphs charts aren’t THAT pretty. We selected Dygraphs for two main reasons:
- We didn’t know D3.js, or have any experience drawing canvas.
- We needed a library that performed fast, really fast.
No one had said anything about pretty charts. But when this became a requirement, we ditched Dygraphs. Then someone asked for different types of charts: spline, bar, candlestick, etc.
Given these new requirements, and the short period of time we had at that moment, we created a table with the pros and cons of the charting libraries we had found that could fulfill most of our new requirements for the charts. In addition to aesthetics and types of charts, we looked at their ability to handle pan, zoom, mouse interactions, large sets of data, and dynamic updates.
We eventually chose Highcharts. It has many types of charts, configurations, colors, and even a timeline! Everyone was happy again. We proceed to create the proof of concept, and boom, it was working: one device, one data storage, one chart, one line. One was the magic number. Sadly, it wasn’t the real number.
We coded everything to use Highcharts and when we started adding more charts, and charts with more than one line, every bit of happiness disappeared. Highcharts was too slow, way too slow. Even though we were triggering updates in the charts once per second, it sometimes took 5 seconds to see the repaint.
What was going on? Well, we were trying to chart too many points in a really short period of time. We tried to fix it. We even got in touch with the creators of the library, but at the end, it was just too much for the library.
With even less time to deliver this part, we realized that the only solution was to create the charts on our own. So we ditched anything that slowed the process, and only left the main parts that would allow us to paint that many points every second. We ended up learning D3.js and coding our own charts.
The result: speed. It was fast. Everything was happiness again. And we also get to have D3.js as one more skill to use in the future.
D3.js is a great tool. It takes some time to grasp the way it does its stuff, but once you understand it, it’s WAY powerful. We are really happy to know D3, and we can also now add any requirement to the charts that is needed. We are no longer limited by the set of features a library can give us: we just code what we need, and take out what we don’t. This tool is something we really recommend to any person that works in the Web industry.
In a follow-up post, we’ll go deeper into how we move and store all this data.
Top image: NASA