Rewrite electricity-logger to use an sqlite3 database

2022-06-25 22:17:46 +02:00
parent 458d824dc8
commit 5b09b06bcf
62 changed files with 5937 additions and 3 deletions
--- a/docs/BENCHMARK.md
+++ b/docs/BENCHMARK.md
@@ -0,0 +1,75 @@
+# solar-server Benchmarks
+
+Comparing a NodeJS REST API using ExpressJS and a file based logging solution against a Pistache REST API using an Sqlite3 logging solution. Measurements are made with Firefox, server is a Raspberry Pi 3 with all content stored on a microsd card (slow!). See the conclusion at the end if you want a TLDR;.
+
+## Raspberry Pi Model 3 B Benchmarks
+Iteration #1: Fetching stuff from the database on a day by day basis (1 day = 1 query)
+
+| Operation | NodeJS | Pistache | Difference |
+|-----------|-------:|---------:|-----------:|
+| Day       | 11ms   | 6ms      | -5ms       |
+| Month     | 38ms   | 80ms     | +80ms      |
+| Year      | 2122ms | 3353ms   | +1231ms    |
+
+Iteration #2: Fetching stuff from the database in a single query for all requests
+ Index on DateTimeUtc
+
+| Operation | NodeJS | Pistache | Difference |
+|-----------|-------:|---------:|-----------:|
+| Day       | 12ms   | 16ms     | +4ms       |
+| Month     | 40ms   | 127ms    | +87ms      |
+| Year      | 1281ms | 6143ms   | +4942ms    |
+
+Iteration #3: Tweaks
+ Index on KilowattHour
+ Reindex
+
+**No difference**
+
+Iteration #4: Split DateTimeUtc into separate TEXT columns
+
+| Operation | NodeJS | Pistache | Difference |
+|-----------|-------:|---------:|-----------:|
+| Day       | 10ms   | 18ms     | +8ms       |
+| Month     | 34ms   | 45ms     | +11ms      |
+| Year      | 1151ms | 2110ms   | +959ms     |
+
+## Pentium G5400 Benchmarks
+
+Iteration #5: Hardware change (Raspberry Pi 3+ to Pentium G5400)
+
+| Operation | NodeJS | Pistache | Difference |
+|-----------|-------:|---------:|-----------:|
+| Day       | -      | 3ms      | -          |
+| Month     | -      | 5ms      | -          |
+| Year      | -      | 83ms     | -          |
+
+Iteration #6: Keep a separate summary table
+
+| Operation | NodeJS | Pistache | Difference |
+|-----------|-------:|---------:|-----------:|
+| Day       | -      | 3ms      | -          |
+| Month     | -      | 1ms      | -          |
+| Year      | -      | 4ms      | -          |
+
+Comparison to the raspberry pi running the same software:
+
+| Operation | RPi    | Pentium  | Difference |
+|-----------|-------:|---------:|-----------:|
+| Day       | 14ms   | 3ms      | -11ms      |
+| Month     | 5ms    | 1ms      | -4ms       |
+| Year      | 16ms   | 4ms      | -12ms      |
+
+## Conclusion
+
+Clarification:
+- All platforms run Debian Buster
+- RPi = Raspbery Pi Model 3 B+
+- NodeJS = NodeJS Express.js server serving the solar logs stored as file per day (/var/log/solar/[yyyy]/[mm_dd].txt)
+- Pistache = Pistache server serving the solar logs stored in a SQLite3 database
+
+| Operation | RPi NodeJS | RPi Pistache | Pentium G5400 Pistache |
+|-----------|-----------:|-------------:|-----------------------:|
+| Day       | 11ms       | 14ms         | 3ms                    |
+| Month     | 38ms       | 5ms          | 1ms                    |
+| Year      | 2122ms     | 16ms         | 4ms                    |
--- a/docs/ELECTRICITY_API.md
+++ b/docs/ELECTRICITY_API.md
@@ -0,0 +1,37 @@
+# The electricity-server REST API
+
+This project depends on the `pistache` HTTP framework, see their [website](http://pistache.io/) for more info.
+
+## Endpoints
+All endpoints respond in JSON. Examples can be found in the Examples section. Dates are always in ISO format, meaning they take the shape of `year-month-day` with leading zeroes for month and day. Example: `2019-01-11` is the eleventh day of January, 2019.
+
+- `/day?start=[yyyy-MM-dd]` This tells the server what date to serve all the collected records from the specified day.
+- `/day?start=[yyyy-MM-dd]&stop=[yyyy-MM-dd]` This tells the server what date range to serve a per day summary for.
+
+### Example Response /day?start=2020-01-10
+Will give you as many objects as there are recorded values for the 10th of January, 2020. Ordered from earliest to latest. If there are no records for the given day an empty array is returned.
+```json
+[
+	{
+		"dateTime":1578614401,
+		"totalPowerUse":3282.95,
+		"totalPowerReturn":2790.88,
+		"totalGasUse":3769.23
+	},
+	...
+]
+```
+
+### Example Response /day?start=2020-01-01&stop=2020-01-19
+Will give you a summary for each recorded day in the range 1st of January 2020 to the 19th of January 2020. If no logs exist for the day on the server it will send an object with a valid dateTime for that day but with 0 values for the other fields.
+```json
+[
+	{
+		"dateTime":1578614401,
+		"totalPowerUse": 8.324,
+		"totalPowerReturn": 3.62,
+		"totalGasUse": 11.05
+	},
+	...
+]
+```
--- a/docs/ELECTRICITY_LOGGER.md
+++ b/docs/ELECTRICITY_LOGGER.md
@@ -0,0 +1,24 @@
+## About
+
+This small tool is meant for logging the electricity values from the `Landis Gyr E350` electricity meter. As a bonus it also stores the gas concumption values.
+
+## Example Usage
+
+```bash
+#!/bin/bash
+
+directoryPath="/var/log/electricity/$(date +%Y/)"
+if ! [ -d "$directoryPath" ]; then
+	mkdir -p "$directoryPath"
+fi
+
+outputFile="$directoryPath$(date +%m_%d.txt)"
+if ! [ -f "$outputFile" ]; then
+	touch "$outputFile"
+fi
+
+datestr="$(date +%Y-%m-%dT%T)"
+electricity-logger -c "$datestr" /dev/ttyUSB0 >> "$outputFile"
+```
+
+Use the `--help` switch for more information about the possible arguments `electricity-logger` can take.
--- a/docs/SERVER.md
+++ b/docs/SERVER.md
@@ -0,0 +1,43 @@
+# Serving Data
+
+This project includes to data servers: the solar server and electricity server. Both host the data collected by their respective logger counterparts.
+
+## API Description
+
+[The solar api description](./SOLAR_API.md)
+[The electricity api description](./ELECTRICITY_API.md)
+
+## Reverse Proxy
+
+It is recommended to use a reverse proxy setup to make all servers and content reachable through standard HTTP(S) ports. When using nginx something like this suffices, using the solar server as example:
+```
+server {
+    server_name solar.valkendaal.duckdns.org;
+		location / {
+			  if ($request_method = 'GET') {
+					  add_header 'Access-Control-Allow-Origin' '*';
+            add_header 'Access-Control-Allow-Methods' 'GET';
+				}
+		    proxy_pass http://localhost:3001;
+		}
+}
+```
+
+The additional if statement within the location scope allows fetching of solar resources by any other domain. This is used to display data exposed by the solar server on the website (`public/`).
+
+## Restricting Access
+
+The electricity server exposes sensitive data, hence it shouldn't be accessible to anyone except users of the household. The easy solution here is to verify the requestee's IP address, since anyone making the request from the household itself should share the same external IP address, including the server. Therefore the electricity server checks the requestee's IP address against its own external one (by resolving the given domain parameter argument to an IP address).
+
+For this a little hackery was necessary in the reverse proxy, since it normally makes the request to the electricity server on its own behalf. This would result in all requests originating from 127.0.0.1 (localhost). To solve this the following line was added before the `proxy_pass` directive:
+```
+proxy_set_header X-Real-IP $remote_addr;
+```
+
+The value of the X-Real-IP HTTP header is then used by the electricity server to validate against the domain resolved IP address. If they match it means the request came from the same network as the server.
+
+This obviously is not watertight, but it serves the purpose well enough and avoids having to lay down more complicated authorization infrastructure.
+
+## Launch Parameters
+
+Both servers use TCLAP for their launch parameters. Simply run either executable without any parameters or use the `--help` switch to get all available commands.
--- a/docs/SOLAR_API.md
+++ b/docs/SOLAR_API.md
@@ -0,0 +1,42 @@
+# The solar-server REST API
+
+This project depends on the `pistache` HTTP framework, see their [website](http://pistache.io/) for more info.
+
+## Endpoints
+All endpoints respond in JSON. Examples can be found in the Examples section. Dates are always in ISO format, meaning they take the shape of `year-month-day` with leading zeroes for month and day. Example: `2019-01-11` is the eleventh day of January, 2019.
+
+- `/day?start=[yyyy-MM-dd]` This tells the server what date to serve all the collected records from the specified day.
+- `/day?start=[yyyy-MM-dd]&stop=[yyyy-MM-dd]` This tells the server what date to serve the highest recorded kilowatt hour collected for each day in the specified date range. Stop date is inclusive.
+- `/month?&start=[yyyy-mm]&stop=[yyyy-mm]` This returns the total kilowatthours generated for a given month. The stop month is inclusive.
+
+## Example Response
+All times are always in UTC. For these examples the current date is 25 December, 2019.
+
+### /day?start=2019-02-15
+Will give you as many objects as there are recorded values for the 15th of Febuary, 2019. Ordered from earliest to latest.
+```json
+[
+	{
+		"time": 1550226776,
+		"watt": 1456,
+		"kwh": 14.53
+	},
+	{
+		"time": 1550226834,
+		"watt": 1456,
+		"kwh": 14.54
+	},
+	...
+]
+```
+
+### /month?start=2019-06&stop=2019-06
+Will give you a total kilowatt hours generated for the month July, 2019. The time component is the first day of the month at 00:00 (midnight) UTC. For multiple months the result is ordered earliest to latest.
+```json
+[
+	{
+		"time": 1559347200,
+		"watt": 0,
+		"kwh": 451.64
+	}
+]
--- a/docs/SOLAR_LOGGER.md
+++ b/docs/SOLAR_LOGGER.md
@@ -0,0 +1,22 @@
+# About
+This software targets linux and has been tested on both x86_64 and armv7 hardware.
+
+## solar-logger
+Simple data collecting program for ZeverSolar's Zeverlution Sxxxx "smart" inverters with a network interface (referred to as "combox" by the manufacturer). It collects all the exposed data of the Zeverlution, including current power generation (watts) and today's cummulative power production (kilowatt/hours).
+
+### Strange output
+Zeverlution Smart Inverters currently have a bug that causes leading zeroes in a decimal number to be ignored. Concretely this means that the value 0.01 becomes 0.10, 3.09 becomes 3.9, etcetera. This is causing strange peaks in the logged data where sequences go from 2.80 to 2.90 to 2.10.
+
+Another bug is the inverter turning itself off and back on again throughout the day. This happens if the yield get too low. This will cause the cumulative power to reset to zero (kilowatt per hour).
+
+# Building
+## Dependencies
+### solar-logger
+Dependencies for this program are:
+- `libcurl` which can be installed using your package manager. It has been tested with the `openSSL` flavour, libcurl version 4.
+- `sqlite3` which is sometimes also referred to as `libsqlite3-dev`.
+
+## Building
+The logger program can be build by running the makefile (`make all`) in the project root. It will create a `bin` folder where it puts the newly created binaries. Move these to wherever you keep your binaries or use the `install` make target.
+
+Refer to the `--help` switch to find out more about supported launch parameters.