tijmen/home-data-collection-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove outdated documentation

Browse Source
This commit is contained in:
Tijmen van Nesselrooij
2023-09-16 22:58:36 +02:00
parent 8e92490c8f
commit 895aaa3a48
5 changed files with 2 additions and 192 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
README.md
View File

@@ -2,20 +2,14 @@
A mono repository containing all the homebrew collectors and REST APIs running on the local server. See it in action [here](https://valkendaal.duckdns.org).
## Project Directory
|- `.vscode` => A folder with Visual Studio Code IDE specific files, for developing this project </br>
|- `docs` => A folder housing all the documentation files (`*.md`) </br>
|- `include` => A folder housing all the header files used by the source files in `src/` </br>
|- `script` => A folder with handy bash scripts to create and migrate databases </br>
|- `src` => A folder with all the source files of the different projects that require compilation </br>
|- `systemd` => A folder with example systemd service files for the servers </br>
## Miscellaneous
A few benchmarks have been done for the solar-server project, which can be found [here](docs/BENCHMARK.md).
## Dependencies
### Runtime (server)
@@ -23,7 +17,7 @@ A few benchmarks have been done for the solar-server project, which can be found
- .NET 6, [website](https://dotnet.microsoft.com/en-us/download/dotnet/6.0)
- Curl: the multiprotocol file tranfser library, [website](https://curl.haxx.se/libcurl/)
- GNU Make, [website](https://www.gnu.org/software/make/)
- Pistache: an elegant C++ REST framework, [website](http://pistache.io/)
- Libfmt, [website](https://github.com/fmtlib/fmt)
- Spdlog, [website](https://github.com/gabime/spdlog)
- Sqlite3: a small SQL database engine, v3.39.2, [website](https://www.sqlite.org/index.html)
@@ -48,4 +42,4 @@ solar-logger --connection-string "/mnt/data0/log/solarpaneloutput.db" -u "http:/
```bash
electricity-logger -d /dev/ttyUSB0 --connection-string /home/pi/logs/electricity.logs
```
```

75
docs/BENCHMARK.md
View File

@@ -1,75 +0,0 @@
# solar-server Benchmarks
Comparing a NodeJS REST API using ExpressJS and a file based logging solution against a Pistache REST API using an Sqlite3 logging solution. Measurements are made with Firefox, server is a Raspberry Pi 3 with all content stored on a microsd card (slow!). See the conclusion at the end if you want a TLDR;.
## Raspberry Pi Model 3 B Benchmarks
Iteration #1: Fetching stuff from the database on a day by day basis (1 day = 1 query)
| Operation | NodeJS | Pistache | Difference |
|-----------|-------:|---------:|-----------:|
| Day | 11ms | 6ms | -5ms |
| Month | 38ms | 80ms | +80ms |
| Year | 2122ms | 3353ms | +1231ms |
Iteration #2: Fetching stuff from the database in a single query for all requests
+ Index on DateTimeUtc
| Operation | NodeJS | Pistache | Difference |
|-----------|-------:|---------:|-----------:|
| Day | 12ms | 16ms | +4ms |
| Month | 40ms | 127ms | +87ms |
| Year | 1281ms | 6143ms | +4942ms |
Iteration #3: Tweaks
+ Index on KilowattHour
+ Reindex
**No difference**
Iteration #4: Split DateTimeUtc into separate TEXT columns
| Operation | NodeJS | Pistache | Difference |
|-----------|-------:|---------:|-----------:|
| Day | 10ms | 18ms | +8ms |
| Month | 34ms | 45ms | +11ms |
| Year | 1151ms | 2110ms | +959ms |
## Pentium G5400 Benchmarks
Iteration #5: Hardware change (Raspberry Pi 3+ to Pentium G5400)
| Operation | NodeJS | Pistache | Difference |
|-----------|-------:|---------:|-----------:|
| Day | - | 3ms | - |
| Month | - | 5ms | - |
| Year | - | 83ms | - |
Iteration #6: Keep a separate summary table
| Operation | NodeJS | Pistache | Difference |
|-----------|-------:|---------:|-----------:|
| Day | - | 3ms | - |
| Month | - | 1ms | - |
| Year | - | 4ms | - |
Comparison to the raspberry pi running the same software:
| Operation | RPi | Pentium | Difference |
|-----------|-------:|---------:|-----------:|
| Day | 14ms | 3ms | -11ms |
| Month | 5ms | 1ms | -4ms |
| Year | 16ms | 4ms | -12ms |
## Conclusion
Clarification:
- All platforms run Debian Buster
- RPi = Raspbery Pi Model 3 B+
- NodeJS = NodeJS Express.js server serving the solar logs stored as file per day (/var/log/solar/[yyyy]/[mm_dd].txt)
- Pistache = Pistache server serving the solar logs stored in a SQLite3 database
| Operation | RPi NodeJS | RPi Pistache | Pentium G5400 Pistache |
|-----------|-----------:|-------------:|-----------------------:|
| Day | 11ms | 14ms | 3ms |
| Month | 38ms | 5ms | 1ms |
| Year | 2122ms | 16ms | 4ms |

24
docs/ELECTRICITY_LOGGER.md
View File

@@ -1,24 +0,0 @@
## About
This small tool is meant for logging the electricity values from the `Landis Gyr E350` electricity meter. As a bonus it also stores the gas concumption values.
## Example Usage
```bash
#!/bin/bash
directoryPath="/var/log/electricity/$(date +%Y/)"
if ! [ -d "$directoryPath" ]; then
mkdir -p "$directoryPath"
fi
outputFile="$directoryPath$(date +%m_%d.txt)"
if ! [ -f "$outputFile" ]; then
touch "$outputFile"
fi
datestr="$(date +%Y-%m-%dT%T)"
electricity-logger -c "$datestr" /dev/ttyUSB0 >> "$outputFile"
```
Use the `--help` switch for more information about the possible arguments `electricity-logger` can take.

43
docs/SERVER.md
View File

@@ -1,43 +0,0 @@
# Serving Data
This project includes to data servers: the solar server and electricity server. Both host the data collected by their respective logger counterparts.
## API Description
[The solar api description](./SOLAR_API.md)
[The electricity api description](./ELECTRICITY_API.md)
## Reverse Proxy
It is recommended to use a reverse proxy setup to make all servers and content reachable through standard HTTP(S) ports. When using nginx something like this suffices, using the solar server as example:
```
server {
server_name solar.valkendaal.duckdns.org;
location / {
if ($request_method = 'GET') {
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Methods' 'GET';
}
proxy_pass http://localhost:3001;
}
}
```
The additional if statement within the location scope allows fetching of solar resources by any other domain. This is used to display data exposed by the solar server on the website (`public/`).
## Restricting Access
The electricity server exposes sensitive data, hence it shouldn't be accessible to anyone except users of the household. The easy solution here is to verify the requestee's IP address, since anyone making the request from the household itself should share the same external IP address, including the server. Therefore the electricity server checks the requestee's IP address against its own external one (by resolving the given domain parameter argument to an IP address).
For this a little hackery was necessary in the reverse proxy, since it normally makes the request to the electricity server on its own behalf. This would result in all requests originating from 127.0.0.1 (localhost). To solve this the following line was added before the `proxy_pass` directive:
```
proxy_set_header X-Real-IP $remote_addr;
```
The value of the X-Real-IP HTTP header is then used by the electricity server to validate against the domain resolved IP address. If they match it means the request came from the same network as the server.
This obviously is not watertight, but it serves the purpose well enough and avoids having to lay down more complicated authorization infrastructure.
## Launch Parameters
Both servers use TCLAP for their launch parameters. Simply run either executable without any parameters or use the `--help` switch to get all available commands.

42
docs/SOLAR_API.md
View File

@@ -1,42 +0,0 @@
# The solar-server REST API
This project depends on the `pistache` HTTP framework, see their [website](http://pistache.io/) for more info.
## Endpoints
All endpoints respond in JSON. Examples can be found in the Examples section. Dates are always in ISO format, meaning they take the shape of `year-month-day` with leading zeroes for month and day. Example: `2019-01-11` is the eleventh day of January, 2019.
- `/day?start=[yyyy-MM-dd]` This tells the server what date to serve all the collected records from the specified day.
- `/day?start=[yyyy-MM-dd]&stop=[yyyy-MM-dd]` This tells the server what date to serve the highest recorded kilowatt hour collected for each day in the specified date range. Stop date is inclusive.
- `/month?&start=[yyyy-mm]&stop=[yyyy-mm]` This returns the total kilowatthours generated for a given month. The stop month is inclusive.
## Example Response
All times are always in UTC. For these examples the current date is 25 December, 2019.
### /day?start=2019-02-15
Will give you as many objects as there are recorded values for the 15th of Febuary, 2019. Ordered from earliest to latest.
```json
[
{
"time": 1550226776,
"watt": 1456,
"kwh": 14.53
},
{
"time": 1550226834,
"watt": 1456,
"kwh": 14.54
},
...
]
```
### /month?start=2019-06&stop=2019-06
Will give you a total kilowatt hours generated for the month July, 2019. The time component is the first day of the month at 00:00 (midnight) UTC. For multiple months the result is ordered earliest to latest.
```json
[
{
"time": 1559347200,
"watt": 0,
"kwh": 451.64
}
]