diff --git a/README.md b/README.md index dedb2a6..025e15b 100644 --- a/README.md +++ b/README.md @@ -2,15 +2,6 @@ A mono repository containing all the homebrew collectors and REST APIs running on the local server. See it in action [here](https://valkendaal.duckdns.org). -## Contents - -This repository houses various projects. All are listed below. - -- The Zeverlution S3000 logger, [documentation](docs/SOLAR_LOGGER.md) -- The Landis Gyr E350 logger, [documentation](docs/ELECTRICITY_LOGGER.md) -- Two pistache REST HTTP APIs, [documentation](docs/SERVER.md) - - A solar panel log server, [documentation](docs/SOLAR_API.md) - - An electricity log server, [documentation](docs/ELECTRICITY_API.md) ## Project Directory @@ -19,10 +10,6 @@ This repository houses various projects. All are listed below. |- `include` => A folder housing all the header files used by the source files in `src/`
|- `script` => A folder with handy bash scripts to create and migrate databases
|- `src` => A folder with all the source files of the different projects that require compilation
-|....|- `electricity-logger` => The Landis Gyr E350 logger source files
-|....|- `electricity-server` => The pistache REST API sources for serving the electricity logs
-|....|- `solar-logger` => The Zeverlution S3000 logger source files
-|....|- `solar-server` => The pistache REST API sources for serving the solar panel logs
|- `systemd` => A folder with example systemd service files for the servers
## Miscellaneous @@ -33,16 +20,17 @@ A few benchmarks have been done for the solar-server project, which can be found ### Runtime (server) +- .NET 6, [website](https://dotnet.microsoft.com/en-us/download/dotnet/6.0) - Curl: the multiprotocol file tranfser library, [website](https://curl.haxx.se/libcurl/) -- cxxopts, [website](https://github.com/jarro2783/cxxopts/tree/v3.0.0) -- Docker CE, [website](https://www.docker.com/) - GNU Make, [website](https://www.gnu.org/software/make/) - Pistache: an elegant C++ REST framework, [website](http://pistache.io/) - Spdlog, [website](https://github.com/gabime/spdlog) -- Sqlite3: a small SQL database engine, [website](https://www.sqlite.org/index.html) +- Sqlite3: a small SQL database engine, v3.39.2, [website](https://www.sqlite.org/index.html) ### Development In addition to all the runtime dependencies the following dependencies are required for development: - A C++20 compatible compiler +- Nlohmann JSON, [website](https://github.com/nlohmann/json/tree/v3.11.2) +- cxxopts, [website](https://github.com/jarro2783/cxxopts/tree/v3.0.0) diff --git a/docs/ELECTRICITY_API.md b/docs/ELECTRICITY_API.md deleted file mode 100644 index 3cdad67..0000000 --- a/docs/ELECTRICITY_API.md +++ /dev/null @@ -1,37 +0,0 @@ -# The electricity-server REST API - -This project depends on the `pistache` HTTP framework, see their [website](http://pistache.io/) for more info. - -## Endpoints -All endpoints respond in JSON. Examples can be found in the Examples section. Dates are always in ISO format, meaning they take the shape of `year-month-day` with leading zeroes for month and day. Example: `2019-01-11` is the eleventh day of January, 2019. - -- `/day?start=[yyyy-MM-dd]` This tells the server what date to serve all the collected records from the specified day. -- `/day?start=[yyyy-MM-dd]&stop=[yyyy-MM-dd]` This tells the server what date range to serve a per day summary for. - -### Example Response /day?start=2020-01-10 -Will give you as many objects as there are recorded values for the 10th of January, 2020. Ordered from earliest to latest. If there are no records for the given day an empty array is returned. -```json -[ - { - "dateTime":1578614401, - "totalPowerUse":3282.95, - "totalPowerReturn":2790.88, - "totalGasUse":3769.23 - }, - ... -] -``` - -### Example Response /day?start=2020-01-01&stop=2020-01-19 -Will give you a summary for each recorded day in the range 1st of January 2020 to the 19th of January 2020. If no logs exist for the day on the server it will send an object with a valid dateTime for that day but with 0 values for the other fields. -```json -[ - { - "dateTime":1578614401, - "totalPowerUse": 8.324, - "totalPowerReturn": 3.62, - "totalGasUse": 11.05 - }, - ... -] -``` \ No newline at end of file diff --git a/docs/SOLAR_LOGGER.md b/docs/SOLAR_LOGGER.md deleted file mode 100644 index c6b3ccf..0000000 --- a/docs/SOLAR_LOGGER.md +++ /dev/null @@ -1,22 +0,0 @@ -# About -This software targets linux and has been tested on both x86_64 and armv7 hardware. - -## solar-logger -Simple data collecting program for ZeverSolar's Zeverlution Sxxxx "smart" inverters with a network interface (referred to as "combox" by the manufacturer). It collects all the exposed data of the Zeverlution, including current power generation (watts) and today's cummulative power production (kilowatt/hours). - -### Strange output -Zeverlution Smart Inverters currently have a bug that causes leading zeroes in a decimal number to be ignored. Concretely this means that the value 0.01 becomes 0.10, 3.09 becomes 3.9, etcetera. This is causing strange peaks in the logged data where sequences go from 2.80 to 2.90 to 2.10. - -Another bug is the inverter turning itself off and back on again throughout the day. This happens if the yield get too low. This will cause the cumulative power to reset to zero (kilowatt per hour). - -# Building -## Dependencies -### solar-logger -Dependencies for this program are: -- `libcurl` which can be installed using your package manager. It has been tested with the `openSSL` flavour, libcurl version 4. -- `sqlite3` which is sometimes also referred to as `libsqlite3-dev`. - -## Building -The logger program can be build by running the makefile (`make all`) in the project root. It will create a `bin` folder where it puts the newly created binaries. Move these to wherever you keep your binaries or use the `install` make target. - -Refer to the `--help` switch to find out more about supported launch parameters. diff --git a/include/solar/logger/database.hpp b/include/solar/logger/database.hpp index 60bc6bf..b357626 100644 --- a/include/solar/logger/database.hpp +++ b/include/solar/logger/database.hpp @@ -3,22 +3,29 @@ #include #include -struct Row +struct ZeverRow { time_t epochTime; std::int32_t watt; double kilowattPerHour; }; +struct EnvoyRow +{ + time_t epochTime; + std::int32_t inverterCount; + std::int32_t currentWatt; + std::int64_t lifetimeWattHour; + std::int64_t inverterTime; +}; + class Database { private: sqlite3 * connectionPtr; - std::string ToSqlInsertStatement(Row const & row) const; - std::string ToSqlUpsertStatement(Row const & row) const; - public: - bool Insert(Row & row); + bool Insert(ZeverRow & row); + bool Insert(EnvoyRow & row); Database(std::string const & databasePath); ~Database(); diff --git a/include/solar/logger/envoydata.hpp b/include/solar/logger/envoydata.hpp new file mode 100644 index 0000000..caef406 --- /dev/null +++ b/include/solar/logger/envoydata.hpp @@ -0,0 +1,14 @@ +#pragma once +#include +#include + +struct EnvoyData +{ + std::string type; + std::int32_t activeCount; + std::int64_t readingTime; + std::int32_t wNow; + std::int64_t whLifetime; + + static std::optional ParseJson(std::string const & json); +}; \ No newline at end of file diff --git a/include/solar/logger/zeverdata.hpp b/include/solar/logger/zeverdata.hpp index 3a2fb8c..6f345e3 100644 --- a/include/solar/logger/zeverdata.hpp +++ b/include/solar/logger/zeverdata.hpp @@ -2,6 +2,7 @@ #include #include #include // tested with libcurl4-openSSH +#include #include #include @@ -25,9 +26,7 @@ struct ZeverData std::string OKmsg, ERRORmsg; // Parses the data string coming from the zeverlution home.cgi webpage - bool ParseString(const std::string & str); - - bool FetchDataFromURL(const std::string & url, const unsigned int timeout); + static std::optional ParseString(const std::string & str); ZeverData(); }; diff --git a/makefile b/makefile index 26b6027..98d9bd1 100644 --- a/makefile +++ b/makefile @@ -1,7 +1,7 @@ CC = g++ CFLAGS = -DSPDLOG_FMT_EXTERNAL -Wall -Wextra -std=c++20 -O2 -Iinclude/ -MMD ELECTRICITY_LOGGER_LFLAGS = -lsqlite3 -lspdlog -lfmt -SOLAR_LOGGER_LFLAGS = -lsqlite3 -lcurl +SOLAR_LOGGER_LFLAGS = -lsqlite3 -lcurl -lspdlog -lfmt SERVER_LFLAGS = -lpistache -lsqlite3 -lstdc++fs -lspdlog -lfmt SOLAR_LOGGER_CPPS = $(shell find src/solar-logger/ -name *.cpp) @@ -16,21 +16,16 @@ ELECT_LOGGER_CPPS = $(shell find src/electricity-logger/ -name *.cpp) ELECT_LOGGER_OBJS = $(patsubst src/%.cpp, build/%.o, ${ELECT_LOGGER_CPPS}) ELECT_LOGGER_DEPS = $(patsubst src/%.cpp, build/%.d, ${ELECT_LOGGER_CPPS}) -ELECT_SRV_CPPS = $(shell find src/electricity-server/ -name *.cpp) -ELECT_SRV_OBJS = $(patsubst src/%.cpp, build/%.o, ${ELECT_SRV_CPPS}) -ELECT_SRV_DEPS = $(patsubst src/%.cpp, build/%.d, ${ELECT_SRV_CPPS}) - SOLAR_LOGGER_BINARY_PATH = bin/solar-logger SOLAR_SRV_BINARY_PATH = bin/solar-server ELECT_LOGGER_BINARY_PATH = bin/electricity-logger -ELECT_SRV_BINARY_PATH = bin/electricity-server INSTALL_PATH = /usr/local/bin/ # Test variables SOLAR_TEST_DATABASE = solarpaneloutput.db -all: ${SOLAR_LOGGER_BINARY_PATH} ${SOLAR_SRV_BINARY_PATH} ${ELECT_LOGGER_BINARY_PATH} ${ELECT_SRV_BINARY_PATH} +all: ${SOLAR_LOGGER_BINARY_PATH} ${SOLAR_SRV_BINARY_PATH} ${ELECT_LOGGER_BINARY_PATH} clean: -rm -rf bin/* build/* @@ -44,10 +39,7 @@ check-solar-server: ${SOLAR_SRV_BINARY_PATH} ${SOLAR_TEST_DATABASE} check-electricity-logger: ${ELECT_LOGGER_BINARY_PATH} ./${ELECT_LOGGER_BINARY_PATH} -d /dev/ttyUSB0 -c additional_column_value -check-electricity-server: ${ELECT_SRV_BINARY_PATH} - ./${ELECT_SRV_BINARY_PATH} -d /var/data0/electricity -p 8081 -s valkendaal.duckdns.org - -install: install-loggers install-servers +install: install-loggers install-solar-server install-loggers: ${SOLAR_LOGGER_BINARY_PATH} ${ELECT_LOGGER_BINARY_PATH} cp -uv ${SOLAR_LOGGER_BINARY_PATH} ${INSTALL_PATH} @@ -56,18 +48,12 @@ install-loggers: ${SOLAR_LOGGER_BINARY_PATH} ${ELECT_LOGGER_BINARY_PATH} install-solar-server: ${SOLAR_SRV_BINARY_PATH} cp -uv ${SOLAR_SRV_BINARY_PATH} ${INSTALL_PATH} -install-electricity-server: ${ELECT_SRV_BINARY_PATH} - cp -uv ${ELECT_SRV_BINARY_PATH} ${INSTALL_PATH} - -install-servers: install-solar-server install-electricity-server - .PHONY: all \ clean \ check-solar-logger check-solar-server \ install \ install-loggers \ -install-solar-server install-electricity-server \ -install-servers +install-solar-server ${SOLAR_TEST_DATABASE}: ./script/createdb.sh ${SOLAR_TEST_DATABASE} @@ -84,16 +70,10 @@ ${ELECT_LOGGER_BINARY_PATH}: ${ELECT_LOGGER_OBJS} mkdir -p ${@D} ${CC} $^ ${ELECTRICITY_LOGGER_LFLAGS} -o $@ -${ELECT_SRV_BINARY_PATH}: ${ELECT_SRV_OBJS} - mkdir -p ${@D} - ${CC} $^ ${SERVER_LFLAGS} -o $@ - build/%.o: src/%.cpp mkdir -p ${@D} ${CC} ${CFLAGS} -c $< -o $@ -include ${SOLAR_SRV_DEPS} -include ${SOLAR_LOGGER_DEPS} --include ${MIGRATOR_DEPS} --include ${ELECT_LOGGER_DEPS} --include ${ELECT_SRV_DEPS} \ No newline at end of file +-include ${ELECT_LOGGER_DEPS} \ No newline at end of file diff --git a/script/solar/README.md b/script/solar/README.md deleted file mode 100644 index 4240fbc..0000000 --- a/script/solar/README.md +++ /dev/null @@ -1,12 +0,0 @@ -# Scripts - -This directory contains the following 2 scripts: -- `createdb.sh` is used to create the SQL database file used by the logger -- `migratedb.sh` is used to fill the newly created SQL database with the old file based logging content - -The created database exists out of 3 rows, ignoring the `RowId` column: -- `DateTimeUtc` is an `INTEGER` and represents the UTC date and time in the Unix epoch format -- `Watts` is an `INTEGER` and represents the power output at the time of logging -- `KilowattHour` is a `REAL` and represents the cumulative power generated that day - -Try running each script without any arguments for more help or read their source code. diff --git a/script/solar/create.sql b/script/solar/create.sql new file mode 100644 index 0000000..c15ebb9 --- /dev/null +++ b/script/solar/create.sql @@ -0,0 +1,18 @@ +BEGIN TRANSACTION; + +-- Create Zever tables +CREATE TABLE ZeverLogs (Id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, Date TEXT NOT NULL, TimeUtc TEXT NOT NULL, CurrentWatts INTEGER NOT NULL, TotalWatts BIGINT NOT NULL); +CREATE INDEX idx_ZeverLogs_Date ON ZeverLogs (Date); +CREATE INDEX idx_ZeverLogs_TimeUtc ON ZeverLogs (TimeUtc); +CREATE INDEX idx_ZeverLogs_TotalWatts ON ZeverLogs (TotalWatts); + +CREATE TABLE ZeverSummary (Id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, Date TEXT NOT NULL UNIQUE, TotalWatts BIGINT NOT NULL); +CREATE UNIQUE INDEX idx_ZeverSummary_Date on ZeverSummary (Date); + +-- Create Envoy table +CREATE TABLE EnvoyLogs (Id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, Date TEXT NOT NULL, TimeUtc TEXT NOT NULL, CurrentWatts INTEGER NOT NULL, TotalWatts BIGINT NOT NULL, Inverters INTEGER NOT NULL); +CREATE INDEX idx_EnvoyLogs_Date ON EnvoyLogs (Date); +CREATE INDEX idx_EnvoyLogs_TimeUtc ON EnvoyLogs (TimeUtc); +CREATE INDEX idx_EnvoyLogs_TotalWatts ON EnvoyLogs (TotalWatts); + +COMMIT; diff --git a/script/solar/createdb.sh b/script/solar/createdb.sh deleted file mode 100755 index 9a29e2f..0000000 --- a/script/solar/createdb.sh +++ /dev/null @@ -1,30 +0,0 @@ -#!/bin/bash - -if [ "$#" -ne 1 ]; then - echo "Usage: $0 [/path/to/db/database].db"; - exit 1; -fi - -echo "Creating database file $1..."; -touch "$1"; -if ! [ -f "$1" ]; then - echo "Cannot open or create database file $1, aborting."; - exit 1; -fi - -TABLE_NAME="SolarPanelOutput"; - -echo "Creating table $TABLE_NAME..."; -sqlite3 $1 "CREATE TABLE IF NOT EXISTS $TABLE_NAME (Date TEXT NOT NULL, TimeUtc TEXT NOT NULL, Watts INTEGER NOT NULL, KilowattHour REAL NOT NULL);"; - -echo "Creating indexes on table $TABLE_NAME..."; -sqlite3 $1 "CREATE INDEX IF NOT EXISTS idx_Date ON $TABLE_NAME (Date);" -sqlite3 $1 "CREATE INDEX IF NOT EXISTS idx_TimeUtc ON $TABLE_NAME (TimeUtc);" - -TABLE2_NAME="SolarPanelSummary"; - -echo "Creating table $TABLE2_NAME..."; -sqlite3 $1 "CREATE TABLE IF NOT EXISTS $TABLE2_NAME (Date TEXT NOT NULL UNIQUE, Kilowatthour REAL NOT NULL);" - -echo "Creating indexes on table $TABLE2_NAME"; -sqlite3 $1 "CREATE UNIQUE INDEX IF NOT EXISTS idx_Date_$TABLE2_NAME on $TABLE2_NAME (Date);" \ No newline at end of file diff --git a/script/solar/migrate.sql b/script/solar/migrate.sql new file mode 100644 index 0000000..25e32b4 --- /dev/null +++ b/script/solar/migrate.sql @@ -0,0 +1,31 @@ +BEGIN TRANSACTION; + +-- Alter metric columns +ALTER TABLE SolarPanelOutput RENAME COLUMN Watts TO CurrentWatts; +ALTER TABLE SolarPanelOutput ADD COLUMN TotalWatts BIGINT; +UPDATE SolarPanelOutput SET TotalWatts = CAST(ROUND(KilowattHour * 1000) AS BIGINT); +ALTER TABLE SolarPanelOutput DROP COLUMN KilowattHour; + +ALTER TABLE SolarPanelSummary ADD COLUMN TotalWatts BIGINT; +UPDATE SolarPanelSummary SET TotalWatts = CAST(ROUND(Kilowatthour * 1000) AS BIGINT); +ALTER TABLE SolarPanelSummary DROP COLUMN Kilowatthour; + +-- Optimize indice +CREATE INDEX idx_TotalWatts ON SolarPanelOutput (TotalWatts); + +-- Fix date bug +UPDATE SolarPanelOutput SET Date = '2018-02-02' WHERE Date = '18-02-02'; + +-- Remove bogus data +DELETE FROM SolarPanelOutput WHERE Date = '2018-02-02' AND TimeUtc = '08:15:10'; +DELETE FROM SolarPanelOutput WHERE Date = '2018-01-02' AND TimeUtc = '13:31:03'; + +-- Copy data +INSERT INTO ZeverLogs (Date, TimeUtc, CurrentWatts, TotalWatts) SELECT * FROM SolarPanelOutput; +INSERT INTO ZeverSummary (Date, TotalWatts) SELECT * FROM SolarPanelSummary; + +-- Delete old table +DROP TABLE SolarPanelOutput; +DROP TABLE SolarPanelSummary; + +COMMIT; \ No newline at end of file diff --git a/script/solar/migratedb.sh b/script/solar/migratedb.sh deleted file mode 100755 index 2b1ef64..0000000 --- a/script/solar/migratedb.sh +++ /dev/null @@ -1,41 +0,0 @@ -#!/bin/bash - -TABLE_NAME="SolarPanelOutput"; - -if [ "$#" -ne 2 ]; then - echo "Usage: $0 [/log/file/directory/] [/path/to/db/database].db"; -fi - -if ! [ -d "$1" ]; then - echo "Error opening log file directory $1, aborting."; - exit 1; -fi - -if ! [ -f "$2" ]; then - echo "Error opening database file $2, aborting."; - exit 1; -fi - -if [ "${$1}" != */ ]; then - $1 = "$1/"; -fi - -for DIRECTORY in $1*/; do - if [ -d "$DIRECTORY" ]; then - echo "Checking out directory $DIRECTORY..."; - for FILE in $DIRECTORY*; do - if [ -f "$FILE" ]; then - echo "Processing file $FILE..."; - while read line; do - PROCESSED_LINE="$(echo "$line" | tr -s ' ')"; - ISODATE="$(echo "$PROCESSED_LINE" | cut -d ' ' -f1)"; - WATT="$(echo "$PROCESSED_LINE" | cut -d ' ' -f2)"; - KILOWATTHR="$(echo "$PROCESSED_LINE" | cut -d ' ' -f3)"; - - EPOCHDATE="$(date -d"$ISODATE" +%s)" - sqlite3 $2 "INSERT INTO $TABLE_NAME VALUES($EPOCHDATE, $WATT, $KILOWATTHR);"; - done < "$FILE" - fi - done; - fi -done \ No newline at end of file diff --git a/src/solar-logger/database.cpp b/src/solar-logger/database.cpp index 6bcd763..28b9faf 100644 --- a/src/solar-logger/database.cpp +++ b/src/solar-logger/database.cpp @@ -1,3 +1,4 @@ +#include #include #include #include @@ -6,27 +7,37 @@ #include #include -std::string Database::ToSqlInsertStatement(Row const & row) const +std::string ToSqlInsertStatement(ZeverRow const & row) { std::stringstream ss; - ss << "INSERT INTO SolarPanelOutput VALUES(" << '\'' << Util::GetSqliteDate(row.epochTime) << "'," << '\'' - << Util::GetSqliteUtcTime(row.epochTime) << "'," << row.watt << ", " << std::fixed << std::setprecision(2) - << row.kilowattPerHour << ");"; + ss << "INSERT INTO ZeverLogs VALUES(" << '\'' << Util::GetSqliteDate(row.epochTime) << "'," << '\'' + << Util::GetSqliteUtcTime(row.epochTime) << "'," << row.watt << ',' + << static_cast(round(row.kilowattPerHour * 1000)) << ");"; return ss.str(); } -std::string Database::ToSqlUpsertStatement(Row const & row) const +std::string ToSqlInsertStatement(EnvoyRow const & row) { std::stringstream ss; - ss << "INSERT INTO SolarPanelSummary(Date,KilowattHour) VALUES(" << '\'' << Util::GetSqliteDate(row.epochTime) - << "'," << std::fixed << std::setprecision(2) << row.kilowattPerHour << ')' - << "ON CONFLICT(Date) DO UPDATE SET KilowattHour = excluded.KilowattHour WHERE excluded.KilowattHour > SolarPanelSummary.KilowattHour;"; + ss << "INSERT INTO EnvoyLogs VALUES(" << '\'' << Util::GetSqliteDate(row.epochTime) << "'," << '\'' + << Util::GetSqliteUtcTime(row.epochTime) << "'," << row.currentWatt << ',' << row.lifetimeWattHour << ',' + << row.inverterCount << ");"; return ss.str(); } -bool Database::Insert(Row & row) +std::string ToSqlUpsertStatement(ZeverRow const & row) +{ + std::stringstream ss; + ss << "INSERT INTO ZeverSummary(Date,TotalWatts) VALUES(" << '\'' << Util::GetSqliteDate(row.epochTime) << "'," + << std::fixed << std::setprecision(2) << static_cast(round(row.kilowattPerHour * 1000)) << ')' + << "ON CONFLICT(Date) DO UPDATE SET TotalWatts = excluded.TotalWatts WHERE excluded.TotalWatts > ZeverSummary.TotalWatts;"; + + return ss.str(); +} + +bool Database::Insert(ZeverRow & row) { std::stringstream transaction; transaction << "BEGIN TRANSACTION;" << ToSqlInsertStatement(row) << ToSqlUpsertStatement(row) << "COMMIT;"; @@ -34,6 +45,11 @@ bool Database::Insert(Row & row) return sqlite3_exec(connectionPtr, transaction.str().c_str(), nullptr, nullptr, nullptr); } +bool Database::Insert(EnvoyRow & row) +{ + return sqlite3_exec(connectionPtr, ToSqlInsertStatement(row).c_str(), nullptr, nullptr, nullptr); +} + Database::Database(std::string const & databasePath) : connectionPtr(nullptr) { if(sqlite3_open(databasePath.c_str(), &connectionPtr) != SQLITE_OK) diff --git a/src/solar-logger/envoydata.cpp b/src/solar-logger/envoydata.cpp new file mode 100644 index 0000000..c4ecc82 --- /dev/null +++ b/src/solar-logger/envoydata.cpp @@ -0,0 +1,69 @@ +#include +#include +#include +#include +#include + +nlohmann::basic_json<> ExtractEnvoyProductionInvertersObject(std::string const & json) +{ + auto const parsed = nlohmann::json::parse(json); + if(!parsed.is_object()) + { + throw std::runtime_error("Invalid JSON"); + } + + auto const production = parsed["production"]; + if(!production.is_array()) + { + throw std::runtime_error("Invalid production property type"); + } + + auto const invertersObject = std::find_if( + production.begin(), + production.end(), + [](nlohmann::basic_json<> const & item) + { return item.is_object() && item["type"].is_string() && item["type"].get() == "inverters"; }); + if(invertersObject == production.end()) + { + throw std::runtime_error("Failed to find JSON object of type inverters in production array"); + } + + if(!(*invertersObject)["activeCount"].is_number_integer()) + { + throw std::runtime_error("Invalid activeCount property in inverters object"); + } + if(!(*invertersObject)["readingTime"].is_number_integer()) + { + throw std::runtime_error("Invalid readingTime property in inverters object"); + } + if(!(*invertersObject)["wNow"].is_number_integer()) + { + throw std::runtime_error("Invalid wNow property in inverters object"); + } + if(!(*invertersObject)["whLifetime"].is_number_integer()) + { + throw std::runtime_error("Invalid whLifetime property in inverters object"); + } + + return *invertersObject; +} + +std::optional EnvoyData::ParseJson(std::string const & json) +{ + try + { + auto const parsed = ExtractEnvoyProductionInvertersObject(json); + return EnvoyData { + .type = parsed["type"].get(), + .activeCount = parsed["activeCount"].get(), + .readingTime = parsed["readingTime"].get(), + .wNow = parsed["wNow"].get(), + .whLifetime = parsed["whLifetime"].get(), + }; + } + catch(std::exception & ex) + { + spdlog::error("Unexpected exception {} whilst parsing JSON {}", ex.what(), json); + return {}; + } +} diff --git a/src/solar-logger/main.cpp b/src/solar-logger/main.cpp index e126a8c..5c20148 100644 --- a/src/solar-logger/main.cpp +++ b/src/solar-logger/main.cpp @@ -1,58 +1,171 @@ #include +#include #include +#include #include +#include #include +#include #include #include -#include + +namespace detail +{ + // libcurl callback + size_t write_to_string(void * ptr, size_t size, size_t nmemb, void * stream) + { + std::string line((char *)ptr, nmemb); + std::string * buffer = (std::string *)stream; + buffer->append(line); + + return nmemb * size; + } +} + +std::optional ExtractArgs(int argc, char ** argv) +{ + cxxopts::Options options( + "solar-logger", + "solar-logger is a small program that retrieves solarpower generation statistics from smart inverters. It supports the Zeverlution Sxxxx and the Envoy S Metered (v5.x.x)"); + + options.add_options()( + "u,url", + "Fully qualified URL path and protocol to the web resource containing the stastics. For the Zeverlution platform this is /home.cgi and for the Envoy S this is /production.json.", + cxxopts::value())( + "connection-string", + "Path to the sqlite3 database file", + cxxopts::value())( + "type", + "Type of web resource to fetch; \"Zeverlution\" or \"Envoy\"", + cxxopts::value())( + "t,timeout", + "Fetch time out in milliseconds", + cxxopts::value()->default_value("1000")); + + if(argc == 1) + { + std::cout << options.help() << std::endl; + return {}; + } + + try + { + auto const parsed = options.parse(argc, argv); + return parsed; + } + catch(cxxopts::OptionException const & e) + { + spdlog::error(e.what()); + return {}; + } +} + +std::optional FetchDataFromURL(const std::string & url, const unsigned int timeout) +{ + curl_global_init(CURL_GLOBAL_ALL); + + CURL * curl = curl_easy_init(); + + curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 1L); + //curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L); + + std::string buffer; + curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, detail::write_to_string); + curl_easy_setopt(curl, CURLOPT_WRITEDATA, &buffer); + + curl_easy_setopt(curl, CURLOPT_TIMEOUT_MS, timeout); + + curl_easy_setopt(curl, CURLOPT_URL, url.c_str()); + + buffer.reserve(256); + const auto result = curl_easy_perform(curl); + if(result) + { + spdlog::error("Failed to fetch data from URL {} with cURL error code {}", url.c_str(), result); + curl_easy_cleanup(curl); + return {}; + } + + buffer.shrink_to_fit(); + + curl_easy_cleanup(curl); + + return buffer; +} int main(int argc, char ** argv) { - TCLAP::CmdLine cmd( - "solar-logger is a small program that retrieves solarpower generation statistics from Zeverlution Sxxxx smart power inverters and stores it into a database", - ' ', - "1.0"); - - TCLAP::ValueArg urlArg( - "u", - "url", - "Fully qualified URL path and protocol to the home.cgi resource", - true, - "", - "Fully qualified URL path and protocol to the home.cgi resource"); - cmd.add(urlArg); - - TCLAP::ValueArg - timeoutArg("t", "timeout", "Fetch time out in milliseconds", false, 1000U, "Fetch time out in milliseconds"); - cmd.add(timeoutArg); - - TCLAP::ValueArg databasePathArg( - "d", - "database-path", - "Absolute path pointing to the solar SQLite *.db file", - true, - "", - "Absolute path pointing to the solar SQLite *.db file"); - cmd.add(databasePathArg); - - cmd.parse(argc, argv); - - ZeverData zeverData; - if(!zeverData.FetchDataFromURL(urlArg.getValue(), timeoutArg.getValue())) + auto const maybeArgs = ExtractArgs(argc, argv); + if(!maybeArgs.has_value()) { - return -1; + return 1; } - Row row; - row.epochTime = std::time(nullptr); // now - row.watt = zeverData.watt; - row.kilowattPerHour = zeverData.kilowattPerHour; + auto const & args = maybeArgs.value(); - Database db(databasePathArg.getValue()); - auto const insertionResult = db.Insert(row); - if(insertionResult) + Database db(args["connection-string"].as()); + auto const now = std::time(nullptr); + auto const resourceType = args["type"].as(); + + if(resourceType == "Zeverlution") { - std::printf("Error %i during insertion of new value into database\n", insertionResult); + auto const webResource = FetchDataFromURL(args["url"].as(), args["timeout"].as()); + if(!webResource.has_value()) + { + return -1; + } + + auto const zeverData = ZeverData::ParseString(webResource.value()); + if(!zeverData.has_value()) + { + return -1; + } + + ZeverRow row = { + .epochTime = now, + .watt = zeverData->watt, + .kilowattPerHour = zeverData->kilowattPerHour, + }; + + auto const insertionResult = db.Insert(row); + if(insertionResult) + { + spdlog::error("Error {} during insertion of new value into database", insertionResult); + return -1; + } + } + else if(resourceType == "Envoy") + { + auto const webResource = FetchDataFromURL(args["url"].as(), args["timeout"].as()); + if(!webResource.has_value()) + { + return -1; + } + + auto const parsed = EnvoyData::ParseJson(webResource.value()); + if(!parsed.has_value()) + { + return -1; + } + + EnvoyRow row = { + .epochTime = now, + .inverterCount = parsed->activeCount, + .currentWatt = parsed->wNow, + .lifetimeWattHour = parsed->whLifetime, + .inverterTime = parsed->readingTime, + }; + + auto const insertionResult = db.Insert(row); + if(insertionResult) + { + spdlog::error("Error {} during insertion of new value into database", insertionResult); + return -1; + } + } + else + { + spdlog::error("Unrecognized Type argument {}", resourceType); return -1; } diff --git a/src/solar-logger/zeverdata.cpp b/src/solar-logger/zeverdata.cpp index 315fc35..06acc02 100644 --- a/src/solar-logger/zeverdata.cpp +++ b/src/solar-logger/zeverdata.cpp @@ -1,88 +1,41 @@ #include +#include -namespace detail -{ - // libcurl callback - size_t write_to_string(void * ptr, size_t size, size_t nmemb, void * stream) - { - std::string line((char *)ptr, nmemb); - std::string * buffer = (std::string *)stream; - buffer->append(line); - - return nmemb * size; - } -} - -bool ZeverData::ParseString(const std::string & str) +std::optional ZeverData::ParseString(const std::string & str) { + ZeverData data; bool isValid = true; std::stringstream ss; ss.str(str); - ss >> number0; - ss >> number1; - ss >> registeryID; - ss >> registeryKey; - ss >> hardwareVersion; + ss >> data.number0; + ss >> data.number1; + ss >> data.registeryID; + ss >> data.registeryKey; + ss >> data.hardwareVersion; std::string versionInfo; ss >> versionInfo; size_t splitPos = versionInfo.find('+'); - appVersion = versionInfo.substr(0, splitPos); + data.appVersion = versionInfo.substr(0, splitPos); ++splitPos; - wifiVersion = versionInfo.substr(splitPos, versionInfo.size()); + data.wifiVersion = versionInfo.substr(splitPos, versionInfo.size()); - ss >> timeValue; - ss >> dateValue; - ss >> zeverCloudStatus; - ss >> number3; - ss >> inverterSN; - isValid = (ss >> watt && ss >> kilowattPerHour); - ss >> OKmsg; - ss >> ERRORmsg; + ss >> data.timeValue; + ss >> data.dateValue; + ss >> data.zeverCloudStatus; + ss >> data.number3; + ss >> data.inverterSN; + isValid = (ss >> data.watt && ss >> data.kilowattPerHour); + ss >> data.OKmsg; + ss >> data.ERRORmsg; if(!isValid) { - std::fprintf(stderr, "Error during parsing of zever data:\n%s\n", str.c_str()); - return false; + spdlog::error("Error parsing zever data {}", str); + return {}; } - return true; -} - -bool ZeverData::FetchDataFromURL(const std::string & url, const unsigned int timeout) -{ - curl_global_init(CURL_GLOBAL_ALL); - - CURL * curl = curl_easy_init(); - - curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 1L); - //curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L); - - std::string buffer; - curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, detail::write_to_string); - curl_easy_setopt(curl, CURLOPT_WRITEDATA, &buffer); - - curl_easy_setopt(curl, CURLOPT_TIMEOUT_MS, timeout); - - curl_easy_setopt(curl, CURLOPT_URL, url.c_str()); - - buffer.reserve(256); - if(curl_easy_perform(curl)) - { - watt = 0; - kilowattPerHour = 0.0; - - std::fprintf(stderr, "Failed to fetch zever data from URL <%s>!\n", url.c_str()); - - curl_easy_cleanup(curl); - return false; - } - - buffer.shrink_to_fit(); - - curl_easy_cleanup(curl); - - return ParseString(buffer); + return data; } ZeverData::ZeverData() : watt(0), kilowattPerHour(0.0) { }